Postgres (#21)

* cluster env without volumeclaims

* add volume

* back to minikube hostpath volume for riak

* add postgres

* pg version and minor fix

* add vault

* initdb

* add readme and sample

* Update README.md

Немного про волт

* autoinitiate vault

* Bump riak chart version and fallback to standard riak image

* Update README.md

Co-authored-by: vilorij <vilorij@ya.ru>

* typo fix

* delete doubled comment

* Comment resolve

* Drop useless comment and back to AppVersion in riak chart

* Delete pv creates from chart.

* delete state from values

Co-authored-by: Dmitry Skokov <d.skokov@rbkmoney.com>
Co-authored-by: Pospolita Nikita <nikita7asics@gmail.com>

README.md

@@ -22,6 +22,10 @@ Helm Charts Repo
 $ kubectl create secret docker-registry dr2reg --docker-server=dr2.rbkmoney.com --docker-username=$USERNAME --docker-password=$PASSWORD
 
 ```
+Для быстрой и автоматической инициализации Vault нужен configMap:
+```
+kubectl apply -f config/vault/init-cm.yaml
+
 
 Добавление в minikube ingress
 ------------------------------
@@ -66,3 +70,73 @@ Phase:          Succeeded
 NOTES:
 You can use machinegun:8022 to connect to the machinegun woody interface.
 ```
+
+Работа с Vault
+----------
+Волт запускается в dev режиме, то есть сразу инициированный и unseal. 
+Референс для работы с секретами в [доке vault](https://www.hashicorp.com/blog/dynamic-database-credentials-with-vault-and-kubernetes/)
+
+<details>
+  <summary>Здесь немного комментов к тому, что происходит автоматом при запуске пода vault</summary>
+
+```
+# kubectl exec -ti vault-0 -- sh
+```
+```
+#Включим движки:
+vault auth enable kubernetes
+vault secrets enable database
+
+#Укажем адрес kube-api, к которому стоит обращаться для проверки токен сервис аккаунта приложения:
+vault write auth/kubernetes/config \
+       token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
+       kubernetes_host=https://${KUBERNETES_PORT_443_TCP_ADDR}:443 \
+       kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
+
+#Создадим роль, которая позволит перечисленным в `bound_service_account_names` сервисаккаунтам получать доступы к БД:
+
+vault write auth/kubernetes/role/db-app \
+    bound_service_account_names="*" \
+    bound_service_account_namespaces=default \
+    policies=db-app \
+    ttl=1h
+
+#теперь настраиваем подключение к постгресу:
+vault write database/config/mydatabase \
+    plugin_name=postgresql-database-plugin \
+    allowed_roles="*" \
+    connection_url="postgresql://{{username}}:{{password}}@postgres-postgresql.default:5432/?sslmode=disable" \
+    username="postgres" \
+    password="uw2dFhY9EP"
+
+vault write database/roles/db-app \
+    db_name=mydatabase \
+    creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
+        GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
+    default_ttl="1h" \
+    max_ttl="24h"
+```
+</details>
+
+Чтобы зайти в вебку волта нужно получить себе новый токен:
+```
+kubectl exec vault-0 -- vault token create
+```
+включить портфорвард на локалхост
+```
+kubectl port-forward vault-0 8200:8200 &
+```
+и с полученым токеном идти в браузере на http://127.0.0.1:8200
+
+Для того, чтобы приложение получило свои секретный логины-пароли к БД нужно добавить к описанию сервиса аннотации, [а тут смотреть целый манифест deployments](docs/service-with-vault-injected-creds-sample.yaml):
+```
+      annotations:
+        vault.hashicorp.com/agent-inject: "true"
+        vault.hashicorp.com/agent-inject-secret-db-creds: "database/creds/db-app"
+        vault.hashicorp.com/agent-inject-template-db-creds: |
+          {{- with secret "database/creds/db-app" -}}
+          "db_connection": "postgresql://{{ .Data.username }}:{{ .Data.password }}@postgres-postgresql:5432/?sslmode=disable"
+          {{- end }}
+        vault.hashicorp.com/role: "db-app"
+```
+После этого в поде с сервисом будет лежать файл `/vault/secrets/db-creds` со строкой подключения к БД

config/postgres/values.yaml

@@ -0,0 +1,11 @@
+replicas: 1
+
+#This is just init password for vault connections.
+postgresqlPassword: "uw2dFhY9EP"
+
+initdbScripts:
+  rbk-dbs.sql: |
+     CREATE DATABASE helm;
+     CREATE DATABASE test1;
+     CREATE DATABASE wizard;
+     

config/vault/init-cm.yaml

@@ -0,0 +1,41 @@
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: vault-cm
+  labels:
+    app: vault
+data:
+  init.vault.sh: |
+    vault secrets enable database
+    vault write database/config/mydatabase \
+        plugin_name=postgresql-database-plugin \
+        allowed_roles="*" \
+        connection_url="postgresql://{{username}}:{{password}}@postgres-postgresql.default:5432/?sslmode=disable" \
+        username="postgres" \
+        password="uw2dFhY9EP"    
+    vault write database/roles/db-app \
+        db_name=mydatabase \
+        creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
+            GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
+        default_ttl="1h" \
+        max_ttl="24h"
+
+    vault auth enable kubernetes
+    vault write auth/kubernetes/config \
+           token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
+           kubernetes_host=https://${KUBERNETES_PORT_443_TCP_ADDR}:443 \
+           kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
+    
+    vault write auth/kubernetes/role/db-app \
+        bound_service_account_names="*" \
+        bound_service_account_namespaces=default \
+        policies=db-app \
+        ttl=1h    
+    
+    vault policy write db-app /vault-init/db-policy.hcl
+  db-policy.hcl: |
+    path "database/creds/db-app" {
+    capabilities = ["read"]
+    }
+          

config/vault/values.yaml

@@ -0,0 +1,18 @@
+server:
+  dev:
+    enabled: true
+
+  volumes:
+    - name: vault-init
+      configMap:
+        name: vault-cm
+ # volumeMounts is a list of volumeMounts for the main server container. These are rendered
+ # via toYaml rather than pre-processed like the extraVolumes value.
+ # The purpose is to make it easy to share volumes between containers.
+  volumeMounts:
+    - mountPath: /vault-init
+      name: vault-init
+  postStart:
+    - /bin/sh
+    - -c
+    - "cat /vault-init/init.vault.sh | sh"

docs/service-with-vault-injected-creds-sample.yaml

@@ -0,0 +1,28 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: web-deployment
+  labels:
+    app: web
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: web
+  template:
+    metadata:
+      labels:
+        app: web
+      annotations:
+        vault.hashicorp.com/agent-inject: "true"
+        vault.hashicorp.com/agent-inject-secret-db-creds: "database/creds/db-app"
+        vault.hashicorp.com/agent-inject-template-db-creds: |
+          {{- with secret "database/creds/db-app" -}}
+          "db_connection": "postgresql://{{ .Data.username }}:{{ .Data.password }}@postgres-postgresql:5432/?sslmode=disable"
+          {{- end }}
+        vault.hashicorp.com/role: "db-app"
+    spec:
+      serviceAccountName: default
+      containers:
+        - name: web
+          image: nicholasjackson/fake-service:v0.7.3

helmfile.lock

@@ -1,4 +1,4 @@
-version: v0.116.0
+version: v0.125.5
 dependencies:
 - name: consul
   repository: https://kubernetes-charts.storage.googleapis.com
@@ -6,8 +6,11 @@ dependencies:
 - name: kafka
   repository: https://kubernetes-charts-incubator.storage.googleapis.com
   version: 0.21.2
+- name: vault
+  repository: https://helm.releases.hashicorp.com
+  version: 0.7.0
 - name: zookeeper
   repository: https://kubernetes-charts-incubator.storage.googleapis.com
   version: 2.1.3
-digest: sha256:0fb454b1a2afa9e8e7e465cd74699976bec14a17642a4d4164a19aa847a4befc
+digest: sha256:65f2c87582a6fee57134df871d6fc07d860248fa99a6c86473195e283089cb00
-generated: "2020-05-20T14:10:13.586616462+03:00"
+generated: "2020-08-26T14:22:40.087884+03:00"

helmfile.yaml

@@ -3,6 +3,10 @@ repositories:
   url: https://kubernetes-charts.storage.googleapis.com
 - name: incubator
   url: https://kubernetes-charts-incubator.storage.googleapis.com
+- name: postgres
+  url: https://charts.bitnami.com/bitnami
+- name: hashicorp
+  url: https://helm.releases.hashicorp.com
 
 templates:
   default: &default
@@ -139,6 +143,14 @@ releases:
     file: config/api-common/keys/capi.privkey.pem
   - name: tokenEnctyptionKey1
     file: config/api-common/keys/token-encryption-keys/1.jwk
+- name: postgres
+  <<: *default
+  chart: bitnami/postgresql
+  version: 9.2.0
+- name: vault
+  <<: *default
+  chart: hashicorp/vault
+  version: 0.7.0
 - name: wapi
   <<: *default
   set:
@@ -155,4 +167,4 @@ releases:
   - name: tokenEnctyptionKey1
     file: config/api-common/keys/token-encryption-keys/1.jwk
   - name: wapiPrivkey
-    file: config/api-common/keys/capi.privkey.pem
+    file: config/api-common/keys/capi.privkey.pem

services/riak/Chart.yaml

@@ -2,5 +2,5 @@ apiVersion: v2
 name: riak
 description: A simple riak cluster for Kubernetes
 type: application
-version: 0.1.0
+version: 0.1.1
-appVersion: c02614659191d4b7793da74355421a32e28c489c
+appVersion: e0d38b2d6e8820bb94f26559a519871bcd789490

services/riak/templates/service.yaml

@@ -1,3 +1,5 @@
+#Headless service for clean dns record of each pod in statefulset.
+#clusterIP: None - is not a fault, it's normal annotations
 apiVersion: v1
 kind: Service
 metadata:
@@ -6,6 +8,7 @@ metadata:
     {{- include "riak.labels" . | nindent 4 }}
 spec:
   type: {{ .Values.service.type }}
+  clusterIP: None
   ports:
     - port: 8098
       targetPort: 8098

services/riak/templates/statefulset.yaml

@@ -24,6 +24,12 @@ spec:
     spec:
       serviceAccountName: {{ include "riak.serviceAccountName" . }}
       terminationGracePeriodSeconds: 10
+{{- if .Values.podSecurityContext.enabled }}
+      securityContext:
+        fsGroup: {{ .Values.podSecurityContext.fsGroup | int64 }}
+        runAsUser: {{ .Values.podSecurityContext.fsGroup | int64 }}
+        runAsGroup: {{ .Values.podSecurityContext.fsGroup | int64 }}
+{{- end }}
       containers:
         - name: {{ .Chart.Name }}
           image: "{{ .Values.image.repository }}:{{ .Chart.AppVersion }}"
@@ -31,6 +37,8 @@ spec:
           env:
             - name: CLUSTER_NAME
               value: "riak"
+            - name: COORDINATOR_NODE
+              value: {{ include "riak.fullname" . }}-0.{{ include "riak.fullname" . }}.{{ .Release.Namespace }}.svc.cluster.local
           ports:
             - name: http
               protocol: TCP
@@ -41,14 +49,14 @@ spec:
           livenessProbe:
             exec:
               command: ["riak", "ping"]
-            initialDelaySeconds: 60
+            initialDelaySeconds: 30
             periodSeconds: 60
             timeoutSeconds: 15
           readinessProbe:
             exec:
               command: ["riak-admin", "test"]
             initialDelaySeconds: 30
-            periodSeconds: 60
+            periodSeconds: 20
             timeoutSeconds: 10
           volumeMounts:
             - name: config-volume
@@ -65,7 +73,7 @@ spec:
             name: {{ include "riak.fullname" . }}-user-config
     {{- with .Values.affinity }}
       affinity:
-          {{- toYaml . | nindent 8 }}
+        {{- toYaml . | nindent 8 }}
     {{- end }}
   volumeClaimTemplates:
     - metadata:

services/riak/values.yaml

@@ -23,10 +23,11 @@ serviceAccount:
   # If not set and create is true, a name is generated using the fullname template
   name:
 
-podAnnotations: {}
+podSecurityContext:
+   enabled: false
+   fsGroup: 102
 
-podSecurityContext: {}
+podAnnotations: {}
-  # fsGroup: 2000
 
 securityContext: {}
   # capabilities:
@@ -62,6 +63,8 @@ config:
 
 storage:
   accessModes: ["ReadWriteOnce"]
+# set right storageClassName at real cluster.
+#  storageClassName: ""
   resources:
      requests:
-       storage: 1Gi
+       storage: 3Gi