Postgres (#21)

* cluster env without volumeclaims * add volume * back to minikube hostpath volume for riak * add postgres * pg version and minor fix * add vault * initdb * add readme and sample * Update README.md Немного про волт * autoinitiate vault * Bump riak chart version and fallback to standard riak image * Update README.md Co-authored-by: vilorij <vilorij@ya.ru> * typo fix * delete doubled comment * Comment resolve * Drop useless comment and back to AppVersion in riak chart * Delete pv creates from chart. * delete state from values Co-authored-by: Dmitry Skokov <d.skokov@rbkmoney.com> Co-authored-by: Pospolita Nikita <nikita7asics@gmail.com>
2024-11-06 00:45:18 +00:00 · 2020-09-02 13:55:06 +03:00 · 2020-09-02 13:55:06 +03:00 · beb05d3c18
commit beb05d3c18
parent 1038fa9957
11 changed files with 214 additions and 13 deletions
--- a/README.md
+++ b/README.md
@ -22,6 +22,10 @@ Helm Charts Repo
 $ kubectl create secret docker-registry dr2reg --docker-server=dr2.rbkmoney.com --docker-username=$USERNAME --docker-password=$PASSWORD

 ```
+Для быстрой и автоматической инициализации Vault нужен configMap:
+```
+kubectl apply -f config/vault/init-cm.yaml
+

 Добавление в minikube ingress
 ------------------------------
@ -66,3 +70,73 @@ Phase:          Succeeded
 NOTES:
 You can use machinegun:8022 to connect to the machinegun woody interface.
 ```
+
+Работа с Vault
+----------
+Волт запускается в dev режиме, то есть сразу инициированный и unseal. 
+Референс для работы с секретами в [доке vault](https://www.hashicorp.com/blog/dynamic-database-credentials-with-vault-and-kubernetes/)
+
+<details>
+  <summary>Здесь немного комментов к тому, что происходит автоматом при запуске пода vault</summary>
+
+```
+# kubectl exec -ti vault-0 -- sh
+```
+```
+#Включим движки:
+vault auth enable kubernetes
+vault secrets enable database
+
+#Укажем адрес kube-api, к которому стоит обращаться для проверки токен сервис аккаунта приложения:
+vault write auth/kubernetes/config \
+       token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
+       kubernetes_host=https://${KUBERNETES_PORT_443_TCP_ADDR}:443 \
+       kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
+
+#Создадим роль, которая позволит перечисленным в `bound_service_account_names` сервисаккаунтам получать доступы к БД:
+
+vault write auth/kubernetes/role/db-app \
+    bound_service_account_names="*" \
+    bound_service_account_namespaces=default \
+    policies=db-app \
+    ttl=1h
+
+#теперь настраиваем подключение к постгресу:
+vault write database/config/mydatabase \
+    plugin_name=postgresql-database-plugin \
+    allowed_roles="*" \
+    connection_url="postgresql://{{username}}:{{password}}@postgres-postgresql.default:5432/?sslmode=disable" \
+    username="postgres" \
+    password="uw2dFhY9EP"
+
+vault write database/roles/db-app \
+    db_name=mydatabase \
+    creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
+        GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
+    default_ttl="1h" \
+    max_ttl="24h"
+```
+</details>
+
+Чтобы зайти в вебку волта нужно получить себе новый токен:
+```
+kubectl exec vault-0 -- vault token create
+```
+включить портфорвард на локалхост
+```
+kubectl port-forward vault-0 8200:8200 &
+```
+и с полученым токеном идти в браузере на http://127.0.0.1:8200
+
+Для того, чтобы приложение получило свои секретный логины-пароли к БД нужно добавить к описанию сервиса аннотации, [а тут смотреть целый манифест deployments](docs/service-with-vault-injected-creds-sample.yaml):
+```
+      annotations:
+        vault.hashicorp.com/agent-inject: "true"
+        vault.hashicorp.com/agent-inject-secret-db-creds: "database/creds/db-app"
+        vault.hashicorp.com/agent-inject-template-db-creds: |
+          {{- with secret "database/creds/db-app" -}}
+          "db_connection": "postgresql://{{ .Data.username }}:{{ .Data.password }}@postgres-postgresql:5432/?sslmode=disable"
+          {{- end }}
+        vault.hashicorp.com/role: "db-app"
+```
+После этого в поде с сервисом будет лежать файл `/vault/secrets/db-creds` со строкой подключения к БД
--- a/config/postgres/values.yaml
+++ b/config/postgres/values.yaml
@ -0,0 +1,11 @@
+replicas: 1
+
+#This is just init password for vault connections.
+postgresqlPassword: "uw2dFhY9EP"
+
+initdbScripts:
+  rbk-dbs.sql: |
+     CREATE DATABASE helm;
+     CREATE DATABASE test1;
+     CREATE DATABASE wizard;
+     
--- a/config/vault/init-cm.yaml
+++ b/config/vault/init-cm.yaml
@ -0,0 +1,41 @@
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: vault-cm
+  labels:
+    app: vault
+data:
+  init.vault.sh: |
+    vault secrets enable database
+    vault write database/config/mydatabase \
+        plugin_name=postgresql-database-plugin \
+        allowed_roles="*" \
+        connection_url="postgresql://{{username}}:{{password}}@postgres-postgresql.default:5432/?sslmode=disable" \
+        username="postgres" \
+        password="uw2dFhY9EP"    
+    vault write database/roles/db-app \
+        db_name=mydatabase \
+        creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
+            GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
+        default_ttl="1h" \
+        max_ttl="24h"
+
+    vault auth enable kubernetes
+    vault write auth/kubernetes/config \
+           token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
+           kubernetes_host=https://${KUBERNETES_PORT_443_TCP_ADDR}:443 \
+           kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
+    
+    vault write auth/kubernetes/role/db-app \
+        bound_service_account_names="*" \
+        bound_service_account_namespaces=default \
+        policies=db-app \
+        ttl=1h    
+    
+    vault policy write db-app /vault-init/db-policy.hcl
+  db-policy.hcl: |
+    path "database/creds/db-app" {
+    capabilities = ["read"]
+    }
+          
--- a/config/vault/values.yaml
+++ b/config/vault/values.yaml
@ -0,0 +1,18 @@
+server:
+  dev:
+    enabled: true
+
+  volumes:
+    - name: vault-init
+      configMap:
+        name: vault-cm
+ # volumeMounts is a list of volumeMounts for the main server container. These are rendered
+ # via toYaml rather than pre-processed like the extraVolumes value.
+ # The purpose is to make it easy to share volumes between containers.
+  volumeMounts:
+    - mountPath: /vault-init
+      name: vault-init
+  postStart:
+    - /bin/sh
+    - -c
+    - "cat /vault-init/init.vault.sh | sh"
--- a/docs/service-with-vault-injected-creds-sample.yaml
+++ b/docs/service-with-vault-injected-creds-sample.yaml
@ -0,0 +1,28 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: web-deployment
+  labels:
+    app: web
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: web
+  template:
+    metadata:
+      labels:
+        app: web
+      annotations:
+        vault.hashicorp.com/agent-inject: "true"
+        vault.hashicorp.com/agent-inject-secret-db-creds: "database/creds/db-app"
+        vault.hashicorp.com/agent-inject-template-db-creds: |
+          {{- with secret "database/creds/db-app" -}}
+          "db_connection": "postgresql://{{ .Data.username }}:{{ .Data.password }}@postgres-postgresql:5432/?sslmode=disable"
+          {{- end }}
+        vault.hashicorp.com/role: "db-app"
+    spec:
+      serviceAccountName: default
+      containers:
+        - name: web
+          image: nicholasjackson/fake-service:v0.7.3
--- a/helmfile.lock
+++ b/helmfile.lock
@ -1,4 +1,4 @@
-version: v0.116.0
+version: v0.125.5
 dependencies:
 - name: consul
  repository: https://kubernetes-charts.storage.googleapis.com
@ -6,8 +6,11 @@ dependencies:
 - name: kafka
  repository: https://kubernetes-charts-incubator.storage.googleapis.com
  version: 0.21.2
+- name: vault
+  repository: https://helm.releases.hashicorp.com
+  version: 0.7.0
 - name: zookeeper
  repository: https://kubernetes-charts-incubator.storage.googleapis.com
  version: 2.1.3
-digest: sha256:0fb454b1a2afa9e8e7e465cd74699976bec14a17642a4d4164a19aa847a4befc
-generated: "2020-05-20T14:10:13.586616462+03:00"
+digest: sha256:65f2c87582a6fee57134df871d6fc07d860248fa99a6c86473195e283089cb00
+generated: "2020-08-26T14:22:40.087884+03:00"
--- a/helmfile.yaml
+++ b/helmfile.yaml
@ -3,6 +3,10 @@ repositories:
  url: https://kubernetes-charts.storage.googleapis.com
 - name: incubator
  url: https://kubernetes-charts-incubator.storage.googleapis.com
+- name: postgres
+  url: https://charts.bitnami.com/bitnami
+- name: hashicorp
+  url: https://helm.releases.hashicorp.com

 templates:
  default: &default
@ -139,6 +143,14 @@ releases:
    file: config/api-common/keys/capi.privkey.pem
  - name: tokenEnctyptionKey1
    file: config/api-common/keys/token-encryption-keys/1.jwk
+- name: postgres
+  <<: *default
+  chart: bitnami/postgresql
+  version: 9.2.0
+- name: vault
+  <<: *default
+  chart: hashicorp/vault
+  version: 0.7.0
 - name: wapi
  <<: *default
  set:
@ -155,4 +167,4 @@ releases:
  - name: tokenEnctyptionKey1
    file: config/api-common/keys/token-encryption-keys/1.jwk
  - name: wapiPrivkey
-    file: config/api-common/keys/capi.privkey.pem
+    file: config/api-common/keys/capi.privkey.pem
--- a/services/riak/Chart.yaml
+++ b/services/riak/Chart.yaml
@ -2,5 +2,5 @@ apiVersion: v2
 name: riak
 description: A simple riak cluster for Kubernetes
 type: application
-version: 0.1.0
-appVersion: c02614659191d4b7793da74355421a32e28c489c
+version: 0.1.1
+appVersion: e0d38b2d6e8820bb94f26559a519871bcd789490
--- a/services/riak/templates/service.yaml
+++ b/services/riak/templates/service.yaml
@ -1,3 +1,5 @@
+#Headless service for clean dns record of each pod in statefulset.
+#clusterIP: None - is not a fault, it's normal annotations
 apiVersion: v1
 kind: Service
 metadata:
@ -6,6 +8,7 @@ metadata:
    {{- include "riak.labels" . | nindent 4 }}
 spec:
  type: {{ .Values.service.type }}
+  clusterIP: None
  ports:
    - port: 8098
      targetPort: 8098
--- a/services/riak/templates/statefulset.yaml
+++ b/services/riak/templates/statefulset.yaml
@ -24,6 +24,12 @@ spec:
    spec:
      serviceAccountName: {{ include "riak.serviceAccountName" . }}
      terminationGracePeriodSeconds: 10
+{{- if .Values.podSecurityContext.enabled }}
+      securityContext:
+        fsGroup: {{ .Values.podSecurityContext.fsGroup | int64 }}
+        runAsUser: {{ .Values.podSecurityContext.fsGroup | int64 }}
+        runAsGroup: {{ .Values.podSecurityContext.fsGroup | int64 }}
+{{- end }}
      containers:
        - name: {{ .Chart.Name }}
          image: "{{ .Values.image.repository }}:{{ .Chart.AppVersion }}"
@ -31,6 +37,8 @@ spec:
          env:
            - name: CLUSTER_NAME
              value: "riak"
+            - name: COORDINATOR_NODE
+              value: {{ include "riak.fullname" . }}-0.{{ include "riak.fullname" . }}.{{ .Release.Namespace }}.svc.cluster.local
          ports:
            - name: http
              protocol: TCP
@ -41,14 +49,14 @@ spec:
          livenessProbe:
            exec:
              command: ["riak", "ping"]
-            initialDelaySeconds: 60
+            initialDelaySeconds: 30
            periodSeconds: 60
            timeoutSeconds: 15
          readinessProbe:
            exec:
              command: ["riak-admin", "test"]
            initialDelaySeconds: 30
-            periodSeconds: 60
+            periodSeconds: 20
            timeoutSeconds: 10
          volumeMounts:
            - name: config-volume
@ -65,7 +73,7 @@ spec:
            name: {{ include "riak.fullname" . }}-user-config
    {{- with .Values.affinity }}
      affinity:
-          {{- toYaml . | nindent 8 }}
+        {{- toYaml . | nindent 8 }}
    {{- end }}
  volumeClaimTemplates:
    - metadata:
--- a/services/riak/values.yaml
+++ b/services/riak/values.yaml
@ -23,10 +23,11 @@ serviceAccount:
  # If not set and create is true, a name is generated using the fullname template
  name:

-podAnnotations: {}
+podSecurityContext:
+   enabled: false
+   fsGroup: 102

-podSecurityContext: {}
-  # fsGroup: 2000
+podAnnotations: {}

 securityContext: {}
  # capabilities:
@ -62,6 +63,8 @@ config:

 storage:
  accessModes: ["ReadWriteOnce"]
+# set right storageClassName at real cluster.
+#  storageClassName: ""
  resources:
     requests:
-       storage: 1Gi
+       storage: 3Gi