Project

General

Profile

The ungleich kubernetes infrastructure » History » Revision 98

Revision 97 (Nico Schottelius, 04/05/2022 06:07 PM) → Revision 98/212 (Nico Schottelius, 04/29/2022 06:03 PM)

h1. The ungleich kubernetes infrastructure and ungleich kubernetes manual 

 {{toc}} 

 h2. Status 

 This document is **pre-production**. 
 This document is to become the ungleich kubernetes infrastructure overview as well as the ungleich kubernetes manual. 

 h2. k8s clusters 

 | Cluster           | Purpose/Setup       | Maintainer | Master(s)                    | argo                                                  | v4 http proxy | last verified | 
 | c0.k8s.ooo        | Dev                 | -            | UNUSED                       |                                                       |                 |      2021-10-05 | 
 | c1.k8s.ooo        | retired             |              | -                            |                                                       |                 |      2022-03-15 | 
 | c2.k8s.ooo        | Dev p7 HW           | Nico         | server47 server53 server54 | "argo":https://argocd-server.argocd.svc.c2.k8s.ooo    |                 |      2021-10-05 | 
 | c3.k8s.ooo        | retired             | -            | -                            |                                                       |                 |      2021-10-05 | 
 | c4.k8s.ooo        | Dev2 p7 HW          | Jin-Guk      | server52 server53 server54 |                                                       |                 |               - | 
 | c5.k8s.ooo        | retired             |              | -                            |                                                       |                 |      2022-03-15 | 
 | c6.k8s.ooo        | Dev p6 VM Jin-Guk | Jin-Guk      |                              |                                                       |                 |                 | 
 | [[p5.k8s.ooo]]    | production          |              | server34 server36 server38 | "argo":https://argocd-server.argocd.svc.p5.k8s.ooo    |               - |                 | 
 | [[p6.k8s.ooo]]    | production          |              | server67 server69 server71 | "argo":https://argocd-server.argocd.svc.p6.k8s.ooo    | 147.78.194.13 |      2021-10-05 | 
 | [[p10.k8s.ooo]] | production          |              | server63 server65 server83 | "argo":https://argocd-server.argocd.svc.p10.k8s.ooo | 147.78.194.12 |      2021-10-05 | 
 | fnnf              | development         | Nico         | server75                     |                                                       |                 |                 | 


 h2. General architecture and components overview 

 * All k8s clusters are IPv6 only 
 * We use BGP peering to propagate podcidr and serviceCidr networks to our infrastructure 
 * The main public testing repository is "ungleich-k8s":https://code.ungleich.ch/ungleich-public/ungleich-k8s 
 ** Private configurations are found in the **k8s-config** repository 

 h3. Cluster types 

 | **Type/Feature**              | **Development**                  | **Production**           | 
 | Min No. nodes                 | 3 (1 master, 3 worker)           | 5 (3 master, 3 worker) | 
 | Recommended minimum           | 4 (dedicated master, 3 worker) | 8 (3 master, 5 worker) | 
 | Separation of control plane | optional                         | recommended              | 
 | Persistent storage            | required                         | required                 | 
 | Number of storage monitors    | 3                                | 5                        | 

 h2. General k8s operations 

 h3. Cheat sheet / external great references 

 * "kubectl cheatsheet":https://kubernetes.io/docs/reference/kubectl/cheatsheet/ 

 h3. Allowing to schedule work on the control plane 

 * Mostly for single node / test / development clusters 
 * Just remove the master taint as follows 

 <pre> 
 kubectl taint nodes --all node-role.kubernetes.io/master- 
 </pre> 


 h3. Get the cluster admin.conf 

 * On the masters of each cluster you can find the file @/etc/kubernetes/admin.conf@ 
 * To be able to administrate the cluster you can copy the admin.conf to your local machine 
 * Multi cluster debugging can very easy if you name the config ~/cX-admin.conf (see example below) 

 <pre> 
 % scp root@server47.place7.ungleich.ch:/etc/kubernetes/admin.conf ~/c2-admin.conf 
 % export KUBECONFIG=~/c2-admin.conf     
 % kubectl get nodes 
 NAME         STATUS                       ROLES                    AGE     VERSION 
 server47     Ready                        control-plane,master     82d     v1.22.0 
 server48     Ready                        control-plane,master     82d     v1.22.0 
 server49     Ready                        <none>                   82d     v1.22.0 
 server50     Ready                        <none>                   82d     v1.22.0 
 server59     Ready                        control-plane,master     82d     v1.22.0 
 server60     Ready,SchedulingDisabled     <none>                   82d     v1.22.0 
 server61     Ready                        <none>                   82d     v1.22.0 
 server62     Ready                        <none>                   82d     v1.22.0                
 </pre> 

 h3. Installing a new k8s cluster 

 * Decide on the cluster name (usually *cX.k8s.ooo*), X counting upwards 
 ** Using pXX.k8s.ooo for production clusters of placeXX 
 * Use cdist to configure the nodes with requirements like crio 
 * Decide between single or multi node control plane setups (see below) 
 ** Single control plane suitable for development clusters 

 Typical init procedure: 

 * Single control plane: @kubeadm init --config bootstrap/XXX/kubeadm.yaml@ 
 * Multi control plane (HA): @kubeadm init --config bootstrap/XXX/kubeadm.yaml --upload-certs@ 

 h3. Deleting a pod that is hanging in terminating state 

 <pre> 
 kubectl delete pod <PODNAME> --grace-period=0 --force --namespace <NAMESPACE> 
 </pre> 

 (from https://stackoverflow.com/questions/35453792/pods-stuck-in-terminating-status) 

 h3. Listing nodes of a cluster 

 <pre> 
 [15:05] bridge:~% kubectl get nodes 
 NAME         STATUS     ROLES                    AGE     VERSION 
 server22     Ready      <none>                   52d     v1.22.0 
 server23     Ready      <none>                   52d     v1.22.2 
 server24     Ready      <none>                   52d     v1.22.0 
 server25     Ready      <none>                   52d     v1.22.0 
 server26     Ready      <none>                   52d     v1.22.0 
 server27     Ready      <none>                   52d     v1.22.0 
 server63     Ready      control-plane,master     52d     v1.22.0 
 server64     Ready      <none>                   52d     v1.22.0 
 server65     Ready      control-plane,master     52d     v1.22.0 
 server66     Ready      <none>                   52d     v1.22.0 
 server83     Ready      control-plane,master     52d     v1.22.0 
 server84     Ready      <none>                   52d     v1.22.0 
 server85     Ready      <none>                   52d     v1.22.0 
 server86     Ready      <none>                   52d     v1.22.0 
 </pre> 


 h3. Removing / draining a node 

 Usually @kubectl drain server@ should do the job, but sometimes we need to be more aggressive: 

 <pre> 
 kubectl drain --delete-emptydir-data --ignore-daemonsets server23 
 </pre> 

 h3. Readding a node after draining 

 <pre> 
 kubectl uncordon serverXX 
 </pre> 

 h3. (Re-)joining worker nodes after creating the cluster 

 * We need to have an up-to-date token 
 * We use different join commands for the workers and control plane nodes 

 Generating the join command on an existing control plane node: 

 <pre> 
 kubeadm token create --print-join-command 
 </pre> 

 h3. (Re-)joining control plane nodes after creating the cluster 

 * We generate the token again 
 * We upload the certificates 
 * We need to combine/create the join command for the control plane node 

 Example session: 

 <pre> 
 % kubeadm token create --print-join-command 
 kubeadm join p10-api.k8s.ooo:6443 --token xmff4i.ABC --discovery-token-ca-cert-hash sha256:longhash  

 % kubeadm init phase upload-certs --upload-certs 
 [upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace 
 [upload-certs] Using certificate key: 
 CERTKEY 

 # Then we use these two outputs on the joining node: 

 kubeadm join p10-api.k8s.ooo:6443 --token xmff4i.ABC --discovery-token-ca-cert-hash sha256:longhash --control-plane --certificate-key CERTKEY 
 </pre> 

 Commands to be used on a control plane node: 

 <pre> 
 kubeadm token create --print-join-command 
 kubeadm init phase upload-certs --upload-certs 
 </pre> 

 Commands to be used on the joining node: 

 <pre> 
 JOINCOMMAND --control-plane --certificate-key CERTKEY 
 </pre> 

 SEE ALSO 

 * https://stackoverflow.com/questions/63936268/how-to-generate-kubeadm-token-for-secondary-control-plane-nodes 
 * https://blog.scottlowe.org/2019/08/15/reconstructing-the-join-command-for-kubeadm/ 

 h3. How to fix etcd does not start when rejoining a kubernetes cluster as a control plane 

 If during the above step etcd does not come up, @kubeadm join@ can hang as follows: 

 <pre> 
 [control-plane] Creating static Pod manifest for "kube-apiserver"                                                               
 [control-plane] Creating static Pod manifest for "kube-controller-manager"                                                      
 [control-plane] Creating static Pod manifest for "kube-scheduler"                                                               
 [check-etcd] Checking that the etcd cluster is healthy                                                                          
 error execution phase check-etcd: etcd cluster is not healthy: failed to dial endpoint https://[2a0a:e5c0:10:1:225:b3ff:fe20:37 
 8a]:2379 with maintenance client: context deadline exceeded                                                                     
 To see the stack trace of this error execute with --v=5 or higher          
 </pre> 

 Then the problem is likely that the etcd server is still a member of the cluster. We first need to remove it from the etcd cluster and then the join works. 

 To fix this we do: 

 * Find a working etcd pod 
 * Find the etcd members / member list 
 * Remove the etcd member that we want to re-join the cluster 


 <pre> 
 # Find the etcd pods 
 kubectl -n kube-system get pods -l component=etcd,tier=control-plane 

 # Get the list of etcd servers with the member id  
 kubectl exec -n kube-system -ti ETCDPODNAME -- etcdctl --endpoints '[::1]:2379' --cacert /etc/kubernetes/pki/etcd/ca.crt --cert    /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member list 

 # Remove the member 
 kubectl exec -n kube-system -ti ETCDPODNAME -- etcdctl --endpoints '[::1]:2379' --cacert /etc/kubernetes/pki/etcd/ca.crt --cert    /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member remove MEMBERID 
 </pre> 

 Sample session: 

 <pre> 
 [10:48] line:~% kubectl -n kube-system get pods -l component=etcd,tier=control-plane 
 NAME              READY     STATUS      RESTARTS       AGE 
 etcd-server63     1/1       Running     0              3m11s 
 etcd-server65     1/1       Running     3              7d2h 
 etcd-server83     1/1       Running     8 (6d ago)     7d2h 
 [10:48] line:~% kubectl exec -n kube-system -ti etcd-server65 -- etcdctl --endpoints '[::1]:2379' --cacert /etc/kubernetes/pki/etcd/ca.crt --cert    /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member list 
 356891cd676df6e4, started, server65, https://[2a0a:e5c0:10:1:225:b3ff:fe20:375c]:2380, https://[2a0a:e5c0:10:1:225:b3ff:fe20:375c]:2379, false 
 371b8a07185dee7e, started, server63, https://[2a0a:e5c0:10:1:225:b3ff:fe20:378a]:2380, https://[2a0a:e5c0:10:1:225:b3ff:fe20:378a]:2379, false 
 5942bc58307f8af9, started, server83, https://[2a0a:e5c0:10:1:3e4a:92ff:fe79:bb98]:2380, https://[2a0a:e5c0:10:1:3e4a:92ff:fe79:bb98]:2379, false 

 [10:48] line:~% kubectl exec -n kube-system -ti etcd-server65 -- etcdctl --endpoints '[::1]:2379' --cacert /etc/kubernetes/pki/etcd/ca.crt --cert    /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member remove 371b8a07185dee7e 
 Member 371b8a07185dee7e removed from cluster e3c0805f592a8f77 

 </pre> 

 SEE ALSO 

 * We found the solution using https://stackoverflow.com/questions/67921552/re-installed-node-cannot-join-kubernetes-cluster 

 h2. Calico CNI 

 h3. Calico Installation 

 * We install "calico using helm":https://docs.projectcalico.org/getting-started/kubernetes/helm 
 * This has the following advantages: 
 ** Easy to upgrade 
 ** Does not require os to configure IPv6/dual stack settings as the tigera operator figures out things on its own 

 Usually plain calico can be installed directly using: 

 <pre> 
 helm repo add projectcalico https://docs.projectcalico.org/charts 
 helm install calico projectcalico/tigera-operator --version v3.20.4 
 </pre> 

 * Check the tags on https://github.com/projectcalico/calico/tags for the latest release 

 h3. Installing calicoctl 

 To be able to manage and configure calico, we need to  
 "install calicoctl (we choose the version as a pod)":https://docs.projectcalico.org/getting-started/clis/calicoctl/install#install-calicoctl-as-a-kubernetes-pod 

 <pre> 
 kubectl apply -f https://docs.projectcalico.org/manifests/calicoctl.yaml 
 </pre> 

 Or version specific: 

 <pre> 
 kubectl apply -f https://github.com/projectcalico/calico/blob/v3.20.4/manifests/calicoctl.yaml 

 # For 3.22 
 kubectl apply -f https://projectcalico.docs.tigera.io/archive/v3.22/manifests/calicoctl.yaml 
 </pre> 

 And making it easier accessible by alias: 

 <pre> 
 alias calicoctl="kubectl exec -i -n kube-system calicoctl -- /calicoctl" 
 </pre> 

 h3. Calico configuration 

 By default our k8s clusters "BGP peer":https://docs.projectcalico.org/networking/bgp 
 with an upstream router to propagate podcidr and servicecidr. 

 Default settings in our infrastructure: 

 * We use a full-mesh using the @nodeToNodeMeshEnabled: true@ option 
 * We keep the original next hop so that *only* the server with the pod is announcing it (instead of ecmp) 
 * We use private ASNs for k8s clusters 
 * We do *not* use any overlay 

 After installing calico and calicoctl the last step of the installation is usually: 

 <pre> 
 calicoctl create -f - < calico-bgp.yaml 
 </pre> 


 A sample BGP configuration: 

 <pre> 
 --- 
 apiVersion: projectcalico.org/v3 
 kind: BGPConfiguration 
 metadata: 
   name: default 
 spec: 
   logSeverityScreen: Info 
   nodeToNodeMeshEnabled: true 
   asNumber: 65534 
   serviceClusterIPs: 
   - cidr: 2a0a:e5c0:10:3::/108 
   serviceExternalIPs: 
   - cidr: 2a0a:e5c0:10:3::/108 
 --- 
 apiVersion: projectcalico.org/v3 
 kind: BGPPeer 
 metadata: 
   name: router1-place10 
 spec: 
   peerIP: 2a0a:e5c0:10:1::50 
   asNumber: 213081 
   keepOriginalNextHop: true 
 </pre> 

 h2. ArgoCD / ArgoWorkFlow 

 h3. Argocd Installation 

 As there is no configuration management present yet, argocd is installed using 

 <pre> 
 kubectl create namespace argocd 

 # Specific Version 
 kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v2.3.2/manifests/install.yaml 

 # OR: latest stable 
 kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml 
 </pre> 

 * See https://argo-cd.readthedocs.io/en/stable/ 

 h3. Get the argocd credentials 

 <pre> 
 kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo "" 
 </pre> 

 h3. Accessing argocd 

 In regular IPv6 clusters: 

 * Navigate to https://argocd-server.argocd.CLUSTERDOMAIN 

 In legacy IPv4 clusters 

 <pre> 
 kubectl --namespace argocd port-forward svc/argocd-server 8080:80 
 </pre> 

 * Navigate to https://localhost:8080 

 h3. Using the argocd webhook to trigger changes 

 * To trigger changes post json https://argocd.example.com/api/webhook 

 h3. Deploying an application 

 * Applications are deployed via git towards gitea (code.ungleich.ch) and then pulled by argo 
 * Always include the *redmine-url* pointing to the (customer) ticket 
 ** Also add the support-url if it exists 

 Application sample 

 <pre> 
 apiVersion: argoproj.io/v1alpha1 
 kind: Application 
 metadata: 
   name: gitea-CUSTOMER 
   namespace: argocd 
 spec: 
   destination: 
     namespace: default 
     server: 'https://kubernetes.default.svc' 
   source: 
     path: apps/prod/gitea 
     repoURL: 'https://code.ungleich.ch/ungleich-intern/k8s-config.git' 
     targetRevision: HEAD 
     helm: 
       parameters: 
         - name: storage.data.storageClass 
           value: rook-ceph-block-hdd 
         - name: storage.data.size 
           value: 200Gi 
         - name: storage.db.storageClass 
           value: rook-ceph-block-ssd 
         - name: storage.db.size 
           value: 10Gi 
         - name: storage.letsencrypt.storageClass 
           value: rook-ceph-block-hdd 
         - name: storage.letsencrypt.size 
           value: 50Mi 
         - name: letsencryptStaging 
           value: 'no' 
         - name: fqdn 
           value: 'code.verua.online' 
   project: default 
   syncPolicy: 
     automated: 
       prune: true 
       selfHeal: true 
   info: 
     - name: 'redmine-url' 
       value: 'https://redmine.ungleich.ch/issues/ISSUEID' 
     - name: 'support-url' 
       value: 'https://support.ungleich.ch/Ticket/Display.html?id=TICKETID' 
 </pre> 

 h2. Helm related operations and conventions 

 We use helm charts extensively. 

 * In production, they are managed via argocd 
 * In development, helm chart can de developed and deployed manually using the helm utility. 

 h3. Installing a helm chart 

 One can use the usual pattern of 

 <pre> 
 helm install <releasename> <chartdirectory> 
 </pre> 

 However often you want to reinstall/update when testing helm charts. The following pattern is "better", because it allows you to reinstall, if it is already installed: 

 <pre> 
 helm upgrade --install <releasename> <chartdirectory> 
 </pre> 

 h3. Naming services and deployments in helm charts [Application labels] 

 * We always have {{ .Release.Name }} to identify the current "instance" 
 * Deployments: 
 ** use @app: <what it is>@, f.i. @app: nginx@, @app: postgres@, ... 
 * See more about standard labels on 
 ** https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/ 
 ** https://helm.sh/docs/chart_best_practices/labels/ 

 h2. Rook / Ceph Related Operations 

 h3. Executing ceph commands 

 Using the ceph-tools pod as follows: 

 <pre> 
 kubectl exec -n rook-ceph -ti $(kubectl -n rook-ceph get pods -l app=rook-ceph-tools -o jsonpath='{.items[*].metadata.name}') -- ceph -s 
 </pre> 

 h3. Inspecting the logs of a specific server 

 <pre> 
 # Get the related pods 
 kubectl -n rook-ceph get pods -l app=rook-ceph-osd-prepare  
 ... 

 # Inspect the logs of a specific pod 
 kubectl -n rook-ceph logs -f rook-ceph-osd-prepare-server23--1-444qx 

 </pre> 

 h3. Inspecting the logs of the rook-ceph-operator 

 <pre> 
 kubectl -n rook-ceph logs -f -l app=rook-ceph-operator 
 </pre> 

 h3. Triggering server prepare / adding new osds 

 The rook-ceph-operator triggers/watches/creates pods to maintain hosts. To trigger a full "re scan", simply delete that pod: 

 <pre> 
 kubectl -n rook-ceph delete pods -l app=rook-ceph-operator 
 </pre> 

 This will cause all the @rook-ceph-osd-prepare-..@ jobs to be recreated and thus OSDs to be created, if new disks have been added. 

 

 h3. Removing an OSD 

 * See "Ceph OSD Management":https://rook.io/docs/rook/v1.7/ceph-osd-mgmt.html 
 * More specifically: https://github.com/rook/rook/blob/release-1.7/cluster/examples/kubernetes/ceph/osd-purge.yaml 

 Set osd id in the osd-purge.yaml and apply it. OSD should be down before. 

 <pre> 
 apiVersion: batch/v1 
 kind: Job 
 metadata: 
   name: rook-ceph-purge-osd 
   namespace: rook-ceph # namespace:cluster 
   labels: 
     app: rook-ceph-purge-osd 
 spec: 
   template: 
     metadata: 
       labels: 
         app: rook-ceph-purge-osd 
     spec: 
       serviceAccountName: rook-ceph-purge-osd 
       containers: 
         - name: osd-removal 
           image: rook/ceph:master 
           # TODO: Insert the OSD ID in the last parameter that is to be removed 
           # The OSD IDs are a comma-separated list. For example: "0" or "0,2". 
           # If you want to preserve the OSD PVCs, set `--preserve-pvc true`. 
           # 
           # A --force-osd-removal option is available if the OSD should be destroyed even though the 
           # removal could lead to data loss. 
           args: 
             - "ceph" 
             - "osd" 
             - "remove" 
             - "--preserve-pvc" 
             - "false" 
             - "--force-osd-removal" 
             - "false" 
             - "--osd-ids" 
             - "SETTHEOSDIDHERE" 
           env: 
             - name: POD_NAMESPACE 
               valueFrom: 
                 fieldRef: 
                   fieldPath: metadata.namespace 
             - name: ROOK_MON_ENDPOINTS 
               valueFrom: 
                 configMapKeyRef: 
                   key: data 
                   name: rook-ceph-mon-endpoints 
             - name: ROOK_CEPH_USERNAME 
               valueFrom: 
                 secretKeyRef: 
                   key: ceph-username 
                   name: rook-ceph-mon 
             - name: ROOK_CEPH_SECRET 
               valueFrom: 
                 secretKeyRef: 
                   key: ceph-secret 
                   name: rook-ceph-mon 
             - name: ROOK_CONFIG_DIR 
               value: /var/lib/rook 
             - name: ROOK_CEPH_CONFIG_OVERRIDE 
               value: /etc/rook/config/override.conf 
             - name: ROOK_FSID 
               valueFrom: 
                 secretKeyRef: 
                   key: fsid 
                   name: rook-ceph-mon 
             - name: ROOK_LOG_LEVEL 
               value: DEBUG 
           volumeMounts: 
             - mountPath: /etc/ceph 
               name: ceph-conf-emptydir 
             - mountPath: /var/lib/rook 
               name: rook-config 
       volumes: 
         - emptyDir: {} 
           name: ceph-conf-emptydir 
         - emptyDir: {} 
           name: rook-config 
       restartPolicy: Never 


 </pre> 

 h2. Harbor 

 * We user "Harbor":https://goharbor.io/ for caching and as an image registry. Internal app reference: apps/prod/harbor. 
 * The admin password is in the password store, auto generated per cluster 
 * At the moment harbor only authenticates against the internal ldap tree 

 h3. LDAP configuration 

 * The url needs to be ldaps://... 
 * uid = uid 
 * rest standard 

 h2. Monitoring / Prometheus 

 * Via "kube-prometheus":https://github.com/prometheus-operator/kube-prometheus/ 

 Access via ... 

 * http://prometheus-k8s.monitoring.svc:9090 
 * http://grafana.monitoring.svc:3000 
 * http://alertmanager.monitoring.svc:9093 



 h2. Nextcloud 

 h3. How to get the nextcloud credentials  

 * The initial username is set to "nextcloud" 
 * The password is autogenerated and saved in a kubernetes secret 

 <pre> 
 kubectl get secret RELEASENAME-nextcloud -o jsonpath="{.data.PASSWORD}" | base64 -d; echo ""  
 </pre> 

 h3. How to fix "Access through untrusted domain" 

 * Nextcloud stores the initial domain configuration 
 * If the FQDN is changed, it will show the error message "Access through untrusted domain" 
 * To fix, edit /var/www/html/config/config.php and correct the domain 
 * Then delete the pods 

 h2. Infrastructure versions 

 h3. ungleich kubernetes infrastructure v5 (2021-10) 

 Clusters are configured / setup in this order: 

 * Bootstrap via kubeadm 
 * "Networking via calico + BGP (non ECMP) using helm":https://docs.projectcalico.org/getting-started/kubernetes/helm 
 * "ArgoCD for CD":https://argo-cd.readthedocs.io/en/stable/ 
 ** "rook for storage via argocd":https://rook.io/ 
 ** haproxy for in IPv6-cluster-IPv4-to-IPv6 proxy via argocd 
 ** "kubernetes-secret-generator for in cluster secrets":https://github.com/mittwald/kubernetes-secret-generator 
 ** "ungleich-certbot managing certs and nginx":https://hub.docker.com/repository/docker/ungleich/ungleich-certbot 


 h3. ungleich kubernetes infrastructure v4 (2021-09) 

 * rook is configured via manifests instead of using the rook-ceph-cluster helm chart 
 * The rook operator is still being installed via helm 

 h3. ungleich kubernetes infrastructure v3 (2021-07) 

 * rook is now installed via helm via argocd instead of directly via manifests 

 h3. ungleich kubernetes infrastructure v2 (2021-05) 

 * Replaced fluxv2 from ungleich k8s v1 with argocd 
 ** argocd can apply helm templates directly without needing to go through Chart releases 
 * We are also using argoflow for build flows 
 * Planned to add "kaniko":https://github.com/GoogleContainerTools/kaniko for image building 

 h3. ungleich kubernetes infrastructure v1 (2021-01) 

 We are using the following components: 

 * "Calico as a CNI":https://www.projectcalico.org/ with BGP, IPv6 only, no encapsulation 
 ** Needed for basic networking 
 * "kubernetes-secret-generator":https://github.com/mittwald/kubernetes-secret-generator for creating secrets 
 ** Needed so that secrets are not stored in the git repository, but only in the cluster 
 * "ungleich-certbot":https://hub.docker.com/repository/docker/ungleich/ungleich-certbot 
 ** Needed to get letsencrypt certificates for services 
 * "rook with ceph rbd + cephfs":https://rook.io/ for storage 
 ** rbd for almost everything, *ReadWriteOnce* 
 ** cephfs for smaller things, multi access *ReadWriteMany* 
 ** Needed for providing persistent storage 
 * "flux v2":https://fluxcd.io/ 
 ** Needed to manage resources automatically