Open Infrastructure

h1. The ungleich kubernetes infrastructure

{{toc}}

h2. Status

This document is **pre-production**

h2. k8s clusters

| Cluster | Purpose | Init/Notes | flux |

| c0.k8s.ooo | Dev | @kubeadm init --config k8s/c0/kubeadm.yaml --upload-certs@ |

| c1.k8s.ooo | Dev p6 | VM based | @kubeadm init --config k8s/c1/kubeadm.yaml@  | 

| c2.k8s.ooo | Demo/Semiprod | @kubeadm init --config k8s/c2/kubeadm.yaml --upload-certs@ | @flux bootstrap git --url=ssh://git@code.ungleich.ch/ungleich-intern/k8s-config.git --path=cluster/c2 --network-policy false --cluster-domain c2.k8s.ooo@ |

| c3.k8s.ooo | Dev p6 | VM based |

| c4.k8s.ooo | Dev p7 | Maintained by Fran | |

| p6.k8s.ooo | Prod | Place6 |

| p7.k8s.ooo | Prod | Place7 |

| p10.k8s.ooo | Prod | Place10: @kubeadm init --config bootstrap/p10/kubeadm.yaml --upload-certs@ |

h2. General architecture and components overview

* All k8s clusters are IPv6 only

* We use BGP peering to propagate podcidr and serviceCidr networks to our infrastructure

* The main / public repository is "ungleich-k8s":https://code.ungleich.ch/ungleich-public/ungleich-k8s

th2. ungleich kubernetes infrastructure v1

We are using the following components:

* "Calico as a CNI":https://www.projectcalico.org/ with BGP, IPv6 only, no encapsulation

** Needed for basic networking

* "kubernetes-secret-generator":https://github.com/mittwald/kubernetes-secret-generator for creating secrets

** Needed so that secrets are not stored in the git repository, but only in the cluster

* "ungleich-certbot":https://hub.docker.com/repository/docker/ungleich/ungleich-certbot 

** Needed to get letsencrypt certificates for services

* "rook with ceph rbd + cephfs":https://rook.io/ for storage

** rbd for almost everything, *ReadWriteOnce*

** cephfs for smaller things, multi access *ReadWriteMany*

** Needed for providing persistent storage

* "flux v2":https://fluxcd.io/ 

** Needed to manage resources automatically

h3. Persistent storage setup 

* 3 or 5 monitors

h3. Cluster types

| **Type/Feature** | **Development** | **Production** |

| Min No. nodes | 3 (1 master, 3 worker) | 5 (3 master, 3 worker) |

| Recommended minimum | 4 (dedicated master, 3 worker) | 8 (3 master, 5 worker) |

| Separation of control plane | optional | recommended |

| Persistent storage | required | required |

| Number of storage monitors | 3 | 5 |

h2. Operations

h3. Installing a new k8s cluster

* Decide on the cluster name (usually *cX.k8s.ooo*), X counting upwards

* Use cdist to configure the nodes with requirements like crio

* Decide between single or multi node control plane setups (see below)

* Setup DNS delegation and glue records:

** kube-dns.kube-system.svc.cX AAAA ...

** kube-dns.kube-system.svc.cX A ...

** cX NS kube-dns.kube-system.svc.cX

h2. Open Issues / To be discussed

* "Maybe add Autoscaling support?":https://github.com/kubernetes-sigs/metrics-server

** https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/

** https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler/

* Certainly deploy in-cluster monitoring

** "prometheus-operator":https://github.com/prometheus-operator/prometheus-operator CR

** "kube-prometheus":https://github.com/prometheus-operator/kube-prometheus complete example, based on prometheus-operator

** "kubernetes dashboard":https://github.com/kubernetes/dashboard generic cluster overview, basically kubectl for a broswer + graphs

** "kube-prometheus-stack via helm":https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack#kube-prometheus-stack

*** Looks most fitting, testing it in #9468

* Matrix/Notification bot

** Informing about changes in the cluster