Open Infrastructure

h1. The ungleich kubernetes infrastructure and ungleich kubernetes manual

{{toc}}

h2. Status

This document is **pre-production**.

This document is to become the ungleich kubernetes infrastructure overview as well as the ungleich kubernetes manual.

h2. k8s clusters

| Cluster     | Purpose       | Master(s) |

| c0.k8s.ooo  | Dev           | |

| c1.k8s.ooo  | Dev p6        | 2a0a-e5c0-2-11-0-62ff-fe0b-1a3d.k8s-1.place6.ungleich.ch |

| c2.k8s.ooo  | Demo place7   | server47 server53 server54 | 

| c3.k8s.ooo  | Test p7       | Raspberry PIs (inactive of 2021-08-20) |

| c4.k8s.ooo  | active-dev p7 | server52 server53 server54 |

| c5.k8s.ooo  | Development Amal | 2a0a-e5c0-2-11-0-62ff-fe0b-1a46.k8s-1.place6.ungleich.ch |

| p6.k8s.ooo  | production    | |

| p10.k8s.ooo | production    | server63 server65 server83 |

h2. General architecture and components overview

* All k8s clusters are IPv6 only

* We use BGP peering to propagate podcidr and serviceCidr networks to our infrastructure

* The main public testing repository is "ungleich-k8s":https://code.ungleich.ch/ungleich-public/ungleich-k8s

** Private configurations are found in the **k8s-config** repository

h3. Cluster types

| **Type/Feature**            | **Development**                | **Production**         |

| Min No. nodes               | 3 (1 master, 3 worker)         | 5 (3 master, 3 worker) |

| Recommended minimum         | 4 (dedicated master, 3 worker) | 8 (3 master, 5 worker) |

| Separation of control plane | optional                       | recommended            |

| Persistent storage          | required                       | required               |

| Number of storage monitors  | 3                              | 5                      |

h2. Operations

h3. Installing a new k8s cluster

* Decide on the cluster name (usually *cX.k8s.ooo*), X counting upwards

** Using pXX.k8s.ooo for production clusters of placeXX

* Use cdist to configure the nodes with requirements like crio

* Decide between single or multi node control plane setups (see below)

** Single control plane suitable for development clusters

Typical init procedure:

* Single control plane: @kubeadm init --config bootstrap/XXX/kubeadm.yaml@

* Multi control plane (HA): @kubeadm init --config bootstrap/XXX/kubeadm.yaml --upload-certs@

h3. Deleting a pod that is hanging in terminating state

<pre>

kubectl delete pod <PODNAME> --grace-period=0 --force --namespace <NAMESPACE>

</pre>

(from https://stackoverflow.com/questions/35453792/pods-stuck-in-terminating-status)

h2. Infrastructure versions

h3. ungleich kubernetes infrastructure v3

* rook is now installed via helm via argocd instead of directly via manifests

h3. ungleich kubernetes infrastructure v2

* Replaced fluxv2 from ungleich k8s v1 with argocd

** argocd can apply helm templates directly without needing to go through Chart releases

* We are also using argoflow for build flows

* Planned to add "kaniko":https://github.com/GoogleContainerTools/kaniko for image building

h3. ungleich kubernetes infrastructure v1

We are using the following components:

* "Calico as a CNI":https://www.projectcalico.org/ with BGP, IPv6 only, no encapsulation

** Needed for basic networking

* "kubernetes-secret-generator":https://github.com/mittwald/kubernetes-secret-generator for creating secrets

** Needed so that secrets are not stored in the git repository, but only in the cluster

* "ungleich-certbot":https://hub.docker.com/repository/docker/ungleich/ungleich-certbot

** Needed to get letsencrypt certificates for services

* "rook with ceph rbd + cephfs":https://rook.io/ for storage

** rbd for almost everything, *ReadWriteOnce*

** cephfs for smaller things, multi access *ReadWriteMany*

** Needed for providing persistent storage

* "flux v2":https://fluxcd.io/

** Needed to manage resources automatically