Project

General

Profile

The ungleich kubernetes infrastructure » History » Version 11

Nico Schottelius, 07/11/2021 01:53 PM

1 1 Nico Schottelius
h1. The ungleich kubernetes infrastructure
2
3 3 Nico Schottelius
{{toc}}
4
5 1 Nico Schottelius
h2. Status
6
7
This document is **pre-production**
8
9 10 Nico Schottelius
h2. k8s clusters
10
11
| Cluster | Init |
12
| c0.k8s.ooo | @kubeadm init --config k8s/c0/kubeadm.yaml --upload-certs@ |
13 11 Nico Schottelius
| c2.k8s.ooo | @kubeadm init --config k8s/c2/kubeadm.yaml@ |
14 10 Nico Schottelius
15 1 Nico Schottelius
h2. General architecture and components overview
16
17
* All k8s clusters are IPv6 only
18
* We use BGP peering to propagate podcidr and serviceCidr networks to our infrastructure
19 2 Nico Schottelius
* The main / public repository is "ungleich-k8s":https://code.ungleich.ch/ungleich-public/ungleich-k8s
20 1 Nico Schottelius
21
h2. ungleich kubernetes infrastructure v1
22
23
We are using the following components:
24
25 6 Nico Schottelius
* "Calico as a CNI":https://www.projectcalico.org/ with BGP, IPv6 only, no encapsulation
26 1 Nico Schottelius
* "kubernetes-secret-generator":https://github.com/mittwald/kubernetes-secret-generator for creating secrets
27
* "ungleich-certbot":https://hub.docker.com/repository/docker/ungleich/ungleich-certbot to get certificates
28 2 Nico Schottelius
* "rook with ceph rbd + cephfs":https://rook.io/ for storage
29
** rbd for almost everything, *ReadWriteOnce*
30
** cephfs for smaller things, multi access *ReadWriteMany*
31 3 Nico Schottelius
32 8 Nico Schottelius
h3. Persistent storage setup 
33
34
* 3 or 5 monitors
35
36 9 Nico Schottelius
h3. Cluster types
37 1 Nico Schottelius
38 9 Nico Schottelius
| **Type/Feature** | **Development** | **Production** |
39
| Min No. nodes | 3 (1 master, 3 worker) | 5 (3 master, 3 worker) |
40
| Recommended minimum | 4 (dedicated master, 3 worker) | 8 (3 master, 5 worker) |
41
| Separation of control plane | optional | recommended |
42
| Persistent storage | required | required |
43
| Number of storage monitors | 3 | 5 |
44 10 Nico Schottelius
45
h2. Operations
46
47
h3. Installing a new k8s cluster
48
49
* Decide on the cluster name (usually *cX.k8s.ooo*), X counting upwards
50
* Use cdist to configure the nodes with requirements like crio
51
* Decide between single or multi node control plane setups (see below)
52
* Setup DNS delegation and glue records:
53
** kube-dns.kube-system.svc.cX AAAA ...
54
** kube-dns.kube-system.svc.cX A ...
55
** cX NS kube-dns.kube-system.svc.cX
56
57 9 Nico Schottelius
58 8 Nico Schottelius
59 3 Nico Schottelius
h2. Open Issues / To be discussed
60
61
* "Maybe add Autoscaling support?":https://github.com/kubernetes-sigs/metrics-server
62
** https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/
63
** https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler/
64
* Certainly deploy in-cluster monitoring
65
** "prometheus-operator":https://github.com/prometheus-operator/prometheus-operator CR
66
** "kube-prometheus":https://github.com/prometheus-operator/kube-prometheus complete example, based on prometheus-operator
67 4 Nico Schottelius
** "kubernetes dashboard":https://github.com/kubernetes/dashboard generic cluster overview, basically kubectl for a broswer + graphs
68 5 Nico Schottelius
** "kube-prometheus-stack via helm":https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack#kube-prometheus-stack
69
*** Looks most fitting, testing it in #9468
70 7 Nico Schottelius
* Matrix/Notification bot
71
** Informing about changes in the cluster