Project

General

Profile

The ungleich kubernetes infrastructure » History » Version 25

Nico Schottelius, 08/14/2021 07:14 PM

1 22 Nico Schottelius
h1. The ungleich kubernetes infrastructure and ungleich kubernetes manual
2 1 Nico Schottelius
3 3 Nico Schottelius
{{toc}}
4
5 1 Nico Schottelius
h2. Status
6
7 22 Nico Schottelius
This document is **pre-production**. This document is to become the ungleich kubernetes infrastructure overview as well as the ungleich kubernetes manual.
8 1 Nico Schottelius
9 10 Nico Schottelius
h2. k8s clusters
10
11 25 Nico Schottelius
| Cluster     | Purpose       | Notes |
12
| c0.k8s.ooo  | Dev           |       |
13
| c1.k8s.ooo  | Dev p6        |       |
14
| c2.k8s.ooo  | Demo/Semiprod |       |
15
| c3.k8s.ooo  | Dev p6        |       |
16
| c4.k8s.ooo  | active-dev p7 |       |
17
| p6.k8s.ooo  | planned       |       |
18
| p10.k8s.ooo | production    |       |
19
20
Typical init procedure:
21
22
* Single control plane: @kubeadm init --config bootstrap/XXX/kubeadm.yaml@
23
* Multi control plane (HA): @kubeadm init --config bootstrap/XXX/kubeadm.yaml --upload-certs@
24 21 Nico Schottelius
25 1 Nico Schottelius
h2. General architecture and components overview
26
27
* All k8s clusters are IPv6 only
28
* We use BGP peering to propagate podcidr and serviceCidr networks to our infrastructure
29 2 Nico Schottelius
* The main / public repository is "ungleich-k8s":https://code.ungleich.ch/ungleich-public/ungleich-k8s
30 1 Nico Schottelius
31 23 Nico Schottelius
h2. ungleich kubernetes infrastructure v1
32 1 Nico Schottelius
33
We are using the following components:
34
35
* "Calico as a CNI":https://www.projectcalico.org/ with BGP, IPv6 only, no encapsulation
36 18 Nico Schottelius
** Needed for basic networking
37 1 Nico Schottelius
* "kubernetes-secret-generator":https://github.com/mittwald/kubernetes-secret-generator for creating secrets
38 18 Nico Schottelius
** Needed so that secrets are not stored in the git repository, but only in the cluster
39
* "ungleich-certbot":https://hub.docker.com/repository/docker/ungleich/ungleich-certbot 
40
** Needed to get letsencrypt certificates for services
41 2 Nico Schottelius
* "rook with ceph rbd + cephfs":https://rook.io/ for storage
42 1 Nico Schottelius
** rbd for almost everything, *ReadWriteOnce*
43
** cephfs for smaller things, multi access *ReadWriteMany*
44 18 Nico Schottelius
** Needed for providing persistent storage
45
* "flux v2":https://fluxcd.io/ 
46
** Needed to manage resources automatically
47
48 3 Nico Schottelius
49 8 Nico Schottelius
h3. Persistent storage setup 
50
51
* 3 or 5 monitors
52
53 9 Nico Schottelius
h3. Cluster types
54 1 Nico Schottelius
55 9 Nico Schottelius
| **Type/Feature** | **Development** | **Production** |
56
| Min No. nodes | 3 (1 master, 3 worker) | 5 (3 master, 3 worker) |
57
| Recommended minimum | 4 (dedicated master, 3 worker) | 8 (3 master, 5 worker) |
58
| Separation of control plane | optional | recommended |
59
| Persistent storage | required | required |
60
| Number of storage monitors | 3 | 5 |
61 10 Nico Schottelius
62
h2. Operations
63
64
h3. Installing a new k8s cluster
65
66
* Decide on the cluster name (usually *cX.k8s.ooo*), X counting upwards
67
* Use cdist to configure the nodes with requirements like crio
68
* Decide between single or multi node control plane setups (see below)
69
* Setup DNS delegation and glue records:
70
** kube-dns.kube-system.svc.cX AAAA ...
71
** kube-dns.kube-system.svc.cX A ...
72
** cX NS kube-dns.kube-system.svc.cX
73
74 9 Nico Schottelius
75 8 Nico Schottelius
76 3 Nico Schottelius
h2. Open Issues / To be discussed
77
78
* "Maybe add Autoscaling support?":https://github.com/kubernetes-sigs/metrics-server
79
** https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/
80
** https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler/
81
* Certainly deploy in-cluster monitoring
82
** "prometheus-operator":https://github.com/prometheus-operator/prometheus-operator CR
83
** "kube-prometheus":https://github.com/prometheus-operator/kube-prometheus complete example, based on prometheus-operator
84 4 Nico Schottelius
** "kubernetes dashboard":https://github.com/kubernetes/dashboard generic cluster overview, basically kubectl for a broswer + graphs
85 5 Nico Schottelius
** "kube-prometheus-stack via helm":https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack#kube-prometheus-stack
86
*** Looks most fitting, testing it in #9468
87 7 Nico Schottelius
* Matrix/Notification bot
88
** Informing about changes in the cluster