Project

General

Profile

The ungleich kubernetes infrastructure » History » Version 40

Nico Schottelius, 10/05/2021 09:29 AM

1 22 Nico Schottelius
h1. The ungleich kubernetes infrastructure and ungleich kubernetes manual
2 1 Nico Schottelius
3 3 Nico Schottelius
{{toc}}
4
5 1 Nico Schottelius
h2. Status
6
7 28 Nico Schottelius
This document is **pre-production**.
8
This document is to become the ungleich kubernetes infrastructure overview as well as the ungleich kubernetes manual.
9 1 Nico Schottelius
10 10 Nico Schottelius
h2. k8s clusters
11
12 40 Nico Schottelius
| Cluster     | Purpose/Setup  | Maintainer   | Master(s)                                                | last verified |
13
| c0.k8s.ooo  | Dev            | -            | UNUSED                                                   |    2021-10-05 |
14
| c1.k8s.ooo  | Dev p6 VM      | Nico         | 2a0a-e5c0-2-11-0-62ff-fe0b-1a3d.k8s-1.place6.ungleich.ch |    2021-10-05 |
15
| c2.k8s.ooo  | Dev p7 HW      | Nico         | server47 server53 server54                               |    2021-10-05 |
16
| c3.k8s.ooo  | Test p7 PI     | -            | UNUSED                                                   |    2021-10-05 |
17
| c4.k8s.ooo  | Dev2 p7 HW     | Fran/Jin-Guk | server52 server53 server54                               |             - |
18
| c5.k8s.ooo  | Dev p6 VM Amal | Nico/Amal    | 2a0a-e5c0-2-11-0-62ff-fe0b-1a46.k8s-1.place6.ungleich.ch |               |
19
| p6.k8s.ooo  | production     |              | server67 server69 server71                               |    2021-10-05 |
20
| p10.k8s.ooo | production     |              | server63 server65 server83                               |    2021-10-05 |
21
|             |                |              |                                                          |               |
22 39 Nico Schottelius
23 21 Nico Schottelius
24 1 Nico Schottelius
h2. General architecture and components overview
25
26
* All k8s clusters are IPv6 only
27
* We use BGP peering to propagate podcidr and serviceCidr networks to our infrastructure
28
* The main public testing repository is "ungleich-k8s":https://code.ungleich.ch/ungleich-public/ungleich-k8s
29 18 Nico Schottelius
** Private configurations are found in the **k8s-config** repository
30 1 Nico Schottelius
31
h3. Cluster types
32
33 28 Nico Schottelius
| **Type/Feature**            | **Development**                | **Production**         |
34
| Min No. nodes               | 3 (1 master, 3 worker)         | 5 (3 master, 3 worker) |
35
| Recommended minimum         | 4 (dedicated master, 3 worker) | 8 (3 master, 5 worker) |
36
| Separation of control plane | optional                       | recommended            |
37
| Persistent storage          | required                       | required               |
38
| Number of storage monitors  | 3                              | 5                      |
39 1 Nico Schottelius
40
h2. Operations
41
42 36 Nico Schottelius
h3. Get the argocd credentials
43
44
<pre>
45 37 Nico Schottelius
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo ""
46 36 Nico Schottelius
</pre>
47
48 18 Nico Schottelius
h3. Installing a new k8s cluster
49 8 Nico Schottelius
50 9 Nico Schottelius
* Decide on the cluster name (usually *cX.k8s.ooo*), X counting upwards
51 28 Nico Schottelius
** Using pXX.k8s.ooo for production clusters of placeXX
52 9 Nico Schottelius
* Use cdist to configure the nodes with requirements like crio
53
* Decide between single or multi node control plane setups (see below)
54 28 Nico Schottelius
** Single control plane suitable for development clusters
55 9 Nico Schottelius
56 28 Nico Schottelius
Typical init procedure:
57 9 Nico Schottelius
58 28 Nico Schottelius
* Single control plane: @kubeadm init --config bootstrap/XXX/kubeadm.yaml@
59
* Multi control plane (HA): @kubeadm init --config bootstrap/XXX/kubeadm.yaml --upload-certs@
60 10 Nico Schottelius
61 29 Nico Schottelius
h3. Deleting a pod that is hanging in terminating state
62
63
<pre>
64
kubectl delete pod <PODNAME> --grace-period=0 --force --namespace <NAMESPACE>
65
</pre>
66
67
(from https://stackoverflow.com/questions/35453792/pods-stuck-in-terminating-status)
68
69 1 Nico Schottelius
h2. Infrastructure versions
70 35 Nico Schottelius
71
h3. ungleich kubernetes infrastructure v3
72
73
* rook is now installed via helm via argocd instead of directly via manifests
74 10 Nico Schottelius
75 28 Nico Schottelius
h3. ungleich kubernetes infrastructure v2
76
77
* Replaced fluxv2 from ungleich k8s v1 with argocd
78
** argocd can apply helm templates directly without needing to go through Chart releases
79
* We are also using argoflow for build flows
80
* Planned to add "kaniko":https://github.com/GoogleContainerTools/kaniko for image building
81
82
h3. ungleich kubernetes infrastructure v1
83
84
We are using the following components:
85
86
* "Calico as a CNI":https://www.projectcalico.org/ with BGP, IPv6 only, no encapsulation
87
** Needed for basic networking
88
* "kubernetes-secret-generator":https://github.com/mittwald/kubernetes-secret-generator for creating secrets
89
** Needed so that secrets are not stored in the git repository, but only in the cluster
90
* "ungleich-certbot":https://hub.docker.com/repository/docker/ungleich/ungleich-certbot
91
** Needed to get letsencrypt certificates for services
92
* "rook with ceph rbd + cephfs":https://rook.io/ for storage
93
** rbd for almost everything, *ReadWriteOnce*
94
** cephfs for smaller things, multi access *ReadWriteMany*
95
** Needed for providing persistent storage
96
* "flux v2":https://fluxcd.io/
97
** Needed to manage resources automatically