Project

General

Profile

The ungleich kubernetes infrastructure » History » Version 36

Nico Schottelius, 09/04/2021 08:54 PM

1 22 Nico Schottelius
h1. The ungleich kubernetes infrastructure and ungleich kubernetes manual
2 1 Nico Schottelius
3 3 Nico Schottelius
{{toc}}
4
5 1 Nico Schottelius
h2. Status
6
7 28 Nico Schottelius
This document is **pre-production**.
8
This document is to become the ungleich kubernetes infrastructure overview as well as the ungleich kubernetes manual.
9 1 Nico Schottelius
10 10 Nico Schottelius
h2. k8s clusters
11
12 30 Nico Schottelius
| Cluster     | Purpose       | Master(s) |
13
| c0.k8s.ooo  | Dev           | |
14
| c1.k8s.ooo  | Dev p6        | 2a0a-e5c0-2-11-0-62ff-fe0b-1a3d.k8s-1.place6.ungleich.ch |
15
| c2.k8s.ooo  | Demo place7   | server47 server53 server54 | 
16 33 Nico Schottelius
| c3.k8s.ooo  | Test p7       | Raspberry PIs (inactive of 2021-08-20) |
17 1 Nico Schottelius
| c4.k8s.ooo  | active-dev p7 | server52 server53 server54 |
18 32 Nico Schottelius
| c5.k8s.ooo  | Development Amal | 2a0a-e5c0-2-11-0-62ff-fe0b-1a46.k8s-1.place6.ungleich.ch |
19 34 Nico Schottelius
| p6.k8s.ooo  | production    | |
20
| p10.k8s.ooo | production    | server63 server65 server83 |
21 21 Nico Schottelius
22 1 Nico Schottelius
h2. General architecture and components overview
23
24
* All k8s clusters are IPv6 only
25
* We use BGP peering to propagate podcidr and serviceCidr networks to our infrastructure
26
* The main public testing repository is "ungleich-k8s":https://code.ungleich.ch/ungleich-public/ungleich-k8s
27 18 Nico Schottelius
** Private configurations are found in the **k8s-config** repository
28 1 Nico Schottelius
29
h3. Cluster types
30
31 28 Nico Schottelius
| **Type/Feature**            | **Development**                | **Production**         |
32
| Min No. nodes               | 3 (1 master, 3 worker)         | 5 (3 master, 3 worker) |
33
| Recommended minimum         | 4 (dedicated master, 3 worker) | 8 (3 master, 5 worker) |
34
| Separation of control plane | optional                       | recommended            |
35
| Persistent storage          | required                       | required               |
36
| Number of storage monitors  | 3                              | 5                      |
37 1 Nico Schottelius
38
h2. Operations
39
40 36 Nico Schottelius
h3. Get the argocd credentials
41
42
<pre>
43
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d
44
</pre>
45
46 18 Nico Schottelius
h3. Installing a new k8s cluster
47 8 Nico Schottelius
48 9 Nico Schottelius
* Decide on the cluster name (usually *cX.k8s.ooo*), X counting upwards
49 28 Nico Schottelius
** Using pXX.k8s.ooo for production clusters of placeXX
50 9 Nico Schottelius
* Use cdist to configure the nodes with requirements like crio
51
* Decide between single or multi node control plane setups (see below)
52 28 Nico Schottelius
** Single control plane suitable for development clusters
53 9 Nico Schottelius
54 28 Nico Schottelius
Typical init procedure:
55 9 Nico Schottelius
56 28 Nico Schottelius
* Single control plane: @kubeadm init --config bootstrap/XXX/kubeadm.yaml@
57
* Multi control plane (HA): @kubeadm init --config bootstrap/XXX/kubeadm.yaml --upload-certs@
58 10 Nico Schottelius
59 29 Nico Schottelius
h3. Deleting a pod that is hanging in terminating state
60
61
<pre>
62
kubectl delete pod <PODNAME> --grace-period=0 --force --namespace <NAMESPACE>
63
</pre>
64
65
(from https://stackoverflow.com/questions/35453792/pods-stuck-in-terminating-status)
66
67 1 Nico Schottelius
h2. Infrastructure versions
68 35 Nico Schottelius
69
h3. ungleich kubernetes infrastructure v3
70
71
* rook is now installed via helm via argocd instead of directly via manifests
72 10 Nico Schottelius
73 28 Nico Schottelius
h3. ungleich kubernetes infrastructure v2
74
75
* Replaced fluxv2 from ungleich k8s v1 with argocd
76
** argocd can apply helm templates directly without needing to go through Chart releases
77
* We are also using argoflow for build flows
78
* Planned to add "kaniko":https://github.com/GoogleContainerTools/kaniko for image building
79
80
h3. ungleich kubernetes infrastructure v1
81
82
We are using the following components:
83
84
* "Calico as a CNI":https://www.projectcalico.org/ with BGP, IPv6 only, no encapsulation
85
** Needed for basic networking
86
* "kubernetes-secret-generator":https://github.com/mittwald/kubernetes-secret-generator for creating secrets
87
** Needed so that secrets are not stored in the git repository, but only in the cluster
88
* "ungleich-certbot":https://hub.docker.com/repository/docker/ungleich/ungleich-certbot
89
** Needed to get letsencrypt certificates for services
90
* "rook with ceph rbd + cephfs":https://rook.io/ for storage
91
** rbd for almost everything, *ReadWriteOnce*
92
** cephfs for smaller things, multi access *ReadWriteMany*
93
** Needed for providing persistent storage
94
* "flux v2":https://fluxcd.io/
95
** Needed to manage resources automatically