Project

General

Profile

The ungleich kubernetes infrastructure » History » Version 43

Nico Schottelius, 10/06/2021 09:30 AM

1 22 Nico Schottelius
h1. The ungleich kubernetes infrastructure and ungleich kubernetes manual
2 1 Nico Schottelius
3 3 Nico Schottelius
{{toc}}
4
5 1 Nico Schottelius
h2. Status
6
7 28 Nico Schottelius
This document is **pre-production**.
8
This document is to become the ungleich kubernetes infrastructure overview as well as the ungleich kubernetes manual.
9 1 Nico Schottelius
10 10 Nico Schottelius
h2. k8s clusters
11
12 40 Nico Schottelius
| Cluster     | Purpose/Setup  | Maintainer   | Master(s)                                                | last verified |
13
| c0.k8s.ooo  | Dev            | -            | UNUSED                                                   |    2021-10-05 |
14
| c1.k8s.ooo  | Dev p6 VM      | Nico         | 2a0a-e5c0-2-11-0-62ff-fe0b-1a3d.k8s-1.place6.ungleich.ch |    2021-10-05 |
15
| c2.k8s.ooo  | Dev p7 HW      | Nico         | server47 server53 server54                               |    2021-10-05 |
16
| c3.k8s.ooo  | Test p7 PI     | -            | UNUSED                                                   |    2021-10-05 |
17
| c4.k8s.ooo  | Dev2 p7 HW     | Fran/Jin-Guk | server52 server53 server54                               |             - |
18
| c5.k8s.ooo  | Dev p6 VM Amal | Nico/Amal    | 2a0a-e5c0-2-11-0-62ff-fe0b-1a46.k8s-1.place6.ungleich.ch |               |
19
| p6.k8s.ooo  | production     |              | server67 server69 server71                               |    2021-10-05 |
20
| p10.k8s.ooo | production     |              | server63 server65 server83                               |    2021-10-05 |
21
|             |                |              |                                                          |               |
22 39 Nico Schottelius
23 21 Nico Schottelius
24 1 Nico Schottelius
h2. General architecture and components overview
25
26
* All k8s clusters are IPv6 only
27
* We use BGP peering to propagate podcidr and serviceCidr networks to our infrastructure
28
* The main public testing repository is "ungleich-k8s":https://code.ungleich.ch/ungleich-public/ungleich-k8s
29 18 Nico Schottelius
** Private configurations are found in the **k8s-config** repository
30 1 Nico Schottelius
31
h3. Cluster types
32
33 28 Nico Schottelius
| **Type/Feature**            | **Development**                | **Production**         |
34
| Min No. nodes               | 3 (1 master, 3 worker)         | 5 (3 master, 3 worker) |
35
| Recommended minimum         | 4 (dedicated master, 3 worker) | 8 (3 master, 5 worker) |
36
| Separation of control plane | optional                       | recommended            |
37
| Persistent storage          | required                       | required               |
38
| Number of storage monitors  | 3                              | 5                      |
39 1 Nico Schottelius
40 43 Nico Schottelius
h2. General k8s operations
41 1 Nico Schottelius
42 36 Nico Schottelius
h3. Get the argocd credentials
43
44
<pre>
45 37 Nico Schottelius
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo ""
46 36 Nico Schottelius
</pre>
47
48 18 Nico Schottelius
h3. Installing a new k8s cluster
49 8 Nico Schottelius
50 9 Nico Schottelius
* Decide on the cluster name (usually *cX.k8s.ooo*), X counting upwards
51 28 Nico Schottelius
** Using pXX.k8s.ooo for production clusters of placeXX
52 9 Nico Schottelius
* Use cdist to configure the nodes with requirements like crio
53
* Decide between single or multi node control plane setups (see below)
54 28 Nico Schottelius
** Single control plane suitable for development clusters
55 9 Nico Schottelius
56 28 Nico Schottelius
Typical init procedure:
57 9 Nico Schottelius
58 28 Nico Schottelius
* Single control plane: @kubeadm init --config bootstrap/XXX/kubeadm.yaml@
59
* Multi control plane (HA): @kubeadm init --config bootstrap/XXX/kubeadm.yaml --upload-certs@
60 10 Nico Schottelius
61 29 Nico Schottelius
h3. Deleting a pod that is hanging in terminating state
62
63
<pre>
64
kubectl delete pod <PODNAME> --grace-period=0 --force --namespace <NAMESPACE>
65
</pre>
66
67
(from https://stackoverflow.com/questions/35453792/pods-stuck-in-terminating-status)
68
69 42 Nico Schottelius
h3. Listing nodes of a cluster
70
71
<pre>
72
[15:05] bridge:~% kubectl get nodes
73
NAME       STATUS   ROLES                  AGE   VERSION
74
server22   Ready    <none>                 52d   v1.22.0
75
server23   Ready    <none>                 52d   v1.22.2
76
server24   Ready    <none>                 52d   v1.22.0
77
server25   Ready    <none>                 52d   v1.22.0
78
server26   Ready    <none>                 52d   v1.22.0
79
server27   Ready    <none>                 52d   v1.22.0
80
server63   Ready    control-plane,master   52d   v1.22.0
81
server64   Ready    <none>                 52d   v1.22.0
82
server65   Ready    control-plane,master   52d   v1.22.0
83
server66   Ready    <none>                 52d   v1.22.0
84
server83   Ready    control-plane,master   52d   v1.22.0
85
server84   Ready    <none>                 52d   v1.22.0
86
server85   Ready    <none>                 52d   v1.22.0
87
server86   Ready    <none>                 52d   v1.22.0
88
</pre>
89
90
91 41 Nico Schottelius
h3. Removing / draining a node
92
93
Usually @kubectl drain server@ should do the job, but sometimes we need to be more aggressive:
94
95
<pre>
96
kubectl drain --delete-emptydir-data --ignore-daemonsets server23
97 42 Nico Schottelius
</pre>
98
99
h3. Readding a node after draining
100
101
<pre>
102
kubectl uncordon serverXX
103 1 Nico Schottelius
</pre>
104 43 Nico Schottelius
105
h2. Rook / Ceph Related Operations
106
107
h3. Inspecting the logs of a specific server
108
109
<pre>
110
# Get the related pods
111
kubectl -n rook-ceph get pods -l app=rook-ceph-osd-prepare 
112
...
113
114
# Inspect the logs of a specific pod
115
kubectl -n rook-ceph logs -f rook-ceph-osd-prepare-server23--1-444qx
116
117
</pre>
118
119
h3. Triggering server prepare / adding new osds
120
121
The rook-ceph-operator triggers/watches/creates pods to maintain hosts. To trigger a full "re scan", simply delete that pod:
122
123
<pre>
124
kubectl -n rook-ceph delete pods -l app=rook-ceph-operator
125
</pre>
126
127
This will cause all the @rook-ceph-osd-prepare-..@ jobs to be recreated and thus OSDs to be created, if new disks have been added.
128
129
h3. Removing an OSD
130
131
* See "Ceph OSD Management":https://rook.io/docs/rook/v1.7/ceph-osd-mgmt.html
132 41 Nico Schottelius
133 1 Nico Schottelius
h2. Infrastructure versions
134 35 Nico Schottelius
135
h3. ungleich kubernetes infrastructure v3
136
137
* rook is now installed via helm via argocd instead of directly via manifests
138 10 Nico Schottelius
139 28 Nico Schottelius
h3. ungleich kubernetes infrastructure v2
140
141
* Replaced fluxv2 from ungleich k8s v1 with argocd
142
** argocd can apply helm templates directly without needing to go through Chart releases
143
* We are also using argoflow for build flows
144
* Planned to add "kaniko":https://github.com/GoogleContainerTools/kaniko for image building
145
146
h3. ungleich kubernetes infrastructure v1
147
148
We are using the following components:
149
150
* "Calico as a CNI":https://www.projectcalico.org/ with BGP, IPv6 only, no encapsulation
151
** Needed for basic networking
152
* "kubernetes-secret-generator":https://github.com/mittwald/kubernetes-secret-generator for creating secrets
153
** Needed so that secrets are not stored in the git repository, but only in the cluster
154
* "ungleich-certbot":https://hub.docker.com/repository/docker/ungleich/ungleich-certbot
155
** Needed to get letsencrypt certificates for services
156
* "rook with ceph rbd + cephfs":https://rook.io/ for storage
157
** rbd for almost everything, *ReadWriteOnce*
158
** cephfs for smaller things, multi access *ReadWriteMany*
159
** Needed for providing persistent storage
160
* "flux v2":https://fluxcd.io/
161
** Needed to manage resources automatically