Project

General

Profile

The ungleich kubernetes infrastructure » History » Version 65

Nico Schottelius, 11/06/2021 09:50 PM

1 22 Nico Schottelius
h1. The ungleich kubernetes infrastructure and ungleich kubernetes manual
2 1 Nico Schottelius
3 3 Nico Schottelius
{{toc}}
4
5 1 Nico Schottelius
h2. Status
6
7 28 Nico Schottelius
This document is **pre-production**.
8
This document is to become the ungleich kubernetes infrastructure overview as well as the ungleich kubernetes manual.
9 1 Nico Schottelius
10 10 Nico Schottelius
h2. k8s clusters
11
12 65 Nico Schottelius
| Cluster         | Purpose/Setup     | Maintainer   | Master(s)                                                | argo | rook |      v4 proxy | last verified |
13
| c0.k8s.ooo      | Dev               | -            | UNUSED                                                   |      |      |               |    2021-10-05 |
14
| c1.k8s.ooo      | Dev p6 VM         | Nico         | 2a0a-e5c0-2-11-0-62ff-fe0b-1a3d.k8s-1.place6.ungleich.ch |      |      |               |    2021-10-05 |
15
| c2.k8s.ooo      | Dev p7 HW         | Nico         | server47 server53 server54                               | x    | x    |               |    2021-10-05 |
16
| c3.k8s.ooo      | Test p7 PI        | -            | UNUSED                                                   |      |      |               |    2021-10-05 |
17
| c4.k8s.ooo      | Dev2 p7 HW        | Fran/Jin-Guk | server52 server53 server54                               |      |      |               |             - |
18
| c5.k8s.ooo      | Dev p6 VM Amal    | Nico/Amal    | 2a0a-e5c0-2-11-0-62ff-fe0b-1a46.k8s-1.place6.ungleich.ch |      |      |               |               |
19
| c6.k8s.ooo      | Dev p6 VM Jin-Guk | Jin-Guk      |                                                          |      |      |               |               |
20
| [[p6.k8s.ooo]]  | production        |              | server67 server69 server71                               | x    | x    | 147.78.194.13 |    2021-10-05 |
21
| [[p10.k8s.ooo]] | production        |              | server63 server65 server83                               | x    | x    | 147.78.194.12 |    2021-10-05 |
22 48 Nico Schottelius
23 21 Nico Schottelius
24 1 Nico Schottelius
h2. General architecture and components overview
25
26
* All k8s clusters are IPv6 only
27
* We use BGP peering to propagate podcidr and serviceCidr networks to our infrastructure
28
* The main public testing repository is "ungleich-k8s":https://code.ungleich.ch/ungleich-public/ungleich-k8s
29 18 Nico Schottelius
** Private configurations are found in the **k8s-config** repository
30 1 Nico Schottelius
31
h3. Cluster types
32
33 28 Nico Schottelius
| **Type/Feature**            | **Development**                | **Production**         |
34
| Min No. nodes               | 3 (1 master, 3 worker)         | 5 (3 master, 3 worker) |
35
| Recommended minimum         | 4 (dedicated master, 3 worker) | 8 (3 master, 5 worker) |
36
| Separation of control plane | optional                       | recommended            |
37
| Persistent storage          | required                       | required               |
38
| Number of storage monitors  | 3                              | 5                      |
39 1 Nico Schottelius
40 43 Nico Schottelius
h2. General k8s operations
41 1 Nico Schottelius
42 46 Nico Schottelius
h3. Cheat sheet / external great references
43
44
* "kubectl cheatsheet":https://kubernetes.io/docs/reference/kubectl/cheatsheet/
45
46 44 Nico Schottelius
h3. Get the cluster admin.conf
47
48
* On the masters of each cluster you can find the file @/etc/kubernetes/admin.conf@
49
* To be able to administrate the cluster you can copy the admin.conf to your local machine
50
* Multi cluster debugging can very easy if you name the config ~/cX-admin.conf (see example below)
51
52
<pre>
53
% scp root@server47.place7.ungleich.ch:/etc/kubernetes/admin.conf ~/c2-admin.conf
54
% export KUBECONFIG=~/c2-admin.conf    
55
% kubectl get nodes
56
NAME       STATUS                     ROLES                  AGE   VERSION
57
server47   Ready                      control-plane,master   82d   v1.22.0
58
server48   Ready                      control-plane,master   82d   v1.22.0
59
server49   Ready                      <none>                 82d   v1.22.0
60
server50   Ready                      <none>                 82d   v1.22.0
61
server59   Ready                      control-plane,master   82d   v1.22.0
62
server60   Ready,SchedulingDisabled   <none>                 82d   v1.22.0
63
server61   Ready                      <none>                 82d   v1.22.0
64
server62   Ready                      <none>                 82d   v1.22.0               
65
</pre>
66
67 18 Nico Schottelius
h3. Installing a new k8s cluster
68 8 Nico Schottelius
69 9 Nico Schottelius
* Decide on the cluster name (usually *cX.k8s.ooo*), X counting upwards
70 28 Nico Schottelius
** Using pXX.k8s.ooo for production clusters of placeXX
71 9 Nico Schottelius
* Use cdist to configure the nodes with requirements like crio
72
* Decide between single or multi node control plane setups (see below)
73 28 Nico Schottelius
** Single control plane suitable for development clusters
74 9 Nico Schottelius
75 28 Nico Schottelius
Typical init procedure:
76 9 Nico Schottelius
77 28 Nico Schottelius
* Single control plane: @kubeadm init --config bootstrap/XXX/kubeadm.yaml@
78
* Multi control plane (HA): @kubeadm init --config bootstrap/XXX/kubeadm.yaml --upload-certs@
79 10 Nico Schottelius
80 29 Nico Schottelius
h3. Deleting a pod that is hanging in terminating state
81
82
<pre>
83
kubectl delete pod <PODNAME> --grace-period=0 --force --namespace <NAMESPACE>
84
</pre>
85
86
(from https://stackoverflow.com/questions/35453792/pods-stuck-in-terminating-status)
87
88 42 Nico Schottelius
h3. Listing nodes of a cluster
89
90
<pre>
91
[15:05] bridge:~% kubectl get nodes
92
NAME       STATUS   ROLES                  AGE   VERSION
93
server22   Ready    <none>                 52d   v1.22.0
94
server23   Ready    <none>                 52d   v1.22.2
95
server24   Ready    <none>                 52d   v1.22.0
96
server25   Ready    <none>                 52d   v1.22.0
97
server26   Ready    <none>                 52d   v1.22.0
98
server27   Ready    <none>                 52d   v1.22.0
99
server63   Ready    control-plane,master   52d   v1.22.0
100
server64   Ready    <none>                 52d   v1.22.0
101
server65   Ready    control-plane,master   52d   v1.22.0
102
server66   Ready    <none>                 52d   v1.22.0
103
server83   Ready    control-plane,master   52d   v1.22.0
104
server84   Ready    <none>                 52d   v1.22.0
105
server85   Ready    <none>                 52d   v1.22.0
106
server86   Ready    <none>                 52d   v1.22.0
107
</pre>
108
109
110 41 Nico Schottelius
h3. Removing / draining a node
111
112
Usually @kubectl drain server@ should do the job, but sometimes we need to be more aggressive:
113
114
<pre>
115
kubectl drain --delete-emptydir-data --ignore-daemonsets server23
116 42 Nico Schottelius
</pre>
117
118
h3. Readding a node after draining
119
120
<pre>
121
kubectl uncordon serverXX
122 1 Nico Schottelius
</pre>
123 43 Nico Schottelius
124 50 Nico Schottelius
h3. (Re-)joining worker nodes after creating the cluster
125 49 Nico Schottelius
126
* We need to have an up-to-date token
127
* We use different join commands for the workers and control plane nodes
128
129
Generating the join command on an existing control plane node:
130
131
<pre>
132
kubeadm token create --print-join-command
133
</pre>
134
135 50 Nico Schottelius
h3. (Re-)joining control plane nodes after creating the cluster
136 1 Nico Schottelius
137 50 Nico Schottelius
* We generate the token again
138
* We upload the certificates
139
* We need to combine/create the join command for the control plane node
140
141
Example session:
142
143
<pre>
144
% kubeadm token create --print-join-command
145
kubeadm join p10-api.k8s.ooo:6443 --token xmff4i.ABC --discovery-token-ca-cert-hash sha256:longhash 
146
147
% kubeadm init phase upload-certs --upload-certs
148
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
149
[upload-certs] Using certificate key:
150
CERTKEY
151
152
# Then we use these two outputs on the joining node:
153
154
kubeadm join p10-api.k8s.ooo:6443 --token xmff4i.ABC --discovery-token-ca-cert-hash sha256:longhash --control-plane --certificate-key CERTKEY
155
</pre>
156
157
Commands to be used on a control plane node:
158
159
<pre>
160
kubeadm token create --print-join-command
161
kubeadm init phase upload-certs --upload-certs
162
</pre>
163
164
Commands to be used on the joining node:
165
166
<pre>
167
JOINCOMMAND --control-plane --certificate-key CERTKEY
168
</pre>
169 49 Nico Schottelius
170 51 Nico Schottelius
SEE ALSO
171
172
* https://stackoverflow.com/questions/63936268/how-to-generate-kubeadm-token-for-secondary-control-plane-nodes
173
* https://blog.scottlowe.org/2019/08/15/reconstructing-the-join-command-for-kubeadm/
174
175 53 Nico Schottelius
h3. How to fix etcd does not start when rejoining a kubernetes cluster as a control plane
176 52 Nico Schottelius
177
If during the above step etcd does not come up, @kubeadm join@ can hang as follows:
178
179
<pre>
180
[control-plane] Creating static Pod manifest for "kube-apiserver"                                                              
181
[control-plane] Creating static Pod manifest for "kube-controller-manager"                                                     
182
[control-plane] Creating static Pod manifest for "kube-scheduler"                                                              
183
[check-etcd] Checking that the etcd cluster is healthy                                                                         
184
error execution phase check-etcd: etcd cluster is not healthy: failed to dial endpoint https://[2a0a:e5c0:10:1:225:b3ff:fe20:37
185
8a]:2379 with maintenance client: context deadline exceeded                                                                    
186
To see the stack trace of this error execute with --v=5 or higher         
187
</pre>
188
189
Then the problem is likely that the etcd server is still a member of the cluster. We first need to remove it from the etcd cluster and then the join works.
190
191
To fix this we do:
192
193
* Find a working etcd pod
194
* Find the etcd members / member list
195
* Remove the etcd member that we want to re-join the cluster
196
197
198
<pre>
199
# Find the etcd pods
200
kubectl -n kube-system get pods -l component=etcd,tier=control-plane
201
202
# Get the list of etcd servers with the member id 
203
kubectl exec -n kube-system -ti ETCDPODNAME -- etcdctl --endpoints '[::1]:2379' --cacert /etc/kubernetes/pki/etcd/ca.crt --cert  /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member list
204
205
# Remove the member
206
kubectl exec -n kube-system -ti ETCDPODNAME -- etcdctl --endpoints '[::1]:2379' --cacert /etc/kubernetes/pki/etcd/ca.crt --cert  /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member remove MEMBERID
207
</pre>
208
209
Sample session:
210
211
<pre>
212
[10:48] line:~% kubectl -n kube-system get pods -l component=etcd,tier=control-plane
213
NAME            READY   STATUS    RESTARTS     AGE
214
etcd-server63   1/1     Running   0            3m11s
215
etcd-server65   1/1     Running   3            7d2h
216
etcd-server83   1/1     Running   8 (6d ago)   7d2h
217
[10:48] line:~% kubectl exec -n kube-system -ti etcd-server65 -- etcdctl --endpoints '[::1]:2379' --cacert /etc/kubernetes/pki/etcd/ca.crt --cert  /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member list
218
356891cd676df6e4, started, server65, https://[2a0a:e5c0:10:1:225:b3ff:fe20:375c]:2380, https://[2a0a:e5c0:10:1:225:b3ff:fe20:375c]:2379, false
219
371b8a07185dee7e, started, server63, https://[2a0a:e5c0:10:1:225:b3ff:fe20:378a]:2380, https://[2a0a:e5c0:10:1:225:b3ff:fe20:378a]:2379, false
220
5942bc58307f8af9, started, server83, https://[2a0a:e5c0:10:1:3e4a:92ff:fe79:bb98]:2380, https://[2a0a:e5c0:10:1:3e4a:92ff:fe79:bb98]:2379, false
221
222
[10:48] line:~% kubectl exec -n kube-system -ti etcd-server65 -- etcdctl --endpoints '[::1]:2379' --cacert /etc/kubernetes/pki/etcd/ca.crt --cert  /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member remove 371b8a07185dee7e
223
Member 371b8a07185dee7e removed from cluster e3c0805f592a8f77
224 1 Nico Schottelius
225
</pre>
226
227
SEE ALSO
228
229
* We found the solution using https://stackoverflow.com/questions/67921552/re-installed-node-cannot-join-kubernetes-cluster
230 56 Nico Schottelius
231 62 Nico Schottelius
h2. Calico CNI
232
233
234
h3. Calico Installation
235
236
* We install "calico using helm":https://docs.projectcalico.org/getting-started/kubernetes/helm
237
* This has the following advantages:
238
** Easy to upgrade
239
** Does not require os to configure IPv6/dual stack settings as the tigera operator figures out things on its own
240
241
Usually plain calico can be installed directly using:
242
243
<pre>
244
helm repo add projectcalico https://docs.projectcalico.org/charts
245
helm install calico projectcalico/tigera-operator --version v3.20.2
246
</pre>
247
248
h3. Installing calicoctl
249
250
To be able to manage and configure calico, we need to 
251
"install calicoctl (we choose the version as a pod)":https://docs.projectcalico.org/getting-started/clis/calicoctl/install#install-calicoctl-as-a-kubernetes-pod
252
253
<pre>
254
kubectl apply -f https://docs.projectcalico.org/manifests/calicoctl.yaml
255
</pre>
256
257
h3. Calico configuration
258
259 63 Nico Schottelius
By default our k8s clusters "BGP peer":https://docs.projectcalico.org/networking/bgp
260
with an upstream router to propagate podcidr and servicecidr.
261 62 Nico Schottelius
262
Default settings in our infrastructure:
263
264
* We use a full-mesh using the @nodeToNodeMeshEnabled: true@ option
265
* We keep the original next hop so that *only* the server with the pod is announcing it (instead of ecmp)
266 1 Nico Schottelius
* We use private ASNs for k8s clusters
267 63 Nico Schottelius
* We do *not* use any overlay
268 62 Nico Schottelius
269
After installing calico and calicoctl the last step of the installation is usually:
270
271
<pre>
272
calicoctl create -f - < bgp-config-this-cluster.yaml
273
</pre>
274
275
276
A sample BGP configuration:
277
278
<pre>
279
---
280
apiVersion: projectcalico.org/v3
281
kind: BGPConfiguration
282
metadata:
283
  name: default
284
spec:
285
  logSeverityScreen: Info
286
  nodeToNodeMeshEnabled: true
287
  asNumber: 65534
288
  serviceClusterIPs:
289
  - cidr: 2a0a:e5c0:10:3::/108
290
  serviceExternalIPs:
291
  - cidr: 2a0a:e5c0:10:3::/108
292
---
293
apiVersion: projectcalico.org/v3
294
kind: BGPPeer
295
metadata:
296
  name: router1-place10
297
spec:
298
  peerIP: 2a0a:e5c0:10:1::50
299
  asNumber: 213081
300
  keepOriginalNextHop: true
301
</pre>
302
303 64 Nico Schottelius
h2. ArgoCD / ArgoWorkFlow
304 56 Nico Schottelius
305 60 Nico Schottelius
h3. Argocd Installation
306 1 Nico Schottelius
307 60 Nico Schottelius
As there is no configuration management present yet, argocd is installed using
308
309 1 Nico Schottelius
<pre>
310 60 Nico Schottelius
kubectl create namespace argocd
311
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
312 1 Nico Schottelius
</pre>
313 56 Nico Schottelius
314 60 Nico Schottelius
* See https://argo-cd.readthedocs.io/en/stable/
315 1 Nico Schottelius
316 60 Nico Schottelius
h3. Get the argocd credentials
317
318
<pre>
319
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo ""
320
</pre>
321 52 Nico Schottelius
322 55 Nico Schottelius
h2. Helm related operations
323
324 61 Nico Schottelius
We use helm charts extensively.
325
326
* In production, they are managed via argocd
327
* In development, helm chart can de developed and deployed manually using the helm utility.
328
329 55 Nico Schottelius
h3. Installing a helm chart
330
331
One can use the usual pattern of
332
333
<pre>
334
helm install <releasename> <chartdirectory>
335
</pre>
336
337
However often you want to reinstall/update when testing helm charts. The following pattern is "better", because it allows you to reinstall, if it is already installed:
338
339
<pre>
340
helm upgrade --install <releasename> <chartdirectory>
341
</pre>
342
343 43 Nico Schottelius
h2. Rook / Ceph Related Operations
344
345
h3. Inspecting the logs of a specific server
346
347
<pre>
348
# Get the related pods
349
kubectl -n rook-ceph get pods -l app=rook-ceph-osd-prepare 
350
...
351
352
# Inspect the logs of a specific pod
353
kubectl -n rook-ceph logs -f rook-ceph-osd-prepare-server23--1-444qx
354
355
</pre>
356
357
h3. Triggering server prepare / adding new osds
358
359
The rook-ceph-operator triggers/watches/creates pods to maintain hosts. To trigger a full "re scan", simply delete that pod:
360
361
<pre>
362
kubectl -n rook-ceph delete pods -l app=rook-ceph-operator
363
</pre>
364
365
This will cause all the @rook-ceph-osd-prepare-..@ jobs to be recreated and thus OSDs to be created, if new disks have been added.
366
367
h3. Removing an OSD
368
369
* See "Ceph OSD Management":https://rook.io/docs/rook/v1.7/ceph-osd-mgmt.html
370 41 Nico Schottelius
371 1 Nico Schottelius
h2. Infrastructure versions
372 35 Nico Schottelius
373 57 Nico Schottelius
h3. ungleich kubernetes infrastructure v5 (2021-10)
374 1 Nico Schottelius
375 57 Nico Schottelius
Clusters are configured / setup in this order:
376
377
* Bootstrap via kubeadm
378 59 Nico Schottelius
* "Networking via calico + BGP (non ECMP) using helm":https://docs.projectcalico.org/getting-started/kubernetes/helm
379
* "ArgoCD for CD":https://argo-cd.readthedocs.io/en/stable/
380
** "rook for storage via argocd":https://rook.io/
381 58 Nico Schottelius
** haproxy for in IPv6-cluster-IPv4-to-IPv6 proxy via argocd
382
** "kubernetes-secret-generator for in cluster secrets":https://github.com/mittwald/kubernetes-secret-generator
383
** "ungleich-certbot managing certs and nginx":https://hub.docker.com/repository/docker/ungleich/ungleich-certbot
384
385 57 Nico Schottelius
386
h3. ungleich kubernetes infrastructure v4 (2021-09)
387
388 54 Nico Schottelius
* rook is configured via manifests instead of using the rook-ceph-cluster helm chart
389 1 Nico Schottelius
* The rook operator is still being installed via helm
390 35 Nico Schottelius
391 57 Nico Schottelius
h3. ungleich kubernetes infrastructure v3 (2021-07)
392 1 Nico Schottelius
393 10 Nico Schottelius
* rook is now installed via helm via argocd instead of directly via manifests
394 28 Nico Schottelius
395 57 Nico Schottelius
h3. ungleich kubernetes infrastructure v2 (2021-05)
396 28 Nico Schottelius
397
* Replaced fluxv2 from ungleich k8s v1 with argocd
398 1 Nico Schottelius
** argocd can apply helm templates directly without needing to go through Chart releases
399 28 Nico Schottelius
* We are also using argoflow for build flows
400
* Planned to add "kaniko":https://github.com/GoogleContainerTools/kaniko for image building
401
402 57 Nico Schottelius
h3. ungleich kubernetes infrastructure v1 (2021-01)
403 28 Nico Schottelius
404
We are using the following components:
405
406
* "Calico as a CNI":https://www.projectcalico.org/ with BGP, IPv6 only, no encapsulation
407
** Needed for basic networking
408
* "kubernetes-secret-generator":https://github.com/mittwald/kubernetes-secret-generator for creating secrets
409
** Needed so that secrets are not stored in the git repository, but only in the cluster
410
* "ungleich-certbot":https://hub.docker.com/repository/docker/ungleich/ungleich-certbot
411
** Needed to get letsencrypt certificates for services
412
* "rook with ceph rbd + cephfs":https://rook.io/ for storage
413
** rbd for almost everything, *ReadWriteOnce*
414
** cephfs for smaller things, multi access *ReadWriteMany*
415
** Needed for providing persistent storage
416
* "flux v2":https://fluxcd.io/
417
** Needed to manage resources automatically