Project

General

Profile

The ungleich kubernetes infrastructure » History » Version 170

Nico Schottelius, 12/26/2022 09:48 PM

1 22 Nico Schottelius
h1. The ungleich kubernetes infrastructure and ungleich kubernetes manual
2 1 Nico Schottelius
3 3 Nico Schottelius
{{toc}}
4
5 1 Nico Schottelius
h2. Status
6
7 28 Nico Schottelius
This document is **pre-production**.
8
This document is to become the ungleich kubernetes infrastructure overview as well as the ungleich kubernetes manual.
9 1 Nico Schottelius
10 10 Nico Schottelius
h2. k8s clusters
11
12 123 Nico Schottelius
| Cluster            | Purpose/Setup     | Maintainer | Master(s)                     | argo                                                   | v4 http proxy | last verified |
13
| c0.k8s.ooo         | Dev               | -          | UNUSED                        |                                                        |               |    2021-10-05 |
14
| c1.k8s.ooo         | retired           |            | -                             |                                                        |               |    2022-03-15 |
15
| c2.k8s.ooo         | Dev p7 HW         | Nico       | server47 server53 server54    | "argo":https://argocd-server.argocd.svc.c2.k8s.ooo     |               |    2021-10-05 |
16
| c3.k8s.ooo         | retired           | -          | -                             |                                                        |               |    2021-10-05 |
17
| c4.k8s.ooo         | Dev2 p7 HW        | Jin-Guk    | server52 server53 server54    |                                                        |               |             - |
18
| c5.k8s.ooo         | retired           |            | -                             |                                                        |               |    2022-03-15 |
19
| c6.k8s.ooo         | Dev p6 VM Jin-Guk | Jin-Guk    |                               |                                                        |               |               |
20
| [[p5.k8s.ooo]]     | production        |            | server34 server36 server38    | "argo":https://argocd-server.argocd.svc.p5.k8s.ooo     | -             |               |
21
| [[p5-cow.k8s.ooo]] | production        | Nico       | server47 server51 server55    | "argo":https://argocd-server.argocd.svc.p5-cow.k8s.ooo |               |    2022-08-27 |
22
| [[p6.k8s.ooo]]     | production        |            | server67 server69 server71    | "argo":https://argocd-server.argocd.svc.p6.k8s.ooo     | 147.78.194.13 |    2021-10-05 |
23
| [[p10.k8s.ooo]]    | production        |            | server63 server65 server83    | "argo":https://argocd-server.argocd.svc.p10.k8s.ooo    | 147.78.194.12 |    2021-10-05 |
24
| [[k8s.ge.nau.so]]  | development       |            | server107 server108 server109 | "argo":https://argocd-server.argocd.svc.k8s.ge.nau.so  |               |               |
25
| [[dev.k8s.ooo]]    | development       |            | server110 server111 server112 | "argo":https://argocd-server.argocd.svc.dev.k8s.ooo    | -             |    2022-07-08 |
26 164 Nico Schottelius
| [[r1r2p15k8sooo|r1.p15.k8s.ooo]] | production | Nico | server120 | | | 2022-10-30 |
27
| [[r1r2p15k8sooo|r2.p15.k8s.ooo]] | production | Nico | server121 | | | 2022-09-06 |
28 162 Nico Schottelius
| [[r1r2p10k8sooo|r1.p10.k8s.ooo]] | production | Nico | server122 | | | 2022-10-30 |
29
| [[r1r2p10k8sooo|r2.p10.k8s.ooo]] | production | Nico | server123 | | | 2022-10-15 |
30
| [[r1r2p5k8sooo|r1.p5.k8s.ooo]] | production | Nico | server137 | | | 2022-10-30 |
31
| [[r1r2p5k8sooo|r2.p5.k8s.ooo]] | production | Nico | server138 | | | 2022-10-30 |
32
| [[r1r2p6k8sooo|r1.p6.k8s.ooo]] | production | Nico | server139 | | | 2022-10-30 |
33
| [[r1r2p6k8sooo|r2.p6.k8s.ooo]] | production | Nico | server140 | | | 2022-10-30 |
34 21 Nico Schottelius
35 1 Nico Schottelius
h2. General architecture and components overview
36
37
* All k8s clusters are IPv6 only
38
* We use BGP peering to propagate podcidr and serviceCidr networks to our infrastructure
39
* The main public testing repository is "ungleich-k8s":https://code.ungleich.ch/ungleich-public/ungleich-k8s
40 18 Nico Schottelius
** Private configurations are found in the **k8s-config** repository
41 1 Nico Schottelius
42
h3. Cluster types
43
44 28 Nico Schottelius
| **Type/Feature**            | **Development**                | **Production**         |
45
| Min No. nodes               | 3 (1 master, 3 worker)         | 5 (3 master, 3 worker) |
46
| Recommended minimum         | 4 (dedicated master, 3 worker) | 8 (3 master, 5 worker) |
47
| Separation of control plane | optional                       | recommended            |
48
| Persistent storage          | required                       | required               |
49
| Number of storage monitors  | 3                              | 5                      |
50 1 Nico Schottelius
51 43 Nico Schottelius
h2. General k8s operations
52 1 Nico Schottelius
53 46 Nico Schottelius
h3. Cheat sheet / external great references
54
55
* "kubectl cheatsheet":https://kubernetes.io/docs/reference/kubectl/cheatsheet/
56
57 117 Nico Schottelius
h3. Allowing to schedule work on the control plane / removing node taints
58 69 Nico Schottelius
59
* Mostly for single node / test / development clusters
60
* Just remove the master taint as follows
61
62
<pre>
63
kubectl taint nodes --all node-role.kubernetes.io/master-
64 118 Nico Schottelius
kubectl taint nodes --all node-role.kubernetes.io/control-plane-
65 69 Nico Schottelius
</pre>
66 1 Nico Schottelius
67 117 Nico Schottelius
You can check the node taints using @kubectl describe node ...@
68 69 Nico Schottelius
69 44 Nico Schottelius
h3. Get the cluster admin.conf
70
71
* On the masters of each cluster you can find the file @/etc/kubernetes/admin.conf@
72
* To be able to administrate the cluster you can copy the admin.conf to your local machine
73
* Multi cluster debugging can very easy if you name the config ~/cX-admin.conf (see example below)
74
75
<pre>
76
% scp root@server47.place7.ungleich.ch:/etc/kubernetes/admin.conf ~/c2-admin.conf
77
% export KUBECONFIG=~/c2-admin.conf    
78
% kubectl get nodes
79
NAME       STATUS                     ROLES                  AGE   VERSION
80
server47   Ready                      control-plane,master   82d   v1.22.0
81
server48   Ready                      control-plane,master   82d   v1.22.0
82
server49   Ready                      <none>                 82d   v1.22.0
83
server50   Ready                      <none>                 82d   v1.22.0
84
server59   Ready                      control-plane,master   82d   v1.22.0
85
server60   Ready,SchedulingDisabled   <none>                 82d   v1.22.0
86
server61   Ready                      <none>                 82d   v1.22.0
87
server62   Ready                      <none>                 82d   v1.22.0               
88
</pre>
89
90 18 Nico Schottelius
h3. Installing a new k8s cluster
91 8 Nico Schottelius
92 9 Nico Schottelius
* Decide on the cluster name (usually *cX.k8s.ooo*), X counting upwards
93 28 Nico Schottelius
** Using pXX.k8s.ooo for production clusters of placeXX
94 9 Nico Schottelius
* Use cdist to configure the nodes with requirements like crio
95
* Decide between single or multi node control plane setups (see below)
96 28 Nico Schottelius
** Single control plane suitable for development clusters
97 9 Nico Schottelius
98 28 Nico Schottelius
Typical init procedure:
99 9 Nico Schottelius
100 28 Nico Schottelius
* Single control plane: @kubeadm init --config bootstrap/XXX/kubeadm.yaml@
101
* Multi control plane (HA): @kubeadm init --config bootstrap/XXX/kubeadm.yaml --upload-certs@
102 10 Nico Schottelius
103 29 Nico Schottelius
h3. Deleting a pod that is hanging in terminating state
104
105
<pre>
106
kubectl delete pod <PODNAME> --grace-period=0 --force --namespace <NAMESPACE>
107
</pre>
108
109
(from https://stackoverflow.com/questions/35453792/pods-stuck-in-terminating-status)
110
111 42 Nico Schottelius
h3. Listing nodes of a cluster
112
113
<pre>
114
[15:05] bridge:~% kubectl get nodes
115
NAME       STATUS   ROLES                  AGE   VERSION
116
server22   Ready    <none>                 52d   v1.22.0
117
server23   Ready    <none>                 52d   v1.22.2
118
server24   Ready    <none>                 52d   v1.22.0
119
server25   Ready    <none>                 52d   v1.22.0
120
server26   Ready    <none>                 52d   v1.22.0
121
server27   Ready    <none>                 52d   v1.22.0
122
server63   Ready    control-plane,master   52d   v1.22.0
123
server64   Ready    <none>                 52d   v1.22.0
124
server65   Ready    control-plane,master   52d   v1.22.0
125
server66   Ready    <none>                 52d   v1.22.0
126
server83   Ready    control-plane,master   52d   v1.22.0
127
server84   Ready    <none>                 52d   v1.22.0
128
server85   Ready    <none>                 52d   v1.22.0
129
server86   Ready    <none>                 52d   v1.22.0
130
</pre>
131
132 41 Nico Schottelius
h3. Removing / draining a node
133
134
Usually @kubectl drain server@ should do the job, but sometimes we need to be more aggressive:
135
136 1 Nico Schottelius
<pre>
137 103 Nico Schottelius
kubectl drain --delete-emptydir-data --ignore-daemonsets serverXX
138 42 Nico Schottelius
</pre>
139
140
h3. Readding a node after draining
141
142
<pre>
143
kubectl uncordon serverXX
144 1 Nico Schottelius
</pre>
145 43 Nico Schottelius
146 50 Nico Schottelius
h3. (Re-)joining worker nodes after creating the cluster
147 49 Nico Schottelius
148
* We need to have an up-to-date token
149
* We use different join commands for the workers and control plane nodes
150
151
Generating the join command on an existing control plane node:
152
153
<pre>
154
kubeadm token create --print-join-command
155
</pre>
156
157 50 Nico Schottelius
h3. (Re-)joining control plane nodes after creating the cluster
158 1 Nico Schottelius
159 50 Nico Schottelius
* We generate the token again
160
* We upload the certificates
161
* We need to combine/create the join command for the control plane node
162
163
Example session:
164
165
<pre>
166
% kubeadm token create --print-join-command
167
kubeadm join p10-api.k8s.ooo:6443 --token xmff4i.ABC --discovery-token-ca-cert-hash sha256:longhash 
168
169
% kubeadm init phase upload-certs --upload-certs
170
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
171
[upload-certs] Using certificate key:
172
CERTKEY
173
174
# Then we use these two outputs on the joining node:
175
176
kubeadm join p10-api.k8s.ooo:6443 --token xmff4i.ABC --discovery-token-ca-cert-hash sha256:longhash --control-plane --certificate-key CERTKEY
177
</pre>
178
179
Commands to be used on a control plane node:
180
181
<pre>
182
kubeadm token create --print-join-command
183
kubeadm init phase upload-certs --upload-certs
184
</pre>
185
186
Commands to be used on the joining node:
187
188
<pre>
189
JOINCOMMAND --control-plane --certificate-key CERTKEY
190
</pre>
191 49 Nico Schottelius
192 51 Nico Schottelius
SEE ALSO
193
194
* https://stackoverflow.com/questions/63936268/how-to-generate-kubeadm-token-for-secondary-control-plane-nodes
195
* https://blog.scottlowe.org/2019/08/15/reconstructing-the-join-command-for-kubeadm/
196
197 53 Nico Schottelius
h3. How to fix etcd does not start when rejoining a kubernetes cluster as a control plane
198 52 Nico Schottelius
199
If during the above step etcd does not come up, @kubeadm join@ can hang as follows:
200
201
<pre>
202
[control-plane] Creating static Pod manifest for "kube-apiserver"                                                              
203
[control-plane] Creating static Pod manifest for "kube-controller-manager"                                                     
204
[control-plane] Creating static Pod manifest for "kube-scheduler"                                                              
205
[check-etcd] Checking that the etcd cluster is healthy                                                                         
206
error execution phase check-etcd: etcd cluster is not healthy: failed to dial endpoint https://[2a0a:e5c0:10:1:225:b3ff:fe20:37
207
8a]:2379 with maintenance client: context deadline exceeded                                                                    
208
To see the stack trace of this error execute with --v=5 or higher         
209
</pre>
210
211
Then the problem is likely that the etcd server is still a member of the cluster. We first need to remove it from the etcd cluster and then the join works.
212
213
To fix this we do:
214
215
* Find a working etcd pod
216
* Find the etcd members / member list
217
* Remove the etcd member that we want to re-join the cluster
218
219
220
<pre>
221
# Find the etcd pods
222
kubectl -n kube-system get pods -l component=etcd,tier=control-plane
223
224
# Get the list of etcd servers with the member id 
225
kubectl exec -n kube-system -ti ETCDPODNAME -- etcdctl --endpoints '[::1]:2379' --cacert /etc/kubernetes/pki/etcd/ca.crt --cert  /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member list
226
227
# Remove the member
228
kubectl exec -n kube-system -ti ETCDPODNAME -- etcdctl --endpoints '[::1]:2379' --cacert /etc/kubernetes/pki/etcd/ca.crt --cert  /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member remove MEMBERID
229
</pre>
230
231
Sample session:
232
233
<pre>
234
[10:48] line:~% kubectl -n kube-system get pods -l component=etcd,tier=control-plane
235
NAME            READY   STATUS    RESTARTS     AGE
236
etcd-server63   1/1     Running   0            3m11s
237
etcd-server65   1/1     Running   3            7d2h
238
etcd-server83   1/1     Running   8 (6d ago)   7d2h
239
[10:48] line:~% kubectl exec -n kube-system -ti etcd-server65 -- etcdctl --endpoints '[::1]:2379' --cacert /etc/kubernetes/pki/etcd/ca.crt --cert  /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member list
240
356891cd676df6e4, started, server65, https://[2a0a:e5c0:10:1:225:b3ff:fe20:375c]:2380, https://[2a0a:e5c0:10:1:225:b3ff:fe20:375c]:2379, false
241
371b8a07185dee7e, started, server63, https://[2a0a:e5c0:10:1:225:b3ff:fe20:378a]:2380, https://[2a0a:e5c0:10:1:225:b3ff:fe20:378a]:2379, false
242
5942bc58307f8af9, started, server83, https://[2a0a:e5c0:10:1:3e4a:92ff:fe79:bb98]:2380, https://[2a0a:e5c0:10:1:3e4a:92ff:fe79:bb98]:2379, false
243
244
[10:48] line:~% kubectl exec -n kube-system -ti etcd-server65 -- etcdctl --endpoints '[::1]:2379' --cacert /etc/kubernetes/pki/etcd/ca.crt --cert  /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member remove 371b8a07185dee7e
245
Member 371b8a07185dee7e removed from cluster e3c0805f592a8f77
246 1 Nico Schottelius
247
</pre>
248
249
SEE ALSO
250
251
* We found the solution using https://stackoverflow.com/questions/67921552/re-installed-node-cannot-join-kubernetes-cluster
252 56 Nico Schottelius
253 147 Nico Schottelius
h3. Node labels (adding, showing, removing)
254
255
Listing the labels:
256
257
<pre>
258
kubectl get nodes --show-labels
259
</pre>
260
261
Adding labels:
262
263
<pre>
264
kubectl label nodes LIST-OF-NODES label1=value1 
265
266
</pre>
267
268
For instance:
269
270
<pre>
271
kubectl label nodes router2 router3 hosttype=router 
272
</pre>
273
274
Selecting nodes in pods:
275
276
<pre>
277
apiVersion: v1
278
kind: Pod
279
...
280
spec:
281
  nodeSelector:
282
    hosttype: router
283
</pre>
284
285 148 Nico Schottelius
Removing labels by adding a minus at the end of the label name:
286
287
<pre>
288
kubectl label node <nodename> <labelname>-
289
</pre>
290
291
For instance:
292
293
<pre>
294
kubectl label nodes router2 router3 hosttype- 
295
</pre>
296
297 147 Nico Schottelius
SEE ALSO
298 1 Nico Schottelius
299 148 Nico Schottelius
* https://kubernetes.io/docs/tasks/configure-pod-container/assign-pods-nodes/
300
* https://stackoverflow.com/questions/34067979/how-to-delete-a-node-label-by-command-and-api
301 147 Nico Schottelius
302 101 Nico Schottelius
h3. Hardware Maintenance using ungleich-hardware
303
304
Use the following manifest and replace the HOST with the actual host:
305
306
<pre>
307
apiVersion: v1
308
kind: Pod
309
metadata:
310
  name: ungleich-hardware-HOST
311
spec:
312
  containers:
313
  - name: ungleich-hardware
314
    image: ungleich/ungleich-hardware:0.0.5
315
    args:
316
    - sleep
317
    - "1000000"
318
    volumeMounts:
319
      - mountPath: /dev
320
        name: dev
321
    securityContext:
322
      privileged: true
323
  nodeSelector:
324
    kubernetes.io/hostname: "HOST"
325
326
  volumes:
327
    - name: dev
328
      hostPath:
329
        path: /dev
330
</pre>
331
332 102 Nico Schottelius
Also see: [[The_ungleich_hardware_maintenance_guide]]
333
334 105 Nico Schottelius
h3. Triggering a cronjob / creating a job from a cronjob
335 104 Nico Schottelius
336
To test a cronjob, we can create a job from a cronjob:
337
338
<pre>
339
kubectl create job --from=cronjob/volume2-daily-backup volume2-manual
340
</pre>
341
342
This creates a job volume2-manual based on the cronjob  volume2-daily
343
344 112 Nico Schottelius
h3. su-ing into a user that has nologin shell set
345
346
Many times users are having nologin as their shell inside the container. To be able to execute maintenance commands within the
347
container, we can use @su -s /bin/sh@ like this:
348
349
<pre>
350
su -s /bin/sh -c '/path/to/your/script' testuser
351
</pre>
352
353
Found on https://serverfault.com/questions/351046/how-to-run-command-as-user-who-has-usr-sbin-nologin-as-shell
354
355 113 Nico Schottelius
h3. How to print a secret value
356
357
Assuming you want the "password" item from a secret, use:
358
359
<pre>
360
kubectl get secret SECRETNAME -o jsonpath="{.data.password}" | base64 -d; echo "" 
361
</pre>
362
363 157 Nico Schottelius
h2. Reference CNI
364
365
* Mainly "stupid", but effective plugins
366
* Main documentation on https://www.cni.dev/plugins/current/
367 158 Nico Schottelius
* Plugins
368
** bridge
369
*** Can create the bridge on the host
370
*** But seems not to be able to add host interfaces to it as well
371
*** Has support for vlan tags
372
** vlan
373
*** creates vlan tagged sub interface on the host
374 160 Nico Schottelius
*** "It's a 1:1 mapping (i.e. no bridge in between)":https://github.com/k8snetworkplumbingwg/multus-cni/issues/569
375 158 Nico Schottelius
** host-device
376
*** moves the interface from the host into the container
377
*** very easy for physical connections to containers
378 159 Nico Schottelius
** ipvlan
379
*** "virtualisation" of a host device
380
*** routing based on IP
381
*** Same MAC for everyone
382
*** Cannot reach the master interface
383
** maclvan
384
*** With mac addresses
385
*** Supports various modes (to be checked)
386
** ptp ("point to point")
387
*** Creates a host device and connects it to the container
388
** win*
389 158 Nico Schottelius
*** Windows implementations
390 157 Nico Schottelius
391 62 Nico Schottelius
h2. Calico CNI
392
393
h3. Calico Installation
394
395
* We install "calico using helm":https://docs.projectcalico.org/getting-started/kubernetes/helm
396
* This has the following advantages:
397
** Easy to upgrade
398
** Does not require os to configure IPv6/dual stack settings as the tigera operator figures out things on its own
399
400
Usually plain calico can be installed directly using:
401
402
<pre>
403 167 Nico Schottelius
VERSION=v3.24.5
404 149 Nico Schottelius
405 1 Nico Schottelius
helm repo add projectcalico https://docs.projectcalico.org/charts
406 167 Nico Schottelius
helm repo update
407 124 Nico Schottelius
helm upgrade --install --namespace tigera calico projectcalico/tigera-operator --version $VERSION --create-namespace
408 1 Nico Schottelius
</pre>
409 92 Nico Schottelius
410
* Check the tags on https://github.com/projectcalico/calico/tags for the latest release
411 62 Nico Schottelius
412
h3. Installing calicoctl
413
414 115 Nico Schottelius
* General installation instructions, including binary download: https://projectcalico.docs.tigera.io/maintenance/clis/calicoctl/install
415
416 62 Nico Schottelius
To be able to manage and configure calico, we need to 
417
"install calicoctl (we choose the version as a pod)":https://docs.projectcalico.org/getting-started/clis/calicoctl/install#install-calicoctl-as-a-kubernetes-pod
418
419
<pre>
420
kubectl apply -f https://docs.projectcalico.org/manifests/calicoctl.yaml
421
</pre>
422
423 93 Nico Schottelius
Or version specific:
424
425
<pre>
426
kubectl apply -f https://github.com/projectcalico/calico/blob/v3.20.4/manifests/calicoctl.yaml
427 97 Nico Schottelius
428
# For 3.22
429
kubectl apply -f https://projectcalico.docs.tigera.io/archive/v3.22/manifests/calicoctl.yaml
430 93 Nico Schottelius
</pre>
431
432 70 Nico Schottelius
And making it easier accessible by alias:
433
434
<pre>
435
alias calicoctl="kubectl exec -i -n kube-system calicoctl -- /calicoctl"
436
</pre>
437
438 62 Nico Schottelius
h3. Calico configuration
439
440 63 Nico Schottelius
By default our k8s clusters "BGP peer":https://docs.projectcalico.org/networking/bgp
441
with an upstream router to propagate podcidr and servicecidr.
442 62 Nico Schottelius
443
Default settings in our infrastructure:
444
445
* We use a full-mesh using the @nodeToNodeMeshEnabled: true@ option
446
* We keep the original next hop so that *only* the server with the pod is announcing it (instead of ecmp)
447 1 Nico Schottelius
* We use private ASNs for k8s clusters
448 63 Nico Schottelius
* We do *not* use any overlay
449 62 Nico Schottelius
450
After installing calico and calicoctl the last step of the installation is usually:
451
452 1 Nico Schottelius
<pre>
453 79 Nico Schottelius
calicoctl create -f - < calico-bgp.yaml
454 62 Nico Schottelius
</pre>
455
456
457
A sample BGP configuration:
458
459
<pre>
460
---
461
apiVersion: projectcalico.org/v3
462
kind: BGPConfiguration
463
metadata:
464
  name: default
465
spec:
466
  logSeverityScreen: Info
467
  nodeToNodeMeshEnabled: true
468
  asNumber: 65534
469
  serviceClusterIPs:
470
  - cidr: 2a0a:e5c0:10:3::/108
471
  serviceExternalIPs:
472
  - cidr: 2a0a:e5c0:10:3::/108
473
---
474
apiVersion: projectcalico.org/v3
475
kind: BGPPeer
476
metadata:
477
  name: router1-place10
478
spec:
479
  peerIP: 2a0a:e5c0:10:1::50
480
  asNumber: 213081
481
  keepOriginalNextHop: true
482
</pre>
483
484 126 Nico Schottelius
h2. Cilium CNI (experimental)
485
486 137 Nico Schottelius
h3. Status
487
488 138 Nico Schottelius
*NO WORKING CILIUM CONFIGURATION FOR IPV6 only modes*
489 137 Nico Schottelius
490 146 Nico Schottelius
h3. Latest error
491
492
It seems cilium does not run on IPv6 only hosts:
493
494
<pre>
495
level=info msg="Validating configured node address ranges" subsys=daemon
496
level=fatal msg="postinit failed" error="external IPv4 node address could not be derived, please configure via --ipv4-node" subsys=daemon
497
level=info msg="Starting IP identity watcher" subsys=ipcache
498
</pre>
499
500
It crashes after that log entry
501
502 128 Nico Schottelius
h3. BGP configuration
503
504
* The cilium-operator will not start without a correct configmap being present beforehand (see error message below)
505
* Creating the bgp config beforehand as a configmap is thus required.
506
507
The error one gets without the configmap present:
508
509
Pods are hanging with:
510
511
<pre>
512
cilium-bpqm6                       0/1     Init:0/4            0             9s
513
cilium-operator-5947d94f7f-5bmh2   0/1     ContainerCreating   0             9s
514
</pre>
515
516
The error message in the cilium-*perator is:
517
518
<pre>
519
Events:
520
  Type     Reason       Age                From               Message
521
  ----     ------       ----               ----               -------
522
  Normal   Scheduled    80s                default-scheduler  Successfully assigned kube-system/cilium-operator-5947d94f7f-lqcsp to server56
523
  Warning  FailedMount  16s (x8 over 80s)  kubelet            MountVolume.SetUp failed for volume "bgp-config-path" : configmap "bgp-config" not found
524
</pre>
525
526
A correct bgp config looks like this:
527
528
<pre>
529
apiVersion: v1
530
kind: ConfigMap
531
metadata:
532
  name: bgp-config
533
  namespace: kube-system
534
data:
535
  config.yaml: |
536
    peers:
537
      - peer-address: 2a0a:e5c0::46
538
        peer-asn: 209898
539
        my-asn: 65533
540
      - peer-address: 2a0a:e5c0::47
541
        peer-asn: 209898
542
        my-asn: 65533
543
    address-pools:
544
      - name: default
545
        protocol: bgp
546
        addresses:
547
          - 2a0a:e5c0:0:14::/64
548
</pre>
549 127 Nico Schottelius
550
h3. Installation
551 130 Nico Schottelius
552 127 Nico Schottelius
Adding the repo
553 1 Nico Schottelius
<pre>
554 127 Nico Schottelius
555 129 Nico Schottelius
helm repo add cilium https://helm.cilium.io/
556 130 Nico Schottelius
helm repo update
557
</pre>
558 129 Nico Schottelius
559 135 Nico Schottelius
Installing + configuring cilium
560 129 Nico Schottelius
<pre>
561 130 Nico Schottelius
ipv6pool=2a0a:e5c0:0:14::/112
562 1 Nico Schottelius
563 146 Nico Schottelius
version=1.12.2
564 129 Nico Schottelius
565
helm upgrade --install cilium cilium/cilium --version $version \
566 1 Nico Schottelius
  --namespace kube-system \
567
  --set ipv4.enabled=false \
568
  --set ipv6.enabled=true \
569 146 Nico Schottelius
  --set enableIPv6Masquerade=false \
570
  --set bgpControlPlane.enabled=true 
571 1 Nico Schottelius
572 146 Nico Schottelius
#  --set ipam.operator.clusterPoolIPv6PodCIDRList=$ipv6pool
573
574
# Old style bgp?
575 136 Nico Schottelius
#   --set bgp.enabled=true --set bgp.announce.podCIDR=true \
576 127 Nico Schottelius
577
# Show possible configuration options
578
helm show values cilium/cilium
579
580 1 Nico Schottelius
</pre>
581 132 Nico Schottelius
582
Using a /64 for ipam.operator.clusterPoolIPv6PodCIDRList fails with:
583
584
<pre>
585
level=fatal msg="Unable to init cluster-pool allocator" error="unable to initialize IPv6 allocator New CIDR set failed; the node CIDR size is too big" subsys=cilium-operator-generic
586
</pre>
587
588 126 Nico Schottelius
589 1 Nico Schottelius
See also https://github.com/cilium/cilium/issues/20756
590 135 Nico Schottelius
591
Seems a /112 is actually working.
592
593
h3. Kernel modules
594
595
Cilium requires the following modules to be loaded on the host (not loaded by default):
596
597
<pre>
598 1 Nico Schottelius
modprobe  ip6table_raw
599
modprobe  ip6table_filter
600
</pre>
601 146 Nico Schottelius
602
h3. Interesting helm flags
603
604
* autoDirectNodeRoutes
605
* bgpControlPlane.enabled = true
606
607
h3. SEE ALSO
608
609
* https://docs.cilium.io/en/v1.12/helm-reference/
610 133 Nico Schottelius
611 168 Nico Schottelius
h2. Multus (incomplete/experimental/WIP)
612 1 Nico Schottelius
613 168 Nico Schottelius
614
* https://github.com/k8snetworkplumbingwg/multus-cni
615
* Installing a deployment w/ CRDs
616 150 Nico Schottelius
617 169 Nico Schottelius
<pre>
618
VERSION=v3.9.2
619
620 170 Nico Schottelius
kubectl apply -f https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-cni/${VERSION}/deployments/multus-daemonset-crio.yml
621 1 Nico Schottelius
</pre>
622 170 Nico Schottelius
623
* crio based fails on alpine linux due to:
624
625
626
<pre>
627
[22:07] nb3:~% kubectl logs -n kube-system kube-multus-ds-2g9d5         
628
2022-12-26T21:05:21+00:00 Generating Multus configuration file using files in /host/etc/cni/net.d...
629
2022-12-26T21:05:21+00:00 Using MASTER_PLUGIN: 10-calico.conflist
630
2022-12-26T21:05:25+00:00 Nested capabilities string: "capabilities": {"bandwidth": true, "portMappings": true},
631
2022-12-26T21:05:25+00:00 Using /host/etc/cni/net.d/10-calico.conflist as a source to generate the Multus configuration
632
2022-12-26T21:05:26+00:00 Config file created @ /host/etc/cni/net.d/00-multus.conf
633
{ "cniVersion": "0.3.1", "name": "multus-cni-network", "type": "multus", "capabilities": {"bandwidth": true, "portMappings": true}, "kubeconfig": "/etc/cni/net.d/multus.d/multus.kubeconfig", "delegates": [ { "name": "k8s-pod-network", "cniVersion": "0.3.1", "plugins": [ { "type": "calico", "datastore_type": "kubernetes", "mtu": 0, "nodename_file_optional": false, "log_level": "Info", "log_file_path": "/var/log/calico/cni/cni.log", "ipam": { "type": "calico-ipam", "assign_ipv4" : "false", "assign_ipv6" : "true"}, "container_settings": { "allow_ip_forwarding": false }, "policy": { "type": "k8s" }, "kubernetes": { "k8s_api_root":"https://[2a0a:e5c0:43:bb::1]:443", "kubeconfig": "/etc/cni/net.d/calico-kubeconfig" } }, { "type": "bandwidth", "capabilities": {"bandwidth": true} }, {"type": "portmap", "snat": true, "capabilities": {"portMappings": true}} ] } ] }
634
2022-12-26T21:05:26+00:00 Restarting crio
635
/entrypoint.sh: line 434: systemctl: command not found
636
</pre>
637
638
639 169 Nico Schottelius
640 122 Nico Schottelius
h2. ArgoCD 
641 56 Nico Schottelius
642 60 Nico Schottelius
h3. Argocd Installation
643 1 Nico Schottelius
644 116 Nico Schottelius
* See https://argo-cd.readthedocs.io/en/stable/
645
646 60 Nico Schottelius
As there is no configuration management present yet, argocd is installed using
647
648 1 Nico Schottelius
<pre>
649 60 Nico Schottelius
kubectl create namespace argocd
650 86 Nico Schottelius
651 96 Nico Schottelius
# Specific Version
652
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v2.3.2/manifests/install.yaml
653 86 Nico Schottelius
654
# OR: latest stable
655 60 Nico Schottelius
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
656 56 Nico Schottelius
</pre>
657 1 Nico Schottelius
658 116 Nico Schottelius
659 1 Nico Schottelius
660 60 Nico Schottelius
h3. Get the argocd credentials
661
662
<pre>
663
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo ""
664
</pre>
665 52 Nico Schottelius
666 87 Nico Schottelius
h3. Accessing argocd
667
668
In regular IPv6 clusters:
669
670
* Navigate to https://argocd-server.argocd.CLUSTERDOMAIN
671
672
In legacy IPv4 clusters
673
674
<pre>
675
kubectl --namespace argocd port-forward svc/argocd-server 8080:80
676
</pre>
677
678 88 Nico Schottelius
* Navigate to https://localhost:8080
679
680 68 Nico Schottelius
h3. Using the argocd webhook to trigger changes
681 67 Nico Schottelius
682
* To trigger changes post json https://argocd.example.com/api/webhook
683
684 72 Nico Schottelius
h3. Deploying an application
685
686
* Applications are deployed via git towards gitea (code.ungleich.ch) and then pulled by argo
687 73 Nico Schottelius
* Always include the *redmine-url* pointing to the (customer) ticket
688
** Also add the support-url if it exists
689 72 Nico Schottelius
690
Application sample
691
692
<pre>
693
apiVersion: argoproj.io/v1alpha1
694
kind: Application
695
metadata:
696
  name: gitea-CUSTOMER
697
  namespace: argocd
698
spec:
699
  destination:
700
    namespace: default
701
    server: 'https://kubernetes.default.svc'
702
  source:
703
    path: apps/prod/gitea
704
    repoURL: 'https://code.ungleich.ch/ungleich-intern/k8s-config.git'
705
    targetRevision: HEAD
706
    helm:
707
      parameters:
708
        - name: storage.data.storageClass
709
          value: rook-ceph-block-hdd
710
        - name: storage.data.size
711
          value: 200Gi
712
        - name: storage.db.storageClass
713
          value: rook-ceph-block-ssd
714
        - name: storage.db.size
715
          value: 10Gi
716
        - name: storage.letsencrypt.storageClass
717
          value: rook-ceph-block-hdd
718
        - name: storage.letsencrypt.size
719
          value: 50Mi
720
        - name: letsencryptStaging
721
          value: 'no'
722
        - name: fqdn
723
          value: 'code.verua.online'
724
  project: default
725
  syncPolicy:
726
    automated:
727
      prune: true
728
      selfHeal: true
729
  info:
730
    - name: 'redmine-url'
731
      value: 'https://redmine.ungleich.ch/issues/ISSUEID'
732
    - name: 'support-url'
733
      value: 'https://support.ungleich.ch/Ticket/Display.html?id=TICKETID'
734
</pre>
735
736 80 Nico Schottelius
h2. Helm related operations and conventions
737 55 Nico Schottelius
738 61 Nico Schottelius
We use helm charts extensively.
739
740
* In production, they are managed via argocd
741
* In development, helm chart can de developed and deployed manually using the helm utility.
742
743 55 Nico Schottelius
h3. Installing a helm chart
744
745
One can use the usual pattern of
746
747
<pre>
748
helm install <releasename> <chartdirectory>
749
</pre>
750
751
However often you want to reinstall/update when testing helm charts. The following pattern is "better", because it allows you to reinstall, if it is already installed:
752
753
<pre>
754
helm upgrade --install <releasename> <chartdirectory>
755 1 Nico Schottelius
</pre>
756 80 Nico Schottelius
757
h3. Naming services and deployments in helm charts [Application labels]
758
759
* We always have {{ .Release.Name }} to identify the current "instance"
760
* Deployments:
761
** use @app: <what it is>@, f.i. @app: nginx@, @app: postgres@, ...
762 81 Nico Schottelius
* See more about standard labels on
763
** https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/
764
** https://helm.sh/docs/chart_best_practices/labels/
765 55 Nico Schottelius
766 151 Nico Schottelius
h3. Show all versions of a helm chart
767
768
<pre>
769
helm search repo -l repo/chart
770
</pre>
771
772
For example:
773
774
<pre>
775
% helm search repo -l projectcalico/tigera-operator 
776
NAME                         	CHART VERSION	APP VERSION	DESCRIPTION                            
777
projectcalico/tigera-operator	v3.23.3      	v3.23.3    	Installs the Tigera operator for Calico
778
projectcalico/tigera-operator	v3.23.2      	v3.23.2    	Installs the Tigera operator for Calico
779
....
780
</pre>
781
782 152 Nico Schottelius
h3. Show possible values of a chart
783
784
<pre>
785
helm show values <repo/chart>
786
</pre>
787
788
Example:
789
790
<pre>
791
helm show values ingress-nginx/ingress-nginx
792
</pre>
793
794
795 139 Nico Schottelius
h2. Rook + Ceph
796
797
h3. Installation
798
799
* Usually directly via argocd
800
801
Manual steps:
802
803
<pre>
804
805
</pre>
806 43 Nico Schottelius
807 71 Nico Schottelius
h3. Executing ceph commands
808
809
Using the ceph-tools pod as follows:
810
811
<pre>
812
kubectl exec -n rook-ceph -ti $(kubectl -n rook-ceph get pods -l app=rook-ceph-tools -o jsonpath='{.items[*].metadata.name}') -- ceph -s
813
</pre>
814
815 43 Nico Schottelius
h3. Inspecting the logs of a specific server
816
817
<pre>
818
# Get the related pods
819
kubectl -n rook-ceph get pods -l app=rook-ceph-osd-prepare 
820
...
821
822
# Inspect the logs of a specific pod
823
kubectl -n rook-ceph logs -f rook-ceph-osd-prepare-server23--1-444qx
824
825 71 Nico Schottelius
</pre>
826
827
h3. Inspecting the logs of the rook-ceph-operator
828
829
<pre>
830
kubectl -n rook-ceph logs -f -l app=rook-ceph-operator
831 43 Nico Schottelius
</pre>
832
833 121 Nico Schottelius
h3. Restarting the rook operator
834
835
<pre>
836
kubectl -n rook-ceph delete pods  -l app=rook-ceph-operator
837
</pre>
838
839 43 Nico Schottelius
h3. Triggering server prepare / adding new osds
840
841
The rook-ceph-operator triggers/watches/creates pods to maintain hosts. To trigger a full "re scan", simply delete that pod:
842
843
<pre>
844
kubectl -n rook-ceph delete pods -l app=rook-ceph-operator
845
</pre>
846
847
This will cause all the @rook-ceph-osd-prepare-..@ jobs to be recreated and thus OSDs to be created, if new disks have been added.
848
849
h3. Removing an OSD
850
851
* See "Ceph OSD Management":https://rook.io/docs/rook/v1.7/ceph-osd-mgmt.html
852 77 Nico Schottelius
* More specifically: https://github.com/rook/rook/blob/release-1.7/cluster/examples/kubernetes/ceph/osd-purge.yaml
853 99 Nico Schottelius
* Then delete the related deployment
854 41 Nico Schottelius
855 98 Nico Schottelius
Set osd id in the osd-purge.yaml and apply it. OSD should be down before.
856
857
<pre>
858
apiVersion: batch/v1
859
kind: Job
860
metadata:
861
  name: rook-ceph-purge-osd
862
  namespace: rook-ceph # namespace:cluster
863
  labels:
864
    app: rook-ceph-purge-osd
865
spec:
866
  template:
867
    metadata:
868
      labels:
869
        app: rook-ceph-purge-osd
870
    spec:
871
      serviceAccountName: rook-ceph-purge-osd
872
      containers:
873
        - name: osd-removal
874
          image: rook/ceph:master
875
          # TODO: Insert the OSD ID in the last parameter that is to be removed
876
          # The OSD IDs are a comma-separated list. For example: "0" or "0,2".
877
          # If you want to preserve the OSD PVCs, set `--preserve-pvc true`.
878
          #
879
          # A --force-osd-removal option is available if the OSD should be destroyed even though the
880
          # removal could lead to data loss.
881
          args:
882
            - "ceph"
883
            - "osd"
884
            - "remove"
885
            - "--preserve-pvc"
886
            - "false"
887
            - "--force-osd-removal"
888
            - "false"
889
            - "--osd-ids"
890
            - "SETTHEOSDIDHERE"
891
          env:
892
            - name: POD_NAMESPACE
893
              valueFrom:
894
                fieldRef:
895
                  fieldPath: metadata.namespace
896
            - name: ROOK_MON_ENDPOINTS
897
              valueFrom:
898
                configMapKeyRef:
899
                  key: data
900
                  name: rook-ceph-mon-endpoints
901
            - name: ROOK_CEPH_USERNAME
902
              valueFrom:
903
                secretKeyRef:
904
                  key: ceph-username
905
                  name: rook-ceph-mon
906
            - name: ROOK_CEPH_SECRET
907
              valueFrom:
908
                secretKeyRef:
909
                  key: ceph-secret
910
                  name: rook-ceph-mon
911
            - name: ROOK_CONFIG_DIR
912
              value: /var/lib/rook
913
            - name: ROOK_CEPH_CONFIG_OVERRIDE
914
              value: /etc/rook/config/override.conf
915
            - name: ROOK_FSID
916
              valueFrom:
917
                secretKeyRef:
918
                  key: fsid
919
                  name: rook-ceph-mon
920
            - name: ROOK_LOG_LEVEL
921
              value: DEBUG
922
          volumeMounts:
923
            - mountPath: /etc/ceph
924
              name: ceph-conf-emptydir
925
            - mountPath: /var/lib/rook
926
              name: rook-config
927
      volumes:
928
        - emptyDir: {}
929
          name: ceph-conf-emptydir
930
        - emptyDir: {}
931
          name: rook-config
932
      restartPolicy: Never
933
934
935 99 Nico Schottelius
</pre>
936
937
Deleting the deployment:
938
939
<pre>
940
[18:05] bridge:~% kubectl -n rook-ceph delete deployment rook-ceph-osd-6
941
deployment.apps "rook-ceph-osd-6" deleted
942 98 Nico Schottelius
</pre>
943
944 145 Nico Schottelius
h2. Ingress + Cert Manager
945
946
* We deploy "nginx-ingress":https://docs.nginx.com/nginx-ingress-controller/ to get an ingress
947
* we deploy "cert-manager":https://cert-manager.io/ to handle certificates
948
* We independently deploy @ClusterIssuer@ to allow the cert-manager app to deploy and the issuer to be created once the CRDs from cert manager are in place
949
950
h3. IPv4 reachability 
951
952
The ingress is by default IPv6 only. To make it reachable from the IPv4 world, get its IPv6 address and configure a NAT64 mapping in Jool.
953
954
Steps:
955
956
h4. Get the ingress IPv6 address
957
958
Use @kubectl -n ingress-nginx get svc ingress-nginx-controller -o jsonpath='{.spec.clusterIP}'; echo ''@
959
960
Example:
961
962
<pre>
963
kubectl -n ingress-nginx get svc ingress-nginx-controller -o jsonpath='{.spec.clusterIP}'; echo ''
964
2a0a:e5c0:10:1b::ce11
965
</pre>
966
967
h4. Add NAT64 mapping
968
969
* Update the __dcl_jool_siit cdist type
970
* Record the two IPs (IPv6 and IPv4)
971
* Configure all routers
972
973
974
h4. Add DNS record
975
976
To use the ingress capable as a CNAME destination, create an "ingress" DNS record, such as:
977
978
<pre>
979
; k8s ingress for dev
980
dev-ingress                 AAAA 2a0a:e5c0:10:1b::ce11
981
dev-ingress                 A 147.78.194.23
982
983
</pre> 
984
985
h4. Add supporting wildcard DNS
986
987
If you plan to add various sites under a specific domain, we can add a wildcard DNS entry, such as *.k8s-dev.django-hosting.ch:
988
989
<pre>
990
*.k8s-dev         CNAME dev-ingress.ungleich.ch.
991
</pre>
992
993 76 Nico Schottelius
h2. Harbor
994
995
* We user "Harbor":https://goharbor.io/ for caching and as an image registry. Internal app reference: apps/prod/harbor.
996
* The admin password is in the password store, auto generated per cluster
997
* At the moment harbor only authenticates against the internal ldap tree
998
999
h3. LDAP configuration
1000
1001
* The url needs to be ldaps://...
1002
* uid = uid
1003
* rest standard
1004 75 Nico Schottelius
1005 89 Nico Schottelius
h2. Monitoring / Prometheus
1006
1007 90 Nico Schottelius
* Via "kube-prometheus":https://github.com/prometheus-operator/kube-prometheus/
1008 89 Nico Schottelius
1009 91 Nico Schottelius
Access via ...
1010
1011
* http://prometheus-k8s.monitoring.svc:9090
1012
* http://grafana.monitoring.svc:3000
1013
* http://alertmanager.monitoring.svc:9093
1014
1015
1016 100 Nico Schottelius
h3. Prometheus Options
1017
1018
* "helm/kube-prometheus-stack":https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack
1019
** Includes dashboards and co.
1020
* "manifest based kube-prometheus":https://github.com/prometheus-operator/kube-prometheus
1021
** Includes dashboards and co.
1022
* "Prometheus Operator (mainly CRD manifest":https://github.com/prometheus-operator/prometheus-operator
1023
1024 82 Nico Schottelius
h2. Nextcloud
1025
1026 85 Nico Schottelius
h3. How to get the nextcloud credentials 
1027 84 Nico Schottelius
1028
* The initial username is set to "nextcloud"
1029
* The password is autogenerated and saved in a kubernetes secret
1030
1031
<pre>
1032 85 Nico Schottelius
kubectl get secret RELEASENAME-nextcloud -o jsonpath="{.data.PASSWORD}" | base64 -d; echo "" 
1033 84 Nico Schottelius
</pre>
1034
1035 83 Nico Schottelius
h3. How to fix "Access through untrusted domain"
1036
1037 82 Nico Schottelius
* Nextcloud stores the initial domain configuration
1038 1 Nico Schottelius
* If the FQDN is changed, it will show the error message "Access through untrusted domain"
1039 82 Nico Schottelius
* To fix, edit /var/www/html/config/config.php and correct the domain
1040 1 Nico Schottelius
* Then delete the pods
1041 165 Nico Schottelius
1042
h3. Running occ commands inside the nextcloud container
1043
1044
* Find the pod in the right namespace
1045
1046
Exec:
1047
1048
<pre>
1049
su www-data -s /bin/sh -c ./occ
1050
</pre>
1051
1052
* -s /bin/sh is needed as the default shell is set to /bin/false
1053
1054 166 Nico Schottelius
h4. Rescanning files
1055 165 Nico Schottelius
1056 166 Nico Schottelius
* If files have been added without nextcloud's knowledge
1057
1058
<pre>
1059
su www-data -s /bin/sh -c "./occ files:scan --all"
1060
</pre>
1061 82 Nico Schottelius
1062 1 Nico Schottelius
h2. Infrastructure versions
1063 35 Nico Schottelius
1064 57 Nico Schottelius
h3. ungleich kubernetes infrastructure v5 (2021-10)
1065 1 Nico Schottelius
1066 57 Nico Schottelius
Clusters are configured / setup in this order:
1067
1068
* Bootstrap via kubeadm
1069 59 Nico Schottelius
* "Networking via calico + BGP (non ECMP) using helm":https://docs.projectcalico.org/getting-started/kubernetes/helm
1070
* "ArgoCD for CD":https://argo-cd.readthedocs.io/en/stable/
1071
** "rook for storage via argocd":https://rook.io/
1072 58 Nico Schottelius
** haproxy for in IPv6-cluster-IPv4-to-IPv6 proxy via argocd
1073
** "kubernetes-secret-generator for in cluster secrets":https://github.com/mittwald/kubernetes-secret-generator
1074
** "ungleich-certbot managing certs and nginx":https://hub.docker.com/repository/docker/ungleich/ungleich-certbot
1075
1076 57 Nico Schottelius
1077
h3. ungleich kubernetes infrastructure v4 (2021-09)
1078
1079 54 Nico Schottelius
* rook is configured via manifests instead of using the rook-ceph-cluster helm chart
1080 1 Nico Schottelius
* The rook operator is still being installed via helm
1081 35 Nico Schottelius
1082 57 Nico Schottelius
h3. ungleich kubernetes infrastructure v3 (2021-07)
1083 1 Nico Schottelius
1084 10 Nico Schottelius
* rook is now installed via helm via argocd instead of directly via manifests
1085 28 Nico Schottelius
1086 57 Nico Schottelius
h3. ungleich kubernetes infrastructure v2 (2021-05)
1087 28 Nico Schottelius
1088
* Replaced fluxv2 from ungleich k8s v1 with argocd
1089 1 Nico Schottelius
** argocd can apply helm templates directly without needing to go through Chart releases
1090 28 Nico Schottelius
* We are also using argoflow for build flows
1091
* Planned to add "kaniko":https://github.com/GoogleContainerTools/kaniko for image building
1092
1093 57 Nico Schottelius
h3. ungleich kubernetes infrastructure v1 (2021-01)
1094 28 Nico Schottelius
1095
We are using the following components:
1096
1097
* "Calico as a CNI":https://www.projectcalico.org/ with BGP, IPv6 only, no encapsulation
1098
** Needed for basic networking
1099
* "kubernetes-secret-generator":https://github.com/mittwald/kubernetes-secret-generator for creating secrets
1100
** Needed so that secrets are not stored in the git repository, but only in the cluster
1101
* "ungleich-certbot":https://hub.docker.com/repository/docker/ungleich/ungleich-certbot
1102
** Needed to get letsencrypt certificates for services
1103
* "rook with ceph rbd + cephfs":https://rook.io/ for storage
1104
** rbd for almost everything, *ReadWriteOnce*
1105
** cephfs for smaller things, multi access *ReadWriteMany*
1106
** Needed for providing persistent storage
1107
* "flux v2":https://fluxcd.io/
1108
** Needed to manage resources automatically