Project

General

Profile

The ungleich kubernetes infrastructure » History » Version 188

Nico Schottelius, 06/25/2023 11:06 AM

1 22 Nico Schottelius
h1. The ungleich kubernetes infrastructure and ungleich kubernetes manual
2 1 Nico Schottelius
3 3 Nico Schottelius
{{toc}}
4
5 1 Nico Schottelius
h2. Status
6
7 28 Nico Schottelius
This document is **pre-production**.
8
This document is to become the ungleich kubernetes infrastructure overview as well as the ungleich kubernetes manual.
9 1 Nico Schottelius
10 10 Nico Schottelius
h2. k8s clusters
11
12 123 Nico Schottelius
| Cluster            | Purpose/Setup     | Maintainer | Master(s)                     | argo                                                   | v4 http proxy | last verified |
13
| c0.k8s.ooo         | Dev               | -          | UNUSED                        |                                                        |               |    2021-10-05 |
14
| c1.k8s.ooo         | retired           |            | -                             |                                                        |               |    2022-03-15 |
15
| c2.k8s.ooo         | Dev p7 HW         | Nico       | server47 server53 server54    | "argo":https://argocd-server.argocd.svc.c2.k8s.ooo     |               |    2021-10-05 |
16
| c3.k8s.ooo         | retired           | -          | -                             |                                                        |               |    2021-10-05 |
17
| c4.k8s.ooo         | Dev2 p7 HW        | Jin-Guk    | server52 server53 server54    |                                                        |               |             - |
18
| c5.k8s.ooo         | retired           |            | -                             |                                                        |               |    2022-03-15 |
19
| c6.k8s.ooo         | Dev p6 VM Jin-Guk | Jin-Guk    |                               |                                                        |               |               |
20
| [[p5.k8s.ooo]]     | production        |            | server34 server36 server38    | "argo":https://argocd-server.argocd.svc.p5.k8s.ooo     | -             |               |
21
| [[p5-cow.k8s.ooo]] | production        | Nico       | server47 server51 server55    | "argo":https://argocd-server.argocd.svc.p5-cow.k8s.ooo |               |    2022-08-27 |
22
| [[p6.k8s.ooo]]     | production        |            | server67 server69 server71    | "argo":https://argocd-server.argocd.svc.p6.k8s.ooo     | 147.78.194.13 |    2021-10-05 |
23 184 Nico Schottelius
| [[p6-cow.k8s.ooo]] | production        |            | server134 server135 server136 | "argo":https://argocd-server.argocd.svc.p6in10.k8s.ooo | ?             |    2023-05-17 |
24 177 Nico Schottelius
| [[p10.k8s.ooo]]    | production        |            | server131 server132 server133 | "argo":https://argocd-server.argocd.svc.p10.k8s.ooo    | 147.78.194.12 |    2021-10-05 |
25 123 Nico Schottelius
| [[k8s.ge.nau.so]]  | development       |            | server107 server108 server109 | "argo":https://argocd-server.argocd.svc.k8s.ge.nau.so  |               |               |
26
| [[dev.k8s.ooo]]    | development       |            | server110 server111 server112 | "argo":https://argocd-server.argocd.svc.dev.k8s.ooo    | -             |    2022-07-08 |
27 164 Nico Schottelius
| [[r1r2p15k8sooo|r1.p15.k8s.ooo]] | production | Nico | server120 | | | 2022-10-30 |
28
| [[r1r2p15k8sooo|r2.p15.k8s.ooo]] | production | Nico | server121 | | | 2022-09-06 |
29 162 Nico Schottelius
| [[r1r2p10k8sooo|r1.p10.k8s.ooo]] | production | Nico | server122 | | | 2022-10-30 |
30
| [[r1r2p10k8sooo|r2.p10.k8s.ooo]] | production | Nico | server123 | | | 2022-10-15 |
31
| [[r1r2p5k8sooo|r1.p5.k8s.ooo]] | production | Nico | server137 | | | 2022-10-30 |
32
| [[r1r2p5k8sooo|r2.p5.k8s.ooo]] | production | Nico | server138 | | | 2022-10-30 |
33
| [[r1r2p6k8sooo|r1.p6.k8s.ooo]] | production | Nico | server139 | | | 2022-10-30 |
34
| [[r1r2p6k8sooo|r2.p6.k8s.ooo]] | production | Nico | server140 | | | 2022-10-30 |
35 21 Nico Schottelius
36 1 Nico Schottelius
h2. General architecture and components overview
37
38
* All k8s clusters are IPv6 only
39
* We use BGP peering to propagate podcidr and serviceCidr networks to our infrastructure
40
* The main public testing repository is "ungleich-k8s":https://code.ungleich.ch/ungleich-public/ungleich-k8s
41 18 Nico Schottelius
** Private configurations are found in the **k8s-config** repository
42 1 Nico Schottelius
43
h3. Cluster types
44
45 28 Nico Schottelius
| **Type/Feature**            | **Development**                | **Production**         |
46
| Min No. nodes               | 3 (1 master, 3 worker)         | 5 (3 master, 3 worker) |
47
| Recommended minimum         | 4 (dedicated master, 3 worker) | 8 (3 master, 5 worker) |
48
| Separation of control plane | optional                       | recommended            |
49
| Persistent storage          | required                       | required               |
50
| Number of storage monitors  | 3                              | 5                      |
51 1 Nico Schottelius
52 43 Nico Schottelius
h2. General k8s operations
53 1 Nico Schottelius
54 46 Nico Schottelius
h3. Cheat sheet / external great references
55
56
* "kubectl cheatsheet":https://kubernetes.io/docs/reference/kubectl/cheatsheet/
57
58 117 Nico Schottelius
h3. Allowing to schedule work on the control plane / removing node taints
59 69 Nico Schottelius
60
* Mostly for single node / test / development clusters
61
* Just remove the master taint as follows
62
63
<pre>
64
kubectl taint nodes --all node-role.kubernetes.io/master-
65 118 Nico Schottelius
kubectl taint nodes --all node-role.kubernetes.io/control-plane-
66 69 Nico Schottelius
</pre>
67 1 Nico Schottelius
68 117 Nico Schottelius
You can check the node taints using @kubectl describe node ...@
69 69 Nico Schottelius
70 44 Nico Schottelius
h3. Get the cluster admin.conf
71
72
* On the masters of each cluster you can find the file @/etc/kubernetes/admin.conf@
73
* To be able to administrate the cluster you can copy the admin.conf to your local machine
74
* Multi cluster debugging can very easy if you name the config ~/cX-admin.conf (see example below)
75
76
<pre>
77
% scp root@server47.place7.ungleich.ch:/etc/kubernetes/admin.conf ~/c2-admin.conf
78
% export KUBECONFIG=~/c2-admin.conf    
79
% kubectl get nodes
80
NAME       STATUS                     ROLES                  AGE   VERSION
81
server47   Ready                      control-plane,master   82d   v1.22.0
82
server48   Ready                      control-plane,master   82d   v1.22.0
83
server49   Ready                      <none>                 82d   v1.22.0
84
server50   Ready                      <none>                 82d   v1.22.0
85
server59   Ready                      control-plane,master   82d   v1.22.0
86
server60   Ready,SchedulingDisabled   <none>                 82d   v1.22.0
87
server61   Ready                      <none>                 82d   v1.22.0
88
server62   Ready                      <none>                 82d   v1.22.0               
89
</pre>
90
91 18 Nico Schottelius
h3. Installing a new k8s cluster
92 8 Nico Schottelius
93 9 Nico Schottelius
* Decide on the cluster name (usually *cX.k8s.ooo*), X counting upwards
94 28 Nico Schottelius
** Using pXX.k8s.ooo for production clusters of placeXX
95 9 Nico Schottelius
* Use cdist to configure the nodes with requirements like crio
96
* Decide between single or multi node control plane setups (see below)
97 28 Nico Schottelius
** Single control plane suitable for development clusters
98 9 Nico Schottelius
99 28 Nico Schottelius
Typical init procedure:
100 9 Nico Schottelius
101 28 Nico Schottelius
* Single control plane: @kubeadm init --config bootstrap/XXX/kubeadm.yaml@
102
* Multi control plane (HA): @kubeadm init --config bootstrap/XXX/kubeadm.yaml --upload-certs@
103 10 Nico Schottelius
104 29 Nico Schottelius
h3. Deleting a pod that is hanging in terminating state
105
106
<pre>
107
kubectl delete pod <PODNAME> --grace-period=0 --force --namespace <NAMESPACE>
108
</pre>
109
110
(from https://stackoverflow.com/questions/35453792/pods-stuck-in-terminating-status)
111
112 42 Nico Schottelius
h3. Listing nodes of a cluster
113
114
<pre>
115
[15:05] bridge:~% kubectl get nodes
116
NAME       STATUS   ROLES                  AGE   VERSION
117
server22   Ready    <none>                 52d   v1.22.0
118
server23   Ready    <none>                 52d   v1.22.2
119
server24   Ready    <none>                 52d   v1.22.0
120
server25   Ready    <none>                 52d   v1.22.0
121
server26   Ready    <none>                 52d   v1.22.0
122
server27   Ready    <none>                 52d   v1.22.0
123
server63   Ready    control-plane,master   52d   v1.22.0
124
server64   Ready    <none>                 52d   v1.22.0
125
server65   Ready    control-plane,master   52d   v1.22.0
126
server66   Ready    <none>                 52d   v1.22.0
127
server83   Ready    control-plane,master   52d   v1.22.0
128
server84   Ready    <none>                 52d   v1.22.0
129
server85   Ready    <none>                 52d   v1.22.0
130
server86   Ready    <none>                 52d   v1.22.0
131
</pre>
132
133 41 Nico Schottelius
h3. Removing / draining a node
134
135
Usually @kubectl drain server@ should do the job, but sometimes we need to be more aggressive:
136
137 1 Nico Schottelius
<pre>
138 103 Nico Schottelius
kubectl drain --delete-emptydir-data --ignore-daemonsets serverXX
139 42 Nico Schottelius
</pre>
140
141
h3. Readding a node after draining
142
143
<pre>
144
kubectl uncordon serverXX
145 1 Nico Schottelius
</pre>
146 43 Nico Schottelius
147 50 Nico Schottelius
h3. (Re-)joining worker nodes after creating the cluster
148 49 Nico Schottelius
149
* We need to have an up-to-date token
150
* We use different join commands for the workers and control plane nodes
151
152
Generating the join command on an existing control plane node:
153
154
<pre>
155
kubeadm token create --print-join-command
156
</pre>
157
158 50 Nico Schottelius
h3. (Re-)joining control plane nodes after creating the cluster
159 1 Nico Schottelius
160 50 Nico Schottelius
* We generate the token again
161
* We upload the certificates
162
* We need to combine/create the join command for the control plane node
163
164
Example session:
165
166
<pre>
167
% kubeadm token create --print-join-command
168
kubeadm join p10-api.k8s.ooo:6443 --token xmff4i.ABC --discovery-token-ca-cert-hash sha256:longhash 
169
170
% kubeadm init phase upload-certs --upload-certs
171
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
172
[upload-certs] Using certificate key:
173
CERTKEY
174
175
# Then we use these two outputs on the joining node:
176
177
kubeadm join p10-api.k8s.ooo:6443 --token xmff4i.ABC --discovery-token-ca-cert-hash sha256:longhash --control-plane --certificate-key CERTKEY
178
</pre>
179
180
Commands to be used on a control plane node:
181
182
<pre>
183
kubeadm token create --print-join-command
184
kubeadm init phase upload-certs --upload-certs
185
</pre>
186
187
Commands to be used on the joining node:
188
189
<pre>
190
JOINCOMMAND --control-plane --certificate-key CERTKEY
191
</pre>
192 49 Nico Schottelius
193 51 Nico Schottelius
SEE ALSO
194
195
* https://stackoverflow.com/questions/63936268/how-to-generate-kubeadm-token-for-secondary-control-plane-nodes
196
* https://blog.scottlowe.org/2019/08/15/reconstructing-the-join-command-for-kubeadm/
197
198 53 Nico Schottelius
h3. How to fix etcd does not start when rejoining a kubernetes cluster as a control plane
199 52 Nico Schottelius
200
If during the above step etcd does not come up, @kubeadm join@ can hang as follows:
201
202
<pre>
203
[control-plane] Creating static Pod manifest for "kube-apiserver"                                                              
204
[control-plane] Creating static Pod manifest for "kube-controller-manager"                                                     
205
[control-plane] Creating static Pod manifest for "kube-scheduler"                                                              
206
[check-etcd] Checking that the etcd cluster is healthy                                                                         
207
error execution phase check-etcd: etcd cluster is not healthy: failed to dial endpoint https://[2a0a:e5c0:10:1:225:b3ff:fe20:37
208
8a]:2379 with maintenance client: context deadline exceeded                                                                    
209
To see the stack trace of this error execute with --v=5 or higher         
210
</pre>
211
212
Then the problem is likely that the etcd server is still a member of the cluster. We first need to remove it from the etcd cluster and then the join works.
213
214
To fix this we do:
215
216
* Find a working etcd pod
217
* Find the etcd members / member list
218
* Remove the etcd member that we want to re-join the cluster
219
220
221
<pre>
222
# Find the etcd pods
223
kubectl -n kube-system get pods -l component=etcd,tier=control-plane
224
225
# Get the list of etcd servers with the member id 
226
kubectl exec -n kube-system -ti ETCDPODNAME -- etcdctl --endpoints '[::1]:2379' --cacert /etc/kubernetes/pki/etcd/ca.crt --cert  /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member list
227
228
# Remove the member
229
kubectl exec -n kube-system -ti ETCDPODNAME -- etcdctl --endpoints '[::1]:2379' --cacert /etc/kubernetes/pki/etcd/ca.crt --cert  /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member remove MEMBERID
230
</pre>
231
232
Sample session:
233
234
<pre>
235
[10:48] line:~% kubectl -n kube-system get pods -l component=etcd,tier=control-plane
236
NAME            READY   STATUS    RESTARTS     AGE
237
etcd-server63   1/1     Running   0            3m11s
238
etcd-server65   1/1     Running   3            7d2h
239
etcd-server83   1/1     Running   8 (6d ago)   7d2h
240
[10:48] line:~% kubectl exec -n kube-system -ti etcd-server65 -- etcdctl --endpoints '[::1]:2379' --cacert /etc/kubernetes/pki/etcd/ca.crt --cert  /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member list
241
356891cd676df6e4, started, server65, https://[2a0a:e5c0:10:1:225:b3ff:fe20:375c]:2380, https://[2a0a:e5c0:10:1:225:b3ff:fe20:375c]:2379, false
242
371b8a07185dee7e, started, server63, https://[2a0a:e5c0:10:1:225:b3ff:fe20:378a]:2380, https://[2a0a:e5c0:10:1:225:b3ff:fe20:378a]:2379, false
243
5942bc58307f8af9, started, server83, https://[2a0a:e5c0:10:1:3e4a:92ff:fe79:bb98]:2380, https://[2a0a:e5c0:10:1:3e4a:92ff:fe79:bb98]:2379, false
244
245
[10:48] line:~% kubectl exec -n kube-system -ti etcd-server65 -- etcdctl --endpoints '[::1]:2379' --cacert /etc/kubernetes/pki/etcd/ca.crt --cert  /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member remove 371b8a07185dee7e
246
Member 371b8a07185dee7e removed from cluster e3c0805f592a8f77
247 1 Nico Schottelius
248
</pre>
249
250
SEE ALSO
251
252
* We found the solution using https://stackoverflow.com/questions/67921552/re-installed-node-cannot-join-kubernetes-cluster
253 56 Nico Schottelius
254 147 Nico Schottelius
h3. Node labels (adding, showing, removing)
255
256
Listing the labels:
257
258
<pre>
259
kubectl get nodes --show-labels
260
</pre>
261
262
Adding labels:
263
264
<pre>
265
kubectl label nodes LIST-OF-NODES label1=value1 
266
267
</pre>
268
269
For instance:
270
271
<pre>
272
kubectl label nodes router2 router3 hosttype=router 
273
</pre>
274
275
Selecting nodes in pods:
276
277
<pre>
278
apiVersion: v1
279
kind: Pod
280
...
281
spec:
282
  nodeSelector:
283
    hosttype: router
284
</pre>
285
286 148 Nico Schottelius
Removing labels by adding a minus at the end of the label name:
287
288
<pre>
289
kubectl label node <nodename> <labelname>-
290
</pre>
291
292
For instance:
293
294
<pre>
295
kubectl label nodes router2 router3 hosttype- 
296
</pre>
297
298 147 Nico Schottelius
SEE ALSO
299 1 Nico Schottelius
300 148 Nico Schottelius
* https://kubernetes.io/docs/tasks/configure-pod-container/assign-pods-nodes/
301
* https://stackoverflow.com/questions/34067979/how-to-delete-a-node-label-by-command-and-api
302 147 Nico Schottelius
303 101 Nico Schottelius
h3. Hardware Maintenance using ungleich-hardware
304
305
Use the following manifest and replace the HOST with the actual host:
306
307
<pre>
308
apiVersion: v1
309
kind: Pod
310
metadata:
311
  name: ungleich-hardware-HOST
312
spec:
313
  containers:
314
  - name: ungleich-hardware
315
    image: ungleich/ungleich-hardware:0.0.5
316
    args:
317
    - sleep
318
    - "1000000"
319
    volumeMounts:
320
      - mountPath: /dev
321
        name: dev
322
    securityContext:
323
      privileged: true
324
  nodeSelector:
325
    kubernetes.io/hostname: "HOST"
326
327
  volumes:
328
    - name: dev
329
      hostPath:
330
        path: /dev
331
</pre>
332
333 102 Nico Schottelius
Also see: [[The_ungleich_hardware_maintenance_guide]]
334
335 105 Nico Schottelius
h3. Triggering a cronjob / creating a job from a cronjob
336 104 Nico Schottelius
337
To test a cronjob, we can create a job from a cronjob:
338
339
<pre>
340
kubectl create job --from=cronjob/volume2-daily-backup volume2-manual
341
</pre>
342
343
This creates a job volume2-manual based on the cronjob  volume2-daily
344
345 112 Nico Schottelius
h3. su-ing into a user that has nologin shell set
346
347
Many times users are having nologin as their shell inside the container. To be able to execute maintenance commands within the
348
container, we can use @su -s /bin/sh@ like this:
349
350
<pre>
351
su -s /bin/sh -c '/path/to/your/script' testuser
352
</pre>
353
354
Found on https://serverfault.com/questions/351046/how-to-run-command-as-user-who-has-usr-sbin-nologin-as-shell
355
356 113 Nico Schottelius
h3. How to print a secret value
357
358
Assuming you want the "password" item from a secret, use:
359
360
<pre>
361
kubectl get secret SECRETNAME -o jsonpath="{.data.password}" | base64 -d; echo "" 
362
</pre>
363
364 173 Nico Schottelius
h3. How to upgrade a kubernetes cluster
365 172 Nico Schottelius
366
h4. General
367
368
* Should be done every X months to stay up-to-date
369
** X probably something like 3-6
370
* kubeadm based clusters
371
* Needs specific kubeadm versions for upgrade
372
* Follow instructions on https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/
373
374
h4. Getting a specific kubeadm or kubelet version
375
376
<pre>
377
RELEASE=v1.24.9
378 181 Nico Schottelius
RELEASE=v1.25.9
379 1 Nico Schottelius
RELEASE=v1.23.17
380 187 Nico Schottelius
RELEASE=v1.26.6
381
ARCH=amd64
382 172 Nico Schottelius
383
curl -L --remote-name-all https://dl.k8s.io/release/${RELEASE}/bin/linux/${ARCH}/{kubeadm,kubelet}
384 182 Nico Schottelius
chmod u+x kubeadm kubelet
385 172 Nico Schottelius
</pre>
386
387
h4. Steps
388
389
* kubeadm upgrade plan
390
** On one control plane node
391
* kubeadm upgrade apply vXX.YY.ZZ
392
** On one control plane node
393
394 173 Nico Schottelius
Repeat for all control planes nodes. The upgrade kubelet on all other nodes via package manager.
395 172 Nico Schottelius
396 188 Nico Schottelius
h4. Upgrading to 1.26.6
397
398
* https://v1-26.docs.kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/
399
400
401 186 Nico Schottelius
h4. Upgrade to kubernetes 1.27
402
403
* kubelet will not start anymore
404
* reason: @"command failed" err="failed to parse kubelet flag: unknown flag: --container-runtime"@
405
* /var/lib/kubelet/kubeadm-flags.env contains that parameter
406
* remove it, start kubelet
407
408
h4. Upgrade to crio 1.27: missing crun
409
410
Error message
411
412
<pre>
413
level=fatal msg="validating runtime config: runtime validation: \"crun\" not found in $PATH: exec: \"crun\": executable file not found in $PATH"
414
</pre>
415
416
Fix:
417
418
<pre>
419
apk add crun
420
</pre>
421
422
423 157 Nico Schottelius
h2. Reference CNI
424
425
* Mainly "stupid", but effective plugins
426
* Main documentation on https://www.cni.dev/plugins/current/
427 158 Nico Schottelius
* Plugins
428
** bridge
429
*** Can create the bridge on the host
430
*** But seems not to be able to add host interfaces to it as well
431
*** Has support for vlan tags
432
** vlan
433
*** creates vlan tagged sub interface on the host
434 160 Nico Schottelius
*** "It's a 1:1 mapping (i.e. no bridge in between)":https://github.com/k8snetworkplumbingwg/multus-cni/issues/569
435 158 Nico Schottelius
** host-device
436
*** moves the interface from the host into the container
437
*** very easy for physical connections to containers
438 159 Nico Schottelius
** ipvlan
439
*** "virtualisation" of a host device
440
*** routing based on IP
441
*** Same MAC for everyone
442
*** Cannot reach the master interface
443
** maclvan
444
*** With mac addresses
445
*** Supports various modes (to be checked)
446
** ptp ("point to point")
447
*** Creates a host device and connects it to the container
448
** win*
449 158 Nico Schottelius
*** Windows implementations
450 157 Nico Schottelius
451 62 Nico Schottelius
h2. Calico CNI
452
453
h3. Calico Installation
454
455
* We install "calico using helm":https://docs.projectcalico.org/getting-started/kubernetes/helm
456
* This has the following advantages:
457
** Easy to upgrade
458
** Does not require os to configure IPv6/dual stack settings as the tigera operator figures out things on its own
459
460
Usually plain calico can be installed directly using:
461
462
<pre>
463 174 Nico Schottelius
VERSION=v3.25.0
464 149 Nico Schottelius
465 1 Nico Schottelius
helm repo add projectcalico https://docs.projectcalico.org/charts
466 167 Nico Schottelius
helm repo update
467 124 Nico Schottelius
helm upgrade --install --namespace tigera calico projectcalico/tigera-operator --version $VERSION --create-namespace
468 1 Nico Schottelius
</pre>
469 92 Nico Schottelius
470
* Check the tags on https://github.com/projectcalico/calico/tags for the latest release
471 62 Nico Schottelius
472
h3. Installing calicoctl
473
474 115 Nico Schottelius
* General installation instructions, including binary download: https://projectcalico.docs.tigera.io/maintenance/clis/calicoctl/install
475
476 62 Nico Schottelius
To be able to manage and configure calico, we need to 
477
"install calicoctl (we choose the version as a pod)":https://docs.projectcalico.org/getting-started/clis/calicoctl/install#install-calicoctl-as-a-kubernetes-pod
478
479
<pre>
480
kubectl apply -f https://docs.projectcalico.org/manifests/calicoctl.yaml
481
</pre>
482
483 93 Nico Schottelius
Or version specific:
484
485
<pre>
486
kubectl apply -f https://github.com/projectcalico/calico/blob/v3.20.4/manifests/calicoctl.yaml
487 97 Nico Schottelius
488
# For 3.22
489
kubectl apply -f https://projectcalico.docs.tigera.io/archive/v3.22/manifests/calicoctl.yaml
490 93 Nico Schottelius
</pre>
491
492 70 Nico Schottelius
And making it easier accessible by alias:
493
494
<pre>
495
alias calicoctl="kubectl exec -i -n kube-system calicoctl -- /calicoctl"
496
</pre>
497
498 62 Nico Schottelius
h3. Calico configuration
499
500 63 Nico Schottelius
By default our k8s clusters "BGP peer":https://docs.projectcalico.org/networking/bgp
501
with an upstream router to propagate podcidr and servicecidr.
502 62 Nico Schottelius
503
Default settings in our infrastructure:
504
505
* We use a full-mesh using the @nodeToNodeMeshEnabled: true@ option
506
* We keep the original next hop so that *only* the server with the pod is announcing it (instead of ecmp)
507 1 Nico Schottelius
* We use private ASNs for k8s clusters
508 63 Nico Schottelius
* We do *not* use any overlay
509 62 Nico Schottelius
510
After installing calico and calicoctl the last step of the installation is usually:
511
512 1 Nico Schottelius
<pre>
513 79 Nico Schottelius
calicoctl create -f - < calico-bgp.yaml
514 62 Nico Schottelius
</pre>
515
516
517
A sample BGP configuration:
518
519
<pre>
520
---
521
apiVersion: projectcalico.org/v3
522
kind: BGPConfiguration
523
metadata:
524
  name: default
525
spec:
526
  logSeverityScreen: Info
527
  nodeToNodeMeshEnabled: true
528
  asNumber: 65534
529
  serviceClusterIPs:
530
  - cidr: 2a0a:e5c0:10:3::/108
531
  serviceExternalIPs:
532
  - cidr: 2a0a:e5c0:10:3::/108
533
---
534
apiVersion: projectcalico.org/v3
535
kind: BGPPeer
536
metadata:
537
  name: router1-place10
538
spec:
539
  peerIP: 2a0a:e5c0:10:1::50
540
  asNumber: 213081
541
  keepOriginalNextHop: true
542
</pre>
543
544 126 Nico Schottelius
h2. Cilium CNI (experimental)
545
546 137 Nico Schottelius
h3. Status
547
548 138 Nico Schottelius
*NO WORKING CILIUM CONFIGURATION FOR IPV6 only modes*
549 137 Nico Schottelius
550 146 Nico Schottelius
h3. Latest error
551
552
It seems cilium does not run on IPv6 only hosts:
553
554
<pre>
555
level=info msg="Validating configured node address ranges" subsys=daemon
556
level=fatal msg="postinit failed" error="external IPv4 node address could not be derived, please configure via --ipv4-node" subsys=daemon
557
level=info msg="Starting IP identity watcher" subsys=ipcache
558
</pre>
559
560
It crashes after that log entry
561
562 128 Nico Schottelius
h3. BGP configuration
563
564
* The cilium-operator will not start without a correct configmap being present beforehand (see error message below)
565
* Creating the bgp config beforehand as a configmap is thus required.
566
567
The error one gets without the configmap present:
568
569
Pods are hanging with:
570
571
<pre>
572
cilium-bpqm6                       0/1     Init:0/4            0             9s
573
cilium-operator-5947d94f7f-5bmh2   0/1     ContainerCreating   0             9s
574
</pre>
575
576
The error message in the cilium-*perator is:
577
578
<pre>
579
Events:
580
  Type     Reason       Age                From               Message
581
  ----     ------       ----               ----               -------
582
  Normal   Scheduled    80s                default-scheduler  Successfully assigned kube-system/cilium-operator-5947d94f7f-lqcsp to server56
583
  Warning  FailedMount  16s (x8 over 80s)  kubelet            MountVolume.SetUp failed for volume "bgp-config-path" : configmap "bgp-config" not found
584
</pre>
585
586
A correct bgp config looks like this:
587
588
<pre>
589
apiVersion: v1
590
kind: ConfigMap
591
metadata:
592
  name: bgp-config
593
  namespace: kube-system
594
data:
595
  config.yaml: |
596
    peers:
597
      - peer-address: 2a0a:e5c0::46
598
        peer-asn: 209898
599
        my-asn: 65533
600
      - peer-address: 2a0a:e5c0::47
601
        peer-asn: 209898
602
        my-asn: 65533
603
    address-pools:
604
      - name: default
605
        protocol: bgp
606
        addresses:
607
          - 2a0a:e5c0:0:14::/64
608
</pre>
609 127 Nico Schottelius
610
h3. Installation
611 130 Nico Schottelius
612 127 Nico Schottelius
Adding the repo
613 1 Nico Schottelius
<pre>
614 127 Nico Schottelius
615 129 Nico Schottelius
helm repo add cilium https://helm.cilium.io/
616 130 Nico Schottelius
helm repo update
617
</pre>
618 129 Nico Schottelius
619 135 Nico Schottelius
Installing + configuring cilium
620 129 Nico Schottelius
<pre>
621 130 Nico Schottelius
ipv6pool=2a0a:e5c0:0:14::/112
622 1 Nico Schottelius
623 146 Nico Schottelius
version=1.12.2
624 129 Nico Schottelius
625
helm upgrade --install cilium cilium/cilium --version $version \
626 1 Nico Schottelius
  --namespace kube-system \
627
  --set ipv4.enabled=false \
628
  --set ipv6.enabled=true \
629 146 Nico Schottelius
  --set enableIPv6Masquerade=false \
630
  --set bgpControlPlane.enabled=true 
631 1 Nico Schottelius
632 146 Nico Schottelius
#  --set ipam.operator.clusterPoolIPv6PodCIDRList=$ipv6pool
633
634
# Old style bgp?
635 136 Nico Schottelius
#   --set bgp.enabled=true --set bgp.announce.podCIDR=true \
636 127 Nico Schottelius
637
# Show possible configuration options
638
helm show values cilium/cilium
639
640 1 Nico Schottelius
</pre>
641 132 Nico Schottelius
642
Using a /64 for ipam.operator.clusterPoolIPv6PodCIDRList fails with:
643
644
<pre>
645
level=fatal msg="Unable to init cluster-pool allocator" error="unable to initialize IPv6 allocator New CIDR set failed; the node CIDR size is too big" subsys=cilium-operator-generic
646
</pre>
647
648 126 Nico Schottelius
649 1 Nico Schottelius
See also https://github.com/cilium/cilium/issues/20756
650 135 Nico Schottelius
651
Seems a /112 is actually working.
652
653
h3. Kernel modules
654
655
Cilium requires the following modules to be loaded on the host (not loaded by default):
656
657
<pre>
658 1 Nico Schottelius
modprobe  ip6table_raw
659
modprobe  ip6table_filter
660
</pre>
661 146 Nico Schottelius
662
h3. Interesting helm flags
663
664
* autoDirectNodeRoutes
665
* bgpControlPlane.enabled = true
666
667
h3. SEE ALSO
668
669
* https://docs.cilium.io/en/v1.12/helm-reference/
670 133 Nico Schottelius
671 179 Nico Schottelius
h2. Multus
672 168 Nico Schottelius
673
* https://github.com/k8snetworkplumbingwg/multus-cni
674
* Installing a deployment w/ CRDs
675 150 Nico Schottelius
676 169 Nico Schottelius
<pre>
677 176 Nico Schottelius
VERSION=v4.0.1
678 169 Nico Schottelius
679 170 Nico Schottelius
kubectl apply -f https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-cni/${VERSION}/deployments/multus-daemonset-crio.yml
680
</pre>
681 169 Nico Schottelius
682 122 Nico Schottelius
h2. ArgoCD 
683 56 Nico Schottelius
684 60 Nico Schottelius
h3. Argocd Installation
685 1 Nico Schottelius
686 116 Nico Schottelius
* See https://argo-cd.readthedocs.io/en/stable/
687
688 60 Nico Schottelius
As there is no configuration management present yet, argocd is installed using
689
690 1 Nico Schottelius
<pre>
691 60 Nico Schottelius
kubectl create namespace argocd
692 86 Nico Schottelius
693 96 Nico Schottelius
# Specific Version
694
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v2.3.2/manifests/install.yaml
695 86 Nico Schottelius
696
# OR: latest stable
697 60 Nico Schottelius
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
698 56 Nico Schottelius
</pre>
699 1 Nico Schottelius
700 116 Nico Schottelius
701 1 Nico Schottelius
702 60 Nico Schottelius
h3. Get the argocd credentials
703
704
<pre>
705
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo ""
706
</pre>
707 52 Nico Schottelius
708 87 Nico Schottelius
h3. Accessing argocd
709
710
In regular IPv6 clusters:
711
712
* Navigate to https://argocd-server.argocd.CLUSTERDOMAIN
713
714
In legacy IPv4 clusters
715
716
<pre>
717
kubectl --namespace argocd port-forward svc/argocd-server 8080:80
718
</pre>
719
720 88 Nico Schottelius
* Navigate to https://localhost:8080
721
722 68 Nico Schottelius
h3. Using the argocd webhook to trigger changes
723 67 Nico Schottelius
724
* To trigger changes post json https://argocd.example.com/api/webhook
725
726 72 Nico Schottelius
h3. Deploying an application
727
728
* Applications are deployed via git towards gitea (code.ungleich.ch) and then pulled by argo
729 73 Nico Schottelius
* Always include the *redmine-url* pointing to the (customer) ticket
730
** Also add the support-url if it exists
731 72 Nico Schottelius
732
Application sample
733
734
<pre>
735
apiVersion: argoproj.io/v1alpha1
736
kind: Application
737
metadata:
738
  name: gitea-CUSTOMER
739
  namespace: argocd
740
spec:
741
  destination:
742
    namespace: default
743
    server: 'https://kubernetes.default.svc'
744
  source:
745
    path: apps/prod/gitea
746
    repoURL: 'https://code.ungleich.ch/ungleich-intern/k8s-config.git'
747
    targetRevision: HEAD
748
    helm:
749
      parameters:
750
        - name: storage.data.storageClass
751
          value: rook-ceph-block-hdd
752
        - name: storage.data.size
753
          value: 200Gi
754
        - name: storage.db.storageClass
755
          value: rook-ceph-block-ssd
756
        - name: storage.db.size
757
          value: 10Gi
758
        - name: storage.letsencrypt.storageClass
759
          value: rook-ceph-block-hdd
760
        - name: storage.letsencrypt.size
761
          value: 50Mi
762
        - name: letsencryptStaging
763
          value: 'no'
764
        - name: fqdn
765
          value: 'code.verua.online'
766
  project: default
767
  syncPolicy:
768
    automated:
769
      prune: true
770
      selfHeal: true
771
  info:
772
    - name: 'redmine-url'
773
      value: 'https://redmine.ungleich.ch/issues/ISSUEID'
774
    - name: 'support-url'
775
      value: 'https://support.ungleich.ch/Ticket/Display.html?id=TICKETID'
776
</pre>
777
778 80 Nico Schottelius
h2. Helm related operations and conventions
779 55 Nico Schottelius
780 61 Nico Schottelius
We use helm charts extensively.
781
782
* In production, they are managed via argocd
783
* In development, helm chart can de developed and deployed manually using the helm utility.
784
785 55 Nico Schottelius
h3. Installing a helm chart
786
787
One can use the usual pattern of
788
789
<pre>
790
helm install <releasename> <chartdirectory>
791
</pre>
792
793
However often you want to reinstall/update when testing helm charts. The following pattern is "better", because it allows you to reinstall, if it is already installed:
794
795
<pre>
796
helm upgrade --install <releasename> <chartdirectory>
797 1 Nico Schottelius
</pre>
798 80 Nico Schottelius
799
h3. Naming services and deployments in helm charts [Application labels]
800
801
* We always have {{ .Release.Name }} to identify the current "instance"
802
* Deployments:
803
** use @app: <what it is>@, f.i. @app: nginx@, @app: postgres@, ...
804 81 Nico Schottelius
* See more about standard labels on
805
** https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/
806
** https://helm.sh/docs/chart_best_practices/labels/
807 55 Nico Schottelius
808 151 Nico Schottelius
h3. Show all versions of a helm chart
809
810
<pre>
811
helm search repo -l repo/chart
812
</pre>
813
814
For example:
815
816
<pre>
817
% helm search repo -l projectcalico/tigera-operator 
818
NAME                         	CHART VERSION	APP VERSION	DESCRIPTION                            
819
projectcalico/tigera-operator	v3.23.3      	v3.23.3    	Installs the Tigera operator for Calico
820
projectcalico/tigera-operator	v3.23.2      	v3.23.2    	Installs the Tigera operator for Calico
821
....
822
</pre>
823
824 152 Nico Schottelius
h3. Show possible values of a chart
825
826
<pre>
827
helm show values <repo/chart>
828
</pre>
829
830
Example:
831
832
<pre>
833
helm show values ingress-nginx/ingress-nginx
834
</pre>
835
836 178 Nico Schottelius
h3. Download a chart
837
838
For instance for checking it out locally. Use:
839
840
<pre>
841
helm pull <repo/chart>
842
</pre>
843 152 Nico Schottelius
844 139 Nico Schottelius
h2. Rook + Ceph
845
846
h3. Installation
847
848
* Usually directly via argocd
849
850 71 Nico Schottelius
h3. Executing ceph commands
851
852
Using the ceph-tools pod as follows:
853
854
<pre>
855
kubectl exec -n rook-ceph -ti $(kubectl -n rook-ceph get pods -l app=rook-ceph-tools -o jsonpath='{.items[*].metadata.name}') -- ceph -s
856
</pre>
857
858 43 Nico Schottelius
h3. Inspecting the logs of a specific server
859
860
<pre>
861
# Get the related pods
862
kubectl -n rook-ceph get pods -l app=rook-ceph-osd-prepare 
863
...
864
865
# Inspect the logs of a specific pod
866
kubectl -n rook-ceph logs -f rook-ceph-osd-prepare-server23--1-444qx
867
868 71 Nico Schottelius
</pre>
869
870
h3. Inspecting the logs of the rook-ceph-operator
871
872
<pre>
873
kubectl -n rook-ceph logs -f -l app=rook-ceph-operator
874 43 Nico Schottelius
</pre>
875
876 121 Nico Schottelius
h3. Restarting the rook operator
877
878
<pre>
879
kubectl -n rook-ceph delete pods  -l app=rook-ceph-operator
880
</pre>
881
882 43 Nico Schottelius
h3. Triggering server prepare / adding new osds
883
884
The rook-ceph-operator triggers/watches/creates pods to maintain hosts. To trigger a full "re scan", simply delete that pod:
885
886
<pre>
887
kubectl -n rook-ceph delete pods -l app=rook-ceph-operator
888
</pre>
889
890
This will cause all the @rook-ceph-osd-prepare-..@ jobs to be recreated and thus OSDs to be created, if new disks have been added.
891
892
h3. Removing an OSD
893
894
* See "Ceph OSD Management":https://rook.io/docs/rook/v1.7/ceph-osd-mgmt.html
895 77 Nico Schottelius
* More specifically: https://github.com/rook/rook/blob/release-1.7/cluster/examples/kubernetes/ceph/osd-purge.yaml
896 99 Nico Schottelius
* Then delete the related deployment
897 41 Nico Schottelius
898 98 Nico Schottelius
Set osd id in the osd-purge.yaml and apply it. OSD should be down before.
899
900
<pre>
901
apiVersion: batch/v1
902
kind: Job
903
metadata:
904
  name: rook-ceph-purge-osd
905
  namespace: rook-ceph # namespace:cluster
906
  labels:
907
    app: rook-ceph-purge-osd
908
spec:
909
  template:
910
    metadata:
911
      labels:
912
        app: rook-ceph-purge-osd
913
    spec:
914
      serviceAccountName: rook-ceph-purge-osd
915
      containers:
916
        - name: osd-removal
917
          image: rook/ceph:master
918
          # TODO: Insert the OSD ID in the last parameter that is to be removed
919
          # The OSD IDs are a comma-separated list. For example: "0" or "0,2".
920
          # If you want to preserve the OSD PVCs, set `--preserve-pvc true`.
921
          #
922
          # A --force-osd-removal option is available if the OSD should be destroyed even though the
923
          # removal could lead to data loss.
924
          args:
925
            - "ceph"
926
            - "osd"
927
            - "remove"
928
            - "--preserve-pvc"
929
            - "false"
930
            - "--force-osd-removal"
931
            - "false"
932
            - "--osd-ids"
933
            - "SETTHEOSDIDHERE"
934
          env:
935
            - name: POD_NAMESPACE
936
              valueFrom:
937
                fieldRef:
938
                  fieldPath: metadata.namespace
939
            - name: ROOK_MON_ENDPOINTS
940
              valueFrom:
941
                configMapKeyRef:
942
                  key: data
943
                  name: rook-ceph-mon-endpoints
944
            - name: ROOK_CEPH_USERNAME
945
              valueFrom:
946
                secretKeyRef:
947
                  key: ceph-username
948
                  name: rook-ceph-mon
949
            - name: ROOK_CEPH_SECRET
950
              valueFrom:
951
                secretKeyRef:
952
                  key: ceph-secret
953
                  name: rook-ceph-mon
954
            - name: ROOK_CONFIG_DIR
955
              value: /var/lib/rook
956
            - name: ROOK_CEPH_CONFIG_OVERRIDE
957
              value: /etc/rook/config/override.conf
958
            - name: ROOK_FSID
959
              valueFrom:
960
                secretKeyRef:
961
                  key: fsid
962
                  name: rook-ceph-mon
963
            - name: ROOK_LOG_LEVEL
964
              value: DEBUG
965
          volumeMounts:
966
            - mountPath: /etc/ceph
967
              name: ceph-conf-emptydir
968
            - mountPath: /var/lib/rook
969
              name: rook-config
970
      volumes:
971
        - emptyDir: {}
972
          name: ceph-conf-emptydir
973
        - emptyDir: {}
974
          name: rook-config
975
      restartPolicy: Never
976
977
978 99 Nico Schottelius
</pre>
979
980 1 Nico Schottelius
Deleting the deployment:
981
982
<pre>
983
[18:05] bridge:~% kubectl -n rook-ceph delete deployment rook-ceph-osd-6
984 99 Nico Schottelius
deployment.apps "rook-ceph-osd-6" deleted
985
</pre>
986 185 Nico Schottelius
987
h3. Placement of mons/osds/etc.
988
989
See https://rook.io/docs/rook/v1.11/CRDs/Cluster/ceph-cluster-crd/#placement-configuration-settings
990 98 Nico Schottelius
991 145 Nico Schottelius
h2. Ingress + Cert Manager
992
993
* We deploy "nginx-ingress":https://docs.nginx.com/nginx-ingress-controller/ to get an ingress
994
* we deploy "cert-manager":https://cert-manager.io/ to handle certificates
995
* We independently deploy @ClusterIssuer@ to allow the cert-manager app to deploy and the issuer to be created once the CRDs from cert manager are in place
996
997
h3. IPv4 reachability 
998
999
The ingress is by default IPv6 only. To make it reachable from the IPv4 world, get its IPv6 address and configure a NAT64 mapping in Jool.
1000
1001
Steps:
1002
1003
h4. Get the ingress IPv6 address
1004
1005
Use @kubectl -n ingress-nginx get svc ingress-nginx-controller -o jsonpath='{.spec.clusterIP}'; echo ''@
1006
1007
Example:
1008
1009
<pre>
1010
kubectl -n ingress-nginx get svc ingress-nginx-controller -o jsonpath='{.spec.clusterIP}'; echo ''
1011
2a0a:e5c0:10:1b::ce11
1012
</pre>
1013
1014
h4. Add NAT64 mapping
1015
1016
* Update the __dcl_jool_siit cdist type
1017
* Record the two IPs (IPv6 and IPv4)
1018
* Configure all routers
1019
1020
1021
h4. Add DNS record
1022
1023
To use the ingress capable as a CNAME destination, create an "ingress" DNS record, such as:
1024
1025
<pre>
1026
; k8s ingress for dev
1027
dev-ingress                 AAAA 2a0a:e5c0:10:1b::ce11
1028
dev-ingress                 A 147.78.194.23
1029
1030
</pre> 
1031
1032
h4. Add supporting wildcard DNS
1033
1034
If you plan to add various sites under a specific domain, we can add a wildcard DNS entry, such as *.k8s-dev.django-hosting.ch:
1035
1036
<pre>
1037
*.k8s-dev         CNAME dev-ingress.ungleich.ch.
1038
</pre>
1039
1040 76 Nico Schottelius
h2. Harbor
1041
1042 175 Nico Schottelius
* We user "Harbor":https://goharbor.io/ as an image registry for our own images. Internal app reference: apps/prod/harbor.
1043
* The admin password is in the password store, it is Harbor12345 by default
1044 76 Nico Schottelius
* At the moment harbor only authenticates against the internal ldap tree
1045
1046
h3. LDAP configuration
1047
1048
* The url needs to be ldaps://...
1049
* uid = uid
1050
* rest standard
1051 75 Nico Schottelius
1052 89 Nico Schottelius
h2. Monitoring / Prometheus
1053
1054 90 Nico Schottelius
* Via "kube-prometheus":https://github.com/prometheus-operator/kube-prometheus/
1055 89 Nico Schottelius
1056 91 Nico Schottelius
Access via ...
1057
1058
* http://prometheus-k8s.monitoring.svc:9090
1059
* http://grafana.monitoring.svc:3000
1060
* http://alertmanager.monitoring.svc:9093
1061
1062
1063 100 Nico Schottelius
h3. Prometheus Options
1064
1065
* "helm/kube-prometheus-stack":https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack
1066
** Includes dashboards and co.
1067
* "manifest based kube-prometheus":https://github.com/prometheus-operator/kube-prometheus
1068
** Includes dashboards and co.
1069
* "Prometheus Operator (mainly CRD manifest":https://github.com/prometheus-operator/prometheus-operator
1070
1071 171 Nico Schottelius
h3. Grafana default password
1072
1073
* If not changed: @prom-operator@
1074
1075 82 Nico Schottelius
h2. Nextcloud
1076
1077 85 Nico Schottelius
h3. How to get the nextcloud credentials 
1078 84 Nico Schottelius
1079
* The initial username is set to "nextcloud"
1080
* The password is autogenerated and saved in a kubernetes secret
1081
1082
<pre>
1083 85 Nico Schottelius
kubectl get secret RELEASENAME-nextcloud -o jsonpath="{.data.PASSWORD}" | base64 -d; echo "" 
1084 84 Nico Schottelius
</pre>
1085
1086 83 Nico Schottelius
h3. How to fix "Access through untrusted domain"
1087
1088 82 Nico Schottelius
* Nextcloud stores the initial domain configuration
1089 1 Nico Schottelius
* If the FQDN is changed, it will show the error message "Access through untrusted domain"
1090 82 Nico Schottelius
* To fix, edit /var/www/html/config/config.php and correct the domain
1091 1 Nico Schottelius
* Then delete the pods
1092 165 Nico Schottelius
1093
h3. Running occ commands inside the nextcloud container
1094
1095
* Find the pod in the right namespace
1096
1097
Exec:
1098
1099
<pre>
1100
su www-data -s /bin/sh -c ./occ
1101
</pre>
1102
1103
* -s /bin/sh is needed as the default shell is set to /bin/false
1104
1105 166 Nico Schottelius
h4. Rescanning files
1106 165 Nico Schottelius
1107 166 Nico Schottelius
* If files have been added without nextcloud's knowledge
1108
1109
<pre>
1110
su www-data -s /bin/sh -c "./occ files:scan --all"
1111
</pre>
1112 82 Nico Schottelius
1113 1 Nico Schottelius
h2. Infrastructure versions
1114 35 Nico Schottelius
1115 57 Nico Schottelius
h3. ungleich kubernetes infrastructure v5 (2021-10)
1116 1 Nico Schottelius
1117 57 Nico Schottelius
Clusters are configured / setup in this order:
1118
1119
* Bootstrap via kubeadm
1120 59 Nico Schottelius
* "Networking via calico + BGP (non ECMP) using helm":https://docs.projectcalico.org/getting-started/kubernetes/helm
1121
* "ArgoCD for CD":https://argo-cd.readthedocs.io/en/stable/
1122
** "rook for storage via argocd":https://rook.io/
1123 58 Nico Schottelius
** haproxy for in IPv6-cluster-IPv4-to-IPv6 proxy via argocd
1124
** "kubernetes-secret-generator for in cluster secrets":https://github.com/mittwald/kubernetes-secret-generator
1125
** "ungleich-certbot managing certs and nginx":https://hub.docker.com/repository/docker/ungleich/ungleich-certbot
1126
1127 57 Nico Schottelius
1128
h3. ungleich kubernetes infrastructure v4 (2021-09)
1129
1130 54 Nico Schottelius
* rook is configured via manifests instead of using the rook-ceph-cluster helm chart
1131 1 Nico Schottelius
* The rook operator is still being installed via helm
1132 35 Nico Schottelius
1133 57 Nico Schottelius
h3. ungleich kubernetes infrastructure v3 (2021-07)
1134 1 Nico Schottelius
1135 10 Nico Schottelius
* rook is now installed via helm via argocd instead of directly via manifests
1136 28 Nico Schottelius
1137 57 Nico Schottelius
h3. ungleich kubernetes infrastructure v2 (2021-05)
1138 28 Nico Schottelius
1139
* Replaced fluxv2 from ungleich k8s v1 with argocd
1140 1 Nico Schottelius
** argocd can apply helm templates directly without needing to go through Chart releases
1141 28 Nico Schottelius
* We are also using argoflow for build flows
1142
* Planned to add "kaniko":https://github.com/GoogleContainerTools/kaniko for image building
1143
1144 57 Nico Schottelius
h3. ungleich kubernetes infrastructure v1 (2021-01)
1145 28 Nico Schottelius
1146
We are using the following components:
1147
1148
* "Calico as a CNI":https://www.projectcalico.org/ with BGP, IPv6 only, no encapsulation
1149
** Needed for basic networking
1150
* "kubernetes-secret-generator":https://github.com/mittwald/kubernetes-secret-generator for creating secrets
1151
** Needed so that secrets are not stored in the git repository, but only in the cluster
1152
* "ungleich-certbot":https://hub.docker.com/repository/docker/ungleich/ungleich-certbot
1153
** Needed to get letsencrypt certificates for services
1154
* "rook with ceph rbd + cephfs":https://rook.io/ for storage
1155
** rbd for almost everything, *ReadWriteOnce*
1156
** cephfs for smaller things, multi access *ReadWriteMany*
1157
** Needed for providing persistent storage
1158
* "flux v2":https://fluxcd.io/
1159
** Needed to manage resources automatically