Project

General

Profile

The ungleich kubernetes infrastructure » History » Version 87

Nico Schottelius, 01/28/2022 06:38 PM

1 22 Nico Schottelius
h1. The ungleich kubernetes infrastructure and ungleich kubernetes manual
2 1 Nico Schottelius
3 3 Nico Schottelius
{{toc}}
4
5 1 Nico Schottelius
h2. Status
6
7 28 Nico Schottelius
This document is **pre-production**.
8
This document is to become the ungleich kubernetes infrastructure overview as well as the ungleich kubernetes manual.
9 1 Nico Schottelius
10 10 Nico Schottelius
h2. k8s clusters
11
12 78 Nico Schottelius
| Cluster         | Purpose/Setup     | Maintainer   | Master(s)                                                | argo                                                | rook | v4 http proxy | last verified |
13
| c0.k8s.ooo      | Dev               | -            | UNUSED                                                   |                                                     |      |               |    2021-10-05 |
14
| c1.k8s.ooo      | Dev p6 VM         | Nico         | 2a0a-e5c0-2-11-0-62ff-fe0b-1a3d.k8s-1.place6.ungleich.ch |                                                     |      |               |    2021-10-05 |
15
| c2.k8s.ooo      | Dev p7 HW         | Nico         | server47 server53 server54                               | "argo":https://argocd-server.argocd.svc.c2.k8s.ooo  | x    |               |    2021-10-05 |
16
| c3.k8s.ooo      | Test p7 PI        | -            | UNUSED                                                   |                                                     |      |               |    2021-10-05 |
17
| c4.k8s.ooo      | Dev2 p7 HW        | Fran/Jin-Guk | server52 server53 server54                               |                                                     |      |               |             - |
18
| c5.k8s.ooo      | Dev p6 VM Amal    | Nico/Amal    | 2a0a-e5c0-2-11-0-62ff-fe0b-1a46.k8s-1.place6.ungleich.ch |                                                     |      |               |               |
19
| c6.k8s.ooo      | Dev p6 VM Jin-Guk | Jin-Guk      |                                                          |                                                     |      |               |               |
20
| [[p5.k8s.ooo]]  | production        |              | server34 server36 server38                               | "argo":https://argocd-server.argocd.svc.p5.k8s.ooo  |      |             - |               |
21
| [[p6.k8s.ooo]]  | production        |              | server67 server69 server71                               | "argo":https://argocd-server.argocd.svc.p6.k8s.ooo  | x    | 147.78.194.13 |    2021-10-05 |
22
| [[p10.k8s.ooo]] | production        |              | server63 server65 server83                               | "argo":https://argocd-server.argocd.svc.p10.k8s.ooo | x    | 147.78.194.12 |    2021-10-05 |
23
24 21 Nico Schottelius
25 1 Nico Schottelius
h2. General architecture and components overview
26
27
* All k8s clusters are IPv6 only
28
* We use BGP peering to propagate podcidr and serviceCidr networks to our infrastructure
29
* The main public testing repository is "ungleich-k8s":https://code.ungleich.ch/ungleich-public/ungleich-k8s
30 18 Nico Schottelius
** Private configurations are found in the **k8s-config** repository
31 1 Nico Schottelius
32
h3. Cluster types
33
34 28 Nico Schottelius
| **Type/Feature**            | **Development**                | **Production**         |
35
| Min No. nodes               | 3 (1 master, 3 worker)         | 5 (3 master, 3 worker) |
36
| Recommended minimum         | 4 (dedicated master, 3 worker) | 8 (3 master, 5 worker) |
37
| Separation of control plane | optional                       | recommended            |
38
| Persistent storage          | required                       | required               |
39
| Number of storage monitors  | 3                              | 5                      |
40 1 Nico Schottelius
41 43 Nico Schottelius
h2. General k8s operations
42 1 Nico Schottelius
43 46 Nico Schottelius
h3. Cheat sheet / external great references
44
45
* "kubectl cheatsheet":https://kubernetes.io/docs/reference/kubectl/cheatsheet/
46
47 69 Nico Schottelius
h3. Allowing to schedule work on the control plane
48
49
* Mostly for single node / test / development clusters
50
* Just remove the master taint as follows
51
52
<pre>
53
kubectl taint nodes --all node-role.kubernetes.io/master-
54
</pre>
55
56
57 44 Nico Schottelius
h3. Get the cluster admin.conf
58
59
* On the masters of each cluster you can find the file @/etc/kubernetes/admin.conf@
60
* To be able to administrate the cluster you can copy the admin.conf to your local machine
61
* Multi cluster debugging can very easy if you name the config ~/cX-admin.conf (see example below)
62
63
<pre>
64
% scp root@server47.place7.ungleich.ch:/etc/kubernetes/admin.conf ~/c2-admin.conf
65
% export KUBECONFIG=~/c2-admin.conf    
66
% kubectl get nodes
67
NAME       STATUS                     ROLES                  AGE   VERSION
68
server47   Ready                      control-plane,master   82d   v1.22.0
69
server48   Ready                      control-plane,master   82d   v1.22.0
70
server49   Ready                      <none>                 82d   v1.22.0
71
server50   Ready                      <none>                 82d   v1.22.0
72
server59   Ready                      control-plane,master   82d   v1.22.0
73
server60   Ready,SchedulingDisabled   <none>                 82d   v1.22.0
74
server61   Ready                      <none>                 82d   v1.22.0
75
server62   Ready                      <none>                 82d   v1.22.0               
76
</pre>
77
78 18 Nico Schottelius
h3. Installing a new k8s cluster
79 8 Nico Schottelius
80 9 Nico Schottelius
* Decide on the cluster name (usually *cX.k8s.ooo*), X counting upwards
81 28 Nico Schottelius
** Using pXX.k8s.ooo for production clusters of placeXX
82 9 Nico Schottelius
* Use cdist to configure the nodes with requirements like crio
83
* Decide between single or multi node control plane setups (see below)
84 28 Nico Schottelius
** Single control plane suitable for development clusters
85 9 Nico Schottelius
86 28 Nico Schottelius
Typical init procedure:
87 9 Nico Schottelius
88 28 Nico Schottelius
* Single control plane: @kubeadm init --config bootstrap/XXX/kubeadm.yaml@
89
* Multi control plane (HA): @kubeadm init --config bootstrap/XXX/kubeadm.yaml --upload-certs@
90 10 Nico Schottelius
91 29 Nico Schottelius
h3. Deleting a pod that is hanging in terminating state
92
93
<pre>
94
kubectl delete pod <PODNAME> --grace-period=0 --force --namespace <NAMESPACE>
95
</pre>
96
97
(from https://stackoverflow.com/questions/35453792/pods-stuck-in-terminating-status)
98
99 42 Nico Schottelius
h3. Listing nodes of a cluster
100
101
<pre>
102
[15:05] bridge:~% kubectl get nodes
103
NAME       STATUS   ROLES                  AGE   VERSION
104
server22   Ready    <none>                 52d   v1.22.0
105
server23   Ready    <none>                 52d   v1.22.2
106
server24   Ready    <none>                 52d   v1.22.0
107
server25   Ready    <none>                 52d   v1.22.0
108
server26   Ready    <none>                 52d   v1.22.0
109
server27   Ready    <none>                 52d   v1.22.0
110
server63   Ready    control-plane,master   52d   v1.22.0
111
server64   Ready    <none>                 52d   v1.22.0
112
server65   Ready    control-plane,master   52d   v1.22.0
113
server66   Ready    <none>                 52d   v1.22.0
114
server83   Ready    control-plane,master   52d   v1.22.0
115
server84   Ready    <none>                 52d   v1.22.0
116
server85   Ready    <none>                 52d   v1.22.0
117
server86   Ready    <none>                 52d   v1.22.0
118
</pre>
119
120
121 41 Nico Schottelius
h3. Removing / draining a node
122
123
Usually @kubectl drain server@ should do the job, but sometimes we need to be more aggressive:
124
125
<pre>
126
kubectl drain --delete-emptydir-data --ignore-daemonsets server23
127 42 Nico Schottelius
</pre>
128
129
h3. Readding a node after draining
130
131
<pre>
132
kubectl uncordon serverXX
133 1 Nico Schottelius
</pre>
134 43 Nico Schottelius
135 50 Nico Schottelius
h3. (Re-)joining worker nodes after creating the cluster
136 49 Nico Schottelius
137
* We need to have an up-to-date token
138
* We use different join commands for the workers and control plane nodes
139
140
Generating the join command on an existing control plane node:
141
142
<pre>
143
kubeadm token create --print-join-command
144
</pre>
145
146 50 Nico Schottelius
h3. (Re-)joining control plane nodes after creating the cluster
147 1 Nico Schottelius
148 50 Nico Schottelius
* We generate the token again
149
* We upload the certificates
150
* We need to combine/create the join command for the control plane node
151
152
Example session:
153
154
<pre>
155
% kubeadm token create --print-join-command
156
kubeadm join p10-api.k8s.ooo:6443 --token xmff4i.ABC --discovery-token-ca-cert-hash sha256:longhash 
157
158
% kubeadm init phase upload-certs --upload-certs
159
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
160
[upload-certs] Using certificate key:
161
CERTKEY
162
163
# Then we use these two outputs on the joining node:
164
165
kubeadm join p10-api.k8s.ooo:6443 --token xmff4i.ABC --discovery-token-ca-cert-hash sha256:longhash --control-plane --certificate-key CERTKEY
166
</pre>
167
168
Commands to be used on a control plane node:
169
170
<pre>
171
kubeadm token create --print-join-command
172
kubeadm init phase upload-certs --upload-certs
173
</pre>
174
175
Commands to be used on the joining node:
176
177
<pre>
178
JOINCOMMAND --control-plane --certificate-key CERTKEY
179
</pre>
180 49 Nico Schottelius
181 51 Nico Schottelius
SEE ALSO
182
183
* https://stackoverflow.com/questions/63936268/how-to-generate-kubeadm-token-for-secondary-control-plane-nodes
184
* https://blog.scottlowe.org/2019/08/15/reconstructing-the-join-command-for-kubeadm/
185
186 53 Nico Schottelius
h3. How to fix etcd does not start when rejoining a kubernetes cluster as a control plane
187 52 Nico Schottelius
188
If during the above step etcd does not come up, @kubeadm join@ can hang as follows:
189
190
<pre>
191
[control-plane] Creating static Pod manifest for "kube-apiserver"                                                              
192
[control-plane] Creating static Pod manifest for "kube-controller-manager"                                                     
193
[control-plane] Creating static Pod manifest for "kube-scheduler"                                                              
194
[check-etcd] Checking that the etcd cluster is healthy                                                                         
195
error execution phase check-etcd: etcd cluster is not healthy: failed to dial endpoint https://[2a0a:e5c0:10:1:225:b3ff:fe20:37
196
8a]:2379 with maintenance client: context deadline exceeded                                                                    
197
To see the stack trace of this error execute with --v=5 or higher         
198
</pre>
199
200
Then the problem is likely that the etcd server is still a member of the cluster. We first need to remove it from the etcd cluster and then the join works.
201
202
To fix this we do:
203
204
* Find a working etcd pod
205
* Find the etcd members / member list
206
* Remove the etcd member that we want to re-join the cluster
207
208
209
<pre>
210
# Find the etcd pods
211
kubectl -n kube-system get pods -l component=etcd,tier=control-plane
212
213
# Get the list of etcd servers with the member id 
214
kubectl exec -n kube-system -ti ETCDPODNAME -- etcdctl --endpoints '[::1]:2379' --cacert /etc/kubernetes/pki/etcd/ca.crt --cert  /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member list
215
216
# Remove the member
217
kubectl exec -n kube-system -ti ETCDPODNAME -- etcdctl --endpoints '[::1]:2379' --cacert /etc/kubernetes/pki/etcd/ca.crt --cert  /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member remove MEMBERID
218
</pre>
219
220
Sample session:
221
222
<pre>
223
[10:48] line:~% kubectl -n kube-system get pods -l component=etcd,tier=control-plane
224
NAME            READY   STATUS    RESTARTS     AGE
225
etcd-server63   1/1     Running   0            3m11s
226
etcd-server65   1/1     Running   3            7d2h
227
etcd-server83   1/1     Running   8 (6d ago)   7d2h
228
[10:48] line:~% kubectl exec -n kube-system -ti etcd-server65 -- etcdctl --endpoints '[::1]:2379' --cacert /etc/kubernetes/pki/etcd/ca.crt --cert  /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member list
229
356891cd676df6e4, started, server65, https://[2a0a:e5c0:10:1:225:b3ff:fe20:375c]:2380, https://[2a0a:e5c0:10:1:225:b3ff:fe20:375c]:2379, false
230
371b8a07185dee7e, started, server63, https://[2a0a:e5c0:10:1:225:b3ff:fe20:378a]:2380, https://[2a0a:e5c0:10:1:225:b3ff:fe20:378a]:2379, false
231
5942bc58307f8af9, started, server83, https://[2a0a:e5c0:10:1:3e4a:92ff:fe79:bb98]:2380, https://[2a0a:e5c0:10:1:3e4a:92ff:fe79:bb98]:2379, false
232
233
[10:48] line:~% kubectl exec -n kube-system -ti etcd-server65 -- etcdctl --endpoints '[::1]:2379' --cacert /etc/kubernetes/pki/etcd/ca.crt --cert  /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member remove 371b8a07185dee7e
234
Member 371b8a07185dee7e removed from cluster e3c0805f592a8f77
235 1 Nico Schottelius
236
</pre>
237
238
SEE ALSO
239
240
* We found the solution using https://stackoverflow.com/questions/67921552/re-installed-node-cannot-join-kubernetes-cluster
241 56 Nico Schottelius
242 62 Nico Schottelius
h2. Calico CNI
243
244
245
h3. Calico Installation
246
247
* We install "calico using helm":https://docs.projectcalico.org/getting-started/kubernetes/helm
248
* This has the following advantages:
249
** Easy to upgrade
250
** Does not require os to configure IPv6/dual stack settings as the tigera operator figures out things on its own
251
252
Usually plain calico can be installed directly using:
253
254
<pre>
255
helm repo add projectcalico https://docs.projectcalico.org/charts
256
helm install calico projectcalico/tigera-operator --version v3.20.2
257
</pre>
258
259
h3. Installing calicoctl
260
261
To be able to manage and configure calico, we need to 
262
"install calicoctl (we choose the version as a pod)":https://docs.projectcalico.org/getting-started/clis/calicoctl/install#install-calicoctl-as-a-kubernetes-pod
263
264
<pre>
265
kubectl apply -f https://docs.projectcalico.org/manifests/calicoctl.yaml
266
</pre>
267
268 70 Nico Schottelius
And making it easier accessible by alias:
269
270
<pre>
271
alias calicoctl="kubectl exec -i -n kube-system calicoctl -- /calicoctl"
272
</pre>
273
274 62 Nico Schottelius
h3. Calico configuration
275
276 63 Nico Schottelius
By default our k8s clusters "BGP peer":https://docs.projectcalico.org/networking/bgp
277
with an upstream router to propagate podcidr and servicecidr.
278 62 Nico Schottelius
279
Default settings in our infrastructure:
280
281
* We use a full-mesh using the @nodeToNodeMeshEnabled: true@ option
282
* We keep the original next hop so that *only* the server with the pod is announcing it (instead of ecmp)
283 1 Nico Schottelius
* We use private ASNs for k8s clusters
284 63 Nico Schottelius
* We do *not* use any overlay
285 62 Nico Schottelius
286
After installing calico and calicoctl the last step of the installation is usually:
287
288 1 Nico Schottelius
<pre>
289 79 Nico Schottelius
calicoctl create -f - < calico-bgp.yaml
290 62 Nico Schottelius
</pre>
291
292
293
A sample BGP configuration:
294
295
<pre>
296
---
297
apiVersion: projectcalico.org/v3
298
kind: BGPConfiguration
299
metadata:
300
  name: default
301
spec:
302
  logSeverityScreen: Info
303
  nodeToNodeMeshEnabled: true
304
  asNumber: 65534
305
  serviceClusterIPs:
306
  - cidr: 2a0a:e5c0:10:3::/108
307
  serviceExternalIPs:
308
  - cidr: 2a0a:e5c0:10:3::/108
309
---
310
apiVersion: projectcalico.org/v3
311
kind: BGPPeer
312
metadata:
313
  name: router1-place10
314
spec:
315
  peerIP: 2a0a:e5c0:10:1::50
316
  asNumber: 213081
317
  keepOriginalNextHop: true
318
</pre>
319
320 64 Nico Schottelius
h2. ArgoCD / ArgoWorkFlow
321 56 Nico Schottelius
322 60 Nico Schottelius
h3. Argocd Installation
323 1 Nico Schottelius
324 60 Nico Schottelius
As there is no configuration management present yet, argocd is installed using
325
326 1 Nico Schottelius
<pre>
327 60 Nico Schottelius
kubectl create namespace argocd
328 86 Nico Schottelius
329
# Version 2.2.3
330
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v2.2.3/manifests/install.yaml
331
332
# OR: latest stable
333 60 Nico Schottelius
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
334 1 Nico Schottelius
</pre>
335 56 Nico Schottelius
336 60 Nico Schottelius
* See https://argo-cd.readthedocs.io/en/stable/
337 1 Nico Schottelius
338 60 Nico Schottelius
h3. Get the argocd credentials
339
340
<pre>
341
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo ""
342
</pre>
343 52 Nico Schottelius
344 87 Nico Schottelius
h3. Accessing argocd
345
346
In regular IPv6 clusters:
347
348
* Navigate to https://argocd-server.argocd.CLUSTERDOMAIN
349
350
In legacy IPv4 clusters
351
352
<pre>
353
kubectl --namespace argocd port-forward svc/argocd-server 8080:80
354
</pre>
355
356 68 Nico Schottelius
h3. Using the argocd webhook to trigger changes
357 67 Nico Schottelius
358
* To trigger changes post json https://argocd.example.com/api/webhook
359
360 72 Nico Schottelius
h3. Deploying an application
361
362
* Applications are deployed via git towards gitea (code.ungleich.ch) and then pulled by argo
363 73 Nico Schottelius
* Always include the *redmine-url* pointing to the (customer) ticket
364
** Also add the support-url if it exists
365 72 Nico Schottelius
366
Application sample
367
368
<pre>
369
apiVersion: argoproj.io/v1alpha1
370
kind: Application
371
metadata:
372
  name: gitea-CUSTOMER
373
  namespace: argocd
374
spec:
375
  destination:
376
    namespace: default
377
    server: 'https://kubernetes.default.svc'
378
  source:
379
    path: apps/prod/gitea
380
    repoURL: 'https://code.ungleich.ch/ungleich-intern/k8s-config.git'
381
    targetRevision: HEAD
382
    helm:
383
      parameters:
384
        - name: storage.data.storageClass
385
          value: rook-ceph-block-hdd
386
        - name: storage.data.size
387
          value: 200Gi
388
        - name: storage.db.storageClass
389
          value: rook-ceph-block-ssd
390
        - name: storage.db.size
391
          value: 10Gi
392
        - name: storage.letsencrypt.storageClass
393
          value: rook-ceph-block-hdd
394
        - name: storage.letsencrypt.size
395
          value: 50Mi
396
        - name: letsencryptStaging
397
          value: 'no'
398
        - name: fqdn
399
          value: 'code.verua.online'
400
  project: default
401
  syncPolicy:
402
    automated:
403
      prune: true
404
      selfHeal: true
405
  info:
406
    - name: 'redmine-url'
407
      value: 'https://redmine.ungleich.ch/issues/ISSUEID'
408
    - name: 'support-url'
409
      value: 'https://support.ungleich.ch/Ticket/Display.html?id=TICKETID'
410
</pre>
411
412 80 Nico Schottelius
h2. Helm related operations and conventions
413 55 Nico Schottelius
414 61 Nico Schottelius
We use helm charts extensively.
415
416
* In production, they are managed via argocd
417
* In development, helm chart can de developed and deployed manually using the helm utility.
418
419 55 Nico Schottelius
h3. Installing a helm chart
420
421
One can use the usual pattern of
422
423
<pre>
424
helm install <releasename> <chartdirectory>
425
</pre>
426
427
However often you want to reinstall/update when testing helm charts. The following pattern is "better", because it allows you to reinstall, if it is already installed:
428
429
<pre>
430
helm upgrade --install <releasename> <chartdirectory>
431 1 Nico Schottelius
</pre>
432 80 Nico Schottelius
433
h3. Naming services and deployments in helm charts [Application labels]
434
435
* We always have {{ .Release.Name }} to identify the current "instance"
436
* Deployments:
437
** use @app: <what it is>@, f.i. @app: nginx@, @app: postgres@, ...
438 81 Nico Schottelius
* See more about standard labels on
439
** https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/
440
** https://helm.sh/docs/chart_best_practices/labels/
441 55 Nico Schottelius
442 43 Nico Schottelius
h2. Rook / Ceph Related Operations
443
444 71 Nico Schottelius
h3. Executing ceph commands
445
446
Using the ceph-tools pod as follows:
447
448
<pre>
449
kubectl exec -n rook-ceph -ti $(kubectl -n rook-ceph get pods -l app=rook-ceph-tools -o jsonpath='{.items[*].metadata.name}') -- ceph -s
450
</pre>
451
452 43 Nico Schottelius
h3. Inspecting the logs of a specific server
453
454
<pre>
455
# Get the related pods
456
kubectl -n rook-ceph get pods -l app=rook-ceph-osd-prepare 
457
...
458
459
# Inspect the logs of a specific pod
460
kubectl -n rook-ceph logs -f rook-ceph-osd-prepare-server23--1-444qx
461
462 71 Nico Schottelius
</pre>
463
464
h3. Inspecting the logs of the rook-ceph-operator
465
466
<pre>
467
kubectl -n rook-ceph logs -f -l app=rook-ceph-operator
468 43 Nico Schottelius
</pre>
469
470
h3. Triggering server prepare / adding new osds
471
472
The rook-ceph-operator triggers/watches/creates pods to maintain hosts. To trigger a full "re scan", simply delete that pod:
473
474
<pre>
475
kubectl -n rook-ceph delete pods -l app=rook-ceph-operator
476
</pre>
477
478
This will cause all the @rook-ceph-osd-prepare-..@ jobs to be recreated and thus OSDs to be created, if new disks have been added.
479
480
h3. Removing an OSD
481
482
* See "Ceph OSD Management":https://rook.io/docs/rook/v1.7/ceph-osd-mgmt.html
483 77 Nico Schottelius
* More specifically: https://github.com/rook/rook/blob/release-1.7/cluster/examples/kubernetes/ceph/osd-purge.yaml
484 41 Nico Schottelius
485 76 Nico Schottelius
h2. Harbor
486
487
* We user "Harbor":https://goharbor.io/ for caching and as an image registry. Internal app reference: apps/prod/harbor.
488
* The admin password is in the password store, auto generated per cluster
489
* At the moment harbor only authenticates against the internal ldap tree
490
491
h3. LDAP configuration
492
493
* The url needs to be ldaps://...
494
* uid = uid
495
* rest standard
496 75 Nico Schottelius
497 82 Nico Schottelius
h2. Nextcloud
498
499 85 Nico Schottelius
h3. How to get the nextcloud credentials 
500 84 Nico Schottelius
501
* The initial username is set to "nextcloud"
502
* The password is autogenerated and saved in a kubernetes secret
503
504
<pre>
505 85 Nico Schottelius
kubectl get secret RELEASENAME-nextcloud -o jsonpath="{.data.PASSWORD}" | base64 -d; echo "" 
506 84 Nico Schottelius
</pre>
507
508 83 Nico Schottelius
h3. How to fix "Access through untrusted domain"
509
510 82 Nico Schottelius
* Nextcloud stores the initial domain configuration
511 1 Nico Schottelius
* If the FQDN is changed, it will show the error message "Access through untrusted domain"
512 82 Nico Schottelius
* To fix, edit /var/www/html/config/config.php and correct the domain
513 83 Nico Schottelius
* Then delete the pods
514 82 Nico Schottelius
515 1 Nico Schottelius
h2. Infrastructure versions
516 35 Nico Schottelius
517 57 Nico Schottelius
h3. ungleich kubernetes infrastructure v5 (2021-10)
518 1 Nico Schottelius
519 57 Nico Schottelius
Clusters are configured / setup in this order:
520
521
* Bootstrap via kubeadm
522 59 Nico Schottelius
* "Networking via calico + BGP (non ECMP) using helm":https://docs.projectcalico.org/getting-started/kubernetes/helm
523
* "ArgoCD for CD":https://argo-cd.readthedocs.io/en/stable/
524
** "rook for storage via argocd":https://rook.io/
525 58 Nico Schottelius
** haproxy for in IPv6-cluster-IPv4-to-IPv6 proxy via argocd
526
** "kubernetes-secret-generator for in cluster secrets":https://github.com/mittwald/kubernetes-secret-generator
527
** "ungleich-certbot managing certs and nginx":https://hub.docker.com/repository/docker/ungleich/ungleich-certbot
528
529 57 Nico Schottelius
530
h3. ungleich kubernetes infrastructure v4 (2021-09)
531
532 54 Nico Schottelius
* rook is configured via manifests instead of using the rook-ceph-cluster helm chart
533 1 Nico Schottelius
* The rook operator is still being installed via helm
534 35 Nico Schottelius
535 57 Nico Schottelius
h3. ungleich kubernetes infrastructure v3 (2021-07)
536 1 Nico Schottelius
537 10 Nico Schottelius
* rook is now installed via helm via argocd instead of directly via manifests
538 28 Nico Schottelius
539 57 Nico Schottelius
h3. ungleich kubernetes infrastructure v2 (2021-05)
540 28 Nico Schottelius
541
* Replaced fluxv2 from ungleich k8s v1 with argocd
542 1 Nico Schottelius
** argocd can apply helm templates directly without needing to go through Chart releases
543 28 Nico Schottelius
* We are also using argoflow for build flows
544
* Planned to add "kaniko":https://github.com/GoogleContainerTools/kaniko for image building
545
546 57 Nico Schottelius
h3. ungleich kubernetes infrastructure v1 (2021-01)
547 28 Nico Schottelius
548
We are using the following components:
549
550
* "Calico as a CNI":https://www.projectcalico.org/ with BGP, IPv6 only, no encapsulation
551
** Needed for basic networking
552
* "kubernetes-secret-generator":https://github.com/mittwald/kubernetes-secret-generator for creating secrets
553
** Needed so that secrets are not stored in the git repository, but only in the cluster
554
* "ungleich-certbot":https://hub.docker.com/repository/docker/ungleich/ungleich-certbot
555
** Needed to get letsencrypt certificates for services
556
* "rook with ceph rbd + cephfs":https://rook.io/ for storage
557
** rbd for almost everything, *ReadWriteOnce*
558
** cephfs for smaller things, multi access *ReadWriteMany*
559
** Needed for providing persistent storage
560
* "flux v2":https://fluxcd.io/
561
** Needed to manage resources automatically