Project

General

Profile

The ungleich kubernetes infrastructure » History » Version 144

Nico Schottelius, 09/09/2022 08:36 PM

1 22 Nico Schottelius
h1. The ungleich kubernetes infrastructure and ungleich kubernetes manual
2 1 Nico Schottelius
3 3 Nico Schottelius
{{toc}}
4
5 1 Nico Schottelius
h2. Status
6
7 28 Nico Schottelius
This document is **pre-production**.
8
This document is to become the ungleich kubernetes infrastructure overview as well as the ungleich kubernetes manual.
9 1 Nico Schottelius
10 10 Nico Schottelius
h2. k8s clusters
11
12 123 Nico Schottelius
| Cluster            | Purpose/Setup     | Maintainer | Master(s)                     | argo                                                   | v4 http proxy | last verified |
13
| c0.k8s.ooo         | Dev               | -          | UNUSED                        |                                                        |               |    2021-10-05 |
14
| c1.k8s.ooo         | retired           |            | -                             |                                                        |               |    2022-03-15 |
15
| c2.k8s.ooo         | Dev p7 HW         | Nico       | server47 server53 server54    | "argo":https://argocd-server.argocd.svc.c2.k8s.ooo     |               |    2021-10-05 |
16
| c3.k8s.ooo         | retired           | -          | -                             |                                                        |               |    2021-10-05 |
17
| c4.k8s.ooo         | Dev2 p7 HW        | Jin-Guk    | server52 server53 server54    |                                                        |               |             - |
18
| c5.k8s.ooo         | retired           |            | -                             |                                                        |               |    2022-03-15 |
19
| c6.k8s.ooo         | Dev p6 VM Jin-Guk | Jin-Guk    |                               |                                                        |               |               |
20
| [[p5.k8s.ooo]]     | production        |            | server34 server36 server38    | "argo":https://argocd-server.argocd.svc.p5.k8s.ooo     | -             |               |
21
| [[p5-cow.k8s.ooo]] | production        | Nico       | server47 server51 server55    | "argo":https://argocd-server.argocd.svc.p5-cow.k8s.ooo |               |    2022-08-27 |
22
| [[p6.k8s.ooo]]     | production        |            | server67 server69 server71    | "argo":https://argocd-server.argocd.svc.p6.k8s.ooo     | 147.78.194.13 |    2021-10-05 |
23
| [[p10.k8s.ooo]]    | production        |            | server63 server65 server83    | "argo":https://argocd-server.argocd.svc.p10.k8s.ooo    | 147.78.194.12 |    2021-10-05 |
24
| [[k8s.ge.nau.so]]  | development       |            | server107 server108 server109 | "argo":https://argocd-server.argocd.svc.k8s.ge.nau.so  |               |               |
25
| [[dev.k8s.ooo]]    | development       |            | server110 server111 server112 | "argo":https://argocd-server.argocd.svc.dev.k8s.ooo    | -             |    2022-07-08 |
26 142 Nico Schottelius
| [[server121.k8s.ooo]] | production | Nico | server121 | | | 2022-09-06 |
27 21 Nico Schottelius
28 1 Nico Schottelius
h2. General architecture and components overview
29
30
* All k8s clusters are IPv6 only
31
* We use BGP peering to propagate podcidr and serviceCidr networks to our infrastructure
32
* The main public testing repository is "ungleich-k8s":https://code.ungleich.ch/ungleich-public/ungleich-k8s
33 18 Nico Schottelius
** Private configurations are found in the **k8s-config** repository
34 1 Nico Schottelius
35
h3. Cluster types
36
37 28 Nico Schottelius
| **Type/Feature**            | **Development**                | **Production**         |
38
| Min No. nodes               | 3 (1 master, 3 worker)         | 5 (3 master, 3 worker) |
39
| Recommended minimum         | 4 (dedicated master, 3 worker) | 8 (3 master, 5 worker) |
40
| Separation of control plane | optional                       | recommended            |
41
| Persistent storage          | required                       | required               |
42
| Number of storage monitors  | 3                              | 5                      |
43 1 Nico Schottelius
44 43 Nico Schottelius
h2. General k8s operations
45 1 Nico Schottelius
46 46 Nico Schottelius
h3. Cheat sheet / external great references
47
48
* "kubectl cheatsheet":https://kubernetes.io/docs/reference/kubectl/cheatsheet/
49
50 117 Nico Schottelius
h3. Allowing to schedule work on the control plane / removing node taints
51 69 Nico Schottelius
52
* Mostly for single node / test / development clusters
53
* Just remove the master taint as follows
54
55
<pre>
56
kubectl taint nodes --all node-role.kubernetes.io/master-
57 118 Nico Schottelius
kubectl taint nodes --all node-role.kubernetes.io/control-plane-
58 69 Nico Schottelius
</pre>
59 1 Nico Schottelius
60 117 Nico Schottelius
You can check the node taints using @kubectl describe node ...@
61 69 Nico Schottelius
62 44 Nico Schottelius
h3. Get the cluster admin.conf
63
64
* On the masters of each cluster you can find the file @/etc/kubernetes/admin.conf@
65
* To be able to administrate the cluster you can copy the admin.conf to your local machine
66
* Multi cluster debugging can very easy if you name the config ~/cX-admin.conf (see example below)
67
68
<pre>
69
% scp root@server47.place7.ungleich.ch:/etc/kubernetes/admin.conf ~/c2-admin.conf
70
% export KUBECONFIG=~/c2-admin.conf    
71
% kubectl get nodes
72
NAME       STATUS                     ROLES                  AGE   VERSION
73
server47   Ready                      control-plane,master   82d   v1.22.0
74
server48   Ready                      control-plane,master   82d   v1.22.0
75
server49   Ready                      <none>                 82d   v1.22.0
76
server50   Ready                      <none>                 82d   v1.22.0
77
server59   Ready                      control-plane,master   82d   v1.22.0
78
server60   Ready,SchedulingDisabled   <none>                 82d   v1.22.0
79
server61   Ready                      <none>                 82d   v1.22.0
80
server62   Ready                      <none>                 82d   v1.22.0               
81
</pre>
82
83 18 Nico Schottelius
h3. Installing a new k8s cluster
84 8 Nico Schottelius
85 9 Nico Schottelius
* Decide on the cluster name (usually *cX.k8s.ooo*), X counting upwards
86 28 Nico Schottelius
** Using pXX.k8s.ooo for production clusters of placeXX
87 9 Nico Schottelius
* Use cdist to configure the nodes with requirements like crio
88
* Decide between single or multi node control plane setups (see below)
89 28 Nico Schottelius
** Single control plane suitable for development clusters
90 9 Nico Schottelius
91 28 Nico Schottelius
Typical init procedure:
92 9 Nico Schottelius
93 28 Nico Schottelius
* Single control plane: @kubeadm init --config bootstrap/XXX/kubeadm.yaml@
94
* Multi control plane (HA): @kubeadm init --config bootstrap/XXX/kubeadm.yaml --upload-certs@
95 10 Nico Schottelius
96 29 Nico Schottelius
h3. Deleting a pod that is hanging in terminating state
97
98
<pre>
99
kubectl delete pod <PODNAME> --grace-period=0 --force --namespace <NAMESPACE>
100
</pre>
101
102
(from https://stackoverflow.com/questions/35453792/pods-stuck-in-terminating-status)
103
104 42 Nico Schottelius
h3. Listing nodes of a cluster
105
106
<pre>
107
[15:05] bridge:~% kubectl get nodes
108
NAME       STATUS   ROLES                  AGE   VERSION
109
server22   Ready    <none>                 52d   v1.22.0
110
server23   Ready    <none>                 52d   v1.22.2
111
server24   Ready    <none>                 52d   v1.22.0
112
server25   Ready    <none>                 52d   v1.22.0
113
server26   Ready    <none>                 52d   v1.22.0
114
server27   Ready    <none>                 52d   v1.22.0
115
server63   Ready    control-plane,master   52d   v1.22.0
116
server64   Ready    <none>                 52d   v1.22.0
117
server65   Ready    control-plane,master   52d   v1.22.0
118
server66   Ready    <none>                 52d   v1.22.0
119
server83   Ready    control-plane,master   52d   v1.22.0
120
server84   Ready    <none>                 52d   v1.22.0
121
server85   Ready    <none>                 52d   v1.22.0
122
server86   Ready    <none>                 52d   v1.22.0
123
</pre>
124
125 41 Nico Schottelius
h3. Removing / draining a node
126
127
Usually @kubectl drain server@ should do the job, but sometimes we need to be more aggressive:
128
129 1 Nico Schottelius
<pre>
130 103 Nico Schottelius
kubectl drain --delete-emptydir-data --ignore-daemonsets serverXX
131 42 Nico Schottelius
</pre>
132
133
h3. Readding a node after draining
134
135
<pre>
136
kubectl uncordon serverXX
137 1 Nico Schottelius
</pre>
138 43 Nico Schottelius
139 50 Nico Schottelius
h3. (Re-)joining worker nodes after creating the cluster
140 49 Nico Schottelius
141
* We need to have an up-to-date token
142
* We use different join commands for the workers and control plane nodes
143
144
Generating the join command on an existing control plane node:
145
146
<pre>
147
kubeadm token create --print-join-command
148
</pre>
149
150 50 Nico Schottelius
h3. (Re-)joining control plane nodes after creating the cluster
151 1 Nico Schottelius
152 50 Nico Schottelius
* We generate the token again
153
* We upload the certificates
154
* We need to combine/create the join command for the control plane node
155
156
Example session:
157
158
<pre>
159
% kubeadm token create --print-join-command
160
kubeadm join p10-api.k8s.ooo:6443 --token xmff4i.ABC --discovery-token-ca-cert-hash sha256:longhash 
161
162
% kubeadm init phase upload-certs --upload-certs
163
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
164
[upload-certs] Using certificate key:
165
CERTKEY
166
167
# Then we use these two outputs on the joining node:
168
169
kubeadm join p10-api.k8s.ooo:6443 --token xmff4i.ABC --discovery-token-ca-cert-hash sha256:longhash --control-plane --certificate-key CERTKEY
170
</pre>
171
172
Commands to be used on a control plane node:
173
174
<pre>
175
kubeadm token create --print-join-command
176
kubeadm init phase upload-certs --upload-certs
177
</pre>
178
179
Commands to be used on the joining node:
180
181
<pre>
182
JOINCOMMAND --control-plane --certificate-key CERTKEY
183
</pre>
184 49 Nico Schottelius
185 51 Nico Schottelius
SEE ALSO
186
187
* https://stackoverflow.com/questions/63936268/how-to-generate-kubeadm-token-for-secondary-control-plane-nodes
188
* https://blog.scottlowe.org/2019/08/15/reconstructing-the-join-command-for-kubeadm/
189
190 53 Nico Schottelius
h3. How to fix etcd does not start when rejoining a kubernetes cluster as a control plane
191 52 Nico Schottelius
192
If during the above step etcd does not come up, @kubeadm join@ can hang as follows:
193
194
<pre>
195
[control-plane] Creating static Pod manifest for "kube-apiserver"                                                              
196
[control-plane] Creating static Pod manifest for "kube-controller-manager"                                                     
197
[control-plane] Creating static Pod manifest for "kube-scheduler"                                                              
198
[check-etcd] Checking that the etcd cluster is healthy                                                                         
199
error execution phase check-etcd: etcd cluster is not healthy: failed to dial endpoint https://[2a0a:e5c0:10:1:225:b3ff:fe20:37
200
8a]:2379 with maintenance client: context deadline exceeded                                                                    
201
To see the stack trace of this error execute with --v=5 or higher         
202
</pre>
203
204
Then the problem is likely that the etcd server is still a member of the cluster. We first need to remove it from the etcd cluster and then the join works.
205
206
To fix this we do:
207
208
* Find a working etcd pod
209
* Find the etcd members / member list
210
* Remove the etcd member that we want to re-join the cluster
211
212
213
<pre>
214
# Find the etcd pods
215
kubectl -n kube-system get pods -l component=etcd,tier=control-plane
216
217
# Get the list of etcd servers with the member id 
218
kubectl exec -n kube-system -ti ETCDPODNAME -- etcdctl --endpoints '[::1]:2379' --cacert /etc/kubernetes/pki/etcd/ca.crt --cert  /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member list
219
220
# Remove the member
221
kubectl exec -n kube-system -ti ETCDPODNAME -- etcdctl --endpoints '[::1]:2379' --cacert /etc/kubernetes/pki/etcd/ca.crt --cert  /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member remove MEMBERID
222
</pre>
223
224
Sample session:
225
226
<pre>
227
[10:48] line:~% kubectl -n kube-system get pods -l component=etcd,tier=control-plane
228
NAME            READY   STATUS    RESTARTS     AGE
229
etcd-server63   1/1     Running   0            3m11s
230
etcd-server65   1/1     Running   3            7d2h
231
etcd-server83   1/1     Running   8 (6d ago)   7d2h
232
[10:48] line:~% kubectl exec -n kube-system -ti etcd-server65 -- etcdctl --endpoints '[::1]:2379' --cacert /etc/kubernetes/pki/etcd/ca.crt --cert  /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member list
233
356891cd676df6e4, started, server65, https://[2a0a:e5c0:10:1:225:b3ff:fe20:375c]:2380, https://[2a0a:e5c0:10:1:225:b3ff:fe20:375c]:2379, false
234
371b8a07185dee7e, started, server63, https://[2a0a:e5c0:10:1:225:b3ff:fe20:378a]:2380, https://[2a0a:e5c0:10:1:225:b3ff:fe20:378a]:2379, false
235
5942bc58307f8af9, started, server83, https://[2a0a:e5c0:10:1:3e4a:92ff:fe79:bb98]:2380, https://[2a0a:e5c0:10:1:3e4a:92ff:fe79:bb98]:2379, false
236
237
[10:48] line:~% kubectl exec -n kube-system -ti etcd-server65 -- etcdctl --endpoints '[::1]:2379' --cacert /etc/kubernetes/pki/etcd/ca.crt --cert  /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member remove 371b8a07185dee7e
238
Member 371b8a07185dee7e removed from cluster e3c0805f592a8f77
239 1 Nico Schottelius
240
</pre>
241
242
SEE ALSO
243
244
* We found the solution using https://stackoverflow.com/questions/67921552/re-installed-node-cannot-join-kubernetes-cluster
245 56 Nico Schottelius
246 101 Nico Schottelius
h3. Hardware Maintenance using ungleich-hardware
247
248
Use the following manifest and replace the HOST with the actual host:
249
250
<pre>
251
apiVersion: v1
252
kind: Pod
253
metadata:
254
  name: ungleich-hardware-HOST
255
spec:
256
  containers:
257
  - name: ungleich-hardware
258
    image: ungleich/ungleich-hardware:0.0.5
259
    args:
260
    - sleep
261
    - "1000000"
262
    volumeMounts:
263
      - mountPath: /dev
264
        name: dev
265
    securityContext:
266
      privileged: true
267
  nodeSelector:
268
    kubernetes.io/hostname: "HOST"
269
270
  volumes:
271
    - name: dev
272
      hostPath:
273
        path: /dev
274
</pre>
275
276 102 Nico Schottelius
Also see: [[The_ungleich_hardware_maintenance_guide]]
277
278 105 Nico Schottelius
h3. Triggering a cronjob / creating a job from a cronjob
279 104 Nico Schottelius
280
To test a cronjob, we can create a job from a cronjob:
281
282
<pre>
283
kubectl create job --from=cronjob/volume2-daily-backup volume2-manual
284
</pre>
285
286
This creates a job volume2-manual based on the cronjob  volume2-daily
287
288 112 Nico Schottelius
h3. su-ing into a user that has nologin shell set
289
290
Many times users are having nologin as their shell inside the container. To be able to execute maintenance commands within the
291
container, we can use @su -s /bin/sh@ like this:
292
293
<pre>
294
su -s /bin/sh -c '/path/to/your/script' testuser
295
</pre>
296
297
Found on https://serverfault.com/questions/351046/how-to-run-command-as-user-who-has-usr-sbin-nologin-as-shell
298
299 113 Nico Schottelius
h3. How to print a secret value
300
301
Assuming you want the "password" item from a secret, use:
302
303
<pre>
304
kubectl get secret SECRETNAME -o jsonpath="{.data.password}" | base64 -d; echo "" 
305
</pre>
306
307 62 Nico Schottelius
h2. Calico CNI
308
309
h3. Calico Installation
310
311
* We install "calico using helm":https://docs.projectcalico.org/getting-started/kubernetes/helm
312
* This has the following advantages:
313
** Easy to upgrade
314
** Does not require os to configure IPv6/dual stack settings as the tigera operator figures out things on its own
315
316
Usually plain calico can be installed directly using:
317
318
<pre>
319 125 Nico Schottelius
VERSION=v3.23.3
320 120 Nico Schottelius
helm repo add projectcalico https://docs.projectcalico.org/charts
321 124 Nico Schottelius
helm upgrade --install --namespace tigera calico projectcalico/tigera-operator --version $VERSION --create-namespace
322 1 Nico Schottelius
</pre>
323 92 Nico Schottelius
324
* Check the tags on https://github.com/projectcalico/calico/tags for the latest release
325 62 Nico Schottelius
326
h3. Installing calicoctl
327
328 115 Nico Schottelius
* General installation instructions, including binary download: https://projectcalico.docs.tigera.io/maintenance/clis/calicoctl/install
329
330 62 Nico Schottelius
To be able to manage and configure calico, we need to 
331
"install calicoctl (we choose the version as a pod)":https://docs.projectcalico.org/getting-started/clis/calicoctl/install#install-calicoctl-as-a-kubernetes-pod
332
333
<pre>
334
kubectl apply -f https://docs.projectcalico.org/manifests/calicoctl.yaml
335
</pre>
336
337 93 Nico Schottelius
Or version specific:
338
339
<pre>
340
kubectl apply -f https://github.com/projectcalico/calico/blob/v3.20.4/manifests/calicoctl.yaml
341 97 Nico Schottelius
342
# For 3.22
343
kubectl apply -f https://projectcalico.docs.tigera.io/archive/v3.22/manifests/calicoctl.yaml
344 93 Nico Schottelius
</pre>
345
346 70 Nico Schottelius
And making it easier accessible by alias:
347
348
<pre>
349
alias calicoctl="kubectl exec -i -n kube-system calicoctl -- /calicoctl"
350
</pre>
351
352 62 Nico Schottelius
h3. Calico configuration
353
354 63 Nico Schottelius
By default our k8s clusters "BGP peer":https://docs.projectcalico.org/networking/bgp
355
with an upstream router to propagate podcidr and servicecidr.
356 62 Nico Schottelius
357
Default settings in our infrastructure:
358
359
* We use a full-mesh using the @nodeToNodeMeshEnabled: true@ option
360
* We keep the original next hop so that *only* the server with the pod is announcing it (instead of ecmp)
361 1 Nico Schottelius
* We use private ASNs for k8s clusters
362 63 Nico Schottelius
* We do *not* use any overlay
363 62 Nico Schottelius
364
After installing calico and calicoctl the last step of the installation is usually:
365
366 1 Nico Schottelius
<pre>
367 79 Nico Schottelius
calicoctl create -f - < calico-bgp.yaml
368 62 Nico Schottelius
</pre>
369
370
371
A sample BGP configuration:
372
373
<pre>
374
---
375
apiVersion: projectcalico.org/v3
376
kind: BGPConfiguration
377
metadata:
378
  name: default
379
spec:
380
  logSeverityScreen: Info
381
  nodeToNodeMeshEnabled: true
382
  asNumber: 65534
383
  serviceClusterIPs:
384
  - cidr: 2a0a:e5c0:10:3::/108
385
  serviceExternalIPs:
386
  - cidr: 2a0a:e5c0:10:3::/108
387
---
388
apiVersion: projectcalico.org/v3
389
kind: BGPPeer
390
metadata:
391
  name: router1-place10
392
spec:
393
  peerIP: 2a0a:e5c0:10:1::50
394
  asNumber: 213081
395
  keepOriginalNextHop: true
396
</pre>
397
398 126 Nico Schottelius
h2. Cilium CNI (experimental)
399
400 137 Nico Schottelius
h3. Status
401
402 138 Nico Schottelius
*NO WORKING CILIUM CONFIGURATION FOR IPV6 only modes*
403 137 Nico Schottelius
404 128 Nico Schottelius
h3. BGP configuration
405
406
* The cilium-operator will not start without a correct configmap being present beforehand (see error message below)
407
* Creating the bgp config beforehand as a configmap is thus required.
408
409
The error one gets without the configmap present:
410
411
Pods are hanging with:
412
413
<pre>
414
cilium-bpqm6                       0/1     Init:0/4            0             9s
415
cilium-operator-5947d94f7f-5bmh2   0/1     ContainerCreating   0             9s
416
</pre>
417
418
The error message in the cilium-*perator is:
419
420
<pre>
421
Events:
422
  Type     Reason       Age                From               Message
423
  ----     ------       ----               ----               -------
424
  Normal   Scheduled    80s                default-scheduler  Successfully assigned kube-system/cilium-operator-5947d94f7f-lqcsp to server56
425
  Warning  FailedMount  16s (x8 over 80s)  kubelet            MountVolume.SetUp failed for volume "bgp-config-path" : configmap "bgp-config" not found
426
</pre>
427
428
A correct bgp config looks like this:
429
430
<pre>
431
apiVersion: v1
432
kind: ConfigMap
433
metadata:
434
  name: bgp-config
435
  namespace: kube-system
436
data:
437
  config.yaml: |
438
    peers:
439
      - peer-address: 2a0a:e5c0::46
440
        peer-asn: 209898
441
        my-asn: 65533
442
      - peer-address: 2a0a:e5c0::47
443
        peer-asn: 209898
444
        my-asn: 65533
445
    address-pools:
446
      - name: default
447
        protocol: bgp
448
        addresses:
449
          - 2a0a:e5c0:0:14::/64
450
</pre>
451
452 127 Nico Schottelius
h3. Installation
453
454 130 Nico Schottelius
Adding the repo
455 127 Nico Schottelius
<pre>
456 1 Nico Schottelius
457 127 Nico Schottelius
helm repo add cilium https://helm.cilium.io/
458 129 Nico Schottelius
helm repo update
459 130 Nico Schottelius
</pre>
460 1 Nico Schottelius
461 130 Nico Schottelius
Installing + configuring cilium
462
<pre>
463 129 Nico Schottelius
version=1.12.1
464 135 Nico Schottelius
ipv6pool=2a0a:e5c0:0:14::/112
465 129 Nico Schottelius
466 130 Nico Schottelius
467 1 Nico Schottelius
helm upgrade --install cilium cilium/cilium --version $version \
468 129 Nico Schottelius
  --namespace kube-system \
469
  --set ipv4.enabled=false \
470 1 Nico Schottelius
  --set ipv6.enabled=true \
471 131 Nico Schottelius
  --set ipam.operator.clusterPoolIPv6PodCIDRList=$ipv6pool \
472 1 Nico Schottelius
  --set bgpControlPlane.enabled=true
473 136 Nico Schottelius
474
#   --set bgp.enabled=true --set bgp.announce.podCIDR=true \
475 127 Nico Schottelius
476
# Show possible configuration options
477
helm show values cilium/cilium
478
479 1 Nico Schottelius
</pre>
480 132 Nico Schottelius
481
Using a /64 for ipam.operator.clusterPoolIPv6PodCIDRList fails with:
482
483
<pre>
484
level=fatal msg="Unable to init cluster-pool allocator" error="unable to initialize IPv6 allocator New CIDR set failed; the node CIDR size is too big" subsys=cilium-operator-generic
485
</pre>
486
487 126 Nico Schottelius
488 1 Nico Schottelius
See also https://github.com/cilium/cilium/issues/20756
489 135 Nico Schottelius
490
Seems a /112 is actually working.
491
492
h3. Kernel modules
493
494
Cilium requires the following modules to be loaded on the host (not loaded by default):
495
496
<pre>
497
modprobe  ip6table_raw
498
modprobe  ip6table_filter
499
</pre>
500 133 Nico Schottelius
501 122 Nico Schottelius
h2. ArgoCD 
502 56 Nico Schottelius
503 60 Nico Schottelius
h3. Argocd Installation
504 1 Nico Schottelius
505 116 Nico Schottelius
* See https://argo-cd.readthedocs.io/en/stable/
506
507 60 Nico Schottelius
As there is no configuration management present yet, argocd is installed using
508
509 1 Nico Schottelius
<pre>
510 60 Nico Schottelius
kubectl create namespace argocd
511 86 Nico Schottelius
512 96 Nico Schottelius
# Specific Version
513
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v2.3.2/manifests/install.yaml
514 86 Nico Schottelius
515
# OR: latest stable
516 60 Nico Schottelius
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
517 56 Nico Schottelius
</pre>
518 1 Nico Schottelius
519 116 Nico Schottelius
520 1 Nico Schottelius
521 60 Nico Schottelius
h3. Get the argocd credentials
522
523
<pre>
524
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo ""
525
</pre>
526 52 Nico Schottelius
527 87 Nico Schottelius
h3. Accessing argocd
528
529
In regular IPv6 clusters:
530
531
* Navigate to https://argocd-server.argocd.CLUSTERDOMAIN
532
533
In legacy IPv4 clusters
534
535
<pre>
536
kubectl --namespace argocd port-forward svc/argocd-server 8080:80
537
</pre>
538
539 88 Nico Schottelius
* Navigate to https://localhost:8080
540
541 68 Nico Schottelius
h3. Using the argocd webhook to trigger changes
542 67 Nico Schottelius
543
* To trigger changes post json https://argocd.example.com/api/webhook
544
545 72 Nico Schottelius
h3. Deploying an application
546
547
* Applications are deployed via git towards gitea (code.ungleich.ch) and then pulled by argo
548 73 Nico Schottelius
* Always include the *redmine-url* pointing to the (customer) ticket
549
** Also add the support-url if it exists
550 72 Nico Schottelius
551
Application sample
552
553
<pre>
554
apiVersion: argoproj.io/v1alpha1
555
kind: Application
556
metadata:
557
  name: gitea-CUSTOMER
558
  namespace: argocd
559
spec:
560
  destination:
561
    namespace: default
562
    server: 'https://kubernetes.default.svc'
563
  source:
564
    path: apps/prod/gitea
565
    repoURL: 'https://code.ungleich.ch/ungleich-intern/k8s-config.git'
566
    targetRevision: HEAD
567
    helm:
568
      parameters:
569
        - name: storage.data.storageClass
570
          value: rook-ceph-block-hdd
571
        - name: storage.data.size
572
          value: 200Gi
573
        - name: storage.db.storageClass
574
          value: rook-ceph-block-ssd
575
        - name: storage.db.size
576
          value: 10Gi
577
        - name: storage.letsencrypt.storageClass
578
          value: rook-ceph-block-hdd
579
        - name: storage.letsencrypt.size
580
          value: 50Mi
581
        - name: letsencryptStaging
582
          value: 'no'
583
        - name: fqdn
584
          value: 'code.verua.online'
585
  project: default
586
  syncPolicy:
587
    automated:
588
      prune: true
589
      selfHeal: true
590
  info:
591
    - name: 'redmine-url'
592
      value: 'https://redmine.ungleich.ch/issues/ISSUEID'
593
    - name: 'support-url'
594
      value: 'https://support.ungleich.ch/Ticket/Display.html?id=TICKETID'
595
</pre>
596
597 80 Nico Schottelius
h2. Helm related operations and conventions
598 55 Nico Schottelius
599 61 Nico Schottelius
We use helm charts extensively.
600
601
* In production, they are managed via argocd
602
* In development, helm chart can de developed and deployed manually using the helm utility.
603
604 55 Nico Schottelius
h3. Installing a helm chart
605
606
One can use the usual pattern of
607
608
<pre>
609
helm install <releasename> <chartdirectory>
610
</pre>
611
612
However often you want to reinstall/update when testing helm charts. The following pattern is "better", because it allows you to reinstall, if it is already installed:
613
614
<pre>
615
helm upgrade --install <releasename> <chartdirectory>
616 1 Nico Schottelius
</pre>
617 80 Nico Schottelius
618
h3. Naming services and deployments in helm charts [Application labels]
619
620
* We always have {{ .Release.Name }} to identify the current "instance"
621
* Deployments:
622
** use @app: <what it is>@, f.i. @app: nginx@, @app: postgres@, ...
623 81 Nico Schottelius
* See more about standard labels on
624
** https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/
625
** https://helm.sh/docs/chart_best_practices/labels/
626 55 Nico Schottelius
627 139 Nico Schottelius
h2. Rook + Ceph
628
629
h3. Installation
630
631
* Usually directly via argocd
632
633
Manual steps:
634
635
<pre>
636
637
</pre>
638 43 Nico Schottelius
639 71 Nico Schottelius
h3. Executing ceph commands
640
641
Using the ceph-tools pod as follows:
642
643
<pre>
644
kubectl exec -n rook-ceph -ti $(kubectl -n rook-ceph get pods -l app=rook-ceph-tools -o jsonpath='{.items[*].metadata.name}') -- ceph -s
645
</pre>
646
647 43 Nico Schottelius
h3. Inspecting the logs of a specific server
648
649
<pre>
650
# Get the related pods
651
kubectl -n rook-ceph get pods -l app=rook-ceph-osd-prepare 
652
...
653
654
# Inspect the logs of a specific pod
655
kubectl -n rook-ceph logs -f rook-ceph-osd-prepare-server23--1-444qx
656
657 71 Nico Schottelius
</pre>
658
659
h3. Inspecting the logs of the rook-ceph-operator
660
661
<pre>
662
kubectl -n rook-ceph logs -f -l app=rook-ceph-operator
663 43 Nico Schottelius
</pre>
664
665 121 Nico Schottelius
h3. Restarting the rook operator
666
667
<pre>
668
kubectl -n rook-ceph delete pods  -l app=rook-ceph-operator
669
</pre>
670
671 43 Nico Schottelius
h3. Triggering server prepare / adding new osds
672
673
The rook-ceph-operator triggers/watches/creates pods to maintain hosts. To trigger a full "re scan", simply delete that pod:
674
675
<pre>
676
kubectl -n rook-ceph delete pods -l app=rook-ceph-operator
677
</pre>
678
679
This will cause all the @rook-ceph-osd-prepare-..@ jobs to be recreated and thus OSDs to be created, if new disks have been added.
680
681
h3. Removing an OSD
682
683
* See "Ceph OSD Management":https://rook.io/docs/rook/v1.7/ceph-osd-mgmt.html
684 77 Nico Schottelius
* More specifically: https://github.com/rook/rook/blob/release-1.7/cluster/examples/kubernetes/ceph/osd-purge.yaml
685 99 Nico Schottelius
* Then delete the related deployment
686 41 Nico Schottelius
687 98 Nico Schottelius
Set osd id in the osd-purge.yaml and apply it. OSD should be down before.
688
689
<pre>
690
apiVersion: batch/v1
691
kind: Job
692
metadata:
693
  name: rook-ceph-purge-osd
694
  namespace: rook-ceph # namespace:cluster
695
  labels:
696
    app: rook-ceph-purge-osd
697
spec:
698
  template:
699
    metadata:
700
      labels:
701
        app: rook-ceph-purge-osd
702
    spec:
703
      serviceAccountName: rook-ceph-purge-osd
704
      containers:
705
        - name: osd-removal
706
          image: rook/ceph:master
707
          # TODO: Insert the OSD ID in the last parameter that is to be removed
708
          # The OSD IDs are a comma-separated list. For example: "0" or "0,2".
709
          # If you want to preserve the OSD PVCs, set `--preserve-pvc true`.
710
          #
711
          # A --force-osd-removal option is available if the OSD should be destroyed even though the
712
          # removal could lead to data loss.
713
          args:
714
            - "ceph"
715
            - "osd"
716
            - "remove"
717
            - "--preserve-pvc"
718
            - "false"
719
            - "--force-osd-removal"
720
            - "false"
721
            - "--osd-ids"
722
            - "SETTHEOSDIDHERE"
723
          env:
724
            - name: POD_NAMESPACE
725
              valueFrom:
726
                fieldRef:
727
                  fieldPath: metadata.namespace
728
            - name: ROOK_MON_ENDPOINTS
729
              valueFrom:
730
                configMapKeyRef:
731
                  key: data
732
                  name: rook-ceph-mon-endpoints
733
            - name: ROOK_CEPH_USERNAME
734
              valueFrom:
735
                secretKeyRef:
736
                  key: ceph-username
737
                  name: rook-ceph-mon
738
            - name: ROOK_CEPH_SECRET
739
              valueFrom:
740
                secretKeyRef:
741
                  key: ceph-secret
742
                  name: rook-ceph-mon
743
            - name: ROOK_CONFIG_DIR
744
              value: /var/lib/rook
745
            - name: ROOK_CEPH_CONFIG_OVERRIDE
746
              value: /etc/rook/config/override.conf
747
            - name: ROOK_FSID
748
              valueFrom:
749
                secretKeyRef:
750
                  key: fsid
751
                  name: rook-ceph-mon
752
            - name: ROOK_LOG_LEVEL
753
              value: DEBUG
754
          volumeMounts:
755
            - mountPath: /etc/ceph
756
              name: ceph-conf-emptydir
757
            - mountPath: /var/lib/rook
758
              name: rook-config
759
      volumes:
760
        - emptyDir: {}
761
          name: ceph-conf-emptydir
762
        - emptyDir: {}
763
          name: rook-config
764
      restartPolicy: Never
765
766
767 99 Nico Schottelius
</pre>
768
769
Deleting the deployment:
770
771
<pre>
772
[18:05] bridge:~% kubectl -n rook-ceph delete deployment rook-ceph-osd-6
773
deployment.apps "rook-ceph-osd-6" deleted
774 98 Nico Schottelius
</pre>
775
776 76 Nico Schottelius
h2. Harbor
777
778
* We user "Harbor":https://goharbor.io/ for caching and as an image registry. Internal app reference: apps/prod/harbor.
779
* The admin password is in the password store, auto generated per cluster
780
* At the moment harbor only authenticates against the internal ldap tree
781
782
h3. LDAP configuration
783
784
* The url needs to be ldaps://...
785
* uid = uid
786
* rest standard
787 75 Nico Schottelius
788 89 Nico Schottelius
h2. Monitoring / Prometheus
789
790 90 Nico Schottelius
* Via "kube-prometheus":https://github.com/prometheus-operator/kube-prometheus/
791 89 Nico Schottelius
792 91 Nico Schottelius
Access via ...
793
794
* http://prometheus-k8s.monitoring.svc:9090
795
* http://grafana.monitoring.svc:3000
796
* http://alertmanager.monitoring.svc:9093
797
798
799 100 Nico Schottelius
h3. Prometheus Options
800
801
* "helm/kube-prometheus-stack":https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack
802
** Includes dashboards and co.
803
* "manifest based kube-prometheus":https://github.com/prometheus-operator/kube-prometheus
804
** Includes dashboards and co.
805
* "Prometheus Operator (mainly CRD manifest":https://github.com/prometheus-operator/prometheus-operator
806
807 91 Nico Schottelius
808 82 Nico Schottelius
h2. Nextcloud
809
810 85 Nico Schottelius
h3. How to get the nextcloud credentials 
811 84 Nico Schottelius
812
* The initial username is set to "nextcloud"
813
* The password is autogenerated and saved in a kubernetes secret
814
815
<pre>
816 85 Nico Schottelius
kubectl get secret RELEASENAME-nextcloud -o jsonpath="{.data.PASSWORD}" | base64 -d; echo "" 
817 84 Nico Schottelius
</pre>
818
819 83 Nico Schottelius
h3. How to fix "Access through untrusted domain"
820
821 82 Nico Schottelius
* Nextcloud stores the initial domain configuration
822 1 Nico Schottelius
* If the FQDN is changed, it will show the error message "Access through untrusted domain"
823 82 Nico Schottelius
* To fix, edit /var/www/html/config/config.php and correct the domain
824 83 Nico Schottelius
* Then delete the pods
825 82 Nico Schottelius
826 1 Nico Schottelius
h2. Infrastructure versions
827 35 Nico Schottelius
828 57 Nico Schottelius
h3. ungleich kubernetes infrastructure v5 (2021-10)
829 1 Nico Schottelius
830 57 Nico Schottelius
Clusters are configured / setup in this order:
831
832
* Bootstrap via kubeadm
833 59 Nico Schottelius
* "Networking via calico + BGP (non ECMP) using helm":https://docs.projectcalico.org/getting-started/kubernetes/helm
834
* "ArgoCD for CD":https://argo-cd.readthedocs.io/en/stable/
835
** "rook for storage via argocd":https://rook.io/
836 58 Nico Schottelius
** haproxy for in IPv6-cluster-IPv4-to-IPv6 proxy via argocd
837
** "kubernetes-secret-generator for in cluster secrets":https://github.com/mittwald/kubernetes-secret-generator
838
** "ungleich-certbot managing certs and nginx":https://hub.docker.com/repository/docker/ungleich/ungleich-certbot
839
840 57 Nico Schottelius
841
h3. ungleich kubernetes infrastructure v4 (2021-09)
842
843 54 Nico Schottelius
* rook is configured via manifests instead of using the rook-ceph-cluster helm chart
844 1 Nico Schottelius
* The rook operator is still being installed via helm
845 35 Nico Schottelius
846 57 Nico Schottelius
h3. ungleich kubernetes infrastructure v3 (2021-07)
847 1 Nico Schottelius
848 10 Nico Schottelius
* rook is now installed via helm via argocd instead of directly via manifests
849 28 Nico Schottelius
850 57 Nico Schottelius
h3. ungleich kubernetes infrastructure v2 (2021-05)
851 28 Nico Schottelius
852
* Replaced fluxv2 from ungleich k8s v1 with argocd
853 1 Nico Schottelius
** argocd can apply helm templates directly without needing to go through Chart releases
854 28 Nico Schottelius
* We are also using argoflow for build flows
855
* Planned to add "kaniko":https://github.com/GoogleContainerTools/kaniko for image building
856
857 57 Nico Schottelius
h3. ungleich kubernetes infrastructure v1 (2021-01)
858 28 Nico Schottelius
859
We are using the following components:
860
861
* "Calico as a CNI":https://www.projectcalico.org/ with BGP, IPv6 only, no encapsulation
862
** Needed for basic networking
863
* "kubernetes-secret-generator":https://github.com/mittwald/kubernetes-secret-generator for creating secrets
864
** Needed so that secrets are not stored in the git repository, but only in the cluster
865
* "ungleich-certbot":https://hub.docker.com/repository/docker/ungleich/ungleich-certbot
866
** Needed to get letsencrypt certificates for services
867
* "rook with ceph rbd + cephfs":https://rook.io/ for storage
868
** rbd for almost everything, *ReadWriteOnce*
869
** cephfs for smaller things, multi access *ReadWriteMany*
870
** Needed for providing persistent storage
871
* "flux v2":https://fluxcd.io/
872
** Needed to manage resources automatically