Project

General

Profile

The ungleich kubernetes infrastructure » History » Version 137

Nico Schottelius, 09/03/2022 12:49 PM

1 22 Nico Schottelius
h1. The ungleich kubernetes infrastructure and ungleich kubernetes manual
2 1 Nico Schottelius
3 3 Nico Schottelius
{{toc}}
4
5 1 Nico Schottelius
h2. Status
6
7 28 Nico Schottelius
This document is **pre-production**.
8
This document is to become the ungleich kubernetes infrastructure overview as well as the ungleich kubernetes manual.
9 1 Nico Schottelius
10 10 Nico Schottelius
h2. k8s clusters
11
12 123 Nico Schottelius
| Cluster            | Purpose/Setup     | Maintainer | Master(s)                     | argo                                                   | v4 http proxy | last verified |
13
| c0.k8s.ooo         | Dev               | -          | UNUSED                        |                                                        |               |    2021-10-05 |
14
| c1.k8s.ooo         | retired           |            | -                             |                                                        |               |    2022-03-15 |
15
| c2.k8s.ooo         | Dev p7 HW         | Nico       | server47 server53 server54    | "argo":https://argocd-server.argocd.svc.c2.k8s.ooo     |               |    2021-10-05 |
16
| c3.k8s.ooo         | retired           | -          | -                             |                                                        |               |    2021-10-05 |
17
| c4.k8s.ooo         | Dev2 p7 HW        | Jin-Guk    | server52 server53 server54    |                                                        |               |             - |
18
| c5.k8s.ooo         | retired           |            | -                             |                                                        |               |    2022-03-15 |
19
| c6.k8s.ooo         | Dev p6 VM Jin-Guk | Jin-Guk    |                               |                                                        |               |               |
20
| [[p5.k8s.ooo]]     | production        |            | server34 server36 server38    | "argo":https://argocd-server.argocd.svc.p5.k8s.ooo     | -             |               |
21
| [[p5-cow.k8s.ooo]] | production        | Nico       | server47 server51 server55    | "argo":https://argocd-server.argocd.svc.p5-cow.k8s.ooo |               |    2022-08-27 |
22
| [[p6.k8s.ooo]]     | production        |            | server67 server69 server71    | "argo":https://argocd-server.argocd.svc.p6.k8s.ooo     | 147.78.194.13 |    2021-10-05 |
23
| [[p10.k8s.ooo]]    | production        |            | server63 server65 server83    | "argo":https://argocd-server.argocd.svc.p10.k8s.ooo    | 147.78.194.12 |    2021-10-05 |
24
| [[k8s.ge.nau.so]]  | development       |            | server107 server108 server109 | "argo":https://argocd-server.argocd.svc.k8s.ge.nau.so  |               |               |
25
| [[dev.k8s.ooo]]    | development       |            | server110 server111 server112 | "argo":https://argocd-server.argocd.svc.dev.k8s.ooo    | -             |    2022-07-08 |
26
27
28
29 21 Nico Schottelius
30 1 Nico Schottelius
h2. General architecture and components overview
31
32
* All k8s clusters are IPv6 only
33
* We use BGP peering to propagate podcidr and serviceCidr networks to our infrastructure
34
* The main public testing repository is "ungleich-k8s":https://code.ungleich.ch/ungleich-public/ungleich-k8s
35 18 Nico Schottelius
** Private configurations are found in the **k8s-config** repository
36 1 Nico Schottelius
37
h3. Cluster types
38
39 28 Nico Schottelius
| **Type/Feature**            | **Development**                | **Production**         |
40
| Min No. nodes               | 3 (1 master, 3 worker)         | 5 (3 master, 3 worker) |
41
| Recommended minimum         | 4 (dedicated master, 3 worker) | 8 (3 master, 5 worker) |
42
| Separation of control plane | optional                       | recommended            |
43
| Persistent storage          | required                       | required               |
44
| Number of storage monitors  | 3                              | 5                      |
45 1 Nico Schottelius
46 43 Nico Schottelius
h2. General k8s operations
47 1 Nico Schottelius
48 46 Nico Schottelius
h3. Cheat sheet / external great references
49
50
* "kubectl cheatsheet":https://kubernetes.io/docs/reference/kubectl/cheatsheet/
51
52 117 Nico Schottelius
h3. Allowing to schedule work on the control plane / removing node taints
53 69 Nico Schottelius
54
* Mostly for single node / test / development clusters
55
* Just remove the master taint as follows
56
57
<pre>
58
kubectl taint nodes --all node-role.kubernetes.io/master-
59 118 Nico Schottelius
kubectl taint nodes --all node-role.kubernetes.io/control-plane-
60 69 Nico Schottelius
</pre>
61 1 Nico Schottelius
62 117 Nico Schottelius
You can check the node taints using @kubectl describe node ...@
63 69 Nico Schottelius
64 44 Nico Schottelius
h3. Get the cluster admin.conf
65
66
* On the masters of each cluster you can find the file @/etc/kubernetes/admin.conf@
67
* To be able to administrate the cluster you can copy the admin.conf to your local machine
68
* Multi cluster debugging can very easy if you name the config ~/cX-admin.conf (see example below)
69
70
<pre>
71
% scp root@server47.place7.ungleich.ch:/etc/kubernetes/admin.conf ~/c2-admin.conf
72
% export KUBECONFIG=~/c2-admin.conf    
73
% kubectl get nodes
74
NAME       STATUS                     ROLES                  AGE   VERSION
75
server47   Ready                      control-plane,master   82d   v1.22.0
76
server48   Ready                      control-plane,master   82d   v1.22.0
77
server49   Ready                      <none>                 82d   v1.22.0
78
server50   Ready                      <none>                 82d   v1.22.0
79
server59   Ready                      control-plane,master   82d   v1.22.0
80
server60   Ready,SchedulingDisabled   <none>                 82d   v1.22.0
81
server61   Ready                      <none>                 82d   v1.22.0
82
server62   Ready                      <none>                 82d   v1.22.0               
83
</pre>
84
85 18 Nico Schottelius
h3. Installing a new k8s cluster
86 8 Nico Schottelius
87 9 Nico Schottelius
* Decide on the cluster name (usually *cX.k8s.ooo*), X counting upwards
88 28 Nico Schottelius
** Using pXX.k8s.ooo for production clusters of placeXX
89 9 Nico Schottelius
* Use cdist to configure the nodes with requirements like crio
90
* Decide between single or multi node control plane setups (see below)
91 28 Nico Schottelius
** Single control plane suitable for development clusters
92 9 Nico Schottelius
93 28 Nico Schottelius
Typical init procedure:
94 9 Nico Schottelius
95 28 Nico Schottelius
* Single control plane: @kubeadm init --config bootstrap/XXX/kubeadm.yaml@
96
* Multi control plane (HA): @kubeadm init --config bootstrap/XXX/kubeadm.yaml --upload-certs@
97 10 Nico Schottelius
98 29 Nico Schottelius
h3. Deleting a pod that is hanging in terminating state
99
100
<pre>
101
kubectl delete pod <PODNAME> --grace-period=0 --force --namespace <NAMESPACE>
102
</pre>
103
104
(from https://stackoverflow.com/questions/35453792/pods-stuck-in-terminating-status)
105
106 42 Nico Schottelius
h3. Listing nodes of a cluster
107
108
<pre>
109
[15:05] bridge:~% kubectl get nodes
110
NAME       STATUS   ROLES                  AGE   VERSION
111
server22   Ready    <none>                 52d   v1.22.0
112
server23   Ready    <none>                 52d   v1.22.2
113
server24   Ready    <none>                 52d   v1.22.0
114
server25   Ready    <none>                 52d   v1.22.0
115
server26   Ready    <none>                 52d   v1.22.0
116
server27   Ready    <none>                 52d   v1.22.0
117
server63   Ready    control-plane,master   52d   v1.22.0
118
server64   Ready    <none>                 52d   v1.22.0
119
server65   Ready    control-plane,master   52d   v1.22.0
120
server66   Ready    <none>                 52d   v1.22.0
121
server83   Ready    control-plane,master   52d   v1.22.0
122
server84   Ready    <none>                 52d   v1.22.0
123
server85   Ready    <none>                 52d   v1.22.0
124
server86   Ready    <none>                 52d   v1.22.0
125
</pre>
126
127 41 Nico Schottelius
h3. Removing / draining a node
128
129
Usually @kubectl drain server@ should do the job, but sometimes we need to be more aggressive:
130
131 1 Nico Schottelius
<pre>
132 103 Nico Schottelius
kubectl drain --delete-emptydir-data --ignore-daemonsets serverXX
133 42 Nico Schottelius
</pre>
134
135
h3. Readding a node after draining
136
137
<pre>
138
kubectl uncordon serverXX
139 1 Nico Schottelius
</pre>
140 43 Nico Schottelius
141 50 Nico Schottelius
h3. (Re-)joining worker nodes after creating the cluster
142 49 Nico Schottelius
143
* We need to have an up-to-date token
144
* We use different join commands for the workers and control plane nodes
145
146
Generating the join command on an existing control plane node:
147
148
<pre>
149
kubeadm token create --print-join-command
150
</pre>
151
152 50 Nico Schottelius
h3. (Re-)joining control plane nodes after creating the cluster
153 1 Nico Schottelius
154 50 Nico Schottelius
* We generate the token again
155
* We upload the certificates
156
* We need to combine/create the join command for the control plane node
157
158
Example session:
159
160
<pre>
161
% kubeadm token create --print-join-command
162
kubeadm join p10-api.k8s.ooo:6443 --token xmff4i.ABC --discovery-token-ca-cert-hash sha256:longhash 
163
164
% kubeadm init phase upload-certs --upload-certs
165
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
166
[upload-certs] Using certificate key:
167
CERTKEY
168
169
# Then we use these two outputs on the joining node:
170
171
kubeadm join p10-api.k8s.ooo:6443 --token xmff4i.ABC --discovery-token-ca-cert-hash sha256:longhash --control-plane --certificate-key CERTKEY
172
</pre>
173
174
Commands to be used on a control plane node:
175
176
<pre>
177
kubeadm token create --print-join-command
178
kubeadm init phase upload-certs --upload-certs
179
</pre>
180
181
Commands to be used on the joining node:
182
183
<pre>
184
JOINCOMMAND --control-plane --certificate-key CERTKEY
185
</pre>
186 49 Nico Schottelius
187 51 Nico Schottelius
SEE ALSO
188
189
* https://stackoverflow.com/questions/63936268/how-to-generate-kubeadm-token-for-secondary-control-plane-nodes
190
* https://blog.scottlowe.org/2019/08/15/reconstructing-the-join-command-for-kubeadm/
191
192 53 Nico Schottelius
h3. How to fix etcd does not start when rejoining a kubernetes cluster as a control plane
193 52 Nico Schottelius
194
If during the above step etcd does not come up, @kubeadm join@ can hang as follows:
195
196
<pre>
197
[control-plane] Creating static Pod manifest for "kube-apiserver"                                                              
198
[control-plane] Creating static Pod manifest for "kube-controller-manager"                                                     
199
[control-plane] Creating static Pod manifest for "kube-scheduler"                                                              
200
[check-etcd] Checking that the etcd cluster is healthy                                                                         
201
error execution phase check-etcd: etcd cluster is not healthy: failed to dial endpoint https://[2a0a:e5c0:10:1:225:b3ff:fe20:37
202
8a]:2379 with maintenance client: context deadline exceeded                                                                    
203
To see the stack trace of this error execute with --v=5 or higher         
204
</pre>
205
206
Then the problem is likely that the etcd server is still a member of the cluster. We first need to remove it from the etcd cluster and then the join works.
207
208
To fix this we do:
209
210
* Find a working etcd pod
211
* Find the etcd members / member list
212
* Remove the etcd member that we want to re-join the cluster
213
214
215
<pre>
216
# Find the etcd pods
217
kubectl -n kube-system get pods -l component=etcd,tier=control-plane
218
219
# Get the list of etcd servers with the member id 
220
kubectl exec -n kube-system -ti ETCDPODNAME -- etcdctl --endpoints '[::1]:2379' --cacert /etc/kubernetes/pki/etcd/ca.crt --cert  /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member list
221
222
# Remove the member
223
kubectl exec -n kube-system -ti ETCDPODNAME -- etcdctl --endpoints '[::1]:2379' --cacert /etc/kubernetes/pki/etcd/ca.crt --cert  /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member remove MEMBERID
224
</pre>
225
226
Sample session:
227
228
<pre>
229
[10:48] line:~% kubectl -n kube-system get pods -l component=etcd,tier=control-plane
230
NAME            READY   STATUS    RESTARTS     AGE
231
etcd-server63   1/1     Running   0            3m11s
232
etcd-server65   1/1     Running   3            7d2h
233
etcd-server83   1/1     Running   8 (6d ago)   7d2h
234
[10:48] line:~% kubectl exec -n kube-system -ti etcd-server65 -- etcdctl --endpoints '[::1]:2379' --cacert /etc/kubernetes/pki/etcd/ca.crt --cert  /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member list
235
356891cd676df6e4, started, server65, https://[2a0a:e5c0:10:1:225:b3ff:fe20:375c]:2380, https://[2a0a:e5c0:10:1:225:b3ff:fe20:375c]:2379, false
236
371b8a07185dee7e, started, server63, https://[2a0a:e5c0:10:1:225:b3ff:fe20:378a]:2380, https://[2a0a:e5c0:10:1:225:b3ff:fe20:378a]:2379, false
237
5942bc58307f8af9, started, server83, https://[2a0a:e5c0:10:1:3e4a:92ff:fe79:bb98]:2380, https://[2a0a:e5c0:10:1:3e4a:92ff:fe79:bb98]:2379, false
238
239
[10:48] line:~% kubectl exec -n kube-system -ti etcd-server65 -- etcdctl --endpoints '[::1]:2379' --cacert /etc/kubernetes/pki/etcd/ca.crt --cert  /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member remove 371b8a07185dee7e
240
Member 371b8a07185dee7e removed from cluster e3c0805f592a8f77
241 1 Nico Schottelius
242
</pre>
243
244
SEE ALSO
245
246
* We found the solution using https://stackoverflow.com/questions/67921552/re-installed-node-cannot-join-kubernetes-cluster
247 56 Nico Schottelius
248 101 Nico Schottelius
h3. Hardware Maintenance using ungleich-hardware
249
250
Use the following manifest and replace the HOST with the actual host:
251
252
<pre>
253
apiVersion: v1
254
kind: Pod
255
metadata:
256
  name: ungleich-hardware-HOST
257
spec:
258
  containers:
259
  - name: ungleich-hardware
260
    image: ungleich/ungleich-hardware:0.0.5
261
    args:
262
    - sleep
263
    - "1000000"
264
    volumeMounts:
265
      - mountPath: /dev
266
        name: dev
267
    securityContext:
268
      privileged: true
269
  nodeSelector:
270
    kubernetes.io/hostname: "HOST"
271
272
  volumes:
273
    - name: dev
274
      hostPath:
275
        path: /dev
276
</pre>
277
278 102 Nico Schottelius
Also see: [[The_ungleich_hardware_maintenance_guide]]
279
280 105 Nico Schottelius
h3. Triggering a cronjob / creating a job from a cronjob
281 104 Nico Schottelius
282
To test a cronjob, we can create a job from a cronjob:
283
284
<pre>
285
kubectl create job --from=cronjob/volume2-daily-backup volume2-manual
286
</pre>
287
288
This creates a job volume2-manual based on the cronjob  volume2-daily
289
290 112 Nico Schottelius
h3. su-ing into a user that has nologin shell set
291
292
Many times users are having nologin as their shell inside the container. To be able to execute maintenance commands within the
293
container, we can use @su -s /bin/sh@ like this:
294
295
<pre>
296
su -s /bin/sh -c '/path/to/your/script' testuser
297
</pre>
298
299
Found on https://serverfault.com/questions/351046/how-to-run-command-as-user-who-has-usr-sbin-nologin-as-shell
300
301 113 Nico Schottelius
h3. How to print a secret value
302
303
Assuming you want the "password" item from a secret, use:
304
305
<pre>
306
kubectl get secret SECRETNAME -o jsonpath="{.data.password}" | base64 -d; echo "" 
307
</pre>
308
309 62 Nico Schottelius
h2. Calico CNI
310
311
h3. Calico Installation
312
313
* We install "calico using helm":https://docs.projectcalico.org/getting-started/kubernetes/helm
314
* This has the following advantages:
315
** Easy to upgrade
316
** Does not require os to configure IPv6/dual stack settings as the tigera operator figures out things on its own
317
318
Usually plain calico can be installed directly using:
319
320
<pre>
321 125 Nico Schottelius
VERSION=v3.23.3
322 120 Nico Schottelius
helm repo add projectcalico https://docs.projectcalico.org/charts
323 124 Nico Schottelius
helm upgrade --install --namespace tigera calico projectcalico/tigera-operator --version $VERSION --create-namespace
324 1 Nico Schottelius
</pre>
325 92 Nico Schottelius
326
* Check the tags on https://github.com/projectcalico/calico/tags for the latest release
327 62 Nico Schottelius
328
h3. Installing calicoctl
329
330 115 Nico Schottelius
* General installation instructions, including binary download: https://projectcalico.docs.tigera.io/maintenance/clis/calicoctl/install
331
332 62 Nico Schottelius
To be able to manage and configure calico, we need to 
333
"install calicoctl (we choose the version as a pod)":https://docs.projectcalico.org/getting-started/clis/calicoctl/install#install-calicoctl-as-a-kubernetes-pod
334
335
<pre>
336
kubectl apply -f https://docs.projectcalico.org/manifests/calicoctl.yaml
337
</pre>
338
339 93 Nico Schottelius
Or version specific:
340
341
<pre>
342
kubectl apply -f https://github.com/projectcalico/calico/blob/v3.20.4/manifests/calicoctl.yaml
343 97 Nico Schottelius
344
# For 3.22
345
kubectl apply -f https://projectcalico.docs.tigera.io/archive/v3.22/manifests/calicoctl.yaml
346 93 Nico Schottelius
</pre>
347
348 70 Nico Schottelius
And making it easier accessible by alias:
349
350
<pre>
351
alias calicoctl="kubectl exec -i -n kube-system calicoctl -- /calicoctl"
352
</pre>
353
354 62 Nico Schottelius
h3. Calico configuration
355
356 63 Nico Schottelius
By default our k8s clusters "BGP peer":https://docs.projectcalico.org/networking/bgp
357
with an upstream router to propagate podcidr and servicecidr.
358 62 Nico Schottelius
359
Default settings in our infrastructure:
360
361
* We use a full-mesh using the @nodeToNodeMeshEnabled: true@ option
362
* We keep the original next hop so that *only* the server with the pod is announcing it (instead of ecmp)
363 1 Nico Schottelius
* We use private ASNs for k8s clusters
364 63 Nico Schottelius
* We do *not* use any overlay
365 62 Nico Schottelius
366
After installing calico and calicoctl the last step of the installation is usually:
367
368 1 Nico Schottelius
<pre>
369 79 Nico Schottelius
calicoctl create -f - < calico-bgp.yaml
370 62 Nico Schottelius
</pre>
371
372
373
A sample BGP configuration:
374
375
<pre>
376
---
377
apiVersion: projectcalico.org/v3
378
kind: BGPConfiguration
379
metadata:
380
  name: default
381
spec:
382
  logSeverityScreen: Info
383
  nodeToNodeMeshEnabled: true
384
  asNumber: 65534
385
  serviceClusterIPs:
386
  - cidr: 2a0a:e5c0:10:3::/108
387
  serviceExternalIPs:
388
  - cidr: 2a0a:e5c0:10:3::/108
389
---
390
apiVersion: projectcalico.org/v3
391
kind: BGPPeer
392
metadata:
393
  name: router1-place10
394
spec:
395
  peerIP: 2a0a:e5c0:10:1::50
396
  asNumber: 213081
397
  keepOriginalNextHop: true
398
</pre>
399
400 126 Nico Schottelius
h2. Cilium CNI (experimental)
401
402 137 Nico Schottelius
h3. Status
403
404
** NO WORKING CILIUM CONFIGURATION FOR IPV6 only modes**
405
406 128 Nico Schottelius
h3. BGP configuration
407
408
* The cilium-operator will not start without a correct configmap being present beforehand (see error message below)
409
* Creating the bgp config beforehand as a configmap is thus required.
410
411
The error one gets without the configmap present:
412
413
Pods are hanging with:
414
415
<pre>
416
cilium-bpqm6                       0/1     Init:0/4            0             9s
417
cilium-operator-5947d94f7f-5bmh2   0/1     ContainerCreating   0             9s
418
</pre>
419
420
The error message in the cilium-*perator is:
421
422
<pre>
423
Events:
424
  Type     Reason       Age                From               Message
425
  ----     ------       ----               ----               -------
426
  Normal   Scheduled    80s                default-scheduler  Successfully assigned kube-system/cilium-operator-5947d94f7f-lqcsp to server56
427
  Warning  FailedMount  16s (x8 over 80s)  kubelet            MountVolume.SetUp failed for volume "bgp-config-path" : configmap "bgp-config" not found
428
</pre>
429
430
A correct bgp config looks like this:
431
432
<pre>
433
apiVersion: v1
434
kind: ConfigMap
435
metadata:
436
  name: bgp-config
437
  namespace: kube-system
438
data:
439
  config.yaml: |
440
    peers:
441
      - peer-address: 2a0a:e5c0::46
442
        peer-asn: 209898
443
        my-asn: 65533
444
      - peer-address: 2a0a:e5c0::47
445
        peer-asn: 209898
446
        my-asn: 65533
447
    address-pools:
448
      - name: default
449
        protocol: bgp
450
        addresses:
451
          - 2a0a:e5c0:0:14::/64
452
</pre>
453
454 127 Nico Schottelius
h3. Installation
455
456 130 Nico Schottelius
Adding the repo
457 127 Nico Schottelius
<pre>
458 1 Nico Schottelius
459 127 Nico Schottelius
helm repo add cilium https://helm.cilium.io/
460 129 Nico Schottelius
helm repo update
461 130 Nico Schottelius
</pre>
462 1 Nico Schottelius
463 130 Nico Schottelius
Installing + configuring cilium
464
<pre>
465 129 Nico Schottelius
version=1.12.1
466 135 Nico Schottelius
ipv6pool=2a0a:e5c0:0:14::/112
467 129 Nico Schottelius
468 130 Nico Schottelius
469 1 Nico Schottelius
helm upgrade --install cilium cilium/cilium --version $version \
470 129 Nico Schottelius
  --namespace kube-system \
471
  --set ipv4.enabled=false \
472 1 Nico Schottelius
  --set ipv6.enabled=true \
473 131 Nico Schottelius
  --set ipam.operator.clusterPoolIPv6PodCIDRList=$ipv6pool \
474 1 Nico Schottelius
  --set bgpControlPlane.enabled=true
475 136 Nico Schottelius
476
#   --set bgp.enabled=true --set bgp.announce.podCIDR=true \
477 127 Nico Schottelius
478
# Show possible configuration options
479
helm show values cilium/cilium
480
481 1 Nico Schottelius
</pre>
482 132 Nico Schottelius
483
Using a /64 for ipam.operator.clusterPoolIPv6PodCIDRList fails with:
484
485
<pre>
486
level=fatal msg="Unable to init cluster-pool allocator" error="unable to initialize IPv6 allocator New CIDR set failed; the node CIDR size is too big" subsys=cilium-operator-generic
487
</pre>
488
489 126 Nico Schottelius
490 1 Nico Schottelius
See also https://github.com/cilium/cilium/issues/20756
491 135 Nico Schottelius
492
Seems a /112 is actually working.
493
494
h3. Kernel modules
495
496
Cilium requires the following modules to be loaded on the host (not loaded by default):
497
498
<pre>
499
modprobe  ip6table_raw
500
modprobe  ip6table_filter
501
</pre>
502 133 Nico Schottelius
503 122 Nico Schottelius
h2. ArgoCD 
504 56 Nico Schottelius
505 60 Nico Schottelius
h3. Argocd Installation
506 1 Nico Schottelius
507 116 Nico Schottelius
* See https://argo-cd.readthedocs.io/en/stable/
508
509 60 Nico Schottelius
As there is no configuration management present yet, argocd is installed using
510
511 1 Nico Schottelius
<pre>
512 60 Nico Schottelius
kubectl create namespace argocd
513 86 Nico Schottelius
514 96 Nico Schottelius
# Specific Version
515
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v2.3.2/manifests/install.yaml
516 86 Nico Schottelius
517
# OR: latest stable
518 60 Nico Schottelius
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
519 56 Nico Schottelius
</pre>
520 1 Nico Schottelius
521 116 Nico Schottelius
522 1 Nico Schottelius
523 60 Nico Schottelius
h3. Get the argocd credentials
524
525
<pre>
526
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo ""
527
</pre>
528 52 Nico Schottelius
529 87 Nico Schottelius
h3. Accessing argocd
530
531
In regular IPv6 clusters:
532
533
* Navigate to https://argocd-server.argocd.CLUSTERDOMAIN
534
535
In legacy IPv4 clusters
536
537
<pre>
538
kubectl --namespace argocd port-forward svc/argocd-server 8080:80
539
</pre>
540
541 88 Nico Schottelius
* Navigate to https://localhost:8080
542
543 68 Nico Schottelius
h3. Using the argocd webhook to trigger changes
544 67 Nico Schottelius
545
* To trigger changes post json https://argocd.example.com/api/webhook
546
547 72 Nico Schottelius
h3. Deploying an application
548
549
* Applications are deployed via git towards gitea (code.ungleich.ch) and then pulled by argo
550 73 Nico Schottelius
* Always include the *redmine-url* pointing to the (customer) ticket
551
** Also add the support-url if it exists
552 72 Nico Schottelius
553
Application sample
554
555
<pre>
556
apiVersion: argoproj.io/v1alpha1
557
kind: Application
558
metadata:
559
  name: gitea-CUSTOMER
560
  namespace: argocd
561
spec:
562
  destination:
563
    namespace: default
564
    server: 'https://kubernetes.default.svc'
565
  source:
566
    path: apps/prod/gitea
567
    repoURL: 'https://code.ungleich.ch/ungleich-intern/k8s-config.git'
568
    targetRevision: HEAD
569
    helm:
570
      parameters:
571
        - name: storage.data.storageClass
572
          value: rook-ceph-block-hdd
573
        - name: storage.data.size
574
          value: 200Gi
575
        - name: storage.db.storageClass
576
          value: rook-ceph-block-ssd
577
        - name: storage.db.size
578
          value: 10Gi
579
        - name: storage.letsencrypt.storageClass
580
          value: rook-ceph-block-hdd
581
        - name: storage.letsencrypt.size
582
          value: 50Mi
583
        - name: letsencryptStaging
584
          value: 'no'
585
        - name: fqdn
586
          value: 'code.verua.online'
587
  project: default
588
  syncPolicy:
589
    automated:
590
      prune: true
591
      selfHeal: true
592
  info:
593
    - name: 'redmine-url'
594
      value: 'https://redmine.ungleich.ch/issues/ISSUEID'
595
    - name: 'support-url'
596
      value: 'https://support.ungleich.ch/Ticket/Display.html?id=TICKETID'
597
</pre>
598
599 80 Nico Schottelius
h2. Helm related operations and conventions
600 55 Nico Schottelius
601 61 Nico Schottelius
We use helm charts extensively.
602
603
* In production, they are managed via argocd
604
* In development, helm chart can de developed and deployed manually using the helm utility.
605
606 55 Nico Schottelius
h3. Installing a helm chart
607
608
One can use the usual pattern of
609
610
<pre>
611
helm install <releasename> <chartdirectory>
612
</pre>
613
614
However often you want to reinstall/update when testing helm charts. The following pattern is "better", because it allows you to reinstall, if it is already installed:
615
616
<pre>
617
helm upgrade --install <releasename> <chartdirectory>
618 1 Nico Schottelius
</pre>
619 80 Nico Schottelius
620
h3. Naming services and deployments in helm charts [Application labels]
621
622
* We always have {{ .Release.Name }} to identify the current "instance"
623
* Deployments:
624
** use @app: <what it is>@, f.i. @app: nginx@, @app: postgres@, ...
625 81 Nico Schottelius
* See more about standard labels on
626
** https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/
627
** https://helm.sh/docs/chart_best_practices/labels/
628 55 Nico Schottelius
629 43 Nico Schottelius
h2. Rook / Ceph Related Operations
630
631 71 Nico Schottelius
h3. Executing ceph commands
632
633
Using the ceph-tools pod as follows:
634
635
<pre>
636
kubectl exec -n rook-ceph -ti $(kubectl -n rook-ceph get pods -l app=rook-ceph-tools -o jsonpath='{.items[*].metadata.name}') -- ceph -s
637
</pre>
638
639 43 Nico Schottelius
h3. Inspecting the logs of a specific server
640
641
<pre>
642
# Get the related pods
643
kubectl -n rook-ceph get pods -l app=rook-ceph-osd-prepare 
644
...
645
646
# Inspect the logs of a specific pod
647
kubectl -n rook-ceph logs -f rook-ceph-osd-prepare-server23--1-444qx
648
649 71 Nico Schottelius
</pre>
650
651
h3. Inspecting the logs of the rook-ceph-operator
652
653
<pre>
654
kubectl -n rook-ceph logs -f -l app=rook-ceph-operator
655 43 Nico Schottelius
</pre>
656
657 121 Nico Schottelius
h3. Restarting the rook operator
658
659
<pre>
660
kubectl -n rook-ceph delete pods  -l app=rook-ceph-operator
661
</pre>
662
663 43 Nico Schottelius
h3. Triggering server prepare / adding new osds
664
665
The rook-ceph-operator triggers/watches/creates pods to maintain hosts. To trigger a full "re scan", simply delete that pod:
666
667
<pre>
668
kubectl -n rook-ceph delete pods -l app=rook-ceph-operator
669
</pre>
670
671
This will cause all the @rook-ceph-osd-prepare-..@ jobs to be recreated and thus OSDs to be created, if new disks have been added.
672
673
h3. Removing an OSD
674
675
* See "Ceph OSD Management":https://rook.io/docs/rook/v1.7/ceph-osd-mgmt.html
676 77 Nico Schottelius
* More specifically: https://github.com/rook/rook/blob/release-1.7/cluster/examples/kubernetes/ceph/osd-purge.yaml
677 99 Nico Schottelius
* Then delete the related deployment
678 41 Nico Schottelius
679 98 Nico Schottelius
Set osd id in the osd-purge.yaml and apply it. OSD should be down before.
680
681
<pre>
682
apiVersion: batch/v1
683
kind: Job
684
metadata:
685
  name: rook-ceph-purge-osd
686
  namespace: rook-ceph # namespace:cluster
687
  labels:
688
    app: rook-ceph-purge-osd
689
spec:
690
  template:
691
    metadata:
692
      labels:
693
        app: rook-ceph-purge-osd
694
    spec:
695
      serviceAccountName: rook-ceph-purge-osd
696
      containers:
697
        - name: osd-removal
698
          image: rook/ceph:master
699
          # TODO: Insert the OSD ID in the last parameter that is to be removed
700
          # The OSD IDs are a comma-separated list. For example: "0" or "0,2".
701
          # If you want to preserve the OSD PVCs, set `--preserve-pvc true`.
702
          #
703
          # A --force-osd-removal option is available if the OSD should be destroyed even though the
704
          # removal could lead to data loss.
705
          args:
706
            - "ceph"
707
            - "osd"
708
            - "remove"
709
            - "--preserve-pvc"
710
            - "false"
711
            - "--force-osd-removal"
712
            - "false"
713
            - "--osd-ids"
714
            - "SETTHEOSDIDHERE"
715
          env:
716
            - name: POD_NAMESPACE
717
              valueFrom:
718
                fieldRef:
719
                  fieldPath: metadata.namespace
720
            - name: ROOK_MON_ENDPOINTS
721
              valueFrom:
722
                configMapKeyRef:
723
                  key: data
724
                  name: rook-ceph-mon-endpoints
725
            - name: ROOK_CEPH_USERNAME
726
              valueFrom:
727
                secretKeyRef:
728
                  key: ceph-username
729
                  name: rook-ceph-mon
730
            - name: ROOK_CEPH_SECRET
731
              valueFrom:
732
                secretKeyRef:
733
                  key: ceph-secret
734
                  name: rook-ceph-mon
735
            - name: ROOK_CONFIG_DIR
736
              value: /var/lib/rook
737
            - name: ROOK_CEPH_CONFIG_OVERRIDE
738
              value: /etc/rook/config/override.conf
739
            - name: ROOK_FSID
740
              valueFrom:
741
                secretKeyRef:
742
                  key: fsid
743
                  name: rook-ceph-mon
744
            - name: ROOK_LOG_LEVEL
745
              value: DEBUG
746
          volumeMounts:
747
            - mountPath: /etc/ceph
748
              name: ceph-conf-emptydir
749
            - mountPath: /var/lib/rook
750
              name: rook-config
751
      volumes:
752
        - emptyDir: {}
753
          name: ceph-conf-emptydir
754
        - emptyDir: {}
755
          name: rook-config
756
      restartPolicy: Never
757
758
759 99 Nico Schottelius
</pre>
760
761
Deleting the deployment:
762
763
<pre>
764
[18:05] bridge:~% kubectl -n rook-ceph delete deployment rook-ceph-osd-6
765
deployment.apps "rook-ceph-osd-6" deleted
766 98 Nico Schottelius
</pre>
767
768 76 Nico Schottelius
h2. Harbor
769
770
* We user "Harbor":https://goharbor.io/ for caching and as an image registry. Internal app reference: apps/prod/harbor.
771
* The admin password is in the password store, auto generated per cluster
772
* At the moment harbor only authenticates against the internal ldap tree
773
774
h3. LDAP configuration
775
776
* The url needs to be ldaps://...
777
* uid = uid
778
* rest standard
779 75 Nico Schottelius
780 89 Nico Schottelius
h2. Monitoring / Prometheus
781
782 90 Nico Schottelius
* Via "kube-prometheus":https://github.com/prometheus-operator/kube-prometheus/
783 89 Nico Schottelius
784 91 Nico Schottelius
Access via ...
785
786
* http://prometheus-k8s.monitoring.svc:9090
787
* http://grafana.monitoring.svc:3000
788
* http://alertmanager.monitoring.svc:9093
789
790
791 100 Nico Schottelius
h3. Prometheus Options
792
793
* "helm/kube-prometheus-stack":https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack
794
** Includes dashboards and co.
795
* "manifest based kube-prometheus":https://github.com/prometheus-operator/kube-prometheus
796
** Includes dashboards and co.
797
* "Prometheus Operator (mainly CRD manifest":https://github.com/prometheus-operator/prometheus-operator
798
799 91 Nico Schottelius
800 82 Nico Schottelius
h2. Nextcloud
801
802 85 Nico Schottelius
h3. How to get the nextcloud credentials 
803 84 Nico Schottelius
804
* The initial username is set to "nextcloud"
805
* The password is autogenerated and saved in a kubernetes secret
806
807
<pre>
808 85 Nico Schottelius
kubectl get secret RELEASENAME-nextcloud -o jsonpath="{.data.PASSWORD}" | base64 -d; echo "" 
809 84 Nico Schottelius
</pre>
810
811 83 Nico Schottelius
h3. How to fix "Access through untrusted domain"
812
813 82 Nico Schottelius
* Nextcloud stores the initial domain configuration
814 1 Nico Schottelius
* If the FQDN is changed, it will show the error message "Access through untrusted domain"
815 82 Nico Schottelius
* To fix, edit /var/www/html/config/config.php and correct the domain
816 83 Nico Schottelius
* Then delete the pods
817 82 Nico Schottelius
818 1 Nico Schottelius
h2. Infrastructure versions
819 35 Nico Schottelius
820 57 Nico Schottelius
h3. ungleich kubernetes infrastructure v5 (2021-10)
821 1 Nico Schottelius
822 57 Nico Schottelius
Clusters are configured / setup in this order:
823
824
* Bootstrap via kubeadm
825 59 Nico Schottelius
* "Networking via calico + BGP (non ECMP) using helm":https://docs.projectcalico.org/getting-started/kubernetes/helm
826
* "ArgoCD for CD":https://argo-cd.readthedocs.io/en/stable/
827
** "rook for storage via argocd":https://rook.io/
828 58 Nico Schottelius
** haproxy for in IPv6-cluster-IPv4-to-IPv6 proxy via argocd
829
** "kubernetes-secret-generator for in cluster secrets":https://github.com/mittwald/kubernetes-secret-generator
830
** "ungleich-certbot managing certs and nginx":https://hub.docker.com/repository/docker/ungleich/ungleich-certbot
831
832 57 Nico Schottelius
833
h3. ungleich kubernetes infrastructure v4 (2021-09)
834
835 54 Nico Schottelius
* rook is configured via manifests instead of using the rook-ceph-cluster helm chart
836 1 Nico Schottelius
* The rook operator is still being installed via helm
837 35 Nico Schottelius
838 57 Nico Schottelius
h3. ungleich kubernetes infrastructure v3 (2021-07)
839 1 Nico Schottelius
840 10 Nico Schottelius
* rook is now installed via helm via argocd instead of directly via manifests
841 28 Nico Schottelius
842 57 Nico Schottelius
h3. ungleich kubernetes infrastructure v2 (2021-05)
843 28 Nico Schottelius
844
* Replaced fluxv2 from ungleich k8s v1 with argocd
845 1 Nico Schottelius
** argocd can apply helm templates directly without needing to go through Chart releases
846 28 Nico Schottelius
* We are also using argoflow for build flows
847
* Planned to add "kaniko":https://github.com/GoogleContainerTools/kaniko for image building
848
849 57 Nico Schottelius
h3. ungleich kubernetes infrastructure v1 (2021-01)
850 28 Nico Schottelius
851
We are using the following components:
852
853
* "Calico as a CNI":https://www.projectcalico.org/ with BGP, IPv6 only, no encapsulation
854
** Needed for basic networking
855
* "kubernetes-secret-generator":https://github.com/mittwald/kubernetes-secret-generator for creating secrets
856
** Needed so that secrets are not stored in the git repository, but only in the cluster
857
* "ungleich-certbot":https://hub.docker.com/repository/docker/ungleich/ungleich-certbot
858
** Needed to get letsencrypt certificates for services
859
* "rook with ceph rbd + cephfs":https://rook.io/ for storage
860
** rbd for almost everything, *ReadWriteOnce*
861
** cephfs for smaller things, multi access *ReadWriteMany*
862
** Needed for providing persistent storage
863
* "flux v2":https://fluxcd.io/
864
** Needed to manage resources automatically