Project

General

Profile

The ungleich kubernetes infrastructure » History » Version 105

Nico Schottelius, 06/02/2022 05:46 AM

1 22 Nico Schottelius
h1. The ungleich kubernetes infrastructure and ungleich kubernetes manual
2 1 Nico Schottelius
3 3 Nico Schottelius
{{toc}}
4
5 1 Nico Schottelius
h2. Status
6
7 28 Nico Schottelius
This document is **pre-production**.
8
This document is to become the ungleich kubernetes infrastructure overview as well as the ungleich kubernetes manual.
9 1 Nico Schottelius
10 10 Nico Schottelius
h2. k8s clusters
11
12 95 Nico Schottelius
| Cluster         | Purpose/Setup     | Maintainer | Master(s)                  | argo                                                | v4 http proxy | last verified |
13
| c0.k8s.ooo      | Dev               | -          | UNUSED                     |                                                     |               |    2021-10-05 |
14
| c1.k8s.ooo      | retired           |            | -                          |                                                     |               |    2022-03-15 |
15
| c2.k8s.ooo      | Dev p7 HW         | Nico       | server47 server53 server54 | "argo":https://argocd-server.argocd.svc.c2.k8s.ooo  |               |    2021-10-05 |
16
| c3.k8s.ooo      | retired           | -          | -                          |                                                     |               |    2021-10-05 |
17
| c4.k8s.ooo      | Dev2 p7 HW        | Jin-Guk    | server52 server53 server54 |                                                     |               |             - |
18
| c5.k8s.ooo      | retired           |            | -                          |                                                     |               |    2022-03-15 |
19
| c6.k8s.ooo      | Dev p6 VM Jin-Guk | Jin-Guk    |                            |                                                     |               |               |
20
| [[p5.k8s.ooo]]  | production        |            | server34 server36 server38 | "argo":https://argocd-server.argocd.svc.p5.k8s.ooo  |             - |               |
21
| [[p6.k8s.ooo]]  | production        |            | server67 server69 server71 | "argo":https://argocd-server.argocd.svc.p6.k8s.ooo  | 147.78.194.13 |    2021-10-05 |
22
| [[p10.k8s.ooo]] | production        |            | server63 server65 server83 | "argo":https://argocd-server.argocd.svc.p10.k8s.ooo | 147.78.194.12 |    2021-10-05 |
23
| fnnf            | development       | Nico       | server75                   |                                                     |               |               |
24 78 Nico Schottelius
25 21 Nico Schottelius
26 1 Nico Schottelius
h2. General architecture and components overview
27
28
* All k8s clusters are IPv6 only
29
* We use BGP peering to propagate podcidr and serviceCidr networks to our infrastructure
30
* The main public testing repository is "ungleich-k8s":https://code.ungleich.ch/ungleich-public/ungleich-k8s
31 18 Nico Schottelius
** Private configurations are found in the **k8s-config** repository
32 1 Nico Schottelius
33
h3. Cluster types
34
35 28 Nico Schottelius
| **Type/Feature**            | **Development**                | **Production**         |
36
| Min No. nodes               | 3 (1 master, 3 worker)         | 5 (3 master, 3 worker) |
37
| Recommended minimum         | 4 (dedicated master, 3 worker) | 8 (3 master, 5 worker) |
38
| Separation of control plane | optional                       | recommended            |
39
| Persistent storage          | required                       | required               |
40
| Number of storage monitors  | 3                              | 5                      |
41 1 Nico Schottelius
42 43 Nico Schottelius
h2. General k8s operations
43 1 Nico Schottelius
44 46 Nico Schottelius
h3. Cheat sheet / external great references
45
46
* "kubectl cheatsheet":https://kubernetes.io/docs/reference/kubectl/cheatsheet/
47
48 69 Nico Schottelius
h3. Allowing to schedule work on the control plane
49
50
* Mostly for single node / test / development clusters
51
* Just remove the master taint as follows
52
53
<pre>
54
kubectl taint nodes --all node-role.kubernetes.io/master-
55
</pre>
56
57
58 44 Nico Schottelius
h3. Get the cluster admin.conf
59
60
* On the masters of each cluster you can find the file @/etc/kubernetes/admin.conf@
61
* To be able to administrate the cluster you can copy the admin.conf to your local machine
62
* Multi cluster debugging can very easy if you name the config ~/cX-admin.conf (see example below)
63
64
<pre>
65
% scp root@server47.place7.ungleich.ch:/etc/kubernetes/admin.conf ~/c2-admin.conf
66
% export KUBECONFIG=~/c2-admin.conf    
67
% kubectl get nodes
68
NAME       STATUS                     ROLES                  AGE   VERSION
69
server47   Ready                      control-plane,master   82d   v1.22.0
70
server48   Ready                      control-plane,master   82d   v1.22.0
71
server49   Ready                      <none>                 82d   v1.22.0
72
server50   Ready                      <none>                 82d   v1.22.0
73
server59   Ready                      control-plane,master   82d   v1.22.0
74
server60   Ready,SchedulingDisabled   <none>                 82d   v1.22.0
75
server61   Ready                      <none>                 82d   v1.22.0
76
server62   Ready                      <none>                 82d   v1.22.0               
77
</pre>
78
79 18 Nico Schottelius
h3. Installing a new k8s cluster
80 8 Nico Schottelius
81 9 Nico Schottelius
* Decide on the cluster name (usually *cX.k8s.ooo*), X counting upwards
82 28 Nico Schottelius
** Using pXX.k8s.ooo for production clusters of placeXX
83 9 Nico Schottelius
* Use cdist to configure the nodes with requirements like crio
84
* Decide between single or multi node control plane setups (see below)
85 28 Nico Schottelius
** Single control plane suitable for development clusters
86 9 Nico Schottelius
87 28 Nico Schottelius
Typical init procedure:
88 9 Nico Schottelius
89 28 Nico Schottelius
* Single control plane: @kubeadm init --config bootstrap/XXX/kubeadm.yaml@
90
* Multi control plane (HA): @kubeadm init --config bootstrap/XXX/kubeadm.yaml --upload-certs@
91 10 Nico Schottelius
92 29 Nico Schottelius
h3. Deleting a pod that is hanging in terminating state
93
94
<pre>
95
kubectl delete pod <PODNAME> --grace-period=0 --force --namespace <NAMESPACE>
96
</pre>
97
98
(from https://stackoverflow.com/questions/35453792/pods-stuck-in-terminating-status)
99
100 42 Nico Schottelius
h3. Listing nodes of a cluster
101
102
<pre>
103
[15:05] bridge:~% kubectl get nodes
104
NAME       STATUS   ROLES                  AGE   VERSION
105
server22   Ready    <none>                 52d   v1.22.0
106
server23   Ready    <none>                 52d   v1.22.2
107
server24   Ready    <none>                 52d   v1.22.0
108
server25   Ready    <none>                 52d   v1.22.0
109
server26   Ready    <none>                 52d   v1.22.0
110
server27   Ready    <none>                 52d   v1.22.0
111
server63   Ready    control-plane,master   52d   v1.22.0
112
server64   Ready    <none>                 52d   v1.22.0
113
server65   Ready    control-plane,master   52d   v1.22.0
114
server66   Ready    <none>                 52d   v1.22.0
115
server83   Ready    control-plane,master   52d   v1.22.0
116
server84   Ready    <none>                 52d   v1.22.0
117
server85   Ready    <none>                 52d   v1.22.0
118
server86   Ready    <none>                 52d   v1.22.0
119
</pre>
120
121 41 Nico Schottelius
h3. Removing / draining a node
122
123
Usually @kubectl drain server@ should do the job, but sometimes we need to be more aggressive:
124
125 1 Nico Schottelius
<pre>
126 103 Nico Schottelius
kubectl drain --delete-emptydir-data --ignore-daemonsets serverXX
127 42 Nico Schottelius
</pre>
128
129
h3. Readding a node after draining
130
131
<pre>
132
kubectl uncordon serverXX
133 1 Nico Schottelius
</pre>
134 43 Nico Schottelius
135 50 Nico Schottelius
h3. (Re-)joining worker nodes after creating the cluster
136 49 Nico Schottelius
137
* We need to have an up-to-date token
138
* We use different join commands for the workers and control plane nodes
139
140
Generating the join command on an existing control plane node:
141
142
<pre>
143
kubeadm token create --print-join-command
144
</pre>
145
146 50 Nico Schottelius
h3. (Re-)joining control plane nodes after creating the cluster
147 1 Nico Schottelius
148 50 Nico Schottelius
* We generate the token again
149
* We upload the certificates
150
* We need to combine/create the join command for the control plane node
151
152
Example session:
153
154
<pre>
155
% kubeadm token create --print-join-command
156
kubeadm join p10-api.k8s.ooo:6443 --token xmff4i.ABC --discovery-token-ca-cert-hash sha256:longhash 
157
158
% kubeadm init phase upload-certs --upload-certs
159
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
160
[upload-certs] Using certificate key:
161
CERTKEY
162
163
# Then we use these two outputs on the joining node:
164
165
kubeadm join p10-api.k8s.ooo:6443 --token xmff4i.ABC --discovery-token-ca-cert-hash sha256:longhash --control-plane --certificate-key CERTKEY
166
</pre>
167
168
Commands to be used on a control plane node:
169
170
<pre>
171
kubeadm token create --print-join-command
172
kubeadm init phase upload-certs --upload-certs
173
</pre>
174
175
Commands to be used on the joining node:
176
177
<pre>
178
JOINCOMMAND --control-plane --certificate-key CERTKEY
179
</pre>
180 49 Nico Schottelius
181 51 Nico Schottelius
SEE ALSO
182
183
* https://stackoverflow.com/questions/63936268/how-to-generate-kubeadm-token-for-secondary-control-plane-nodes
184
* https://blog.scottlowe.org/2019/08/15/reconstructing-the-join-command-for-kubeadm/
185
186 53 Nico Schottelius
h3. How to fix etcd does not start when rejoining a kubernetes cluster as a control plane
187 52 Nico Schottelius
188
If during the above step etcd does not come up, @kubeadm join@ can hang as follows:
189
190
<pre>
191
[control-plane] Creating static Pod manifest for "kube-apiserver"                                                              
192
[control-plane] Creating static Pod manifest for "kube-controller-manager"                                                     
193
[control-plane] Creating static Pod manifest for "kube-scheduler"                                                              
194
[check-etcd] Checking that the etcd cluster is healthy                                                                         
195
error execution phase check-etcd: etcd cluster is not healthy: failed to dial endpoint https://[2a0a:e5c0:10:1:225:b3ff:fe20:37
196
8a]:2379 with maintenance client: context deadline exceeded                                                                    
197
To see the stack trace of this error execute with --v=5 or higher         
198
</pre>
199
200
Then the problem is likely that the etcd server is still a member of the cluster. We first need to remove it from the etcd cluster and then the join works.
201
202
To fix this we do:
203
204
* Find a working etcd pod
205
* Find the etcd members / member list
206
* Remove the etcd member that we want to re-join the cluster
207
208
209
<pre>
210
# Find the etcd pods
211
kubectl -n kube-system get pods -l component=etcd,tier=control-plane
212
213
# Get the list of etcd servers with the member id 
214
kubectl exec -n kube-system -ti ETCDPODNAME -- etcdctl --endpoints '[::1]:2379' --cacert /etc/kubernetes/pki/etcd/ca.crt --cert  /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member list
215
216
# Remove the member
217
kubectl exec -n kube-system -ti ETCDPODNAME -- etcdctl --endpoints '[::1]:2379' --cacert /etc/kubernetes/pki/etcd/ca.crt --cert  /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member remove MEMBERID
218
</pre>
219
220
Sample session:
221
222
<pre>
223
[10:48] line:~% kubectl -n kube-system get pods -l component=etcd,tier=control-plane
224
NAME            READY   STATUS    RESTARTS     AGE
225
etcd-server63   1/1     Running   0            3m11s
226
etcd-server65   1/1     Running   3            7d2h
227
etcd-server83   1/1     Running   8 (6d ago)   7d2h
228
[10:48] line:~% kubectl exec -n kube-system -ti etcd-server65 -- etcdctl --endpoints '[::1]:2379' --cacert /etc/kubernetes/pki/etcd/ca.crt --cert  /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member list
229
356891cd676df6e4, started, server65, https://[2a0a:e5c0:10:1:225:b3ff:fe20:375c]:2380, https://[2a0a:e5c0:10:1:225:b3ff:fe20:375c]:2379, false
230
371b8a07185dee7e, started, server63, https://[2a0a:e5c0:10:1:225:b3ff:fe20:378a]:2380, https://[2a0a:e5c0:10:1:225:b3ff:fe20:378a]:2379, false
231
5942bc58307f8af9, started, server83, https://[2a0a:e5c0:10:1:3e4a:92ff:fe79:bb98]:2380, https://[2a0a:e5c0:10:1:3e4a:92ff:fe79:bb98]:2379, false
232
233
[10:48] line:~% kubectl exec -n kube-system -ti etcd-server65 -- etcdctl --endpoints '[::1]:2379' --cacert /etc/kubernetes/pki/etcd/ca.crt --cert  /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member remove 371b8a07185dee7e
234
Member 371b8a07185dee7e removed from cluster e3c0805f592a8f77
235 1 Nico Schottelius
236
</pre>
237
238
SEE ALSO
239
240
* We found the solution using https://stackoverflow.com/questions/67921552/re-installed-node-cannot-join-kubernetes-cluster
241 56 Nico Schottelius
242 101 Nico Schottelius
h3. Hardware Maintenance using ungleich-hardware
243
244
Use the following manifest and replace the HOST with the actual host:
245
246
<pre>
247
apiVersion: v1
248
kind: Pod
249
metadata:
250
  name: ungleich-hardware-HOST
251
spec:
252
  containers:
253
  - name: ungleich-hardware
254
    image: ungleich/ungleich-hardware:0.0.5
255
    args:
256
    - sleep
257
    - "1000000"
258
    volumeMounts:
259
      - mountPath: /dev
260
        name: dev
261
    securityContext:
262
      privileged: true
263
  nodeSelector:
264
    kubernetes.io/hostname: "HOST"
265
266
  volumes:
267
    - name: dev
268
      hostPath:
269
        path: /dev
270
</pre>
271
272 102 Nico Schottelius
Also see: [[The_ungleich_hardware_maintenance_guide]]
273
274 105 Nico Schottelius
h3. Triggering a cronjob / creating a job from a cronjob
275 104 Nico Schottelius
276
To test a cronjob, we can create a job from a cronjob:
277
278
<pre>
279
kubectl create job --from=cronjob/volume2-daily-backup volume2-manual
280
</pre>
281
282
This creates a job volume2-manual based on the cronjob  volume2-daily
283
284 62 Nico Schottelius
h2. Calico CNI
285
286
h3. Calico Installation
287
288
* We install "calico using helm":https://docs.projectcalico.org/getting-started/kubernetes/helm
289
* This has the following advantages:
290
** Easy to upgrade
291
** Does not require os to configure IPv6/dual stack settings as the tigera operator figures out things on its own
292
293
Usually plain calico can be installed directly using:
294
295
<pre>
296
helm repo add projectcalico https://docs.projectcalico.org/charts
297 94 Nico Schottelius
helm install calico projectcalico/tigera-operator --version v3.20.4
298 1 Nico Schottelius
</pre>
299 92 Nico Schottelius
300
* Check the tags on https://github.com/projectcalico/calico/tags for the latest release
301 62 Nico Schottelius
302
h3. Installing calicoctl
303
304
To be able to manage and configure calico, we need to 
305
"install calicoctl (we choose the version as a pod)":https://docs.projectcalico.org/getting-started/clis/calicoctl/install#install-calicoctl-as-a-kubernetes-pod
306
307
<pre>
308
kubectl apply -f https://docs.projectcalico.org/manifests/calicoctl.yaml
309
</pre>
310
311 93 Nico Schottelius
Or version specific:
312
313
<pre>
314
kubectl apply -f https://github.com/projectcalico/calico/blob/v3.20.4/manifests/calicoctl.yaml
315 97 Nico Schottelius
316
# For 3.22
317
kubectl apply -f https://projectcalico.docs.tigera.io/archive/v3.22/manifests/calicoctl.yaml
318 93 Nico Schottelius
</pre>
319
320 70 Nico Schottelius
And making it easier accessible by alias:
321
322
<pre>
323
alias calicoctl="kubectl exec -i -n kube-system calicoctl -- /calicoctl"
324
</pre>
325
326 62 Nico Schottelius
h3. Calico configuration
327
328 63 Nico Schottelius
By default our k8s clusters "BGP peer":https://docs.projectcalico.org/networking/bgp
329
with an upstream router to propagate podcidr and servicecidr.
330 62 Nico Schottelius
331
Default settings in our infrastructure:
332
333
* We use a full-mesh using the @nodeToNodeMeshEnabled: true@ option
334
* We keep the original next hop so that *only* the server with the pod is announcing it (instead of ecmp)
335 1 Nico Schottelius
* We use private ASNs for k8s clusters
336 63 Nico Schottelius
* We do *not* use any overlay
337 62 Nico Schottelius
338
After installing calico and calicoctl the last step of the installation is usually:
339
340 1 Nico Schottelius
<pre>
341 79 Nico Schottelius
calicoctl create -f - < calico-bgp.yaml
342 62 Nico Schottelius
</pre>
343
344
345
A sample BGP configuration:
346
347
<pre>
348
---
349
apiVersion: projectcalico.org/v3
350
kind: BGPConfiguration
351
metadata:
352
  name: default
353
spec:
354
  logSeverityScreen: Info
355
  nodeToNodeMeshEnabled: true
356
  asNumber: 65534
357
  serviceClusterIPs:
358
  - cidr: 2a0a:e5c0:10:3::/108
359
  serviceExternalIPs:
360
  - cidr: 2a0a:e5c0:10:3::/108
361
---
362
apiVersion: projectcalico.org/v3
363
kind: BGPPeer
364
metadata:
365
  name: router1-place10
366
spec:
367
  peerIP: 2a0a:e5c0:10:1::50
368
  asNumber: 213081
369
  keepOriginalNextHop: true
370
</pre>
371
372 64 Nico Schottelius
h2. ArgoCD / ArgoWorkFlow
373 56 Nico Schottelius
374 60 Nico Schottelius
h3. Argocd Installation
375 1 Nico Schottelius
376 60 Nico Schottelius
As there is no configuration management present yet, argocd is installed using
377
378 1 Nico Schottelius
<pre>
379 60 Nico Schottelius
kubectl create namespace argocd
380 86 Nico Schottelius
381 96 Nico Schottelius
# Specific Version
382
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v2.3.2/manifests/install.yaml
383 86 Nico Schottelius
384
# OR: latest stable
385 60 Nico Schottelius
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
386 1 Nico Schottelius
</pre>
387 56 Nico Schottelius
388 60 Nico Schottelius
* See https://argo-cd.readthedocs.io/en/stable/
389 1 Nico Schottelius
390 60 Nico Schottelius
h3. Get the argocd credentials
391
392
<pre>
393
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo ""
394
</pre>
395 52 Nico Schottelius
396 87 Nico Schottelius
h3. Accessing argocd
397
398
In regular IPv6 clusters:
399
400
* Navigate to https://argocd-server.argocd.CLUSTERDOMAIN
401
402
In legacy IPv4 clusters
403
404
<pre>
405
kubectl --namespace argocd port-forward svc/argocd-server 8080:80
406
</pre>
407
408 88 Nico Schottelius
* Navigate to https://localhost:8080
409
410 68 Nico Schottelius
h3. Using the argocd webhook to trigger changes
411 67 Nico Schottelius
412
* To trigger changes post json https://argocd.example.com/api/webhook
413
414 72 Nico Schottelius
h3. Deploying an application
415
416
* Applications are deployed via git towards gitea (code.ungleich.ch) and then pulled by argo
417 73 Nico Schottelius
* Always include the *redmine-url* pointing to the (customer) ticket
418
** Also add the support-url if it exists
419 72 Nico Schottelius
420
Application sample
421
422
<pre>
423
apiVersion: argoproj.io/v1alpha1
424
kind: Application
425
metadata:
426
  name: gitea-CUSTOMER
427
  namespace: argocd
428
spec:
429
  destination:
430
    namespace: default
431
    server: 'https://kubernetes.default.svc'
432
  source:
433
    path: apps/prod/gitea
434
    repoURL: 'https://code.ungleich.ch/ungleich-intern/k8s-config.git'
435
    targetRevision: HEAD
436
    helm:
437
      parameters:
438
        - name: storage.data.storageClass
439
          value: rook-ceph-block-hdd
440
        - name: storage.data.size
441
          value: 200Gi
442
        - name: storage.db.storageClass
443
          value: rook-ceph-block-ssd
444
        - name: storage.db.size
445
          value: 10Gi
446
        - name: storage.letsencrypt.storageClass
447
          value: rook-ceph-block-hdd
448
        - name: storage.letsencrypt.size
449
          value: 50Mi
450
        - name: letsencryptStaging
451
          value: 'no'
452
        - name: fqdn
453
          value: 'code.verua.online'
454
  project: default
455
  syncPolicy:
456
    automated:
457
      prune: true
458
      selfHeal: true
459
  info:
460
    - name: 'redmine-url'
461
      value: 'https://redmine.ungleich.ch/issues/ISSUEID'
462
    - name: 'support-url'
463
      value: 'https://support.ungleich.ch/Ticket/Display.html?id=TICKETID'
464
</pre>
465
466 80 Nico Schottelius
h2. Helm related operations and conventions
467 55 Nico Schottelius
468 61 Nico Schottelius
We use helm charts extensively.
469
470
* In production, they are managed via argocd
471
* In development, helm chart can de developed and deployed manually using the helm utility.
472
473 55 Nico Schottelius
h3. Installing a helm chart
474
475
One can use the usual pattern of
476
477
<pre>
478
helm install <releasename> <chartdirectory>
479
</pre>
480
481
However often you want to reinstall/update when testing helm charts. The following pattern is "better", because it allows you to reinstall, if it is already installed:
482
483
<pre>
484
helm upgrade --install <releasename> <chartdirectory>
485 1 Nico Schottelius
</pre>
486 80 Nico Schottelius
487
h3. Naming services and deployments in helm charts [Application labels]
488
489
* We always have {{ .Release.Name }} to identify the current "instance"
490
* Deployments:
491
** use @app: <what it is>@, f.i. @app: nginx@, @app: postgres@, ...
492 81 Nico Schottelius
* See more about standard labels on
493
** https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/
494
** https://helm.sh/docs/chart_best_practices/labels/
495 55 Nico Schottelius
496 43 Nico Schottelius
h2. Rook / Ceph Related Operations
497
498 71 Nico Schottelius
h3. Executing ceph commands
499
500
Using the ceph-tools pod as follows:
501
502
<pre>
503
kubectl exec -n rook-ceph -ti $(kubectl -n rook-ceph get pods -l app=rook-ceph-tools -o jsonpath='{.items[*].metadata.name}') -- ceph -s
504
</pre>
505
506 43 Nico Schottelius
h3. Inspecting the logs of a specific server
507
508
<pre>
509
# Get the related pods
510
kubectl -n rook-ceph get pods -l app=rook-ceph-osd-prepare 
511
...
512
513
# Inspect the logs of a specific pod
514
kubectl -n rook-ceph logs -f rook-ceph-osd-prepare-server23--1-444qx
515
516 71 Nico Schottelius
</pre>
517
518
h3. Inspecting the logs of the rook-ceph-operator
519
520
<pre>
521
kubectl -n rook-ceph logs -f -l app=rook-ceph-operator
522 43 Nico Schottelius
</pre>
523
524
h3. Triggering server prepare / adding new osds
525
526
The rook-ceph-operator triggers/watches/creates pods to maintain hosts. To trigger a full "re scan", simply delete that pod:
527
528
<pre>
529
kubectl -n rook-ceph delete pods -l app=rook-ceph-operator
530
</pre>
531
532
This will cause all the @rook-ceph-osd-prepare-..@ jobs to be recreated and thus OSDs to be created, if new disks have been added.
533
534
h3. Removing an OSD
535
536
* See "Ceph OSD Management":https://rook.io/docs/rook/v1.7/ceph-osd-mgmt.html
537 77 Nico Schottelius
* More specifically: https://github.com/rook/rook/blob/release-1.7/cluster/examples/kubernetes/ceph/osd-purge.yaml
538 99 Nico Schottelius
* Then delete the related deployment
539 41 Nico Schottelius
540 98 Nico Schottelius
Set osd id in the osd-purge.yaml and apply it. OSD should be down before.
541
542
<pre>
543
apiVersion: batch/v1
544
kind: Job
545
metadata:
546
  name: rook-ceph-purge-osd
547
  namespace: rook-ceph # namespace:cluster
548
  labels:
549
    app: rook-ceph-purge-osd
550
spec:
551
  template:
552
    metadata:
553
      labels:
554
        app: rook-ceph-purge-osd
555
    spec:
556
      serviceAccountName: rook-ceph-purge-osd
557
      containers:
558
        - name: osd-removal
559
          image: rook/ceph:master
560
          # TODO: Insert the OSD ID in the last parameter that is to be removed
561
          # The OSD IDs are a comma-separated list. For example: "0" or "0,2".
562
          # If you want to preserve the OSD PVCs, set `--preserve-pvc true`.
563
          #
564
          # A --force-osd-removal option is available if the OSD should be destroyed even though the
565
          # removal could lead to data loss.
566
          args:
567
            - "ceph"
568
            - "osd"
569
            - "remove"
570
            - "--preserve-pvc"
571
            - "false"
572
            - "--force-osd-removal"
573
            - "false"
574
            - "--osd-ids"
575
            - "SETTHEOSDIDHERE"
576
          env:
577
            - name: POD_NAMESPACE
578
              valueFrom:
579
                fieldRef:
580
                  fieldPath: metadata.namespace
581
            - name: ROOK_MON_ENDPOINTS
582
              valueFrom:
583
                configMapKeyRef:
584
                  key: data
585
                  name: rook-ceph-mon-endpoints
586
            - name: ROOK_CEPH_USERNAME
587
              valueFrom:
588
                secretKeyRef:
589
                  key: ceph-username
590
                  name: rook-ceph-mon
591
            - name: ROOK_CEPH_SECRET
592
              valueFrom:
593
                secretKeyRef:
594
                  key: ceph-secret
595
                  name: rook-ceph-mon
596
            - name: ROOK_CONFIG_DIR
597
              value: /var/lib/rook
598
            - name: ROOK_CEPH_CONFIG_OVERRIDE
599
              value: /etc/rook/config/override.conf
600
            - name: ROOK_FSID
601
              valueFrom:
602
                secretKeyRef:
603
                  key: fsid
604
                  name: rook-ceph-mon
605
            - name: ROOK_LOG_LEVEL
606
              value: DEBUG
607
          volumeMounts:
608
            - mountPath: /etc/ceph
609
              name: ceph-conf-emptydir
610
            - mountPath: /var/lib/rook
611
              name: rook-config
612
      volumes:
613
        - emptyDir: {}
614
          name: ceph-conf-emptydir
615
        - emptyDir: {}
616
          name: rook-config
617
      restartPolicy: Never
618
619
620 99 Nico Schottelius
</pre>
621
622
Deleting the deployment:
623
624
<pre>
625
[18:05] bridge:~% kubectl -n rook-ceph delete deployment rook-ceph-osd-6
626
deployment.apps "rook-ceph-osd-6" deleted
627 98 Nico Schottelius
</pre>
628
629 76 Nico Schottelius
h2. Harbor
630
631
* We user "Harbor":https://goharbor.io/ for caching and as an image registry. Internal app reference: apps/prod/harbor.
632
* The admin password is in the password store, auto generated per cluster
633
* At the moment harbor only authenticates against the internal ldap tree
634
635
h3. LDAP configuration
636
637
* The url needs to be ldaps://...
638
* uid = uid
639
* rest standard
640 75 Nico Schottelius
641 89 Nico Schottelius
h2. Monitoring / Prometheus
642
643 90 Nico Schottelius
* Via "kube-prometheus":https://github.com/prometheus-operator/kube-prometheus/
644 89 Nico Schottelius
645 91 Nico Schottelius
Access via ...
646
647
* http://prometheus-k8s.monitoring.svc:9090
648
* http://grafana.monitoring.svc:3000
649
* http://alertmanager.monitoring.svc:9093
650
651
652 100 Nico Schottelius
h3. Prometheus Options
653
654
* "helm/kube-prometheus-stack":https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack
655
** Includes dashboards and co.
656
* "manifest based kube-prometheus":https://github.com/prometheus-operator/kube-prometheus
657
** Includes dashboards and co.
658
* "Prometheus Operator (mainly CRD manifest":https://github.com/prometheus-operator/prometheus-operator
659
660 91 Nico Schottelius
661 82 Nico Schottelius
h2. Nextcloud
662
663 85 Nico Schottelius
h3. How to get the nextcloud credentials 
664 84 Nico Schottelius
665
* The initial username is set to "nextcloud"
666
* The password is autogenerated and saved in a kubernetes secret
667
668
<pre>
669 85 Nico Schottelius
kubectl get secret RELEASENAME-nextcloud -o jsonpath="{.data.PASSWORD}" | base64 -d; echo "" 
670 84 Nico Schottelius
</pre>
671
672 83 Nico Schottelius
h3. How to fix "Access through untrusted domain"
673
674 82 Nico Schottelius
* Nextcloud stores the initial domain configuration
675 1 Nico Schottelius
* If the FQDN is changed, it will show the error message "Access through untrusted domain"
676 82 Nico Schottelius
* To fix, edit /var/www/html/config/config.php and correct the domain
677 83 Nico Schottelius
* Then delete the pods
678 82 Nico Schottelius
679 1 Nico Schottelius
h2. Infrastructure versions
680 35 Nico Schottelius
681 57 Nico Schottelius
h3. ungleich kubernetes infrastructure v5 (2021-10)
682 1 Nico Schottelius
683 57 Nico Schottelius
Clusters are configured / setup in this order:
684
685
* Bootstrap via kubeadm
686 59 Nico Schottelius
* "Networking via calico + BGP (non ECMP) using helm":https://docs.projectcalico.org/getting-started/kubernetes/helm
687
* "ArgoCD for CD":https://argo-cd.readthedocs.io/en/stable/
688
** "rook for storage via argocd":https://rook.io/
689 58 Nico Schottelius
** haproxy for in IPv6-cluster-IPv4-to-IPv6 proxy via argocd
690
** "kubernetes-secret-generator for in cluster secrets":https://github.com/mittwald/kubernetes-secret-generator
691
** "ungleich-certbot managing certs and nginx":https://hub.docker.com/repository/docker/ungleich/ungleich-certbot
692
693 57 Nico Schottelius
694
h3. ungleich kubernetes infrastructure v4 (2021-09)
695
696 54 Nico Schottelius
* rook is configured via manifests instead of using the rook-ceph-cluster helm chart
697 1 Nico Schottelius
* The rook operator is still being installed via helm
698 35 Nico Schottelius
699 57 Nico Schottelius
h3. ungleich kubernetes infrastructure v3 (2021-07)
700 1 Nico Schottelius
701 10 Nico Schottelius
* rook is now installed via helm via argocd instead of directly via manifests
702 28 Nico Schottelius
703 57 Nico Schottelius
h3. ungleich kubernetes infrastructure v2 (2021-05)
704 28 Nico Schottelius
705
* Replaced fluxv2 from ungleich k8s v1 with argocd
706 1 Nico Schottelius
** argocd can apply helm templates directly without needing to go through Chart releases
707 28 Nico Schottelius
* We are also using argoflow for build flows
708
* Planned to add "kaniko":https://github.com/GoogleContainerTools/kaniko for image building
709
710 57 Nico Schottelius
h3. ungleich kubernetes infrastructure v1 (2021-01)
711 28 Nico Schottelius
712
We are using the following components:
713
714
* "Calico as a CNI":https://www.projectcalico.org/ with BGP, IPv6 only, no encapsulation
715
** Needed for basic networking
716
* "kubernetes-secret-generator":https://github.com/mittwald/kubernetes-secret-generator for creating secrets
717
** Needed so that secrets are not stored in the git repository, but only in the cluster
718
* "ungleich-certbot":https://hub.docker.com/repository/docker/ungleich/ungleich-certbot
719
** Needed to get letsencrypt certificates for services
720
* "rook with ceph rbd + cephfs":https://rook.io/ for storage
721
** rbd for almost everything, *ReadWriteOnce*
722
** cephfs for smaller things, multi access *ReadWriteMany*
723
** Needed for providing persistent storage
724
* "flux v2":https://fluxcd.io/
725
** Needed to manage resources automatically