Project

General

Profile

The ungleich kubernetes infrastructure » History » Version 48

Nico Schottelius, 10/12/2021 08:19 AM

1 22 Nico Schottelius
h1. The ungleich kubernetes infrastructure and ungleich kubernetes manual
2 1 Nico Schottelius
3 3 Nico Schottelius
{{toc}}
4
5 1 Nico Schottelius
h2. Status
6
7 28 Nico Schottelius
This document is **pre-production**.
8
This document is to become the ungleich kubernetes infrastructure overview as well as the ungleich kubernetes manual.
9 1 Nico Schottelius
10 10 Nico Schottelius
h2. k8s clusters
11
12 48 Nico Schottelius
| Cluster         | Purpose/Setup     | Maintainer   | Master(s)                                                | argo | rook | last verified |
13
| c0.k8s.ooo      | Dev               | -            | UNUSED                                                   |      |      |    2021-10-05 |
14
| c1.k8s.ooo      | Dev p6 VM         | Nico         | 2a0a-e5c0-2-11-0-62ff-fe0b-1a3d.k8s-1.place6.ungleich.ch |      |      |    2021-10-05 |
15
| c2.k8s.ooo      | Dev p7 HW         | Nico         | server47 server53 server54                               | x    | x    |    2021-10-05 |
16
| c3.k8s.ooo      | Test p7 PI        | -            | UNUSED                                                   |      |      |    2021-10-05 |
17
| c4.k8s.ooo      | Dev2 p7 HW        | Fran/Jin-Guk | server52 server53 server54                               |      |      |             - |
18
| c5.k8s.ooo      | Dev p6 VM Amal    | Nico/Amal    | 2a0a-e5c0-2-11-0-62ff-fe0b-1a46.k8s-1.place6.ungleich.ch |      |      |               |
19
| c6.k8s.ooo      | Dev p6 VM Jin-Guk | Jin-Guk      |                                                          |      |      |               |
20
| [[p6.k8s.ooo]]  | production        |              | server67 server69 server71                               | x    | x    |    2021-10-05 |
21
| [[p10.k8s.ooo]] | production        |              | server63 server65 server83                               | x    | x    |    2021-10-05 |
22
|                 |                   |              |                                                          |      |      |               |
23
24 21 Nico Schottelius
25 1 Nico Schottelius
h2. General architecture and components overview
26
27
* All k8s clusters are IPv6 only
28
* We use BGP peering to propagate podcidr and serviceCidr networks to our infrastructure
29
* The main public testing repository is "ungleich-k8s":https://code.ungleich.ch/ungleich-public/ungleich-k8s
30 18 Nico Schottelius
** Private configurations are found in the **k8s-config** repository
31 1 Nico Schottelius
32
h3. Cluster types
33
34 28 Nico Schottelius
| **Type/Feature**            | **Development**                | **Production**         |
35
| Min No. nodes               | 3 (1 master, 3 worker)         | 5 (3 master, 3 worker) |
36
| Recommended minimum         | 4 (dedicated master, 3 worker) | 8 (3 master, 5 worker) |
37
| Separation of control plane | optional                       | recommended            |
38
| Persistent storage          | required                       | required               |
39
| Number of storage monitors  | 3                              | 5                      |
40 1 Nico Schottelius
41 43 Nico Schottelius
h2. General k8s operations
42 1 Nico Schottelius
43 46 Nico Schottelius
h3. Cheat sheet / external great references
44
45
* "kubectl cheatsheet":https://kubernetes.io/docs/reference/kubectl/cheatsheet/
46
47 36 Nico Schottelius
h3. Get the argocd credentials
48
49
<pre>
50 37 Nico Schottelius
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo ""
51 36 Nico Schottelius
</pre>
52
53 44 Nico Schottelius
h3. Get the cluster admin.conf
54
55
* On the masters of each cluster you can find the file @/etc/kubernetes/admin.conf@
56
* To be able to administrate the cluster you can copy the admin.conf to your local machine
57
* Multi cluster debugging can very easy if you name the config ~/cX-admin.conf (see example below)
58
59
<pre>
60
% scp root@server47.place7.ungleich.ch:/etc/kubernetes/admin.conf ~/c2-admin.conf
61
% export KUBECONFIG=~/c2-admin.conf    
62
% kubectl get nodes
63
NAME       STATUS                     ROLES                  AGE   VERSION
64
server47   Ready                      control-plane,master   82d   v1.22.0
65
server48   Ready                      control-plane,master   82d   v1.22.0
66
server49   Ready                      <none>                 82d   v1.22.0
67
server50   Ready                      <none>                 82d   v1.22.0
68
server59   Ready                      control-plane,master   82d   v1.22.0
69
server60   Ready,SchedulingDisabled   <none>                 82d   v1.22.0
70
server61   Ready                      <none>                 82d   v1.22.0
71
server62   Ready                      <none>                 82d   v1.22.0               
72
</pre>
73
74 18 Nico Schottelius
h3. Installing a new k8s cluster
75 8 Nico Schottelius
76 9 Nico Schottelius
* Decide on the cluster name (usually *cX.k8s.ooo*), X counting upwards
77 28 Nico Schottelius
** Using pXX.k8s.ooo for production clusters of placeXX
78 9 Nico Schottelius
* Use cdist to configure the nodes with requirements like crio
79
* Decide between single or multi node control plane setups (see below)
80 28 Nico Schottelius
** Single control plane suitable for development clusters
81 9 Nico Schottelius
82 28 Nico Schottelius
Typical init procedure:
83 9 Nico Schottelius
84 28 Nico Schottelius
* Single control plane: @kubeadm init --config bootstrap/XXX/kubeadm.yaml@
85
* Multi control plane (HA): @kubeadm init --config bootstrap/XXX/kubeadm.yaml --upload-certs@
86 10 Nico Schottelius
87 29 Nico Schottelius
h3. Deleting a pod that is hanging in terminating state
88
89
<pre>
90
kubectl delete pod <PODNAME> --grace-period=0 --force --namespace <NAMESPACE>
91
</pre>
92
93
(from https://stackoverflow.com/questions/35453792/pods-stuck-in-terminating-status)
94
95 42 Nico Schottelius
h3. Listing nodes of a cluster
96
97
<pre>
98
[15:05] bridge:~% kubectl get nodes
99
NAME       STATUS   ROLES                  AGE   VERSION
100
server22   Ready    <none>                 52d   v1.22.0
101
server23   Ready    <none>                 52d   v1.22.2
102
server24   Ready    <none>                 52d   v1.22.0
103
server25   Ready    <none>                 52d   v1.22.0
104
server26   Ready    <none>                 52d   v1.22.0
105
server27   Ready    <none>                 52d   v1.22.0
106
server63   Ready    control-plane,master   52d   v1.22.0
107
server64   Ready    <none>                 52d   v1.22.0
108
server65   Ready    control-plane,master   52d   v1.22.0
109
server66   Ready    <none>                 52d   v1.22.0
110
server83   Ready    control-plane,master   52d   v1.22.0
111
server84   Ready    <none>                 52d   v1.22.0
112
server85   Ready    <none>                 52d   v1.22.0
113
server86   Ready    <none>                 52d   v1.22.0
114
</pre>
115
116
117 41 Nico Schottelius
h3. Removing / draining a node
118
119
Usually @kubectl drain server@ should do the job, but sometimes we need to be more aggressive:
120
121
<pre>
122
kubectl drain --delete-emptydir-data --ignore-daemonsets server23
123 42 Nico Schottelius
</pre>
124
125
h3. Readding a node after draining
126
127
<pre>
128
kubectl uncordon serverXX
129 1 Nico Schottelius
</pre>
130 43 Nico Schottelius
131
h2. Rook / Ceph Related Operations
132
133
h3. Inspecting the logs of a specific server
134
135
<pre>
136
# Get the related pods
137
kubectl -n rook-ceph get pods -l app=rook-ceph-osd-prepare 
138
...
139
140
# Inspect the logs of a specific pod
141
kubectl -n rook-ceph logs -f rook-ceph-osd-prepare-server23--1-444qx
142
143
</pre>
144
145
h3. Triggering server prepare / adding new osds
146
147
The rook-ceph-operator triggers/watches/creates pods to maintain hosts. To trigger a full "re scan", simply delete that pod:
148
149
<pre>
150
kubectl -n rook-ceph delete pods -l app=rook-ceph-operator
151
</pre>
152
153
This will cause all the @rook-ceph-osd-prepare-..@ jobs to be recreated and thus OSDs to be created, if new disks have been added.
154
155
h3. Removing an OSD
156
157
* See "Ceph OSD Management":https://rook.io/docs/rook/v1.7/ceph-osd-mgmt.html
158 41 Nico Schottelius
159 1 Nico Schottelius
h2. Infrastructure versions
160 35 Nico Schottelius
161
h3. ungleich kubernetes infrastructure v3
162
163
* rook is now installed via helm via argocd instead of directly via manifests
164 10 Nico Schottelius
165 28 Nico Schottelius
h3. ungleich kubernetes infrastructure v2
166
167
* Replaced fluxv2 from ungleich k8s v1 with argocd
168
** argocd can apply helm templates directly without needing to go through Chart releases
169
* We are also using argoflow for build flows
170
* Planned to add "kaniko":https://github.com/GoogleContainerTools/kaniko for image building
171
172
h3. ungleich kubernetes infrastructure v1
173
174
We are using the following components:
175
176
* "Calico as a CNI":https://www.projectcalico.org/ with BGP, IPv6 only, no encapsulation
177
** Needed for basic networking
178
* "kubernetes-secret-generator":https://github.com/mittwald/kubernetes-secret-generator for creating secrets
179
** Needed so that secrets are not stored in the git repository, but only in the cluster
180
* "ungleich-certbot":https://hub.docker.com/repository/docker/ungleich/ungleich-certbot
181
** Needed to get letsencrypt certificates for services
182
* "rook with ceph rbd + cephfs":https://rook.io/ for storage
183
** rbd for almost everything, *ReadWriteOnce*
184
** cephfs for smaller things, multi access *ReadWriteMany*
185
** Needed for providing persistent storage
186
* "flux v2":https://fluxcd.io/
187
** Needed to manage resources automatically