The ungleich kubernetes infrastructure » History » Revision 51
Revision 50 (Nico Schottelius, 10/14/2021 05:20 AM) → Revision 51/219 (Nico Schottelius, 10/14/2021 05:25 AM)
h1. The ungleich kubernetes infrastructure and ungleich kubernetes manual
{{toc}}
h2. Status
This document is **pre-production**.
This document is to become the ungleich kubernetes infrastructure overview as well as the ungleich kubernetes manual.
h2. k8s clusters
| Cluster | Purpose/Setup | Maintainer | Master(s) | argo | rook | last verified |
| c0.k8s.ooo | Dev | - | UNUSED | | | 2021-10-05 |
| c1.k8s.ooo | Dev p6 VM | Nico | 2a0a-e5c0-2-11-0-62ff-fe0b-1a3d.k8s-1.place6.ungleich.ch | | | 2021-10-05 |
| c2.k8s.ooo | Dev p7 HW | Nico | server47 server53 server54 | x | x | 2021-10-05 |
| c3.k8s.ooo | Test p7 PI | - | UNUSED | | | 2021-10-05 |
| c4.k8s.ooo | Dev2 p7 HW | Fran/Jin-Guk | server52 server53 server54 | | | - |
| c5.k8s.ooo | Dev p6 VM Amal | Nico/Amal | 2a0a-e5c0-2-11-0-62ff-fe0b-1a46.k8s-1.place6.ungleich.ch | | | |
| c6.k8s.ooo | Dev p6 VM Jin-Guk | Jin-Guk | | | | |
| [[p6.k8s.ooo]] | production | | server67 server69 server71 | x | x | 2021-10-05 |
| [[p10.k8s.ooo]] | production | | server63 server65 server83 | x | x | 2021-10-05 |
| | | | | | | |
h2. General architecture and components overview
* All k8s clusters are IPv6 only
* We use BGP peering to propagate podcidr and serviceCidr networks to our infrastructure
* The main public testing repository is "ungleich-k8s":https://code.ungleich.ch/ungleich-public/ungleich-k8s
** Private configurations are found in the **k8s-config** repository
h3. Cluster types
| **Type/Feature** | **Development** | **Production** |
| Min No. nodes | 3 (1 master, 3 worker) | 5 (3 master, 3 worker) |
| Recommended minimum | 4 (dedicated master, 3 worker) | 8 (3 master, 5 worker) |
| Separation of control plane | optional | recommended |
| Persistent storage | required | required |
| Number of storage monitors | 3 | 5 |
h2. General k8s operations
h3. Cheat sheet / external great references
* "kubectl cheatsheet":https://kubernetes.io/docs/reference/kubectl/cheatsheet/
h3. Get the argocd credentials
<pre>
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo ""
</pre>
h3. Get the cluster admin.conf
* On the masters of each cluster you can find the file @/etc/kubernetes/admin.conf@
* To be able to administrate the cluster you can copy the admin.conf to your local machine
* Multi cluster debugging can very easy if you name the config ~/cX-admin.conf (see example below)
<pre>
% scp root@server47.place7.ungleich.ch:/etc/kubernetes/admin.conf ~/c2-admin.conf
% export KUBECONFIG=~/c2-admin.conf
% kubectl get nodes
NAME STATUS ROLES AGE VERSION
server47 Ready control-plane,master 82d v1.22.0
server48 Ready control-plane,master 82d v1.22.0
server49 Ready <none> 82d v1.22.0
server50 Ready <none> 82d v1.22.0
server59 Ready control-plane,master 82d v1.22.0
server60 Ready,SchedulingDisabled <none> 82d v1.22.0
server61 Ready <none> 82d v1.22.0
server62 Ready <none> 82d v1.22.0
</pre>
h3. Installing a new k8s cluster
* Decide on the cluster name (usually *cX.k8s.ooo*), X counting upwards
** Using pXX.k8s.ooo for production clusters of placeXX
* Use cdist to configure the nodes with requirements like crio
* Decide between single or multi node control plane setups (see below)
** Single control plane suitable for development clusters
Typical init procedure:
* Single control plane: @kubeadm init --config bootstrap/XXX/kubeadm.yaml@
* Multi control plane (HA): @kubeadm init --config bootstrap/XXX/kubeadm.yaml --upload-certs@
h3. Deleting a pod that is hanging in terminating state
<pre>
kubectl delete pod <PODNAME> --grace-period=0 --force --namespace <NAMESPACE>
</pre>
(from https://stackoverflow.com/questions/35453792/pods-stuck-in-terminating-status)
h3. Listing nodes of a cluster
<pre>
[15:05] bridge:~% kubectl get nodes
NAME STATUS ROLES AGE VERSION
server22 Ready <none> 52d v1.22.0
server23 Ready <none> 52d v1.22.2
server24 Ready <none> 52d v1.22.0
server25 Ready <none> 52d v1.22.0
server26 Ready <none> 52d v1.22.0
server27 Ready <none> 52d v1.22.0
server63 Ready control-plane,master 52d v1.22.0
server64 Ready <none> 52d v1.22.0
server65 Ready control-plane,master 52d v1.22.0
server66 Ready <none> 52d v1.22.0
server83 Ready control-plane,master 52d v1.22.0
server84 Ready <none> 52d v1.22.0
server85 Ready <none> 52d v1.22.0
server86 Ready <none> 52d v1.22.0
</pre>
h3. Removing / draining a node
Usually @kubectl drain server@ should do the job, but sometimes we need to be more aggressive:
<pre>
kubectl drain --delete-emptydir-data --ignore-daemonsets server23
</pre>
h3. Readding a node after draining
<pre>
kubectl uncordon serverXX
</pre>
h3. (Re-)joining worker nodes after creating the cluster
* We need to have an up-to-date token
* We use different join commands for the workers and control plane nodes
Generating the join command on an existing control plane node:
<pre>
kubeadm token create --print-join-command
</pre>
h3. (Re-)joining control plane nodes after creating the cluster
* We generate the token again
* We upload the certificates
* We need to combine/create the join command for the control plane node
Example session:
<pre>
% kubeadm token create --print-join-command
kubeadm join p10-api.k8s.ooo:6443 --token xmff4i.ABC --discovery-token-ca-cert-hash sha256:longhash
% kubeadm init phase upload-certs --upload-certs
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
CERTKEY
# Then we use these two outputs on the joining node:
kubeadm join p10-api.k8s.ooo:6443 --token xmff4i.ABC --discovery-token-ca-cert-hash sha256:longhash --control-plane --certificate-key CERTKEY
</pre>
Commands to be used on a control plane node:
<pre>
kubeadm token create --print-join-command
kubeadm init phase upload-certs --upload-certs
</pre>
Commands to be used on the joining node:
<pre>
JOINCOMMAND --control-plane --certificate-key CERTKEY
</pre>
SEE ALSO
* https://stackoverflow.com/questions/63936268/how-to-generate-kubeadm-token-for-secondary-control-plane-nodes
* https://blog.scottlowe.org/2019/08/15/reconstructing-the-join-command-for-kubeadm/
h2. Rook / Ceph Related Operations
h3. Inspecting the logs of a specific server
<pre>
# Get the related pods
kubectl -n rook-ceph get pods -l app=rook-ceph-osd-prepare
...
# Inspect the logs of a specific pod
kubectl -n rook-ceph logs -f rook-ceph-osd-prepare-server23--1-444qx
</pre>
h3. Triggering server prepare / adding new osds
The rook-ceph-operator triggers/watches/creates pods to maintain hosts. To trigger a full "re scan", simply delete that pod:
<pre>
kubectl -n rook-ceph delete pods -l app=rook-ceph-operator
</pre>
This will cause all the @rook-ceph-osd-prepare-..@ jobs to be recreated and thus OSDs to be created, if new disks have been added.
h3. Removing an OSD
* See "Ceph OSD Management":https://rook.io/docs/rook/v1.7/ceph-osd-mgmt.html
h2. Infrastructure versions
h3. ungleich kubernetes infrastructure v3
* rook is now installed via helm via argocd instead of directly via manifests
h3. ungleich kubernetes infrastructure v2
* Replaced fluxv2 from ungleich k8s v1 with argocd
** argocd can apply helm templates directly without needing to go through Chart releases
* We are also using argoflow for build flows
* Planned to add "kaniko":https://github.com/GoogleContainerTools/kaniko for image building
h3. ungleich kubernetes infrastructure v1
We are using the following components:
* "Calico as a CNI":https://www.projectcalico.org/ with BGP, IPv6 only, no encapsulation
** Needed for basic networking
* "kubernetes-secret-generator":https://github.com/mittwald/kubernetes-secret-generator for creating secrets
** Needed so that secrets are not stored in the git repository, but only in the cluster
* "ungleich-certbot":https://hub.docker.com/repository/docker/ungleich/ungleich-certbot
** Needed to get letsencrypt certificates for services
* "rook with ceph rbd + cephfs":https://rook.io/ for storage
** rbd for almost everything, *ReadWriteOnce*
** cephfs for smaller things, multi access *ReadWriteMany*
** Needed for providing persistent storage
* "flux v2":https://fluxcd.io/
** Needed to manage resources automatically