Project

General

Profile

The ungleich kubernetes infrastructure » History » Revision 47

Revision 46 (Nico Schottelius, 10/10/2021 10:09 AM) → Revision 47/212 (Nico Schottelius, 10/12/2021 08:07 AM)

h1. The ungleich kubernetes infrastructure and ungleich kubernetes manual 

 {{toc}} 

 h2. Status 

 This document is **pre-production**. 
 This document is to become the ungleich kubernetes infrastructure overview as well as the ungleich kubernetes manual. 

 h2. k8s clusters 

 | Cluster       | Purpose/Setup    | Maintainer     | Master(s)                                                  | last verified | 
 | c0.k8s.ooo    | Dev              | -              | UNUSED                                                     |      2021-10-05 | 
 | c1.k8s.ooo    | Dev p6 VM        | Nico           | 2a0a-e5c0-2-11-0-62ff-fe0b-1a3d.k8s-1.place6.ungleich.ch |      2021-10-05 | 
 | c2.k8s.ooo    | Dev p7 HW        | Nico           | server47 server53 server54                                 |      2021-10-05 | 
 | c3.k8s.ooo    | Test p7 PI       | -              | UNUSED                                                     |      2021-10-05 | 
 | c4.k8s.ooo    | Dev2 p7 HW       | Fran/Jin-Guk | server52 server53 server54                                 |               - | 
 | c5.k8s.ooo    | Dev p6 VM Amal | Nico/Amal      | 2a0a-e5c0-2-11-0-62ff-fe0b-1a46.k8s-1.place6.ungleich.ch |                 | 
 | c6.k8s.ooo    | Dev p6 VM Jin-Guk | Jin-Guk      |    |                 | 
 | [[p6.k8s.ooo]] p6.k8s.ooo    | production       |                | server67 server69 server71                                 |      2021-10-05 | 
 | [[p10.k8s.ooo]] p10.k8s.ooo | production       |                | server63 server65 server83                                 |      2021-10-05 | 
 |               |                  |                |                                                            |                 | 

 h2. General architecture and components overview 

 * All k8s clusters are IPv6 only 
 * We use BGP peering to propagate podcidr and serviceCidr networks to our infrastructure 
 * The main public testing repository is "ungleich-k8s":https://code.ungleich.ch/ungleich-public/ungleich-k8s 
 ** Private configurations are found in the **k8s-config** repository 

 h3. Cluster types 

 | **Type/Feature**              | **Development**                  | **Production**           | 
 | Min No. nodes                 | 3 (1 master, 3 worker)           | 5 (3 master, 3 worker) | 
 | Recommended minimum           | 4 (dedicated master, 3 worker) | 8 (3 master, 5 worker) | 
 | Separation of control plane | optional                         | recommended              | 
 | Persistent storage            | required                         | required                 | 
 | Number of storage monitors    | 3                                | 5                        | 

 h2. General k8s operations 

 h3. Cheat sheet / external great references 

 * "kubectl cheatsheet":https://kubernetes.io/docs/reference/kubectl/cheatsheet/ 

 h3. Get the argocd credentials 

 <pre> 
 kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo "" 
 </pre> 

 h3. Get the cluster admin.conf 

 * On the masters of each cluster you can find the file @/etc/kubernetes/admin.conf@ 
 * To be able to administrate the cluster you can copy the admin.conf to your local machine 
 * Multi cluster debugging can very easy if you name the config ~/cX-admin.conf (see example below) 

 <pre> 
 % scp root@server47.place7.ungleich.ch:/etc/kubernetes/admin.conf ~/c2-admin.conf 
 % export KUBECONFIG=~/c2-admin.conf     
 % kubectl get nodes 
 NAME         STATUS                       ROLES                    AGE     VERSION 
 server47     Ready                        control-plane,master     82d     v1.22.0 
 server48     Ready                        control-plane,master     82d     v1.22.0 
 server49     Ready                        <none>                   82d     v1.22.0 
 server50     Ready                        <none>                   82d     v1.22.0 
 server59     Ready                        control-plane,master     82d     v1.22.0 
 server60     Ready,SchedulingDisabled     <none>                   82d     v1.22.0 
 server61     Ready                        <none>                   82d     v1.22.0 
 server62     Ready                        <none>                   82d     v1.22.0                
 </pre> 

 h3. Installing a new k8s cluster 

 * Decide on the cluster name (usually *cX.k8s.ooo*), X counting upwards 
 ** Using pXX.k8s.ooo for production clusters of placeXX 
 * Use cdist to configure the nodes with requirements like crio 
 * Decide between single or multi node control plane setups (see below) 
 ** Single control plane suitable for development clusters 

 Typical init procedure: 

 * Single control plane: @kubeadm init --config bootstrap/XXX/kubeadm.yaml@ 
 * Multi control plane (HA): @kubeadm init --config bootstrap/XXX/kubeadm.yaml --upload-certs@ 

 h3. Deleting a pod that is hanging in terminating state 

 <pre> 
 kubectl delete pod <PODNAME> --grace-period=0 --force --namespace <NAMESPACE> 
 </pre> 

 (from https://stackoverflow.com/questions/35453792/pods-stuck-in-terminating-status) 

 h3. Listing nodes of a cluster 

 <pre> 
 [15:05] bridge:~% kubectl get nodes 
 NAME         STATUS     ROLES                    AGE     VERSION 
 server22     Ready      <none>                   52d     v1.22.0 
 server23     Ready      <none>                   52d     v1.22.2 
 server24     Ready      <none>                   52d     v1.22.0 
 server25     Ready      <none>                   52d     v1.22.0 
 server26     Ready      <none>                   52d     v1.22.0 
 server27     Ready      <none>                   52d     v1.22.0 
 server63     Ready      control-plane,master     52d     v1.22.0 
 server64     Ready      <none>                   52d     v1.22.0 
 server65     Ready      control-plane,master     52d     v1.22.0 
 server66     Ready      <none>                   52d     v1.22.0 
 server83     Ready      control-plane,master     52d     v1.22.0 
 server84     Ready      <none>                   52d     v1.22.0 
 server85     Ready      <none>                   52d     v1.22.0 
 server86     Ready      <none>                   52d     v1.22.0 
 </pre> 


 h3. Removing / draining a node 

 Usually @kubectl drain server@ should do the job, but sometimes we need to be more aggressive: 

 <pre> 
 kubectl drain --delete-emptydir-data --ignore-daemonsets server23 
 </pre> 

 h3. Readding a node after draining 

 <pre> 
 kubectl uncordon serverXX 
 </pre> 

 h2. Rook / Ceph Related Operations 

 h3. Inspecting the logs of a specific server 

 <pre> 
 # Get the related pods 
 kubectl -n rook-ceph get pods -l app=rook-ceph-osd-prepare  
 ... 

 # Inspect the logs of a specific pod 
 kubectl -n rook-ceph logs -f rook-ceph-osd-prepare-server23--1-444qx 

 </pre> 

 h3. Triggering server prepare / adding new osds 

 The rook-ceph-operator triggers/watches/creates pods to maintain hosts. To trigger a full "re scan", simply delete that pod: 

 <pre> 
 kubectl -n rook-ceph delete pods -l app=rook-ceph-operator 
 </pre> 

 This will cause all the @rook-ceph-osd-prepare-..@ jobs to be recreated and thus OSDs to be created, if new disks have been added. 

 h3. Removing an OSD 

 * See "Ceph OSD Management":https://rook.io/docs/rook/v1.7/ceph-osd-mgmt.html 

 h2. Infrastructure versions 

 h3. ungleich kubernetes infrastructure v3 

 * rook is now installed via helm via argocd instead of directly via manifests 

 h3. ungleich kubernetes infrastructure v2 

 * Replaced fluxv2 from ungleich k8s v1 with argocd 
 ** argocd can apply helm templates directly without needing to go through Chart releases 
 * We are also using argoflow for build flows 
 * Planned to add "kaniko":https://github.com/GoogleContainerTools/kaniko for image building 

 h3. ungleich kubernetes infrastructure v1 

 We are using the following components: 

 * "Calico as a CNI":https://www.projectcalico.org/ with BGP, IPv6 only, no encapsulation 
 ** Needed for basic networking 
 * "kubernetes-secret-generator":https://github.com/mittwald/kubernetes-secret-generator for creating secrets 
 ** Needed so that secrets are not stored in the git repository, but only in the cluster 
 * "ungleich-certbot":https://hub.docker.com/repository/docker/ungleich/ungleich-certbot 
 ** Needed to get letsencrypt certificates for services 
 * "rook with ceph rbd + cephfs":https://rook.io/ for storage 
 ** rbd for almost everything, *ReadWriteOnce* 
 ** cephfs for smaller things, multi access *ReadWriteMany* 
 ** Needed for providing persistent storage 
 * "flux v2":https://fluxcd.io/ 
 ** Needed to manage resources automatically