The ungleich kubernetes infrastructure » History » Revision 199

Revision 198 (Nico Schottelius, 10/08/2023 09:30 PM) → Revision 199/213 (Nico Schottelius, 10/15/2023 04:29 PM)

h1. The ungleich kubernetes infrastructure and ungleich kubernetes manual 


 h2. Status 

 This document is **pre-production**. 
 This document is to become the ungleich kubernetes infrastructure overview as well as the ungleich kubernetes manual. 


 h2. k8s clusters 

 | Cluster              | Purpose/Setup       | Maintainer | Master(s)                       | argo                                                     | v4 http proxy | last verified | 
 |           | Dev                 | -            | UNUSED                          |                                                          |                 |      2021-10-05 | 
 |           | retired             |              | -                               |                                                          |                 |      2022-03-15 | 
 |           | Dev p7 HW           | Nico         | server47 server53 server54      | "argo":       |                 |      2021-10-05 | 
 |           | retired             | -            | -                               |                                                          |                 |      2021-10-05 | 
 |           | Dev2 p7 HW          | Jin-Guk      | server52 server53 server54      |                                                          |                 |               - | 
 |           | retired             |              | -                               |                                                          |                 |      2022-03-15 | 
 |           | Dev p6 VM Jin-Guk | Jin-Guk      |                                 |                                                          |                 |                 | 
 | [[]]       | production          |              | server34 server36 server38      | "argo":       | -               |                 | 
 | [[]] | production          | Nico         | server47 server51 server55      | "argo": |                 |      2022-08-27 | 
 | [[]]       | production          |              | server67 server69 server71      | "argo":       | |      2021-10-05 | 
 | [[]] | production          |              | server134 server135 server136 | "argo": | ?               |      2023-05-17 | 
 | [[]]      | production          |              | server131 server132 server133 | "argo":      | |      2021-10-05 | 
 | [[]]    | development         |              | server107 server108 server109 | "argo":    |                 |                 | 
 | [[]]      | development         |              | server110 server111 server112 | "argo":      | -               |      2022-07-08 | 
 | [[r1r2p15k8sooo|]] | production | Nico | server120 | | | 2022-10-30 | 
 | [[r1r2p15k8sooo|]] | production | Nico | server121 | | | 2022-09-06 | 
 | [[r1r2p10k8sooo|]] | production | Nico | server122 | | | 2022-10-30 | 
 | [[r1r2p10k8sooo|]] | production | Nico | server123 | | | 2022-10-15 | 
 | [[r1r2p5k8sooo|]] | production | Nico | server137 | | | 2022-10-30 | 
 | [[r1r2p5k8sooo|]] | production | Nico | server138 | | | 2022-10-30 | 
 | [[r1r2p6k8sooo|]] | production | Nico | server139 | | | 2022-10-30 | 
 | [[r1r2p6k8sooo|]] | production | Nico | server140 | | | 2022-10-30 | 


 h2. General architecture and components overview 

 * All k8s clusters are IPv6 only 
 * We use BGP peering to propagate podcidr and serviceCidr networks to our infrastructure 
 * The main public testing repository is "ungleich-k8s": 
 ** Private configurations are found in the **k8s-config** repository 

 h3. Cluster types 

 | **Type/Feature**              | **Development**                  | **Production**           | 
 | Min No. nodes                 | 3 (1 master, 3 worker)           | 5 (3 master, 3 worker) | 
 | Recommended minimum           | 4 (dedicated master, 3 worker) | 8 (3 master, 5 worker) | 
 | Separation of control plane | optional                         | recommended              | 
 | Persistent storage            | required                         | required                 | 
 | Number of storage monitors    | 3                                | 5                        | 

 h2. General k8s operations 

 h3. Cheat sheet / external great references 

 * "kubectl cheatsheet": 

 h3. Allowing to schedule work on the control plane / removing node taints 

 * Mostly for single node / test / development clusters 
 * Just remove the master taint as follows 

 kubectl taint nodes --all 
 kubectl taint nodes --all 

 You can check the node taints using @kubectl describe node ...@ 

 h3. Get the cluster admin.conf 

 * On the masters of each cluster you can find the file @/etc/kubernetes/admin.conf@ 
 * To be able to administrate the cluster you can copy the admin.conf to your local machine 
 * Multi cluster debugging can very easy if you name the config ~/cX-admin.conf (see example below) 

 % scp ~/c2-admin.conf 
 % export KUBECONFIG=~/c2-admin.conf     
 % kubectl get nodes 
 NAME         STATUS                       ROLES                    AGE     VERSION 
 server47     Ready                        control-plane,master     82d     v1.22.0 
 server48     Ready                        control-plane,master     82d     v1.22.0 
 server49     Ready                        <none>                   82d     v1.22.0 
 server50     Ready                        <none>                   82d     v1.22.0 
 server59     Ready                        control-plane,master     82d     v1.22.0 
 server60     Ready,SchedulingDisabled     <none>                   82d     v1.22.0 
 server61     Ready                        <none>                   82d     v1.22.0 
 server62     Ready                        <none>                   82d     v1.22.0                

 h3. Installing a new k8s cluster 

 * Decide on the cluster name (usually **), X counting upwards 
 ** Using for production clusters of placeXX 
 * Use cdist to configure the nodes with requirements like crio 
 * Decide between single or multi node control plane setups (see below) 
 ** Single control plane suitable for development clusters 

 Typical init procedure: 

 * Single control plane: @kubeadm init --config bootstrap/XXX/kubeadm.yaml@ 
 * Multi control plane (HA): @kubeadm init --config bootstrap/XXX/kubeadm.yaml --upload-certs@ 

 h3. Deleting a pod that is hanging in terminating state 

 kubectl delete pod <PODNAME> --grace-period=0 --force --namespace <NAMESPACE> 


 h3. Listing nodes of a cluster 

 [15:05] bridge:~% kubectl get nodes 
 NAME         STATUS     ROLES                    AGE     VERSION 
 server22     Ready      <none>                   52d     v1.22.0 
 server23     Ready      <none>                   52d     v1.22.2 
 server24     Ready      <none>                   52d     v1.22.0 
 server25     Ready      <none>                   52d     v1.22.0 
 server26     Ready      <none>                   52d     v1.22.0 
 server27     Ready      <none>                   52d     v1.22.0 
 server63     Ready      control-plane,master     52d     v1.22.0 
 server64     Ready      <none>                   52d     v1.22.0 
 server65     Ready      control-plane,master     52d     v1.22.0 
 server66     Ready      <none>                   52d     v1.22.0 
 server83     Ready      control-plane,master     52d     v1.22.0 
 server84     Ready      <none>                   52d     v1.22.0 
 server85     Ready      <none>                   52d     v1.22.0 
 server86     Ready      <none>                   52d     v1.22.0 

 h3. Removing / draining a node 

 Usually @kubectl drain server@ should do the job, but sometimes we need to be more aggressive: 

 kubectl drain --delete-emptydir-data --ignore-daemonsets serverXX 

 h3. Readding a node after draining 

 kubectl uncordon serverXX 

 h3. (Re-)joining worker nodes after creating the cluster 

 * We need to have an up-to-date token 
 * We use different join commands for the workers and control plane nodes 

 Generating the join command on an existing control plane node: 

 kubeadm token create --print-join-command 

 h3. (Re-)joining control plane nodes after creating the cluster 

 * We generate the token again 
 * We upload the certificates 
 * We need to combine/create the join command for the control plane node 

 Example session: 

 % kubeadm token create --print-join-command 
 kubeadm join --token xmff4i.ABC --discovery-token-ca-cert-hash sha256:longhash  

 % kubeadm init phase upload-certs --upload-certs 
 [upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace 
 [upload-certs] Using certificate key: 

 # Then we use these two outputs on the joining node: 

 kubeadm join --token xmff4i.ABC --discovery-token-ca-cert-hash sha256:longhash --control-plane --certificate-key CERTKEY 

 Commands to be used on a control plane node: 

 kubeadm token create --print-join-command 
 kubeadm init phase upload-certs --upload-certs 

 Commands to be used on the joining node: 

 JOINCOMMAND --control-plane --certificate-key CERTKEY 



 h3. How to fix etcd does not start when rejoining a kubernetes cluster as a control plane 

 If during the above step etcd does not come up, @kubeadm join@ can hang as follows: 

 [control-plane] Creating static Pod manifest for "kube-apiserver"                                                               
 [control-plane] Creating static Pod manifest for "kube-controller-manager"                                                      
 [control-plane] Creating static Pod manifest for "kube-scheduler"                                                               
 [check-etcd] Checking that the etcd cluster is healthy                                                                          
 error execution phase check-etcd: etcd cluster is not healthy: failed to dial endpoint https://[2a0a:e5c0:10:1:225:b3ff:fe20:37 
 8a]:2379 with maintenance client: context deadline exceeded                                                                     
 To see the stack trace of this error execute with --v=5 or higher          

 Then the problem is likely that the etcd server is still a member of the cluster. We first need to remove it from the etcd cluster and then the join works. 

 To fix this we do: 

 * Find a working etcd pod 
 * Find the etcd members / member list 
 * Remove the etcd member that we want to re-join the cluster 

 # Find the etcd pods 
 kubectl -n kube-system get pods -l component=etcd,tier=control-plane 

 # Get the list of etcd servers with the member id  
 kubectl exec -n kube-system -ti ETCDPODNAME -- etcdctl --endpoints '[::1]:2379' --cacert /etc/kubernetes/pki/etcd/ca.crt --cert    /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member list 

 # Remove the member 
 kubectl exec -n kube-system -ti ETCDPODNAME -- etcdctl --endpoints '[::1]:2379' --cacert /etc/kubernetes/pki/etcd/ca.crt --cert    /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member remove MEMBERID 

 Sample session: 

 [10:48] line:~% kubectl -n kube-system get pods -l component=etcd,tier=control-plane 
 NAME              READY     STATUS      RESTARTS       AGE 
 etcd-server63     1/1       Running     0              3m11s 
 etcd-server65     1/1       Running     3              7d2h 
 etcd-server83     1/1       Running     8 (6d ago)     7d2h 
 [10:48] line:~% kubectl exec -n kube-system -ti etcd-server65 -- etcdctl --endpoints '[::1]:2379' --cacert /etc/kubernetes/pki/etcd/ca.crt --cert    /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member list 
 356891cd676df6e4, started, server65, https://[2a0a:e5c0:10:1:225:b3ff:fe20:375c]:2380, https://[2a0a:e5c0:10:1:225:b3ff:fe20:375c]:2379, false 
 371b8a07185dee7e, started, server63, https://[2a0a:e5c0:10:1:225:b3ff:fe20:378a]:2380, https://[2a0a:e5c0:10:1:225:b3ff:fe20:378a]:2379, false 
 5942bc58307f8af9, started, server83, https://[2a0a:e5c0:10:1:3e4a:92ff:fe79:bb98]:2380, https://[2a0a:e5c0:10:1:3e4a:92ff:fe79:bb98]:2379, false 

 [10:48] line:~% kubectl exec -n kube-system -ti etcd-server65 -- etcdctl --endpoints '[::1]:2379' --cacert /etc/kubernetes/pki/etcd/ca.crt --cert    /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member remove 371b8a07185dee7e 
 Member 371b8a07185dee7e removed from cluster e3c0805f592a8f77 



 * We found the solution using 

 h3. Node labels (adding, showing, removing) 

 Listing the labels: 

 kubectl get nodes --show-labels 

 Adding labels: 

 kubectl label nodes LIST-OF-NODES label1=value1  


 For instance: 

 kubectl label nodes router2 router3 hosttype=router  

 Selecting nodes in pods: 

 apiVersion: v1 
 kind: Pod 
     hosttype: router 

 Removing labels by adding a minus at the end of the label name: 

 kubectl label node <nodename> <labelname>- 

 For instance: 

 kubectl label nodes router2 router3 hosttype-  



 h3. Listing all pods on a node 

 kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=serverXX 

 Found on 

 h3. Hardware Maintenance using ungleich-hardware 

 Use the following manifest and replace the HOST with the actual host: 

 apiVersion: v1 
 kind: Pod 
   name: ungleich-hardware-HOST 
   - name: ungleich-hardware 
     image: ungleich/ungleich-hardware:0.0.5 
     - sleep 
     - "1000000" 
       - mountPath: /dev 
         name: dev 
       privileged: true 
   nodeSelector: "HOST" 

     - name: dev 
         path: /dev 

 Also see: [[The_ungleich_hardware_maintenance_guide]] 

 h3. Triggering a cronjob / creating a job from a cronjob 

 To test a cronjob, we can create a job from a cronjob: 

 kubectl create job --from=cronjob/volume2-daily-backup volume2-manual 

 This creates a job volume2-manual based on the cronjob    volume2-daily 

 h3. su-ing into a user that has nologin shell set 

 Many times users are having nologin as their shell inside the container. To be able to execute maintenance commands within the 
 container, we can use @su -s /bin/sh@ like this: 

 su -s /bin/sh -c '/path/to/your/script' testuser 

 Found on 

 h3. How to print a secret value 

 Assuming you want the "password" item from a secret, use: 

 kubectl get secret SECRETNAME -o jsonpath="{.data.password}" | base64 -d; echo ""  


 h3. How to upgrade a kubernetes cluster 

 h4. General 

 * Should be done every X months to stay up-to-date 
 ** X probably something like 3-6 
 * kubeadm based clusters 
 * Needs specific kubeadm versions for upgrade 
 * Follow instructions on 
 * Finding releases: 

 h4. Getting a specific kubeadm or kubelet version 



 curl -L --remote-name-all${RELEASE}/bin/linux/${ARCH}/{kubeadm,kubelet} 
 chmod u+x kubeadm kubelet 

 h4. Steps 

 * kubeadm upgrade plan 
 ** On one control plane node 
 * kubeadm upgrade apply vXX.YY.ZZ 
 ** On one control plane node 
 * kubeadm upgrade node 
 ** On all other control plane nodes 
 ** On all worker nodes afterwards 

 Repeat for all control planes nodes. The upgrade kubelet on all other nodes via package manager. 


 h4. Upgrading to 1.22.17 

 * Need to create a kubeadm config map 
 ** f.i. using the following 
 ** @/usr/local/bin/kubeadm-v1.22.17     upgrade --config kubeadm.yaml --ignore-preflight-errors=CoreDNSUnsupportedPlugins,CoreDNSMigration apply -y v1.22.17@ 
 * Done for p6 on 2023-10-04 


 h4. Upgrading to 1.23.17 

 * No special notes 
 * Done for p6 on 2023-10-04 

 h4. Upgrading to 1.24.17 

 * No special notes 
 * Done for p6 on 2023-10-04 

 h4. Upgrading to 1.25.14 

 * No special notes 
 * Done for p6 on 2023-10-04 

 h4. Upgrading to 1.26.9 

 * No special notes 
 * Done for p6 on 2023-10-04 


 h4. Upgrading to 1.27 

 * kubelet will not start anymore 
 * reason: @"command failed" err="failed to parse kubelet flag: unknown flag: --container-runtime"@ 
 * /var/lib/kubelet/kubeadm-flags.env contains that parameter 
 * remove it, start kubelet 


 h4. Upgrading to 1.28 



 h4. Upgrade to crio 1.27: missing crun 

 Error message 

 level=fatal msg="validating runtime config: runtime validation: \"crun\" not found in $PATH: exec: \"crun\": executable file not found in $PATH" 


 apk add crun 


 h2. Reference CNI 

 * Mainly "stupid", but effective plugins 
 * Main documentation on 
 * Plugins 
 ** bridge 
 *** Can create the bridge on the host 
 *** But seems not to be able to add host interfaces to it as well 
 *** Has support for vlan tags 
 ** vlan 
 *** creates vlan tagged sub interface on the host 
 *** "It's a 1:1 mapping (i.e. no bridge in between)": 
 ** host-device 
 *** moves the interface from the host into the container 
 *** very easy for physical connections to containers 
 ** ipvlan 
 *** "virtualisation" of a host device 
 *** routing based on IP 
 *** Same MAC for everyone 
 *** Cannot reach the master interface 
 ** maclvan 
 *** With mac addresses 
 *** Supports various modes (to be checked) 
 ** ptp ("point to point") 
 *** Creates a host device and connects it to the container 
 ** win* 
 *** Windows implementations 

 h2. Calico CNI 

 h3. Calico Installation 

 * We install "calico using helm": 
 * This has the following advantages: 
 ** Easy to upgrade 
 ** Does not require os to configure IPv6/dual stack settings as the tigera operator figures out things on its own 

 Usually plain calico can be installed directly using: 


 helm repo add projectcalico 
 helm repo update 
 helm upgrade --install --namespace tigera calico projectcalico/tigera-operator --version $VERSION --create-namespace 

 * Check the tags on for the latest release 

 h3. Installing calicoctl 

 * General installation instructions, including binary download: 

 To be able to manage and configure calico, we need to  
 "install calicoctl (we choose the version as a pod)": 

 kubectl apply -f 

 Or version specific: 

 kubectl apply -f 

 # For 3.22 
 kubectl apply -f 

 And making it easier accessible by alias: 

 alias calicoctl="kubectl exec -i -n kube-system calicoctl -- /calicoctl" 

 h3. Calico configuration 

 By default our k8s clusters "BGP peer": 
 with an upstream router to propagate podcidr and servicecidr. 

 Default settings in our infrastructure: 

 * We use a full-mesh using the @nodeToNodeMeshEnabled: true@ option 
 * We keep the original next hop so that *only* the server with the pod is announcing it (instead of ecmp) 
 * We use private ASNs for k8s clusters 
 * We do *not* use any overlay 

 After installing calico and calicoctl the last step of the installation is usually: 

 calicoctl create -f - < calico-bgp.yaml 

 A sample BGP configuration: 

 kind: BGPConfiguration 
   name: default 
   logSeverityScreen: Info 
   nodeToNodeMeshEnabled: true 
   asNumber: 65534 
   - cidr: 2a0a:e5c0:10:3::/108 
   - cidr: 2a0a:e5c0:10:3::/108 
 kind: BGPPeer 
   name: router1-place10 
   peerIP: 2a0a:e5c0:10:1::50 
   asNumber: 213081 
   keepOriginalNextHop: true 

 h2. Cilium CNI (experimental) 

 h3. Status 


 h3. Latest error 

 It seems cilium does not run on IPv6 only hosts: 

 level=info msg="Validating configured node address ranges" subsys=daemon 
 level=fatal msg="postinit failed" error="external IPv4 node address could not be derived, please configure via --ipv4-node" subsys=daemon 
 level=info msg="Starting IP identity watcher" subsys=ipcache 

 It crashes after that log entry 

 h3. BGP configuration 

 * The cilium-operator will not start without a correct configmap being present beforehand (see error message below) 
 * Creating the bgp config beforehand as a configmap is thus required. 

 The error one gets without the configmap present: 

 Pods are hanging with: 

 cilium-bpqm6                         0/1       Init:0/4              0               9s 
 cilium-operator-5947d94f7f-5bmh2     0/1       ContainerCreating     0               9s 

 The error message in the cilium-*perator is: 

   Type       Reason         Age                  From                 Message 
   ----       ------         ----                 ----                 ------- 
   Normal     Scheduled      80s                  default-scheduler    Successfully assigned kube-system/cilium-operator-5947d94f7f-lqcsp to server56 
   Warning    FailedMount    16s (x8 over 80s)    kubelet              MountVolume.SetUp failed for volume "bgp-config-path" : configmap "bgp-config" not found 

 A correct bgp config looks like this: 

 apiVersion: v1 
 kind: ConfigMap 
   name: bgp-config 
   namespace: kube-system 
   config.yaml: | 
       - peer-address: 2a0a:e5c0::46 
         peer-asn: 209898 
         my-asn: 65533 
       - peer-address: 2a0a:e5c0::47 
         peer-asn: 209898 
         my-asn: 65533 
       - name: default 
         protocol: bgp 
           - 2a0a:e5c0:0:14::/64 

 h3. Installation 

 Adding the repo 

 helm repo add cilium 
 helm repo update 

 Installing + configuring cilium 


 helm upgrade --install cilium cilium/cilium --version $version \ 
   --namespace kube-system \ 
   --set ipv4.enabled=false \ 
   --set ipv6.enabled=true \ 
   --set enableIPv6Masquerade=false \ 
   --set bgpControlPlane.enabled=true  

 #    --set ipam.operator.clusterPoolIPv6PodCIDRList=$ipv6pool 

 # Old style bgp? 
 #     --set bgp.enabled=true --set bgp.announce.podCIDR=true \ 

 # Show possible configuration options 
 helm show values cilium/cilium 


 Using a /64 for ipam.operator.clusterPoolIPv6PodCIDRList fails with: 

 level=fatal msg="Unable to init cluster-pool allocator" error="unable to initialize IPv6 allocator New CIDR set failed; the node CIDR size is too big" subsys=cilium-operator-generic 

 See also 

 Seems a /112 is actually working. 

 h3. Kernel modules 

 Cilium requires the following modules to be loaded on the host (not loaded by default): 

 modprobe    ip6table_raw 
 modprobe    ip6table_filter 

 h3. Interesting helm flags 

 * autoDirectNodeRoutes 
 * bgpControlPlane.enabled = true 

 h3. SEE ALSO 


 h2. Multus 

 * Installing a deployment w/ CRDs 


 kubectl apply -f${VERSION}/deployments/multus-daemonset-crio.yml 

 h2. ArgoCD 


 h3. Argocd Installation 

 * See 

 As there is no configuration management present yet, argocd is installed using 

 kubectl create namespace argocd 

 # OR: latest stable 
 kubectl apply -n argocd -f 

 # OR Specific Version 
 kubectl apply -n argocd -f 



 h3. Get the argocd credentials 

 kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo "" 

 h3. Accessing argocd 

 In regular IPv6 clusters: 

 * Navigate to https://argocd-server.argocd.CLUSTERDOMAIN 

 In legacy IPv4 clusters 

 kubectl --namespace argocd port-forward svc/argocd-server 8080:80 

 * Navigate to https://localhost:8080 

 h3. Using the argocd webhook to trigger changes 

 * To trigger changes post json 

 h3. Deploying an application 

 * Applications are deployed via git towards gitea ( and then pulled by argo 
 * Always include the *redmine-url* pointing to the (customer) ticket 
 ** Also add the support-url if it exists 

 Application sample 

 kind: Application 
   name: gitea-CUSTOMER 
   namespace: argocd 
     namespace: default 
     server: 'https://kubernetes.default.svc' 
     path: apps/prod/gitea 
     repoURL: '' 
     targetRevision: HEAD 
         - name: 
           value: rook-ceph-block-hdd 
         - name: 
           value: 200Gi 
         - name: storage.db.storageClass 
           value: rook-ceph-block-ssd 
         - name: storage.db.size 
           value: 10Gi 
         - name: storage.letsencrypt.storageClass 
           value: rook-ceph-block-hdd 
         - name: storage.letsencrypt.size 
           value: 50Mi 
         - name: letsencryptStaging 
           value: 'no' 
         - name: fqdn 
           value: '' 
   project: default 
       prune: true 
       selfHeal: true 
     - name: 'redmine-url' 
       value: '' 
     - name: 'support-url' 
       value: '' 

 h2. Helm related operations and conventions 

 We use helm charts extensively. 

 * In production, they are managed via argocd 
 * In development, helm chart can de developed and deployed manually using the helm utility. 

 h3. Installing a helm chart 

 One can use the usual pattern of 

 helm install <releasename> <chartdirectory> 

 However often you want to reinstall/update when testing helm charts. The following pattern is "better", because it allows you to reinstall, if it is already installed: 

 helm upgrade --install <releasename> <chartdirectory> 

 h3. Naming services and deployments in helm charts [Application labels] 

 * We always have {{ .Release.Name }} to identify the current "instance" 
 * Deployments: 
 ** use @app: <what it is>@, f.i. @app: nginx@, @app: postgres@, ... 
 * See more about standard labels on 

 h3. Show all versions of a helm chart 

 helm search repo -l repo/chart 

 For example: 

 % helm search repo -l projectcalico/tigera-operator  
 NAME                          	 CHART VERSION 	 APP VERSION 	 DESCRIPTION                             
 projectcalico/tigera-operator 	 v3.23.3       	 v3.23.3     	 Installs the Tigera operator for Calico 
 projectcalico/tigera-operator 	 v3.23.2       	 v3.23.2     	 Installs the Tigera operator for Calico 

 h3. Show possible values of a chart 

 helm show values <repo/chart> 


 helm show values ingress-nginx/ingress-nginx 

 h3. Download a chart 

 For instance for checking it out locally. Use: 

 helm pull <repo/chart> 


 h2. Rook + Ceph 

 h3. Installation 

 * Usually directly via argocd 

 h3. Executing ceph commands 

 Using the ceph-tools pod as follows: 

 kubectl exec -n rook-ceph -ti $(kubectl -n rook-ceph get pods -l app=rook-ceph-tools -o jsonpath='{.items[*]}') -- ceph -s 

 h3. Inspecting the logs of a specific server 

 # Get the related pods 
 kubectl -n rook-ceph get pods -l app=rook-ceph-osd-prepare  

 # Inspect the logs of a specific pod 
 kubectl -n rook-ceph logs -f rook-ceph-osd-prepare-server23--1-444qx 


 h3. Inspecting the logs of the rook-ceph-operator 

 kubectl -n rook-ceph logs -f -l app=rook-ceph-operator 

 h3. Restarting the rook operator 

 kubectl -n rook-ceph delete pods    -l app=rook-ceph-operator 

 h3. Triggering server prepare / adding new osds 

 The rook-ceph-operator triggers/watches/creates pods to maintain hosts. To trigger a full "re scan", simply delete that pod: 

 kubectl -n rook-ceph delete pods -l app=rook-ceph-operator 

 This will cause all the @rook-ceph-osd-prepare-..@ jobs to be recreated and thus OSDs to be created, if new disks have been added. 

 h3. Removing an OSD 

 * See "Ceph OSD Management": 
 * More specifically: 
 * Then delete the related deployment 

 Set osd id in the osd-purge.yaml and apply it. OSD should be down before. 

 apiVersion: batch/v1 
 kind: Job 
   name: rook-ceph-purge-osd 
   namespace: rook-ceph # namespace:cluster 
     app: rook-ceph-purge-osd 
         app: rook-ceph-purge-osd 
       serviceAccountName: rook-ceph-purge-osd 
         - name: osd-removal 
           image: rook/ceph:master 
           # TODO: Insert the OSD ID in the last parameter that is to be removed 
           # The OSD IDs are a comma-separated list. For example: "0" or "0,2". 
           # If you want to preserve the OSD PVCs, set `--preserve-pvc true`. 
           # A --force-osd-removal option is available if the OSD should be destroyed even though the 
           # removal could lead to data loss. 
             - "ceph" 
             - "osd" 
             - "remove" 
             - "--preserve-pvc" 
             - "false" 
             - "--force-osd-removal" 
             - "false" 
             - "--osd-ids" 
             - "SETTHEOSDIDHERE" 
             - name: POD_NAMESPACE 
                   fieldPath: metadata.namespace 
             - name: ROOK_MON_ENDPOINTS 
                   key: data 
                   name: rook-ceph-mon-endpoints 
             - name: ROOK_CEPH_USERNAME 
                   key: ceph-username 
                   name: rook-ceph-mon 
             - name: ROOK_CEPH_SECRET 
                   key: ceph-secret 
                   name: rook-ceph-mon 
             - name: ROOK_CONFIG_DIR 
               value: /var/lib/rook 
             - name: ROOK_CEPH_CONFIG_OVERRIDE 
               value: /etc/rook/config/override.conf 
             - name: ROOK_FSID 
                   key: fsid 
                   name: rook-ceph-mon 
             - name: ROOK_LOG_LEVEL 
               value: DEBUG 
             - mountPath: /etc/ceph 
               name: ceph-conf-emptydir 
             - mountPath: /var/lib/rook 
               name: rook-config 
         - emptyDir: {} 
           name: ceph-conf-emptydir 
         - emptyDir: {} 
           name: rook-config 
       restartPolicy: Never 


 Deleting the deployment: 

 [18:05] bridge:~% kubectl -n rook-ceph delete deployment rook-ceph-osd-6 
 deployment.apps "rook-ceph-osd-6" deleted 

 h3. Placement of mons/osds/etc. 



 h2. Ingress + Cert Manager 

 * We deploy "nginx-ingress": to get an ingress 
 * we deploy "cert-manager": to handle certificates 
 * We independently deploy @ClusterIssuer@ to allow the cert-manager app to deploy and the issuer to be created once the CRDs from cert manager are in place 

 h3. IPv4 reachability  

 The ingress is by default IPv6 only. To make it reachable from the IPv4 world, get its IPv6 address and configure a NAT64 mapping in Jool. 


 h4. Get the ingress IPv6 address 

 Use @kubectl -n ingress-nginx get svc ingress-nginx-controller -o jsonpath='{.spec.clusterIP}'; echo ''@ 


 kubectl -n ingress-nginx get svc ingress-nginx-controller -o jsonpath='{.spec.clusterIP}'; echo '' 

 h4. Add NAT64 mapping 

 * Update the __dcl_jool_siit cdist type 
 * Record the two IPs (IPv6 and IPv4) 
 * Configure all routers 

 h4. Add DNS record 

 To use the ingress capable as a CNAME destination, create an "ingress" DNS record, such as: 

 ; k8s ingress for dev 
 dev-ingress                   AAAA 2a0a:e5c0:10:1b::ce11 
 dev-ingress                   A 


 h4. Add supporting wildcard DNS 

 If you plan to add various sites under a specific domain, we can add a wildcard DNS entry, such as * 

 *.k8s-dev           CNAME 

 h2. Harbor 

 * We user "Harbor": as an image registry for our own images. Internal app reference: apps/prod/harbor. 
 * The admin password is in the password store, it is Harbor12345 by default 
 * At the moment harbor only authenticates against the internal ldap tree 

 h3. LDAP configuration 

 * The url needs to be ldaps://... 
 * uid = uid 
 * rest standard 

 h2. Monitoring / Prometheus 

 * Via "kube-prometheus": 

 Access via ... 

 * http://prometheus-k8s.monitoring.svc:9090 
 * http://grafana.monitoring.svc:3000 
 * http://alertmanager.monitoring.svc:9093 

 h3. Prometheus Options 

 * "helm/kube-prometheus-stack": 
 ** Includes dashboards and co. 
 * "manifest based kube-prometheus": 
 ** Includes dashboards and co. 
 * "Prometheus Operator (mainly CRD manifest": 

 h3. Grafana default password 

 * If not changed: @prom-operator@ 

 h2. Nextcloud 

 h3. How to get the nextcloud credentials  

 * The initial username is set to "nextcloud" 
 * The password is autogenerated and saved in a kubernetes secret 

 kubectl get secret RELEASENAME-nextcloud -o jsonpath="{.data.PASSWORD}" | base64 -d; echo ""  

 h3. How to fix "Access through untrusted domain" 

 * Nextcloud stores the initial domain configuration 
 * If the FQDN is changed, it will show the error message "Access through untrusted domain" 
 * To fix, edit /var/www/html/config/config.php and correct the domain 
 * Then delete the pods 

 h3. Running occ commands inside the nextcloud container 

 * Find the pod in the right namespace 


 su www-data -s /bin/sh -c ./occ 

 * -s /bin/sh is needed as the default shell is set to /bin/false 

 h4. Rescanning files 

 * If files have been added without nextcloud's knowledge 

 su www-data -s /bin/sh -c "./occ files:scan --all" 

 h2. Infrastructure versions 

 h3. ungleich kubernetes infrastructure v5 (2021-10) 

 Clusters are configured / setup in this order: 

 * Bootstrap via kubeadm 
 * "Networking via calico + BGP (non ECMP) using helm": 
 * "ArgoCD for CD": 
 ** "rook for storage via argocd": 
 ** haproxy for in IPv6-cluster-IPv4-to-IPv6 proxy via argocd 
 ** "kubernetes-secret-generator for in cluster secrets": 
 ** "ungleich-certbot managing certs and nginx": 

 h3. ungleich kubernetes infrastructure v4 (2021-09) 

 * rook is configured via manifests instead of using the rook-ceph-cluster helm chart 
 * The rook operator is still being installed via helm 

 h3. ungleich kubernetes infrastructure v3 (2021-07) 

 * rook is now installed via helm via argocd instead of directly via manifests 

 h3. ungleich kubernetes infrastructure v2 (2021-05) 

 * Replaced fluxv2 from ungleich k8s v1 with argocd 
 ** argocd can apply helm templates directly without needing to go through Chart releases 
 * We are also using argoflow for build flows 
 * Planned to add "kaniko": for image building 

 h3. ungleich kubernetes infrastructure v1 (2021-01) 

 We are using the following components: 

 * "Calico as a CNI": with BGP, IPv6 only, no encapsulation 
 ** Needed for basic networking 
 * "kubernetes-secret-generator": for creating secrets 
 ** Needed so that secrets are not stored in the git repository, but only in the cluster 
 * "ungleich-certbot": 
 ** Needed to get letsencrypt certificates for services 
 * "rook with ceph rbd + cephfs": for storage 
 ** rbd for almost everything, *ReadWriteOnce* 
 ** cephfs for smaller things, multi access *ReadWriteMany* 
 ** Needed for providing persistent storage 
 * "flux v2": 
 ** Needed to manage resources automatically