The ungleich kubernetes infrastructure and ungleich kubernetes manual¶

• Table of contents
• The ungleich kubernetes infrastructure and ungleich kubernetes manual
  • Status
  • k8s clusters
  • General architecture and components overview
    • Cluster types
  • Operations
    • Get the argocd credentials
    • Installing a new k8s cluster
    • Deleting a pod that is hanging in terminating state
    • Removing / draining a node
  • Infrastructure versions
    • ungleich kubernetes infrastructure v3
    • ungleich kubernetes infrastructure v2
    • ungleich kubernetes infrastructure v1

Status¶

This document is pre-production.
This document is to become the ungleich kubernetes infrastructure overview as well as the ungleich kubernetes manual.

k8s clusters¶

Cluster	Purpose/Setup	Maintainer	Master(s)	last verified
c0.k8s.ooo	Dev	-	UNUSED	2021-10-05
c1.k8s.ooo	Dev p6 VM	Nico	2a0a-e5c0-2-11-0-62ff-fe0b-1a3d.k8s-1.place6.ungleich.ch	2021-10-05
c2.k8s.ooo	Dev p7 HW	Nico	server47 server53 server54	2021-10-05
c3.k8s.ooo	Test p7 PI	-	UNUSED	2021-10-05
c4.k8s.ooo	Dev2 p7 HW	Fran/Jin-Guk	server52 server53 server54	-
c5.k8s.ooo	Dev p6 VM Amal	Nico/Amal	2a0a-e5c0-2-11-0-62ff-fe0b-1a46.k8s-1.place6.ungleich.ch
p6.k8s.ooo	production		server67 server69 server71	2021-10-05
p10.k8s.ooo	production		server63 server65 server83	2021-10-05

General architecture and components overview¶

• All k8s clusters are IPv6 only
• We use BGP peering to propagate podcidr and serviceCidr networks to our infrastructure
• The main public testing repository is ungleich-k8s
  • Private configurations are found in the k8s-config repository

Cluster types¶

Type/Feature	Development	Production
Min No. nodes	3 (1 master, 3 worker)	5 (3 master, 3 worker)
Recommended minimum	4 (dedicated master, 3 worker)	8 (3 master, 5 worker)
Separation of control plane	optional	recommended
Persistent storage	required	required
Number of storage monitors	3	5

Operations¶

Get the argocd credentials¶

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo "" 

Installing a new k8s cluster¶

• Decide on the cluster name (usually cX.k8s.ooo), X counting upwards
  • Using pXX.k8s.ooo for production clusters of placeXX
• Use cdist to configure the nodes with requirements like crio
• Decide between single or multi node control plane setups (see below)
  • Single control plane suitable for development clusters

Typical init procedure:

• Single control plane: kubeadm init --config bootstrap/XXX/kubeadm.yaml
• Multi control plane (HA): kubeadm init --config bootstrap/XXX/kubeadm.yaml --upload-certs

Deleting a pod that is hanging in terminating state¶

kubectl delete pod <PODNAME> --grace-period=0 --force --namespace <NAMESPACE>

(from https://stackoverflow.com/questions/35453792/pods-stuck-in-terminating-status)

Removing / draining a node¶

Usually kubectl drain server should do the job, but sometimes we need to be more aggressive:

kubectl drain --delete-emptydir-data --ignore-daemonsets server23

Infrastructure versions¶

ungleich kubernetes infrastructure v3¶

• rook is now installed via helm via argocd instead of directly via manifests

ungleich kubernetes infrastructure v2¶

• Replaced fluxv2 from ungleich k8s v1 with argocd
  • argocd can apply helm templates directly without needing to go through Chart releases
• We are also using argoflow for build flows
• Planned to add kaniko for image building

ungleich kubernetes infrastructure v1¶

We are using the following components:

• Calico as a CNI with BGP, IPv6 only, no encapsulation
  • Needed for basic networking
• kubernetes-secret-generator for creating secrets
  • Needed so that secrets are not stored in the git repository, but only in the cluster
• ungleich-certbot
  • Needed to get letsencrypt certificates for services
• rook with ceph rbd + cephfs for storage
  • rbd for almost everything, ReadWriteOnce
  • cephfs for smaller things, multi access ReadWriteMany
  • Needed for providing persistent storage
• flux v2
  • Needed to manage resources automatically