The ungleich kubernetes infrastructure and ungleich kubernetes manual¶

• Table of contents
• The ungleich kubernetes infrastructure and ungleich kubernetes manual
  • Status
  • k8s clusters
  • General architecture and components overview
    • Cluster types
  • Operations
    • Installing a new k8s cluster
    • Deleting a pod that is hanging in terminating state
  • Infrastructure versions
    • ungleich kubernetes infrastructure v2
    • ungleich kubernetes infrastructure v1

Status¶

This document is pre-production.
This document is to become the ungleich kubernetes infrastructure overview as well as the ungleich kubernetes manual.

k8s clusters¶

Cluster	Purpose	Master(s)
c0.k8s.ooo	Dev
c1.k8s.ooo	Dev p6	2a0a-e5c0-2-11-0-62ff-fe0b-1a3d.k8s-1.place6.ungleich.ch
c2.k8s.ooo	Demo place7	server47 server53 server54
c3.k8s.ooo	Test p7	Raspberry PIs (inactive of 2021-08-20)
c4.k8s.ooo	active-dev p7	server52 server53 server54
c5.k8s.ooo	Development Amal	2a0a-e5c0-2-11-0-62ff-fe0b-1a46.k8s-1.place6.ungleich.ch
p6.k8s.ooo	production
p10.k8s.ooo	production	server63 server65 server83

General architecture and components overview¶

• All k8s clusters are IPv6 only
• We use BGP peering to propagate podcidr and serviceCidr networks to our infrastructure
• The main public testing repository is ungleich-k8s
  • Private configurations are found in the k8s-config repository

Cluster types¶

Type/Feature	Development	Production
Min No. nodes	3 (1 master, 3 worker)	5 (3 master, 3 worker)
Recommended minimum	4 (dedicated master, 3 worker)	8 (3 master, 5 worker)
Separation of control plane	optional	recommended
Persistent storage	required	required
Number of storage monitors	3	5

Operations¶

Installing a new k8s cluster¶

• Decide on the cluster name (usually cX.k8s.ooo), X counting upwards
  • Using pXX.k8s.ooo for production clusters of placeXX
• Use cdist to configure the nodes with requirements like crio
• Decide between single or multi node control plane setups (see below)
  • Single control plane suitable for development clusters

Typical init procedure:

• Single control plane: kubeadm init --config bootstrap/XXX/kubeadm.yaml
• Multi control plane (HA): kubeadm init --config bootstrap/XXX/kubeadm.yaml --upload-certs

Deleting a pod that is hanging in terminating state¶

kubectl delete pod <PODNAME> --grace-period=0 --force --namespace <NAMESPACE>

(from https://stackoverflow.com/questions/35453792/pods-stuck-in-terminating-status)

Infrastructure versions¶

ungleich kubernetes infrastructure v2¶

• Replaced fluxv2 from ungleich k8s v1 with argocd
  • argocd can apply helm templates directly without needing to go through Chart releases
• We are also using argoflow for build flows
• Planned to add kaniko for image building

ungleich kubernetes infrastructure v1¶

We are using the following components:

• Calico as a CNI with BGP, IPv6 only, no encapsulation
  • Needed for basic networking
• kubernetes-secret-generator for creating secrets
  • Needed so that secrets are not stored in the git repository, but only in the cluster
• ungleich-certbot
  • Needed to get letsencrypt certificates for services
• rook with ceph rbd + cephfs for storage
  • rbd for almost everything, ReadWriteOnce
  • cephfs for smaller things, multi access ReadWriteMany
  • Needed for providing persistent storage
• flux v2
  • Needed to manage resources automatically