The ungleich kubernetes infrastructure and ungleich kubernetes manual¶

• Table of contents
• The ungleich kubernetes infrastructure and ungleich kubernetes manual
  • Status
  • k8s clusters
  • General architecture and components overview
  • ungleich kubernetes infrastructure v1
    • Persistent storage setup
    • Cluster types
  • Operations
    • Installing a new k8s cluster
  • Open Issues / To be discussed

Status¶

This document is pre-production. This document is to become the ungleich kubernetes infrastructure overview as well as the ungleich kubernetes manual.

k8s clusters¶

Cluster	Purpose	Notes
c0.k8s.ooo	Dev
c1.k8s.ooo	Dev p6
c2.k8s.ooo	Demo/Semiprod
c3.k8s.ooo	Dev p6
c4.k8s.ooo	active-dev p7
p6.k8s.ooo	planned
p10.k8s.ooo	production

Typical init procedure:

• Single control plane: kubeadm init --config bootstrap/XXX/kubeadm.yaml
• Multi control plane (HA): kubeadm init --config bootstrap/XXX/kubeadm.yaml --upload-certs

General architecture and components overview¶

• All k8s clusters are IPv6 only
• We use BGP peering to propagate podcidr and serviceCidr networks to our infrastructure
• The main / public repository is ungleich-k8s

ungleich kubernetes infrastructure v1¶

We are using the following components:

• Calico as a CNI with BGP, IPv6 only, no encapsulation
  • Needed for basic networking
• kubernetes-secret-generator for creating secrets
  • Needed so that secrets are not stored in the git repository, but only in the cluster
• ungleich-certbot
  • Needed to get letsencrypt certificates for services
• rook with ceph rbd + cephfs for storage
  • rbd for almost everything, ReadWriteOnce
  • cephfs for smaller things, multi access ReadWriteMany
  • Needed for providing persistent storage
• flux v2
  • Needed to manage resources automatically

Persistent storage setup¶

• 3 or 5 monitors

Cluster types¶

Type/Feature	Development	Production
Min No. nodes	3 (1 master, 3 worker)	5 (3 master, 3 worker)
Recommended minimum	4 (dedicated master, 3 worker)	8 (3 master, 5 worker)
Separation of control plane	optional	recommended
Persistent storage	required	required
Number of storage monitors	3	5

Operations¶

Installing a new k8s cluster¶

• Decide on the cluster name (usually cX.k8s.ooo), X counting upwards
• Use cdist to configure the nodes with requirements like crio
• Decide between single or multi node control plane setups (see below)
• Setup DNS delegation and glue records:
  • kube-dns.kube-system.svc.cX AAAA ...
  • kube-dns.kube-system.svc.cX A ...
  • cX NS kube-dns.kube-system.svc.cX

Open Issues / To be discussed¶

• Maybe add Autoscaling support?
  • https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/
  • https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler/
• Certainly deploy in-cluster monitoring
  • prometheus-operator CR
  • kube-prometheus complete example, based on prometheus-operator
  • kubernetes dashboard generic cluster overview, basically kubectl for a broswer + graphs
  • kube-prometheus-stack via helm
    • Looks most fitting, testing it in #9468
• Matrix/Notification bot
  • Informing about changes in the cluster