The ungleich kubernetes infrastructure¶

• Table of contents
• The ungleich kubernetes infrastructure
  • Status
  • k8s clusters
  • General architecture and components overview
    • Persistent storage setup
    • Cluster types
  • Operations
    • Installing a new k8s cluster
  • Open Issues / To be discussed

Status¶

This document is pre-production

k8s clusters¶

Cluster	Purpose	Init/Notes	flux
c0.k8s.ooo	Dev	kubeadm init --config k8s/c0/kubeadm.yaml --upload-certs
c1.k8s.ooo	Dev p6	VM based	kubeadm init --config k8s/c1/kubeadm.yaml
c2.k8s.ooo	Demo/Semiprod	kubeadm init --config k8s/c2/kubeadm.yaml --upload-certs	flux bootstrap git --url=ssh://git@code.ungleich.ch/ungleich-intern/k8s-config.git --path=cluster/c2 --network-policy false --cluster-domain c2.k8s.ooo
c3.k8s.ooo	Dev p6	VM based
p5.k8s.ooo	Prod	Place5
p6.k8s.ooo	Prod	Place6
p7.k8s.ooo	Prod	Place7

General architecture and components overview¶

• All k8s clusters are IPv6 only
• We use BGP peering to propagate podcidr and serviceCidr networks to our infrastructure
• The main / public repository is ungleich-k8s

th2. ungleich kubernetes infrastructure v1

We are using the following components:

• Calico as a CNI with BGP, IPv6 only, no encapsulation
  • Needed for basic networking
• kubernetes-secret-generator for creating secrets
  • Needed so that secrets are not stored in the git repository, but only in the cluster
• ungleich-certbot
  • Needed to get letsencrypt certificates for services
• rook with ceph rbd + cephfs for storage
  • rbd for almost everything, ReadWriteOnce
  • cephfs for smaller things, multi access ReadWriteMany
  • Needed for providing persistent storage
• flux v2
  • Needed to manage resources automatically

Persistent storage setup¶

• 3 or 5 monitors

Cluster types¶

Type/Feature	Development	Production
Min No. nodes	3 (1 master, 3 worker)	5 (3 master, 3 worker)
Recommended minimum	4 (dedicated master, 3 worker)	8 (3 master, 5 worker)
Separation of control plane	optional	recommended
Persistent storage	required	required
Number of storage monitors	3	5

Operations¶

Installing a new k8s cluster¶

• Decide on the cluster name (usually cX.k8s.ooo), X counting upwards
• Use cdist to configure the nodes with requirements like crio
• Decide between single or multi node control plane setups (see below)
• Setup DNS delegation and glue records:
  • kube-dns.kube-system.svc.cX AAAA ...
  • kube-dns.kube-system.svc.cX A ...
  • cX NS kube-dns.kube-system.svc.cX

Open Issues / To be discussed¶

• Maybe add Autoscaling support?
  • https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/
  • https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler/
• Certainly deploy in-cluster monitoring
  • prometheus-operator CR
  • kube-prometheus complete example, based on prometheus-operator
  • kubernetes dashboard generic cluster overview, basically kubectl for a broswer + graphs
  • kube-prometheus-stack via helm
    • Looks most fitting, testing it in #9468
• Matrix/Notification bot
  • Informing about changes in the cluster