The ungleich kubernetes infrastructure¶

• Table of contents
• The ungleich kubernetes infrastructure
  • Status
  • General architecture and components overview
  • ungleich kubernetes infrastructure v1
    • Persistent storage setup
    • Cluster types
  • Open Issues / To be discussed

Status¶

This document is pre-production

General architecture and components overview¶

• All k8s clusters are IPv6 only
• We use BGP peering to propagate podcidr and serviceCidr networks to our infrastructure
• The main / public repository is ungleich-k8s

ungleich kubernetes infrastructure v1¶

We are using the following components:

• Calico as a CNI with BGP, IPv6 only, no encapsulation
• kubernetes-secret-generator for creating secrets
• ungleich-certbot to get certificates
• rook with ceph rbd + cephfs for storage
  • rbd for almost everything, ReadWriteOnce
  • cephfs for smaller things, multi access ReadWriteMany

Persistent storage setup¶

• 3 or 5 monitors

Cluster types¶

Open Issues / To be discussed¶

• Maybe add Autoscaling support?
  • https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/
  • https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler/
• Certainly deploy in-cluster monitoring
  • prometheus-operator CR
  • kube-prometheus complete example, based on prometheus-operator
  • kubernetes dashboard generic cluster overview, basically kubectl for a broswer + graphs
  • kube-prometheus-stack via helm
    • Looks most fitting, testing it in #9468
• Matrix/Notification bot
  • Informing about changes in the cluster