The ungleich kubernetes infrastructure » History » Revision 20

« Previous | Revision 20/213 (diff) | Next »
Nico Schottelius, 07/26/2021 09:58 PM

The ungleich kubernetes infrastructure


This document is pre-production

k8s clusters

Cluster Purpose Init/Notes flux Dev kubeadm init --config k8s/c0/kubeadm.yaml --upload-certs Dev p6 VM based kubeadm init --config k8s/c1/kubeadm.yaml Demo/Semiprod kubeadm init --config k8s/c2/kubeadm.yaml --upload-certs flux bootstrap git --url=ssh:// --path=cluster/c2 --network-policy false --cluster-domain Dev p6 VM based Dev p7 Maintained by Fran Prod Place5 Prod Place6 Prod Place7

General architecture and components overview

  • All k8s clusters are IPv6 only
  • We use BGP peering to propagate podcidr and serviceCidr networks to our infrastructure
  • The main / public repository is ungleich-k8s

th2. ungleich kubernetes infrastructure v1

We are using the following components:

  • Calico as a CNI with BGP, IPv6 only, no encapsulation
    • Needed for basic networking
  • kubernetes-secret-generator for creating secrets
    • Needed so that secrets are not stored in the git repository, but only in the cluster
  • ungleich-certbot
    • Needed to get letsencrypt certificates for services
  • rook with ceph rbd + cephfs for storage
    • rbd for almost everything, ReadWriteOnce
    • cephfs for smaller things, multi access ReadWriteMany
    • Needed for providing persistent storage
  • flux v2
    • Needed to manage resources automatically

Persistent storage setup

  • 3 or 5 monitors

Cluster types

Type/Feature Development Production
Min No. nodes 3 (1 master, 3 worker) 5 (3 master, 3 worker)
Recommended minimum 4 (dedicated master, 3 worker) 8 (3 master, 5 worker)
Separation of control plane optional recommended
Persistent storage required required
Number of storage monitors 3 5


Installing a new k8s cluster

  • Decide on the cluster name (usually, X counting upwards
  • Use cdist to configure the nodes with requirements like crio
  • Decide between single or multi node control plane setups (see below)
  • Setup DNS delegation and glue records:
    • kube-dns.kube-system.svc.cX AAAA ...
    • kube-dns.kube-system.svc.cX A ...
    • cX NS kube-dns.kube-system.svc.cX

Open Issues / To be discussed

Updated by Nico Schottelius almost 3 years ago · 20 revisions