Actions
The ungleich kubernetes infrastructure » History » Revision 34
« Previous |
Revision 34/222
(diff)
| Next »
Nico Schottelius, 08/27/2021 09:22 PM
The ungleich kubernetes infrastructure and ungleich kubernetes manual¶
Status¶
This document is pre-production.
This document is to become the ungleich kubernetes infrastructure overview as well as the ungleich kubernetes manual.
k8s clusters¶
Cluster | Purpose | Master(s) |
c0.k8s.ooo | Dev | |
c1.k8s.ooo | Dev p6 | 2a0a-e5c0-2-11-0-62ff-fe0b-1a3d.k8s-1.place6.ungleich.ch |
c2.k8s.ooo | Demo place7 | server47 server53 server54 |
c3.k8s.ooo | Test p7 | Raspberry PIs (inactive of 2021-08-20) |
c4.k8s.ooo | active-dev p7 | server52 server53 server54 |
c5.k8s.ooo | Development Amal | 2a0a-e5c0-2-11-0-62ff-fe0b-1a46.k8s-1.place6.ungleich.ch |
p6.k8s.ooo | production | |
p10.k8s.ooo | production | server63 server65 server83 |
General architecture and components overview¶
- All k8s clusters are IPv6 only
- We use BGP peering to propagate podcidr and serviceCidr networks to our infrastructure
- The main public testing repository is ungleich-k8s
- Private configurations are found in the k8s-config repository
Cluster types¶
Type/Feature | Development | Production |
Min No. nodes | 3 (1 master, 3 worker) | 5 (3 master, 3 worker) |
Recommended minimum | 4 (dedicated master, 3 worker) | 8 (3 master, 5 worker) |
Separation of control plane | optional | recommended |
Persistent storage | required | required |
Number of storage monitors | 3 | 5 |
Operations¶
Installing a new k8s cluster¶
- Decide on the cluster name (usually cX.k8s.ooo), X counting upwards
- Using pXX.k8s.ooo for production clusters of placeXX
- Use cdist to configure the nodes with requirements like crio
- Decide between single or multi node control plane setups (see below)
- Single control plane suitable for development clusters
Typical init procedure:
- Single control plane:
kubeadm init --config bootstrap/XXX/kubeadm.yaml
- Multi control plane (HA):
kubeadm init --config bootstrap/XXX/kubeadm.yaml --upload-certs
Deleting a pod that is hanging in terminating state¶
kubectl delete pod <PODNAME> --grace-period=0 --force --namespace <NAMESPACE>
(from https://stackoverflow.com/questions/35453792/pods-stuck-in-terminating-status)
Infrastructure versions¶
ungleich kubernetes infrastructure v2¶
- Replaced fluxv2 from ungleich k8s v1 with argocd
- argocd can apply helm templates directly without needing to go through Chart releases
- We are also using argoflow for build flows
- Planned to add kaniko for image building
ungleich kubernetes infrastructure v1¶
We are using the following components:
- Calico as a CNI with BGP, IPv6 only, no encapsulation
- Needed for basic networking
- kubernetes-secret-generator for creating secrets
- Needed so that secrets are not stored in the git repository, but only in the cluster
- ungleich-certbot
- Needed to get letsencrypt certificates for services
- rook with ceph rbd + cephfs for storage
- rbd for almost everything, ReadWriteOnce
- cephfs for smaller things, multi access ReadWriteMany
- Needed for providing persistent storage
- flux v2
- Needed to manage resources automatically
Updated by Nico Schottelius over 3 years ago · 34 revisions