Project

General

Profile

The ungleich kubernetes infrastructure » History » Version 7

Nico Schottelius, 07/01/2021 04:27 PM

1 1 Nico Schottelius
h1. The ungleich kubernetes infrastructure
2
3 3 Nico Schottelius
{{toc}}
4
5 1 Nico Schottelius
h2. Status
6
7
This document is **pre-production**
8
9
h2. General architecture and components overview
10
11
* All k8s clusters are IPv6 only
12
* We use BGP peering to propagate podcidr and serviceCidr networks to our infrastructure
13 2 Nico Schottelius
* The main / public repository is "ungleich-k8s":https://code.ungleich.ch/ungleich-public/ungleich-k8s
14 1 Nico Schottelius
15
h2. ungleich kubernetes infrastructure v1
16
17
We are using the following components:
18
19 6 Nico Schottelius
* "Calico as a CNI":https://www.projectcalico.org/ with BGP, IPv6 only, no encapsulation
20 1 Nico Schottelius
* "kubernetes-secret-generator":https://github.com/mittwald/kubernetes-secret-generator for creating secrets
21
* "ungleich-certbot":https://hub.docker.com/repository/docker/ungleich/ungleich-certbot to get certificates
22 2 Nico Schottelius
* "rook with ceph rbd + cephfs":https://rook.io/ for storage
23
** rbd for almost everything, *ReadWriteOnce*
24
** cephfs for smaller things, multi access *ReadWriteMany*
25 3 Nico Schottelius
26
h2. Open Issues / To be discussed
27
28
* "Maybe add Autoscaling support?":https://github.com/kubernetes-sigs/metrics-server
29
** https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/
30
** https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler/
31
* Certainly deploy in-cluster monitoring
32
** "prometheus-operator":https://github.com/prometheus-operator/prometheus-operator CR
33
** "kube-prometheus":https://github.com/prometheus-operator/kube-prometheus complete example, based on prometheus-operator
34 4 Nico Schottelius
** "kubernetes dashboard":https://github.com/kubernetes/dashboard generic cluster overview, basically kubectl for a broswer + graphs
35 5 Nico Schottelius
** "kube-prometheus-stack via helm":https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack#kube-prometheus-stack
36
*** Looks most fitting, testing it in #9468
37 7 Nico Schottelius
* Matrix/Notification bot
38
** Informing about changes in the cluster