Project

General

Profile

Actions
Copy link

Task #8447

closed

Deploy POC IPv6 cluster on DCL (v202009)

Added by Nico Schottelius about 4 years ago. Updated 11 months ago.

Status:
Rejected
Priority:
Normal
Assignee:
Nico Schottelius
Target version:
-
Start date:
09/14/2020
Due date:
% Done:

0%

Estimated time:
PM Check date:

Description

Setup

  • Create 3 Alpine nodes upgraded to edge, set hostnames
    • Resize to at least 2 cores per node
  • Reserve networks
    • 2a0a:e5c0:2:12::/64 = node1
    • 2a0a:e5c0:2:13::/64 = services
    • 2a0a:e5c0:2:14::/64 = node3
  • Configure routers to accept BGP session (done: in cdist)
  • Deploy kubernetes on first node
  • Deploy kube-router: fail, not IPv6 ready
  • Deploy calico: fail
  • Deploy cilium: testing
  • Create BGP peering
  • Verify BGP peering
  • Setup access to CEPH for persistent storage

OS commands


echo node2 > /etc/hostname 

cat > /etc/resolv.conf << EOF
nameserver 2a0a:e5c0:2:12:0:f0ff:fea9:c451
nameserver 2a0a:e5c0:2:12:0:f0ff:fea9:c45d
search k8s.ungleich.ch
EOF
chattr +i /etc/resolv.conf

cat > /etc/apk/repositories << EOF
https://mirror.ungleich.ch/mirror/packages/alpine/edge/main
https://mirror.ungleich.ch/mirror/packages/alpine/edge/community
https://mirror.ungleich.ch/mirror/packages/alpine/edge/testing
EOF

apk upgrade
apk add kubeadm kubelet kubectl docker
rc-update add kubelet default
rc-update add docker default

echo 'net.ipv6.conf.default.forwarding=1' > /etc/sysctl.d/k8s.conf

kubeadm.conf:

localAPIEndpoint:
  advertiseAddress: 2a0a:e5c0:2:2:0:84ff:fe41:f263
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
networking:
  serviceSubnet: 2a0a:e5c0:2:13::/110
  podSubnet: 2a0a:e5c0:2:12::/64

Init cluster:

kubeadm init --config kubeadm.conf 

useradd -m k8s -s /bin/bash 
mkdir ~k8s/.kube
cp /etc/kubernetes/admin.conf ~k8s/.kube/config
chown -R k8s ~k8s

Take aways

  • docker sets ipv4 forwarding, but not ipv6 (needs manual sysctl entry)
  • Reachability by name (node1) w/o fqdn seems to be important

Current results

  • kube-router does not work out-of-the-box "too many colons"
  • calico does not work out of the box "calico-kube-controllers stays in pending" / no network provided
Actions
Copy link
 #1

Updated by Nico Schottelius about 4 years ago

  • Project changed from 45 to Open Infrastructure
  • Description updated (diff)
Actions
Copy link
 #2

Updated by Nico Schottelius about 4 years ago

  • Description updated (diff)
Actions
Copy link
 #3

Updated by Nico Schottelius about 4 years ago

  • Description updated (diff)

try1:

node1:~# kubeadm init --config kubeadm.conf 
W0914 19:04:31.398219    3022 kubelet.go:200] cannot automatically set CgroupDriver when starting the Kubelet: cannot execute 'docker info -f {{.CgroupDriver}}': executable file not found in $PATH
W0914 19:04:31.439357    3022 version.go:102] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get "https://dl.k8s.io/release/stable-1.txt": dial tcp 34.107.204.206:443: connect: network is unreachable
W0914 19:04:31.439492    3022 version.go:103] falling back to the local client version: v1.19.1
W0914 19:04:31.439653    3022 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[init] Using Kubernetes version: v1.19.1
[preflight] Running pre-flight checks
[preflight] WARNING: Couldn't create the interface used for talking to the container runtime: docker is required for container runtime: exec: "docker": executable file not found in $PATH
    [WARNING Hostname]: hostname "node1" could not be reached
    [WARNING Hostname]: hostname "node1": lookup node1 on [2a0a:e5c0:2:a::a]:53: no such host
    [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'rc-update add kubelet default'
error execution phase preflight: [preflight] Some fatal errors occurred:
    [ERROR NumCPU]: the number of available CPUs 1 is less than the required 2
    [ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist
    [ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1
    [ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-ip6tables]: /proc/sys/net/bridge/bridge-nf-call-ip6tables does not exist
    [ERROR FileContent--proc-sys-net-ipv6-conf-default-forwarding]: /proc/sys/net/ipv6/conf/default/forwarding contents are not set to 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher
node1:~#
Actions
Copy link
 #4

Updated by Nico Schottelius about 4 years ago

  • Description updated (diff)
Actions
Copy link
 #5

Updated by Nico Schottelius about 4 years ago

  • Description updated (diff)

try 3:

node1:~# echo "search k8s.ungleich.ch" >> /etc/resolv.conf 
node1:~# kubeadm init --config kubeadm.conf 
W0914 19:08:05.185304    2958 kubelet.go:200] cannot automatically set CgroupDriver when starting the Kubelet: cannot execute 'docker info -f {{.CgroupDriver}}': executable file not found in $PATH
W0914 19:08:05.186504    2958 version.go:102] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get "https://dl.k8s.io/release/stable-1.txt": dial tcp 34.107.204.206:443: connect: network is unreachable
W0914 19:08:05.186520    2958 version.go:103] falling back to the local client version: v1.19.1
W0914 19:08:05.186605    2958 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[init] Using Kubernetes version: v1.19.1
[preflight] Running pre-flight checks
[preflight] WARNING: Couldn't create the interface used for talking to the container runtime: docker is required for container runtime: exec: "docker": executable file not found in $PATH
        [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'rc-update add kubelet default'
error execution phase preflight: [preflight] Some fatal errors occurred:
        [ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist
        [ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1
        [ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-ip6tables]: /proc/sys/net/bridge/bridge-nf-call-ip6tables does not exist
        [ERROR FileContent--proc-sys-net-ipv6-conf-default-forwarding]: /proc/sys/net/ipv6/conf/default/forwarding contents are not set to 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher
node1:~#

try5:

node1:~# kubeadm init --config kubeadm.conf 
W0914 19:16:53.747589    3306 version.go:102] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get "https://dl.k8s.io/release/stable-1.txt": dial tcp 34.107.204.206:443: connect: network is unreachable
W0914 19:16:53.747631    3306 version.go:103] falling back to the local client version: v1.19.1
W0914 19:16:53.747768    3306 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[init] Using Kubernetes version: v1.19.1
[preflight] Running pre-flight checks
    [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
    [WARNING Hostname]: hostname "node1" could not be reached
    [WARNING Hostname]: hostname "node1": lookup node1 on [2a0a:e5c0:2:a::a]:53: no such host
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki" 
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local node1] and IPs [2a0a:e5c0:2:13::1 2a0a:e5c0:2:2:0:84ff:fe41:f263]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost node1] and IPs [2a0a:e5c0:2:2:0:84ff:fe41:f263 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost node1] and IPs [2a0a:e5c0:2:2:0:84ff:fe41:f263 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes" 
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests" 
[control-plane] Creating static Pod manifest for "kube-apiserver" 
[control-plane] Creating static Pod manifest for "kube-controller-manager" 
[control-plane] Creating static Pod manifest for "kube-scheduler" 
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" 
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
^C
Actions
Copy link
 #6

Updated by Nico Schottelius about 4 years ago

  • Description updated (diff)
Actions
Copy link
 #7

Updated by Nico Schottelius about 4 years ago

  • Description updated (diff)

try 6:

...
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests" 
[control-plane] Creating static Pod manifest for "kube-apiserver" 
[control-plane] Creating static Pod manifest for "kube-controller-manager" 
[control-plane] Creating static Pod manifest for "kube-scheduler" 
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" 
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.

    Unfortunately, an error has occurred:
        timed out waiting for the condition

    This error is likely caused by:
        - The kubelet is not running
        - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

    If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
        - 'systemctl status kubelet'
        - 'journalctl -xeu kubelet'

    Additionally, a control plane component may have crashed or exited when started by the container runtime.
    To troubleshoot, list all containers using your preferred container runtimes CLI.

    Here is one example how you may list all Kubernetes containers running in docker:
        - 'docker ps -a | grep kube | grep -v pause'
        Once you have found the failing container, you can inspect its logs with:
        - 'docker logs CONTAINERID'

error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
node1:~#

checks:

  • etcd seems to be ok
  • scheduler tries to connect do :6443 and fails

network is too big:

node1:~# docker logs de26972b1722
Flag --insecure-port has been deprecated, This flag will be removed in a future version.
I0914 19:25:32.735390       1 server.go:625] external host was not specified, using 2a0a:e5c0:2:2:0:84ff:fe41:f263
Error: specified --service-cluster-ip-range is too large; for 128-bit addresses, the mask must be >= 108
node1:~#

Old:

node1:~# cat kubeadm.conf 
apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 2a0a:e5c0:2:2:0:84ff:fe41:f263
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
networking:
  serviceSubnet: 2a0a:e5c0:2:13::/64
  podSubnet: 2a0a:e5c0:2:12::/64
node1:~#

new:

node1:~# cat kubeadm.conf 
apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 2a0a:e5c0:2:2:0:84ff:fe41:f263
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
networking:
  serviceSubnet: 2a0a:e5c0:2:13::/110
  podSubnet: 2a0a:e5c0:2:12::/64
node1:~#

Success afterwards!

node1:~$ kubectl get pods -n kube-system
NAME                            READY   STATUS    RESTARTS   AGE
coredns-f9fd979d6-24t7g         0/1     Pending   0          2m22s
coredns-f9fd979d6-jt6hw         0/1     Pending   0          2m22s
etcd-node1                      1/1     Running   0          2m39s
kube-apiserver-node1            1/1     Running   0          2m39s
kube-controller-manager-node1   1/1     Running   0          2m39s
kube-proxy-6lpbs                1/1     Running   0          2m22s
kube-scheduler-node1            1/1     Running   0          2m39s
node1:~$
Actions
Copy link
 #8

Updated by Nico Schottelius about 4 years ago

  • Description updated (diff)
node1:~# cat /etc/conf.d/kubelet 
command_args="--cni-bin-dir=/usr/libexec/cni --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml" 
node1:~# vi /etc/conf.d/kubelet 
node1:~# cat /etc/conf.d/kubelet 
command_args="--cni-bin-dir=/usr/libexec/cni --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --allow-privileged=true" 
node1:~# /etc/init.d/kubelet restart
 * Caching service dependencies ...                                                                                                     [ ok ]
 * Stopping kubelet ...                                                                                                                 [ ok ]
 * Starting kubelet ...                                                                                                                 [ ok ]
node1:~#

kube-router/peering:

Example:

--peer-router-ips="192.168.1.99,192.168.1.100" 
--peer-router-asns=65000,65000

github fail:

node1:~$ kubectl apply -f https://raw.githubusercontent.com/cloudnativelabs/kube-router/master/daemonset/kubeadm-kuberouter.yaml
Unable to connect to the server: dial tcp 151.101.128.133:443: connect: network is unreachable
node1:~$

Untaint, tries to deploy:

node1:~$ kubectl taint nodes node1 node-role.kubernetes.io/master-
node1:~$ kubectl apply -f https://raw.githubusercontent.com/cloudnativelabs/kube-router/master/daemonset/kubeadm-kuberouter.yaml

^Cnode1:~$ kubectl get pods -n kube-system
NAME                            READY   STATUS              RESTARTS   AGE
coredns-f9fd979d6-jngp6         0/1     ContainerCreating   0          2m30s
coredns-f9fd979d6-kqjcl         0/1     ContainerCreating   0          2m30s
etcd-node1                      1/1     Running             0          2m45s
kube-apiserver-node1            1/1     Running             0          2m45s
kube-controller-manager-node1   1/1     Running             0          2m45s
kube-proxy-ft7t7                1/1     Running             0          2m30s
kube-router-cxfc2               0/1     CrashLoopBackOff    3          85s
kube-scheduler-node1            1/1     Running             0          2m45s
node1:~$

Config is created:

node1:~# cat  /etc/cni/net.d/10-kuberouter.conflist 
{
   "cniVersion":"0.3.0",
   "name":"mynet",
   "plugins":[
      {
         "name":"kubernetes",
         "type":"bridge",
         "bridge":"kube-bridge",
         "isDefaultGateway":true,
         "ipam":{
            "type":"host-local" 
         }
      }
   ]
}

However kubelet does not have the --allow-privileged=true flag at all. (does not support this parameter)

kube-router crashes without a router-id:

node1:~# docker logs e457ef7a933d 
I0914 20:11:45.995122       1 kube-router.go:231] Running /usr/local/bin/kube-router version v1.1.0-rc1-dirty, built on 2020-09-07T21:42:44+0000, go1.13.13
Failed to run kube-router: Failed to create network routing controller: Router-id must be specified in ipv6 operation
node1:~#

Patching it in:

node1:~$ diff -u kubeadm-kuberouter.yaml.orig kubeadm-kuberouter.yaml
--- kubeadm-kuberouter.yaml.orig
+++ kubeadm-kuberouter.yaml
@@ -55,6 +55,8 @@
         - --run-firewall=true
         - --run-service-proxy=false
         - --bgp-graceful-restart=true
+        - --router-id
+        - ${NODE_NAME}
         env:
         - name: NODE_NAME
           valueFrom:
node1:~$

Results in a new error:

node1:~# docker logs e29309159a4a
I0914 20:28:03.510920       1 kube-router.go:231] Running /usr/local/bin/kube-router version v1.1.0-rc1-dirty, built on 2020-09-07T21:42:44+0000, go1.13.13
I0914 20:28:03.638165       1 network_routes_controller.go:1075] Could not find annotation `kube-router.io/bgp-local-addresses` on node object so BGP will listen on node IP: 2a0a:e5c0:2:2:0:84ff:fe41:f263 address.
I0914 20:28:03.784909       1 network_policy_controller.go:148] Starting network policy controller
I0914 20:28:03.799619       1 network_policy_controller.go:156] Starting network policy controller full sync goroutine
E0914 20:28:03.950130       1 network_routes_controller.go:157] Failed to enable required policy based routing: Failed to add ip rule due to: exit status 2
I0914 20:28:03.953857       1 network_routes_controller.go:228] Starting network route controller
time="2020-09-14T20:28:03Z" level=warning msg="listen failed" Error="listen tcp: address 2a0a:e5c0:2:2:0:84ff:fe41:f263:50051: too many colons in address" Key="2a0a:e5c0:2:2:0:84ff:fe41:f263:50051" Topic=grpc
time="2020-09-14T20:28:03Z" level=fatal msg="failed to listen grpc port: listen tcp: address 2a0a:e5c0:2:2:0:84ff:fe41:f263:50051: too many colons in address" 
node1:~#
Actions
Copy link
 #9

Updated by Nico Schottelius about 4 years ago

  • Description updated (diff)

Switching to DNS64 servers:

node1:~# cat > /etc/resolv.conf 
nameserver 2a0a:e5c0:2:12:0:f0ff:fea9:c451
nameserver 2a0a:e5c0:2:12:0:f0ff:fea9:c45d
search k8s.ungleich.ch
node1:~# chattr +i /etc/resolv.conf
node1:~#
Actions
Copy link
 #10

Updated by Nico Schottelius about 4 years ago

  • Description updated (diff)
node1:~$ kubectl apply -f https://raw.githubusercontent.com/cloudnativelabs/kube-router/master/daemonset/kubeadm-kuberouter.yaml
configmap/kube-router-cfg created
daemonset.apps/kube-router created
serviceaccount/kube-router created
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRole is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRole
clusterrole.rbac.authorization.k8s.io/kube-router created
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRoleBinding is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRoleBinding
clusterrolebinding.rbac.authorization.k8s.io/kube-router created
node1:~$ kubectl get pods -n kube-system
NAME                            READY   STATUS    RESTARTS   AGE
coredns-f9fd979d6-24t7g         0/1     Pending   0          14m
coredns-f9fd979d6-jt6hw         0/1     Pending   0          14m
etcd-node1                      1/1     Running   0          14m
kube-apiserver-node1            1/1     Running   0          14m
kube-controller-manager-node1   1/1     Running   0          14m
kube-proxy-6lpbs                1/1     Running   0          14m
kube-scheduler-node1            1/1     Running   0          14m
node1:~$
Actions
Copy link
 #11

Updated by Nico Schottelius about 4 years ago

untaint master node for testing:

node1:~$ kubectl taint nodes node1 node-role.kubernetes.io/master-
node/node1 untainted
Actions
Copy link
 #12

Updated by Nico Schottelius about 4 years ago

Actions
Copy link
 #13

Updated by Nico Schottelius about 4 years ago

Resetting and trying with calico:

In theory:

kubectl create -f https://docs.projectcalico.org/manifests/tigera-operator.yaml
kubectl create -f https://docs.projectcalico.org/manifests/custom-resources.yaml

In reality:

wget https://docs.projectcalico.org/manifests/calico.yaml
cp calico.yaml calico.yaml.orig

vi calico.yaml
node1:~$ diff -u calico.yaml.orig calico.yaml
--- calico.yaml.orig
+++ calico.yaml
@@ -3634,6 +3634,9 @@
             - name: DATASTORE_TYPE
               value: "kubernetes" 
             # Wait for the datastore.
+            - name: CALICO_ROUTER_ID
+              value: "hash" 
+            # Wait for the datastore.
             - name: WAIT_FOR_DATASTORE
               value: "true" 
             # Set based on the k8s node name.
@@ -3652,7 +3655,7 @@
               value: "k8s,bgp" 
             # Auto-detect the BGP IP address.
             - name: IP
-              value: "autodetect" 
+              value: "none" 
             # Enable IPIP
             - name: CALICO_IPV4POOL_IPIP
               value: "Always"

Applying:

node1:~$ kubectl apply -f calico.yaml
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
serviceaccount/calico-node created
deployment.apps/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
node1:~$

calico-node crashes, no error in docker:

calico-node-qfnr9                         0/1     PodInitializing   0          16s
calico-node-qfnr9                         0/1     RunContainerError   0          28s
calico-node-qfnr9                         0/1     RunContainerError   1          36s
calico-node-qfnr9                         0/1     CrashLoopBackOff    1          46s
calico-node-qfnr9                         0/1     RunContainerError   2          59s

node1:~# docker logs d523a007c34b
node1:~# docker logs 577714a283cc
node1:~#

Error in the pod description:

                 CriticalAddonsOnly op=Exists
                 node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                 node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                 node.kubernetes.io/network-unavailable:NoSchedule op=Exists
                 node.kubernetes.io/not-ready:NoExecute op=Exists
                 node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                 node.kubernetes.io/unreachable:NoExecute op=Exists
                 node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  119s               default-scheduler  Successfully assigned kube-system/calico-node-qfnr9 to node1
  Normal   Pulling    118s               kubelet, node1     Pulling image "calico/cni:v3.16.1" 
  Normal   Pulled     113s               kubelet, node1     Successfully pulled image "calico/cni:v3.16.1" in 5.453532909s
  Normal   Created    112s               kubelet, node1     Created container upgrade-ipam
  Normal   Started    111s               kubelet, node1     Started container upgrade-ipam
  Normal   Pulled     110s               kubelet, node1     Container image "calico/cni:v3.16.1" already present on machine
  Normal   Created    110s               kubelet, node1     Created container install-cni
  Normal   Started    110s               kubelet, node1     Started container install-cni
  Normal   Pulling    108s               kubelet, node1     Pulling image "calico/pod2daemon-flexvol:v3.16.1" 
  Normal   Pulled     105s               kubelet, node1     Successfully pulled image "calico/pod2daemon-flexvol:v3.16.1" in 3.802406662s
  Normal   Created    104s               kubelet, node1     Created container flexvol-driver
  Normal   Started    104s               kubelet, node1     Started container flexvol-driver
  Normal   Pulling    103s               kubelet, node1     Pulling image "calico/node:v3.16.1" 
  Normal   Pulled     92s                kubelet, node1     Successfully pulled image "calico/node:v3.16.1" in 11.364803287s
  Normal   Created    72s (x3 over 91s)  kubelet, node1     Created container calico-node
  Warning  Failed     72s (x3 over 91s)  kubelet, node1     Error: failed to start container "calico-node": Error response from daemon: path /sys/fs is mounted on /sys but it is not a shared mount
  Warning  BackOff    53s (x3 over 83s)  kubelet, node1     Back-off restarting failed container
  Normal   Pulled     40s (x3 over 91s)  kubelet, node1     Container image "calico/node:v3.16.1" already present on machine
node1:~$

trying to fix with:

node1:~# mount --make-rshared /

(from: https://github.com/kubernetes/kubernetes/issues/61058)

Fails to start, because of missing autodetection, which was disabled according to https://docs.projectcalico.org/networking/ipv6

node1:~# docker logs 1d28c0b279f9
2020-09-14 21:03:07.982 [INFO][9] startup/startup.go 361: Early log level set to info
2020-09-14 21:03:07.982 [INFO][9] startup/startup.go 377: Using NODENAME environment for node name
2020-09-14 21:03:07.982 [INFO][9] startup/startup.go 389: Determined node name: node1
2020-09-14 21:03:07.984 [INFO][9] startup/startup.go 421: Checking datastore connection
2020-09-14 21:03:07.990 [INFO][9] startup/startup.go 445: Datastore connection verified
2020-09-14 21:03:07.990 [INFO][9] startup/startup.go 109: Datastore is ready
2020-09-14 21:03:07.992 [INFO][9] startup/customresource.go 101: Error getting resource Key=GlobalFelixConfig(name=CalicoVersion) Name="calico
version" Resource="GlobalFelixConfigs" error=the server could not find the requested resource (get GlobalFelixConfigs.crd.projectcalico.org ca
licoversion)
2020-09-14 21:03:07.998 [INFO][9] startup/startup.go 487: Initialize BGP data
2020-09-14 21:03:07.998 [WARNING][9] startup/startup.go 557: No IP Addresses configured, and autodetection is not enabled
2020-09-14 21:03:07.998 [WARNING][9] startup/startup.go 1310: Terminating

Resetting the autodetection fails as follows:

node1:~# docker logs 0efcaf6f9021
2020-09-14 21:08:03.993 [INFO][8] startup/startup.go 361: Early log level set to info
2020-09-14 21:08:03.993 [INFO][8] startup/startup.go 377: Using NODENAME environment for node name
2020-09-14 21:08:03.993 [INFO][8] startup/startup.go 389: Determined node name: node1
2020-09-14 21:08:03.995 [INFO][8] startup/startup.go 421: Checking datastore connection
2020-09-14 21:08:04.000 [INFO][8] startup/startup.go 445: Datastore connection verified
2020-09-14 21:08:04.000 [INFO][8] startup/startup.go 109: Datastore is ready
2020-09-14 21:08:04.004 [INFO][8] startup/customresource.go 101: Error getting resource Key=GlobalFelixConfig(name=CalicoVersion) Name="calicoversion" Resource="GlobalFelixConfigs" error=the server could not find the requested resource (get GlobalFelixConfigs.crd.projectcalico.org calicoversion)
2020-09-14 21:08:04.015 [INFO][8] startup/startup.go 487: Initialize BGP data
2020-09-14 21:08:04.015 [WARNING][8] startup/startup.go 742: Unable to auto-detect an IPv4 address: no valid IPv4 addresses found on the host interfaces
2020-09-14 21:08:04.015 [WARNING][8] startup/startup.go 509: Couldn't autodetect an IPv4 address. If auto-detecting, choose a different autodetection method. Otherwise provide an explicit address.
2020-09-14 21:08:04.015 [INFO][8] startup/startup.go 325: Clearing out-of-date IPv4 address from this node IP="" 
2020-09-14 21:08:04.021 [WARNING][8] startup/startup.go 1310: Terminating
Calico node failed to start
node1:~#

Next try: set IP6 to autodetect / replace IP with IP6

node1:~$ diff -u calico.yaml.orig calico.yaml
--- calico.yaml.orig
+++ calico.yaml
@@ -3634,6 +3634,9 @@
             - name: DATASTORE_TYPE
               value: "kubernetes" 
             # Wait for the datastore.
+            - name: CALICO_ROUTER_ID
+              value: "hash" 
+            # Wait for the datastore.
             - name: WAIT_FOR_DATASTORE
               value: "true" 
             # Set based on the k8s node name.
@@ -3651,7 +3654,7 @@
             - name: CLUSTER_TYPE
               value: "k8s,bgp" 
             # Auto-detect the BGP IP address.
-            - name: IP
+            - name: IP6
               value: "autodetect" 
             # Enable IPIP
             - name: CALICO_IPV4POOL_IPIP
node1:~$

crashes as well with:

node1:~# docker logs edb8a6ca84d0
2020-09-14 21:12:58.952 [INFO][9] startup/startup.go 361: Early log level set to info
2020-09-14 21:12:58.952 [INFO][9] startup/startup.go 377: Using NODENAME environment for node name
2020-09-14 21:12:58.952 [INFO][9] startup/startup.go 389: Determined node name: node1
2020-09-14 21:12:58.954 [INFO][9] startup/startup.go 421: Checking datastore connection
2020-09-14 21:12:58.960 [INFO][9] startup/startup.go 445: Datastore connection verified
2020-09-14 21:12:58.960 [INFO][9] startup/startup.go 109: Datastore is ready
2020-09-14 21:12:58.963 [INFO][9] startup/customresource.go 101: Error getting resource Key=GlobalFelixConfig(name=CalicoVersion) Name="calicoversion" Resource="GlobalFelixConfigs" error=the server could not find the requested resource (get GlobalFelixConfigs.crd.projectcalico.org calicoversion)
2020-09-14 21:12:58.998 [INFO][9] startup/startup.go 487: Initialize BGP data
2020-09-14 21:12:58.998 [WARNING][9] startup/startup.go 742: Unable to auto-detect an IPv4 address: no valid IPv4 addresses found on the host interfaces
2020-09-14 21:12:58.998 [WARNING][9] startup/startup.go 509: Couldn't autodetect an IPv4 address. If auto-detecting, choose a different autodetection method. Otherwise provide an explicit address.
2020-09-14 21:12:58.998 [INFO][9] startup/startup.go 329: Clearing out-of-date IPv6 address from this node IP="" 
2020-09-14 21:12:59.005 [WARNING][9] startup/startup.go 1310: Terminating
Calico node failed to start
node1:~#

Setting IP to none and IP6 to autodetect:

node1:~$ diff -u calico.yaml.orig calico.yaml
--- calico.yaml.orig
+++ calico.yaml
@@ -3634,6 +3634,9 @@
             - name: DATASTORE_TYPE
               value: "kubernetes" 
             # Wait for the datastore.
+            - name: CALICO_ROUTER_ID
+              value: "hash" 
+            # Wait for the datastore.
             - name: WAIT_FOR_DATASTORE
               value: "true" 
             # Set based on the k8s node name.
@@ -3652,6 +3655,8 @@
               value: "k8s,bgp" 
             # Auto-detect the BGP IP address.
             - name: IP
+              value: "none" 
+            - name: IP6
               value: "autodetect" 
             # Enable IPIP
             - name: CALICO_IPV4POOL_IPIP

And calico-node is running!

However calico-kube-controllers stays pending:

node1:~$ kubectl get pods -n kube-system
NAME                                      READY   STATUS    RESTARTS   AGE
calico-kube-controllers-c9784d67d-w46kf   0/1     Pending   0          32s
calico-node-5lhhv                         1/1     Running   0          32s
coredns-f9fd979d6-kptlk                   0/1     Pending   0          44s
coredns-f9fd979d6-shvw6                   0/1     Pending   0          44s
etcd-node1                                0/1     Running   0          59s
kube-apiserver-node1                      1/1     Running   0          59s
kube-controller-manager-node1             0/1     Running   0          59s
kube-proxy-hrxmz                          1/1     Running   0          44s
kube-scheduler-node1                      0/1     Running   0          58s
node1:~$ 

... describe:

  Type     Reason            Age                From               Message
  ----     ------            ----               ----               -------
  Warning  FailedScheduling  75s (x3 over 81s)  default-scheduler  0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.

the node is not ready: https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/

Not ready, because the network is not ready. Hmm.....

Conditions:
  Type                 Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----                 ------  -----------------                 ------------------                ------                       -------
  NetworkUnavailable   False   Mon, 14 Sep 2020 21:17:49 +0000   Mon, 14 Sep 2020 21:17:49 +0000   CalicoIsUp                   Calico is running on this node
  MemoryPressure       False   Mon, 14 Sep 2020 21:22:18 +0000   Mon, 14 Sep 2020 21:17:10 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure         False   Mon, 14 Sep 2020 21:22:18 +0000   Mon, 14 Sep 2020 21:17:10 +0000   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure          False   Mon, 14 Sep 2020 21:22:18 +0000   Mon, 14 Sep 2020 21:17:10 +0000   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready                False   Mon, 14 Sep 2020 21:22:18 +0000   Mon, 14 Sep 2020 21:17:10 +0000   KubeletNotReady              runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

Removing taint based on hint from https://forum.linuxfoundation.org/discussion/855616/calico-plugin-not-working-with-1-12-kubernetes-please-update-the-k8smaster-sh

node1:~$ kubectl taint nodes node1 node.kubernetes.io/not-ready-
node/node1 untainted

Containers are being started now:

calico-kube-controllers-c9784d67d-w46kf   0/1     ContainerCreating   0          10m
coredns-f9fd979d6-kptlk                   0/1     ContainerCreating   0          10m
coredns-f9fd979d6-shvw6                   0/1     ContainerCreating   0          10m

Resetting, trying with typha. Result: same as above.

Actions
Copy link
 #14

Updated by Nico Schottelius about 4 years ago

  • Description updated (diff)
Actions
Copy link
 #15

Updated by Nico Schottelius about 4 years ago

  • Description updated (diff)
Actions
Copy link
 #16

Updated by Nico Schottelius about 4 years ago

Trying cilium, one operator fails with "level=fatal msg="Unable to init cluster-pool allocator" error="IPv4CIDR can not be set if IPv4 is not enabled" subsys=cilium-operator-generic"


node1:~$ diff -u cilium-quick-install.yaml.orig cilium-quick-install.yaml
--- cilium-quick-install.yaml.orig
+++ cilium-quick-install.yaml
@@ -38,11 +38,11 @@

   # Enable IPv4 addressing. If enabled, all endpoints are allocated an IPv4
   # address.
-  enable-ipv4: "true" 
+  enable-ipv4: "false" 

   # Enable IPv6 addressing. If enabled, all endpoints are allocated an IPv6
   # address.
-  enable-ipv6: "false" 
+  enable-ipv6: "true" 
   enable-bpf-clock-probe: "true" 

   # If you want cilium monitor to aggregate tracing for packets, set this level
node1:~$ 

node1:~$ diff -u cilium-quick-install.yaml.orig cilium-quick-install.yaml
--- cilium-quick-install.yaml.orig
+++ cilium-quick-install.yaml
@@ -38,11 +38,11 @@

   # Enable IPv4 addressing. If enabled, all endpoints are allocated an IPv4
   # address.
-  enable-ipv4: "true" 
+  enable-ipv4: "false" 

   # Enable IPv6 addressing. If enabled, all endpoints are allocated an IPv6
   # address.
-  enable-ipv6: "false" 
+  enable-ipv6: "true" 
   enable-bpf-clock-probe: "true" 

   # If you want cilium monitor to aggregate tracing for packets, set this level
node1:~$ kubectl create -f cilium-quick-install.yaml
serviceaccount/cilium created
serviceaccount/cilium-operator created
configmap/cilium-config created
clusterrole.rbac.authorization.k8s.io/cilium created
clusterrole.rbac.authorization.k8s.io/cilium-operator created
clusterrolebinding.rbac.authorization.k8s.io/cilium created
clusterrolebinding.rbac.authorization.k8s.io/cilium-operator created
daemonset.apps/cilium created
deployment.apps/cilium-operator created

node1:~$ kubectl get pods -n kube-system 
NAME                               READY   STATUS                  RESTARTS   AGE
cilium-cm9l6                       0/1     Init:CrashLoopBackOff   1          24s
cilium-operator-7f4dc846b6-7bfkj   0/1     Pending                 0          24s
cilium-operator-7f4dc846b6-r4rv5   0/1     CrashLoopBackOff        1          24s
coredns-f9fd979d6-84rft            1/1     Running                 0          2m16s
coredns-f9fd979d6-rsvrn            1/1     Running                 0          2m16s
etcd-node1                         1/1     Running                 2          2m33s
kube-apiserver-node1               1/1     Running                 2          2m32s
kube-controller-manager-node1      1/1     Running                 2          2m32s
kube-proxy-xtbcv                   1/1     Running                 0          2m16s
kube-scheduler-node1               1/1     Running                 3          2m33s
node1:~$ 

node1:~$ kubectl get pods -n kube-system 
NAME                               READY   STATUS                  RESTARTS   AGE
cilium-cm9l6                       0/1     Init:CrashLoopBackOff   3          76s
cilium-operator-7f4dc846b6-7bfkj   0/1     Pending                 0          76s
cilium-operator-7f4dc846b6-r4rv5   0/1     CrashLoopBackOff        2          76s
coredns-f9fd979d6-84rft            1/1     Running                 0          3m8s
coredns-f9fd979d6-rsvrn            1/1     Running                 0          3m8s
etcd-node1                         1/1     Running                 2          3m25s
kube-apiserver-node1               1/1     Running                 2          3m24s
kube-controller-manager-node1      1/1     Running                 2          3m24s
kube-proxy-xtbcv                   1/1     Running                 0          3m8s
kube-scheduler-node1               1/1     Running                 3          3m25s
node1:~$ 

level=info msg="  --version='false'" subsys=cilium-operator-generic
level=info msg="Cilium Operator 1.8.3 54cf3810d 2020-09-04T14:01:53+02:00 go version go1.14.7 linux/amd64" subsys=cilium-operator-generic
level=info msg="Establishing connection to apiserver" host="https://[2a0a:e5c0:2:13::1]:443" subsys=k8s
level=info msg="Starting apiserver on address 127.0.0.1:9234" subsys=cilium-operator-generic
level=info msg="Connected to apiserver" subsys=k8s
level=info msg="attempting to acquire leader lease  kube-system/cilium-operator-resource-lock..." subsys=klog
level=info msg="Operator with ID \"node1-vPeGhfrZeV\" elected as new leader" operator-id=node1-SkVuCxxsHW subsys=cilium-operator-generic
level=info msg="successfully acquired lease kube-system/cilium-operator-resource-lock" subsys=klog
level=info msg="Leading the operator HA deployment" subsys=cilium-operator-generic
level=fatal msg="Unable to init cluster-pool allocator" error="IPv4CIDR can not be set if IPv4 is not enabled" subsys=cilium-operator-generic
node1:~#

After rebooting a node it does not get out of the nodenotready state, the following problem occurs:

  Ready            False   Tue, 15 Sep 2020 18:38:24 +0000   Tue, 15 Sep 2020 18:37:59 +0000   KubeletNotReady              runtime network no
t ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Actions
Copy link
 #17

Updated by Nico Schottelius about 4 years ago

  • Description updated (diff)
Actions
Copy link
 #18

Updated by Nico Schottelius about 4 years ago

Trying the statically routed approach:

[20:44] router1.place6:~# ip -6 route add 2a0a:e5c0:2:12::/64 via 2a0a:e5c0:2:2:0:84ff:fe41:f268
[20:47] router1.place6:~# ip -6 route add 2a0a:e5c0:2:13::/64 nexthop via 2a0a:e5c0:2:2:0:84ff:fe41:f268 nexthop via 2a0a:e5c0:2:2:0:84ff:fe41
:f263 nexthop via 2a0a:e5c0:2:2:0:84ff:fe41:f269
[20:49] router1.place6:~# ip -6 route add 2a0a:e5c0:2:14::/64 via 2a0a:e5c0:2:2:0:84ff:fe41:f269
[20:49] router1.place6:~#

cni configuration on worker nodes:

node2:~# cat 10-node2-cni.conf 

{
  "cniVersion": "0.3.0",
  "name": "mynet",
  "type": "bridge",
  "bridge": "cbr0",
  "isDefaultGateway": true,
  "ipMasq": false,
  "hairpinMode": true,
  "ipam": {
    "type": "host-local",
    "ranges": [
      [
        {
          "subnet": "2a0a:e5c0:2:12::/64",
          "gateway": "2a0a:e5c0:2:12::1" 
        }
      ]
    ]
  }
}

node2:~# rm /etc/cni/net.d/*
node2:~# cp 10-node2-cni.conf /etc/cni/net.d/

node3:~# cat 10-node3-cni.conf 
{
  "cniVersion": "0.3.0",
  "name": "mynet",
  "type": "bridge",
  "bridge": "cbr0",
  "isDefaultGateway": true,
  "ipMasq": false,
  "hairpinMode": true,
  "ipam": {
    "type": "host-local",
    "ranges": [
      [
        {
          "subnet": "2a0a:e5c0:2:14::/64",  
          "gateway": "2a0a:e5c0:2:14::1"  
        }
      ]
    ]
  }
}

node3:~# mkdir -p /etc/cni/net.d/
node3:~# cp 10-node3-cni.conf /etc/cni/net.d/
node3:~#

Created an nginx deployment:

node1:~$ cat nginxdeploy.yaml 
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2 # tells deployment to run 2 pods matching the template
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80

node1:~$ kubectl get pods
NAME                               READY   STATUS         RESTARTS   AGE
nginx-deployment-585449566-9vjt8   1/1     Running        0          3m1s
nginx-deployment-585449566-gxbdp   0/1     ErrImagePull   0          3m1s
node1:~$ 

[21:06] bridge:~% curl http://[2a0a:e5c0:2:14::6]

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
[21:06] bridge:~%

-> 1 pods is reachable from outside.

Testing services, cmp https://kubernetes.io/docs/concepts/services-networking/connect-applications-service/

node1:~$ kubectl expose deployment/nginx-deployment
service/nginx-deployment exposed

node1:~$ kubectl get svc nginx-deployment 
NAME               TYPE        CLUSTER-IP             EXTERNAL-IP   PORT(S)   AGE
nginx-deployment   ClusterIP   2a0a:e5c0:2:13::1acd   <none>        80/TCP    69s
node1:~$ 

node1:~$ curl http://[2a0a:e5c0:2:13::1acd]
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

Looks good, let's verify it from outside:

[21:12] bridge:~% curl http://[2a0a:e5c0:2:13::1acd]

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

Testing multi path route via bird directly, works:

# Router specific networks
protocol static this_router_v6 {
    ipv6 {};

    route 2a0a:e5c0:e::/48 unreachable;             # Router local test
    route 2a09:2944::/32 unreachable;               # Router local test

    route 2a0a:e5c0:2:13::/64 
          via 2a0a:e5c0:2:2:0:84ff:fe41:f268 
          via 2a0a:e5c0:2:2:0:84ff:fe41:f263 
          via 2a0a:e5c0:2:2:0:84ff:fe41:f269;
}
Actions
Copy link
 #19

Updated by Nico Schottelius about 4 years ago

  • Description updated (diff)
Actions
Copy link
 #20

Updated by Nico Schottelius about 4 years ago

Testing the ceph connection.

[21:55:54] black2.place6:~# ceph osd pool create kubernetes 128
pool 'kubernetes' created
[21:57:25] black2.place6:~# ceph osd pool set kubernetes crush_rule hdd-big
set pool 19 crush_rule to hdd-big
[21:57:46] black2.place6:~# ceph osd pool application enable kubernetes rbd
enabled application 'rbd' on pool 'kubernetes'
[21:58:16] black2.place6:~# 
[22:00:32] black2.place6:~# ceph auth get-or-create client.kubernetes mon "profile rbd" osd "profile rbd pool=kubernetes" mgr "profile rbd pool=kubernetes"

on the nodes:

apk add ceph-common

(adds support for rbd)

Actions
Copy link
 #21

Updated by Nico Schottelius 11 months ago

  • Status changed from In Progress to Rejected
Actions
Copy link

Also available in: Atom PDF