Project

General

Profile

Task #9468

Test helm kube-prometheus-stack

Added by Nico Schottelius 3 months ago. Updated 3 months ago.

Status:
In Progress
Priority:
Normal
Target version:
-
Start date:
07/01/2021
Due date:
% Done:

0%

Estimated time:
PM Check date:

Description

Installing using:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack

History

#1

Updated by Nico Schottelius 3 months ago

It's possible grafana does not listen on IPv6 by default:

                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age               From               Message
  ----     ------     ----              ----               -------
  Normal   Scheduled  5m53s             default-scheduler  Successfully assigned default/kube-prometheus-stack-grafana-5569d59c6d-zk8cw to server48
  Normal   Pulling    5m46s             kubelet            Pulling image "quay.io/kiwigrid/k8s-sidecar:1.12.2" 
  Normal   Pulled     58s               kubelet            Successfully pulled image "quay.io/kiwigrid/k8s-sidecar:1.12.2" in 4m47.048230693s
  Normal   Created    52s               kubelet            Created container grafana-sc-datasources
  Normal   Started    51s               kubelet            Started container grafana-sc-datasources
  Normal   Pulled     49s               kubelet            Container image "quay.io/kiwigrid/k8s-sidecar:1.12.2" already present on machine
  Normal   Created    47s               kubelet            Created container grafana-sc-dashboard
  Normal   Started    47s               kubelet            Started container grafana-sc-dashboard
  Normal   Pulling    47s               kubelet            Pulling image "grafana/grafana:8.0.3" 
  Normal   Pulled     30s               kubelet            Successfully pulled image "grafana/grafana:8.0.3" in 16.883716549s
  Normal   Created    15s               kubelet            Created container grafana
  Normal   Started    14s               kubelet            Started container grafana
  Warning  Unhealthy  3s (x3 over 13s)  kubelet            Readiness probe failed: Get "http://[2a0a:e5c0:13:e1:9b19:7142:bebb:4dac]:3000/api/health": dial tcp [2a0a:e5c0:13:e1:9b19:7142:bebb:4dac]:3000: connect: connection refused

Correction, it just needed time:

[15:58] bridge:generic% curl http://[2a0a:e5c0:13:e1:9b19:7142:bebb:4dac]:3000/api/health
{
  "commit": "cae5c5e46b",
  "database": "ok",
  "version": "8.0.3" 
}%                                                                                                                                            [16:00] bridge:generic%   

OTOH, the kube-prometheus-stack-prometheus-node-exporter does not come up due to shared option missing:

Volumes:
  proc:
    Type:          HostPath (bare host directory volume)
    Path:          /proc
    HostPathType:  
  sys:
    Type:          HostPath (bare host directory volume)
    Path:          /sys
    HostPathType:  
  root:
    Type:          HostPath (bare host directory volume)
    Path:          /
    HostPathType:  
  kube-api-access-4jktf:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 :NoSchedule op=Exists
                             node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/network-unavailable:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type     Reason     Age                     From               Message
  ----     ------     ----                    ----               -------
  Normal   Scheduled  8m7s                    default-scheduler  Successfully assigned default/kube-prometheus-stack-prometheus-node-exporter-zkz77 to server47
  Normal   Pulled     6m50s (x5 over 8m3s)    kubelet            Container image "quay.io/prometheus/node-exporter:v1.1.2" already present on machine
  Normal   Created    6m49s (x5 over 8m1s)    kubelet            Created container node-exporter
  Warning  Failed     6m48s (x5 over 8m)      kubelet            Error: failed to start container "node-exporter": Error response from daemon: path / is mounted on / but it is not a shared or slave mount
  Warning  BackOff    2m54s (x30 over 7m56s)  kubelet            Back-off restarting failed container
[16:01] bridge:~% 

Also available in: Atom PDF