Task #7185: Setup network monitoring system on new off-site VPS - Open Infrastructure - ungleich redmine

Actions

Copy link

Task #7185

closed

Setup network monitoring system on new off-site VPS

Added by Nico Schottelius about 6 years ago. Updated almost 2 years ago.

Status:

Rejected

Priority:

Normal

Assignee:

Nico Schottelius

Target version:

Start date:

09/30/2019

Due date:

% Done:

Estimated time:

PM Check date:

Description

Objective¶

We want to have a live update similar to https://status.ungleich.ch/ running on the VPS.
We want to do this via some open source monitoring.
We want to check ping, http and maybe https reachability
The information should be public

Hints¶

~~Checkout on one of our VMs, if opennms allows to create a public interface.~~
~~Checkout the list of systems on https://www.pcwdld.com/linux-network-monitor-software-and-tools#OpenNMS~~
~~Another interesting project that we already have running is smokeping~~
Easiest way, reproduce what we already have:
- setup prometheus
- setup grafana
- setup blackbox-exporter
- create a public dashboard

-> checkout the tools, discuss with me in the chat which one to use

Files

Ungleich-1570816597910.json (20.9 KB) Ungleich-1570816597910.json

Sample Dashboard

Ahmed Bilal, 10/11/2019 07:56 PM

Related issues 1 (0 open — 1 closed)

Actions

Copy link

Updated by Nico Schottelius about 6 years ago

Related to Task #7184: Create a mailing list for maintenance notifications added

Actions

Copy link

Updated by Nico Schottelius about 6 years ago

Description updated (diff)

Actions

Copy link

Updated by Ahmed Bilal about 6 years ago

Status changed from New to Seen

Actions

Copy link

Updated by Ahmed Bilal about 6 years ago

Assignee changed from Ahmed Bilal to Nico Schottelius

I have setup (prometheus + grafana + blackbox exporter) on debian.sandbox.ahmedbilal.ungleich.cloud

You can see the Grafana at http://debian.sandbox.ahmedbilal.ungleich.cloud:3000

username: admin
password: admin

You can see the Prometheus at http://debian.sandbox.ahmedbilal.ungleich.cloud:9090

What to do next except deploying it to VPS. I assign the DNS name post.ungleich.ch to VPS.

Actions

Copy link

Updated by Nico Schottelius about 6 years ago

you can also add another name to the VPS like
monitoring.place11.ungleich.ch ;-)

redmine@ungleich.ch writes:

Actions

Copy link

Updated by Ahmed Bilal about 6 years ago

Assignee changed from Nico Schottelius to Ahmed Bilal

Actions

Copy link

Updated by Nico Schottelius about 6 years ago

Seems not to be solved:

monitoring.place11.ungleich.ch’s server IP address could not be found.

redmine@ungleich.ch writes:

Actions

Copy link

Updated by Nico Schottelius about 6 years ago

Ah, sorry, thought it was assigned to me!

Actions

Copy link

Updated by Ahmed Bilal about 6 years ago

/etc/prometheus/blackbox.yml

modules:
  http_2xx:
    prober: http
    http:
  http_post_2xx:
    prober: http
    http:
      method: POST
  tcp_connect:
    prober: tcp
  pop3s_banner:
    prober: tcp
    tcp:
      query_response:
      - expect: "^+OK" 
      tls: true
      tls_config:
        insecure_skip_verify: false
  ssh_banner:
    prober: tcp
    tcp:
      query_response:
      - expect: "^SSH-2.0-" 
  irc_banner:
    prober: tcp
    tcp:
      query_response:
      - send: "NICK prober" 
      - send: "USER prober prober prober :prober" 
      - expect: "PING :([^ ]+)" 
        send: "PONG ${1}" 
      - expect: "^:[^ ]+ 001" 
  icmp:
    prober: icmp

/etc/prometheus/prometheus

global:
  scrape_interval: 10s  # can be overridden by setting scrape_interval in a job
  evaluation_interval: 30s  # for rules

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    # monitor: 'place5-prod'

scrape_configs:
  - job_name: 'routers'
    metrics_path: /probe
    params:
      module: [icmp]
    static_configs:
      - targets:
        - router1.place6.ungleich.ch
        - router2.place6.ungleich.ch
        - router1.place5.ungleich.ch
        - router2.place5.ungleich.ch
        - router3.place5.ungleich.ch
        - router4.place5.ungleich.ch
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: post.ungleich.ch:9115

  - job_name: 'core-services'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
        - https://ungleich.ch
        - https://monitoring.place5.ungleich.ch
        - https://monitoring.place6.ungleich.ch
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: post.ungleich:9115

rule_files:
  - /etc/prometheus/*.rules

alerting:
  alertmanagers:
    - consul_sd_configs:
      - server: '127.0.0.1:8500'
        services:
          - alertmanager

Actions

Copy link

#10