Project

General

Profile

Actions

Task #7185

closed

Setup network monitoring system on new off-site VPS

Added by Nico Schottelius over 4 years ago. Updated 3 months ago.

Status:
Rejected
Priority:
Normal
Target version:
-
Start date:
09/30/2019
Due date:
% Done:

0%

Estimated time:
PM Check date:

Description

Objective

  • We want to have a live update similar to https://status.ungleich.ch/ running on the VPS.
  • We want to do this via some open source monitoring.
  • We want to check ping, http and maybe https reachability
  • The information should be public

Hints

  • Checkout on one of our VMs, if opennms allows to create a public interface.
  • Checkout the list of systems on https://www.pcwdld.com/linux-network-monitor-software-and-tools#OpenNMS
  • Another interesting project that we already have running is smokeping
  • Easiest way, reproduce what we already have:
    • setup prometheus
    • setup grafana
    • setup blackbox-exporter
    • create a public dashboard

-> checkout the tools, discuss with me in the chat which one to use


Files

Ungleich-1570816597910.json (20.9 KB) Ungleich-1570816597910.json Sample Dashboard Ahmed Bilal, 10/11/2019 07:56 PM

Related issues 1 (0 open1 closed)

Related to Open Infrastructure - Task #7184: Create a mailing list for maintenance notificationsRejectedSanghee Kim09/30/201910/04/2019

Actions
Actions #1

Updated by Nico Schottelius over 4 years ago

  • Related to Task #7184: Create a mailing list for maintenance notifications added
Actions #2

Updated by Nico Schottelius over 4 years ago

  • Description updated (diff)
Actions #3

Updated by Ahmed Bilal over 4 years ago

  • Status changed from New to Seen
Actions #4

Updated by Ahmed Bilal over 4 years ago

  • Assignee changed from Ahmed Bilal to Nico Schottelius

I have setup (prometheus + grafana + blackbox exporter) on debian.sandbox.ahmedbilal.ungleich.cloud

You can see the Grafana at http://debian.sandbox.ahmedbilal.ungleich.cloud:3000
  • username: admin
  • password: admin

You can see the Prometheus at http://debian.sandbox.ahmedbilal.ungleich.cloud:9090

What to do next except deploying it to VPS. I assign the DNS name post.ungleich.ch to VPS.

Actions #5

Updated by Nico Schottelius over 4 years ago

you can also add another name to the VPS like
monitoring.place11.ungleich.ch ;-)

writes:

Actions #6

Updated by Ahmed Bilal over 4 years ago

  • Assignee changed from Nico Schottelius to Ahmed Bilal
Actions #7

Updated by Nico Schottelius over 4 years ago

Seems not to be solved:

monitoring.place11.ungleich.ch’s server IP address could not be found.

writes:

Actions #8

Updated by Nico Schottelius over 4 years ago

Ah, sorry, thought it was assigned to me!

Actions #9

Updated by Ahmed Bilal over 4 years ago

/etc/prometheus/blackbox.yml

modules:
  http_2xx:
    prober: http
    http:
  http_post_2xx:
    prober: http
    http:
      method: POST
  tcp_connect:
    prober: tcp
  pop3s_banner:
    prober: tcp
    tcp:
      query_response:
      - expect: "^+OK" 
      tls: true
      tls_config:
        insecure_skip_verify: false
  ssh_banner:
    prober: tcp
    tcp:
      query_response:
      - expect: "^SSH-2.0-" 
  irc_banner:
    prober: tcp
    tcp:
      query_response:
      - send: "NICK prober" 
      - send: "USER prober prober prober :prober" 
      - expect: "PING :([^ ]+)" 
        send: "PONG ${1}" 
      - expect: "^:[^ ]+ 001" 
  icmp:
    prober: icmp

/etc/prometheus/prometheus

global:
  scrape_interval: 10s  # can be overridden by setting scrape_interval in a job
  evaluation_interval: 30s  # for rules

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    # monitor: 'place5-prod'

scrape_configs:
  - job_name: 'routers'
    metrics_path: /probe
    params:
      module: [icmp]
    static_configs:
      - targets:
        - router1.place6.ungleich.ch
        - router2.place6.ungleich.ch
        - router1.place5.ungleich.ch
        - router2.place5.ungleich.ch
        - router3.place5.ungleich.ch
        - router4.place5.ungleich.ch
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: post.ungleich.ch:9115

  - job_name: 'core-services'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
        - https://ungleich.ch
        - https://monitoring.place5.ungleich.ch
        - https://monitoring.place6.ungleich.ch
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: post.ungleich:9115

rule_files:
  - /etc/prometheus/*.rules

alerting:
  alertmanagers:
    - consul_sd_configs:
      - server: '127.0.0.1:8500'
        services:
          - alertmanager

Actions #10

Updated by Ahmed Bilal over 4 years ago

How to change admin password? https://grafana.com/docs/administration/cli/

Actions #11

Updated by Ahmed Bilal over 4 years ago

  • File Ungleich-1570814496841.json added
Actions #12

Updated by Ahmed Bilal over 4 years ago

  • File deleted (Ungleich-1570814496841.json)
Actions #14

Updated by Ahmed Bilal over 4 years ago

Monitoring system is accessible at https://monitoring.place11.ungleich.ch/

Actions #15

Updated by Ahmed Bilal over 4 years ago

  • Assignee changed from Ahmed Bilal to Nico Schottelius
Actions #16

Updated by Nico Schottelius over 4 years ago

  • Status changed from Seen to New
  • Assignee changed from Nico Schottelius to ll nu

Well. done. Balazs, can you confirm/ensure that you can

  • reproduce the setup
  • understand how monitoring is done there
  • can make changes to prometheus and grafana

Additionally, please create a new ticket for creating an email account on the external system and configure alertmanager as follows:

Send alerts to

Actions #17

Updated by ll nu over 4 years ago

  • Status changed from New to Seen
Actions #18

Updated by ll nu over 4 years ago

ABK is added to sre@
imap mailbox creation is pending

Actions #19

Updated by Nico Schottelius almost 4 years ago

  • Assignee changed from ll nu to Nico Schottelius
Actions #20

Updated by Nico Schottelius 3 months ago

  • Status changed from Seen to Rejected
Actions

Also available in: Atom PDF