http://localhost:3000/http://localhost:3000/favicon.ico?16699092332019-09-30T11:12:54Zungleich redmineOpen Infrastructure - Task #7185: Setup network monitoring system on new off-site VPShttp://localhost:3000/issues/7185?journal_id=294032019-09-30T11:12:54ZNico Schotteliusnico.schottelius@ungleich.ch
<ul><li><strong>Related to</strong> <i><a class="issue tracker-5 status-6 priority-2 priority-default closed behind-schedule" href="/issues/7184">Task #7184</a>: Create a mailing list for maintenance notifications</i> added</li></ul> Open Infrastructure - Task #7185: Setup network monitoring system on new off-site VPShttp://localhost:3000/issues/7185?journal_id=294092019-09-30T11:54:40ZNico Schotteliusnico.schottelius@ungleich.ch
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/29409/diff?detail_id=29633">diff</a>)</li></ul> Open Infrastructure - Task #7185: Setup network monitoring system on new off-site VPShttp://localhost:3000/issues/7185?journal_id=294862019-10-04T06:49:19ZAhmed Bilal
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Seen</i></li></ul> Open Infrastructure - Task #7185: Setup network monitoring system on new off-site VPShttp://localhost:3000/issues/7185?journal_id=295002019-10-06T12:47:14ZAhmed Bilal
<ul><li><strong>Assignee</strong> changed from <i>Ahmed Bilal</i> to <i>Nico Schottelius</i></li></ul><p>I have setup (prometheus + grafana + blackbox exporter) on debian.sandbox.ahmedbilal.ungleich.cloud</p>
You can see the Grafana at <a class="external" href="http://debian.sandbox.ahmedbilal.ungleich.cloud:3000">http://debian.sandbox.ahmedbilal.ungleich.cloud:3000</a>
<ul>
<li><b>username</b>: admin</li>
<li><b>password</b>: admin</li>
</ul>
<p>You can see the Prometheus at <a class="external" href="http://debian.sandbox.ahmedbilal.ungleich.cloud:9090">http://debian.sandbox.ahmedbilal.ungleich.cloud:9090</a></p>
<p>What to do next except deploying it to VPS. I assign the DNS name post.ungleich.ch to VPS.</p> Open Infrastructure - Task #7185: Setup network monitoring system on new off-site VPShttp://localhost:3000/issues/7185?journal_id=295042019-10-06T14:19:08ZNico Schotteliusnico.schottelius@ungleich.ch
<ul></ul><p>you can also add another name to the VPS like<br />monitoring.place11.ungleich.ch ;-)</p>
<p><a class="email" href="mailto:redmine@ungleich.ch">redmine@ungleich.ch</a> writes:</p> Open Infrastructure - Task #7185: Setup network monitoring system on new off-site VPShttp://localhost:3000/issues/7185?journal_id=295792019-10-11T17:24:28ZAhmed Bilal
<ul><li><strong>Assignee</strong> changed from <i>Nico Schottelius</i> to <i>Ahmed Bilal</i></li></ul> Open Infrastructure - Task #7185: Setup network monitoring system on new off-site VPShttp://localhost:3000/issues/7185?journal_id=295802019-10-11T19:09:03ZNico Schotteliusnico.schottelius@ungleich.ch
<ul></ul><p>Seems not to be solved:</p>
<p>monitoring.place11.ungleich.ch’s server IP address could not be found.</p>
<p><a class="email" href="mailto:redmine@ungleich.ch">redmine@ungleich.ch</a> writes:</p> Open Infrastructure - Task #7185: Setup network monitoring system on new off-site VPShttp://localhost:3000/issues/7185?journal_id=295812019-10-11T19:09:25ZNico Schotteliusnico.schottelius@ungleich.ch
<ul></ul><p>Ah, sorry, thought it was assigned to me!</p> Open Infrastructure - Task #7185: Setup network monitoring system on new off-site VPShttp://localhost:3000/issues/7185?journal_id=295822019-10-11T19:16:20ZAhmed Bilal
<ul></ul><p>/etc/prometheus/blackbox.yml</p>
<pre>
modules:
http_2xx:
prober: http
http:
http_post_2xx:
prober: http
http:
method: POST
tcp_connect:
prober: tcp
pop3s_banner:
prober: tcp
tcp:
query_response:
- expect: "^+OK"
tls: true
tls_config:
insecure_skip_verify: false
ssh_banner:
prober: tcp
tcp:
query_response:
- expect: "^SSH-2.0-"
irc_banner:
prober: tcp
tcp:
query_response:
- send: "NICK prober"
- send: "USER prober prober prober :prober"
- expect: "PING :([^ ]+)"
send: "PONG ${1}"
- expect: "^:[^ ]+ 001"
icmp:
prober: icmp
</pre>
<p>/etc/prometheus/prometheus<br /><pre>
global:
scrape_interval: 10s # can be overridden by setting scrape_interval in a job
evaluation_interval: 30s # for rules
# Attach these labels to any time series or alerts when communicating with
# external systems (federation, remote storage, Alertmanager).
external_labels:
# monitor: 'place5-prod'
scrape_configs:
- job_name: 'routers'
metrics_path: /probe
params:
module: [icmp]
static_configs:
- targets:
- router1.place6.ungleich.ch
- router2.place6.ungleich.ch
- router1.place5.ungleich.ch
- router2.place5.ungleich.ch
- router3.place5.ungleich.ch
- router4.place5.ungleich.ch
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: post.ungleich.ch:9115
- job_name: 'core-services'
metrics_path: /probe
params:
module: [http_2xx]
static_configs:
- targets:
- https://ungleich.ch
- https://monitoring.place5.ungleich.ch
- https://monitoring.place6.ungleich.ch
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: post.ungleich:9115
rule_files:
- /etc/prometheus/*.rules
alerting:
alertmanagers:
- consul_sd_configs:
- server: '127.0.0.1:8500'
services:
- alertmanager
</pre></p> Open Infrastructure - Task #7185: Setup network monitoring system on new off-site VPShttp://localhost:3000/issues/7185?journal_id=295832019-10-11T19:18:57ZAhmed Bilal
<ul></ul><p>How to change admin password? <a class="external" href="https://grafana.com/docs/administration/cli/">https://grafana.com/docs/administration/cli/</a></p> Open Infrastructure - Task #7185: Setup network monitoring system on new off-site VPShttp://localhost:3000/issues/7185?journal_id=295842019-10-11T19:22:10ZAhmed Bilal
<ul><li><strong>File</strong> <i>Ungleich-1570814496841.json</i> added</li></ul> Open Infrastructure - Task #7185: Setup network monitoring system on new off-site VPShttp://localhost:3000/issues/7185?journal_id=295852019-10-11T19:55:00ZAhmed Bilal
<ul><li><strong>File</strong> deleted (<del><i>Ungleich-1570814496841.json</i></del>)</li></ul> Open Infrastructure - Task #7185: Setup network monitoring system on new off-site VPShttp://localhost:3000/issues/7185?journal_id=295862019-10-11T19:57:04ZAhmed Bilal
<ul><li><strong>File</strong> <a href="/attachments/3134">Ungleich-1570816597910.json</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/3134/Ungleich-1570816597910.json">Ungleich-1570816597910.json</a> added</li></ul> Open Infrastructure - Task #7185: Setup network monitoring system on new off-site VPShttp://localhost:3000/issues/7185?journal_id=296252019-10-18T13:13:20ZAhmed Bilal
<ul></ul><p>Monitoring system is accessible at <a class="external" href="https://monitoring.place11.ungleich.ch/">https://monitoring.place11.ungleich.ch/</a></p> Open Infrastructure - Task #7185: Setup network monitoring system on new off-site VPShttp://localhost:3000/issues/7185?journal_id=296322019-10-18T14:59:49ZAhmed Bilal
<ul><li><strong>Assignee</strong> changed from <i>Ahmed Bilal</i> to <i>Nico Schottelius</i></li></ul> Open Infrastructure - Task #7185: Setup network monitoring system on new off-site VPShttp://localhost:3000/issues/7185?journal_id=296332019-10-18T16:14:31ZNico Schotteliusnico.schottelius@ungleich.ch
<ul><li><strong>Status</strong> changed from <i>Seen</i> to <i>New</i></li><li><strong>Assignee</strong> changed from <i>Nico Schottelius</i> to <i>ll nu</i></li></ul><p>Well. done. Balazs, can you confirm/ensure that you can</p>
<ul>
<li>reproduce the setup</li>
<li>understand how monitoring is done there</li>
<li>can make changes to prometheus and grafana</li>
</ul>
<p>Additionally, please create a new ticket for creating an email account on the external system and configure alertmanager as follows:</p>
<p>Send alerts to</p>
<ul>
<li><a class="email" href="mailto:sre@ungleich.ch">sre@ungleich.ch</a> (our standard list; also add ahmed here, if he isn't in it)</li>
<li><a class="email" href="mailto:sre@place11.ungleich.ch">sre@place11.ungleich.ch</a> -> new imap mailbox that we all can add to our phone</li>
</ul> Open Infrastructure - Task #7185: Setup network monitoring system on new off-site VPShttp://localhost:3000/issues/7185?journal_id=302562019-12-09T11:46:37Zll nu
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Seen</i></li></ul> Open Infrastructure - Task #7185: Setup network monitoring system on new off-site VPShttp://localhost:3000/issues/7185?journal_id=302982019-12-09T15:02:58Zll nu
<ul></ul><p>ABK is added to sre@<br />imap mailbox creation is pending</p> Open Infrastructure - Task #7185: Setup network monitoring system on new off-site VPShttp://localhost:3000/issues/7185?journal_id=328622020-06-11T09:36:01ZNico Schotteliusnico.schottelius@ungleich.ch
<ul><li><strong>Assignee</strong> changed from <i>ll nu</i> to <i>Nico Schottelius</i></li></ul> Open Infrastructure - Task #7185: Setup network monitoring system on new off-site VPShttp://localhost:3000/issues/7185?journal_id=521932024-01-03T08:51:39ZNico Schotteliusnico.schottelius@ungleich.ch
<ul><li><strong>Status</strong> changed from <i>Seen</i> to <i>Rejected</i></li></ul>