The ungleich monitoring infrastructure 2024 » History » Revision 6
Revision 5 (Nico Schottelius, 12/21/2023 01:17 PM) → Revision 6/8 (Nico Schottelius, 12/25/2023 12:30 PM)
h1. The ungleich monitoring infrastructure 2024 (WIP)
{{toc}}
h2. Intro
This is a work-in-progress update from [[The_ungleich_monitoring_infrastructure]]. The infrastructure is still based on prometheus + blackbox exporter, but now also makes use of kubernetes native objects.
h2. Monitoring definition
h3. External primary router/link monitoring
* Objective: find out from an external PoV whether the lines are functioning
* Implementation:
** Collecting/alerting with prometheus on place12
** blackbox on place12
** blackbox on place11
* Targets
** ipv6/router1.place10/snr
** ipv4/router1.place10/snr
** ipv6/server12X.place10/snr
** ipv4/server12X.place10/snr
** ipv4/fiberstream/place5
** ipv4/fiberstream/place6
** ipv4/fiberstream/place7
** ipv4/fiberstream/place10
* Status: TBD
h3. External primary router
* Objective: find out whether a router is reachable via any path
* Implementation:
** Collecting/alerting with prometheus on place12
** blackbox on place12
** blackbox on place11
h3. Test external monitoring
* Objective: find out whether the external monitoring is alive
* Implementation:
** Collecting/alerting with prometheus on place10
* Targets
** ipv6/emonitor1.place12/prometheus
** ipv6/emonitor1.place12/blackbox
** ipv6/emonitor1.place12/alertmanager
** ipv6/vm1.place11/blackbox
h3. Test per place monitoring infrastructure (blackbox exporter, prometheus)
Each place should provide a blackbox exporter suitable for monitoring onsite targets.
We need to ensure that these blackbox exporters all function and that prometheus instances are up.
* Objective: find out whether the onsite monitoring is alive
* Implementation:
** Collecting/alerting with prometheus on place12
* Targets
** blackbox-exporter + prometheus/place5
** blackbox-exporter + prometheus/place6
** blackbox-exporter + prometheus/place10
h3. Internal internal router monitoring (TBD)
Per place monitor internal routers
* Objective: find out whether the internal monitoring is alive
* Implementation:
** Collecting/alerting with prometheus on ...
* Targets
** ipv6/apu-router1.place6 (via place10/blackbox)
h3. Internal network device monitoring
* Objective: find out whether all production switches are alive
* Implementation:
** Dedicated blackbox_exporter on a router or similar (needs to be secured)
* Targets
** All Arista in each place
** All Mikrotik in each place