How to become an ungleich system engineer


This document is IN PROGRESS.


If you are interested in the skill set a typical engineer at ungleich needs to have or if you want to improve your skill set for being able to apply at as system engineer, this document is for you.
This document is structured into topics that you should be able to handle. With one part that describes how to learn that section and one part on how to verify you have learned the right things.

Operating System Knowledge

If you are fluent in either Linux or BSD, you should easily find your way. If you are not yet, we recommend:

  • Install your own computer with Ubuntu (any version)
  • Get used to it
  • Wipe Ubuntu, replace it with Arch Linux
  • Use it, break it
  • Switch to Alpine Linux
    • Understand apk and openrc
  • (optional) Use another machine and install Centos
    • Observe the differences to all other operating systems you tried before

Networking / IPv6 Knowledge

You know how to separate networks, you are fluent in IPv6 (hint: man sipcalc) and you know some basics of routing ()

Learning IPv6 / Networking

  • Ensure that your computer has an IPv6 address (read How to get IPv6)
  • Ensure that all your computers have GUA (hint: use a VPN, size >= /48 preferred)
  • Configure a firewall to
    • all forwarding traffic from your internal network(s)
    • all traffic to the ssh port of your firewall/router
    • disallow all other incoming and forwarding traffic

Some questions to check your knowledge:

  • Why does the gateway has to be in the same network?
  • How do IPv6 hosts get their IP addresses?
  • Why is there an fe80:... default gateway in IPv6?
  • Why is IPv4 NAT not a security feature?

Ceph Knowledge

You can install, configure and maintain ceph clusters on IPv6 only networks.


  • Take 3 hosts (if you don't have 3 computers, you can get 3 affordable VMs at IPv6OnlyHosting

Verify questions:

  • Why does ceph need managers?
  • How to move an OSD to another host?

Monitoring Knowledge

Learning monitoring

  • Install nagios or icinga, monitor another host
  • Install collectd and compare what it monitors to icinga and nagios
  • Setup prometheus, node-exporter, grafana and monitor 3+ nodes

Verify questions:

  • Why is nagios alone not a good solution?
  • What is the difference between nagios and icinga?
  • What is the difference between icinga and check_mk?
  • Which things do you monitor? Why?

Updated by Nico Schottelius over 2 years ago · 5 revisions