Open Infrastructure

This service will run on EVERY vm host and will do the heavy lifting / creating of VMs

For v1, implement the following:

• Watch etcd on key prefix /v1/vm/
• If there is a change, check whether
  • an existing VM on the current host should be killed
  • a new VM should be started
  • On event emit an email to sre@ (mail should be injected by env) -- just initially, later this will be disabled
• Create a VM
  • if there is a new json entry in /v1/vm with a VM that is designated for this host and has not been started
  • Use qemu ... for it (sample below)
• Delete a VM
  • If the status json entry of a VM is TO_STOP, stop it
  • After stopping was successful, put it into STOP
  • We need to register the runtime somewhere (v2)
• Monitor a VM
  • If it is still running
  • If it is not running anymore, notify sysadmins
    • Write to /v1/log/
    • And/or send email to sre@
    • And restart the VM

For v2 implement:

• Every 5 minutes poll etcd
• Check whether the VMs that are supposed to run on this host are running
  • If one missing: send mail to sre@
• Check that not OTHER VMs are running on this host
  • If there is one that doesn't belong here: send mail to sre@

Sample qemu commands from production¶

• This is way too long, we can trim it down.
• We will also use the -name attribute
• We will also use ceph and smp
• We will also use uuid
• We will run VNC
  • v1: tcp
  • v4: unix sockets

oneadmin 3641289  0.5  0.1 2577776 407572 ?      Sl   Jun21  59:10 /usr/bin/qemu-system-x86_64 -name guest=one-24442,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-24-one-24442/master-key.aes -machine pc-i440fx-2.8,accel=kvm,usb=off,dump-guest-core=off -m 1024 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid d902efb7-2c66-4685-8914-95d38a717c90 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-24-one-24442/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -object secret,id=virtio-disk0-secret0,data=PR218Q61gRpXffWtRuIgQntokENHbAaBJGDDG66u3lg=,keyid=masterKey0,iv=XUHyWqKdHWGuva4eX+5qPw==,format=base64 -drive file=rbd:one/one-449-24442-0:id=libvirt:auth_supported=cephx\;none:mon_host=ceph1\:6789\;ceph2\:6789\;ceph3\:6789,file.password-secret=virtio-disk0-secret0,format=raw,if=none,id=drive-virtio-disk0,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/var/lib/one//datastores/100/24442/disk.1,format=raw,if=none,id=drive-ide0-0-0,readonly=on -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,fd=35,id=hostnet0,vhost=on,vhostfd=37 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=02:00:b3:39:79:75,bus=pci.0,addr=0x3 -vnc [::]:4162 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on