Open Infrastructure

h1. ucloud (To Be Continued)

{{toc}}

h2. Installation

**Note:** This installation guide assumes that the ~ is /root

* **Install QEMU**

<pre><code class="shell">

sudo apt install qemu qemu-kvm

</code></pre>

* **Create Directory Structure**

<pre><code class="shell">

mkdir /var/vm/

mkdir /var/www/

</code></pre>

* **Create User Directories**

<pre><code class="shell">

mkdir /var/www/ahmedbilal-admin

mkdir /var/www/nico

mkdir /var/www/kjg

cd ~

</code></pre>

* **Download and place Alpine Linux image in _/var/www/ahmedbilal-admin_**

<pre><code class="shell">

wget https://www.dropbox.com/s/5eyryhxun6847hx/alpine.zip?dl=1 -O alpine.zip

unzip alpine.zip

mv alpine.qcow2 /var/www/ahmedbilal-admin

</code></pre>

* **Clone repos**

<pre><code class="shell">

git clone --recurse-submodules git@code.ungleich.ch:ungleich-public/ucloud-api.git

git clone --recurse-submodules git@code.ungleich.ch:ungleich-public/ucloud-scheduler.git

git clone --recurse-submodules git@code.ungleich.ch:ungleich-public/ucloud-vm.git

git clone --recurse-submodules git@code.ungleich.ch:ungleich-public/ucloud-file-scan.git

git clone --recurse-submodules git@code.ungleich.ch:ungleich-public/ucloud-image-scanner.git

</code></pre>

h2. *+Put related .env inside all the above directories+*

* **Run "ucloud-api" as daemon**

<pre><code class="shell">

cd ucloud-api

pipenv install

pipenv run gunicorn --bind [::]:80 main:app --daemon

wget https://www.dropbox.com/s/lyu3atymxyw3846/setup.py?dl=1 -O setup.py

pipenv run python setup.py

cd ~

</code></pre>

* **Run "ucloud-scheduler" as daemon**

<pre><code class="shell">

cd ucloud-scheduler

pipenv install

pipenv run python main.py &>/dev/null &

cd ~

</code></pre>

* **Run "ucloud-vm" as daemon**

<pre><code class="shell">

cd ucloud-vm

pipenv install

pipenv run python main.py /v1/host/1 --vm True &>/dev/null &

cd ~

</code></pre>

* **Install ucloud-file-scan**

<pre><code class="shell">

cd ucloud-file-scan

pipenv install

cd ~

</code></pre>

* **Install ucloud-image-scanner**

<pre><code class="shell">

cd ucloud-image-scanner

pipenv install

cd ~

</code></pre>

* **Run @crontab -e@ and put these entries there and save by pressing @Ctrl+x@ then @y@ then @enter@**

<pre>

*/1 * * * * (cd /root/ucloud-file-scan && /usr/local/bin/pipenv run python main.py)

*/1 * * * * (cd /root/ucloud-image-scanner/ && /usr/local/bin/pipenv run python main.py)

</pre>

h2. ucloud-api

Outside world (indirect) communication with the internal systems.

Its responsibilities are

* Create VM *(Done)* POST _/vm/create_ 

<pre><code class="javascript">

{

	"name": "username",

	"realm": "user_realm",

	"seed": "user_seed",

	"specs": {

		"cpu": 16,

		"ram": 256,

		"hdd": 2000,

		"ssd": 256

	}

}

</code></pre>

* Delete VM *(Done)* POST _/vm/delete_

<pre><code class="javascript">

{

	"name": "username",

	"realm": "user_realm",

	"seed": "user_seed",

	"vmid": "id_of_virtual_machine"

}

</code></pre>

* Status VM *(Done)* GET _/vm/status_

<pre><code class="javascript">

{

	"id": "id_of_virtual_machine"

}

</code></pre>

* Create new network

* Attach network to VM

* Detach network from VM

* Delete network

h2. ucloud-scheduler

It schedules/reschedules VM that are created using ucloud-api. How does it schedules? It does by watching any changes under the _/v1/vm/_ prefix. Basically, it deals with two ETCD entries

1. Virtual Machines Entries that looks like

<pre><code class="javascript">

/v1/vm/1

{

  "owner": "ahmedbilal-admin",

  "specs": {

    "cpu": 20,

    "ram": 2,

    "hdd": 10,

    "ssd": 10

  },

  "hostname": "",

  "status": "REQUESTED_NEW" 

}

</code></pre>

2. Hosts Entries that look like

<pre><code class="javascript">

/v1/host/1

{

    "cpu": 32,

    "ram": 128,

    "hdd": 1024,

    "ssd": 0

    "status": "UP"

}

</code></pre>

It loop through list of host entries found under _/v1/host/_ and check whether we can run the incoming VM on that host. if we can, then it (ucloud-scheduler) sets the hostname key of incoming VM to that host. Suppose, our scheduler decides that we can run the earlier shown VM _/v1/vm/1_ on _/v1/host/1_. Then, our updated entry would look like

<pre><code class="javascript">

/v1/vm/1

{

  "owner": "ahmedbilal-admin",

  "specs": {

    "cpu": 20,

    "ram": 2,

    "hdd": 10,

    "ssd": 10

  },

  "hostname": "/v1/host/1",

  "status": "REQUESTED_NEW" 

}

</code></pre>

*Note* the hostname key/value pair.

h2. ucloud-vm

It is responsible for creating (running) / deleting (stopping) / monitoring virtual machines

It watches the _/v1/vm/_ prefix for any changes like (Virtual Machines with status like *SCHEDULED_DEPLOY* or *REQUEST_DELETE*.

On, *SCHEDULED_DEPLOY* it creates the VM and run it. On, *REQUEST_DELETE* it stops the VM. 

It is also responsible for monitoring VM statues i.e whether they are RUNNING or KILLED (in action). If a VM is killed we put that in log file, notify system administrator and restart it.

h2. Administrator Manual

(being written: how to setup the ucloud infrastructure)

h3. Requirements

ucloud is an opinionated cloud management framework. It is made to solve the problem of "virtual machine hosting" including migrations, scale out, etc. While there are many different ways on how to run, store and distribute VMs, there are not that many ways that work reliable and scale out. For this reason ucloud is based on existing software that has proven to be working for virtual machine hosting.

The following components are required for running ucloud:

1. Python 3.7, pip, pipenv

2. etcd

3. ceph

Setting up these components is out of scope of this manual.

h2. User Manual

(being written: how to use ucloud as a user)

h3. Creating a VM

(to be filled in by ahmed)

h3. Deleting a VM

h2. API reference

(to be moved here by Ahmed)