Activity
From 01/07/2020 to 02/05/2020
02/04/2020
- NS 05:28 PM Task #7688: Disable rp_filter on router2.place5 (alpine sets =1 on .all and .default)
- Added the following to our internal cdist:
- NS 05:19 PM Task #7688 (In Progress): Disable rp_filter on router2.place5 (alpine sets =1 on .all and .default)
- NS 05:15 PM Task #7688 (Closed): Disable rp_filter on router2.place5 (alpine sets =1 on .all and .default)
- DR 05:24 PM Task #7689 (Seen): Update certbot on several VMs
- DR 05:24 PM Task #7689 (Rejected): Update certbot on several VMs
- From Mail (letsencrypt):
02/03/2020
01/31/2020
- AB 07:16 PM Task #7649: Sketch a VM backup & restore
- The functionality part is complete beside `ungleich-cli` and can be found at https://code.ungleich.ch/ahmedbilal/vm-cli-services. I am ready to give a test-run tour whenever you want.
- AB 07:13 PM Task #7650: Synchronise opennebula VMs with etcd
- I changed the whole engine, so it is not slow anymore. Previously, I am using pyone library provided by OpenNebula that is in simple words full of bugs. Now, I am directly using XML-RPC client builtin Python to make remote procedural cal...
- NS 04:16 PM Task #7304 (Rejected): Test NAT64 with distributed routers with joold on alpine
- Active-passive
- NS 04:16 PM Task #7377 (Rejected): Create an active-active NAT64 gateway
- Going active-passive.
01/28/2020
- AB 02:38 PM Task #7650: Synchronise opennebula VMs with etcd
- Nico its done. https://code.ungleich.ch/ungleich-public/ungleich-tools/tree/master/opennebula-vm-etcd
1. Putting VM info into etcd **put-vm-info-into-etcd.py**
2. VM Queries Example Script **vm-queries.py**
although slow as I disc... - NS 10:36 AM Task #7654: Get VMs info from Opennebula and save it in etcd
- ping sre team
- NS 10:35 AM Task #7654: Get VMs info from Opennebula and save it in etcd
- clearly a cron job.
Let's create a new (IPv6 only) service VM based on Alpine, which is
dedicated for running stateless scripts (via cron or other means).
=> service.ungleich.ch
We might eventually also use docker on this VM, just as ... - AB 09:31 AM Task #7654: Get VMs info from Opennebula and save it in etcd
- We can run it either as cron job or put a sleep in the code.
01/27/2020
- NS 09:29 PM Task #7654: Get VMs info from Opennebula and save it in etcd
- Where does it run / update every 10 minutes?
- AB 05:37 PM Task #7654: Get VMs info from Opennebula and save it in etcd
- Done
- AB 10:59 AM Task #7654 (Rejected): Get VMs info from Opennebula and save it in etcd
- Create a service that gets VMs' information from OpenNebula and save it in etcd.
It should also update/sync this information every 10 minutes. - LN 04:58 PM Task #7632: Setup rados / s3 storage on ceph
- NS 01:25 PM Task #7650: Synchronise opennebula VMs with etcd
- I actually created this ticket some time ago
01/26/2020
- TF 05:56 PM Task #7641: create images for uncloud
- * The image definition scripts are defined in: https://code.ungleich.ch/uncloud/images
* The images have been uploaded to ONE and are available under the uncloud-* VM templates.
* The Fedora and Ubuntu images works fine but I still hav... - NS 10:02 AM Task #7653 (In Progress): Move VMs with routed /64 into their own /64 [was: Instructions for adding /64 to VM]
- Good point.
Timothee, can you create a new network for VMs with routed networks so that we can again decide on DNS64 in bind by network prefix?
Morris, we will fix it the way that the user does not have to do that manually. Above n...
01/25/2020
- MJ 05:01 PM Task #7653 (Closed): Move VMs with routed /64 into their own /64 [was: Instructions for adding /64 to VM]
- Document process for moving VM to /64 rather than /128.
Importantly: customer should modify their name server config (/etc/resolv.conf) to be as follows:
nameserver 2a0a:e5c0:2:12:0:f0ff:fea9:c451
nameserver 2a0a:e5c0:2:12:0:f0ff:...
01/24/2020
- NS 11:18 AM Task #7649: Sketch a VM backup & restore
- h2. OpenNebula testing
- NS 11:16 AM Task #7649: Sketch a VM backup & restore
- Snapshotting:
- NS 10:49 AM Task #7649: Sketch a VM backup & restore
- h2. Ceph
It seems ceph even has a limit support: - NS 10:16 AM Task #7649 (Rejected): Sketch a VM backup & restore
- * User wants to have possibility to roll back in time
h2. Implementation
* We can use ceph snapshots
* We might need/want to signal qemu before taking the snapshot
** how?
** Difference between uncloud and opennebula?
h2. Use... - NS 11:08 AM Task #7650 (Rejected): Synchronise opennebula VMs with etcd
- * etcd prefix: /opennebula
*
h2. What to synchronise
* /opennebula/vm/<VM id>
** all information about the VM
** also terminated VMs
h2. Use cases
I want to be able to
* Get the current host of the VM
* Get the VNC po... - NS 10:03 AM Task #7636: Find out current retention period for monitoring servers and ensure that data is kept for 5 years
- Go with it
redmine@ungleich.ch writes:
01/23/2020
- DR 10:57 PM Task #7636: Find out current retention period for monitoring servers and ensure that data is kept for 5 years
- it's configured in __dcl_monitoring_server type
- NS 05:53 PM Task #7632: Setup rados / s3 storage on ceph
- Note for production checking: what happens/is required to do if the
certificate is replaced with a new one?
redmine@ungleich.ch writes: - LN 05:46 PM Task #7632: Setup rados / s3 storage on ceph
- updated conf with ssl
[client.rgw.rgw]
host = rgw
#rgw socket path = /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock
rgw socket path = /tmp/radosgw.sock
rgw frontends = beast ssl_port=443 ssl_certificate=/etc/letsencrypt/live/rgw.... - LN 05:30 PM Task #7632: Setup rados / s3 storage on ceph
- Radosgw is running. The correct way to start is1:
/usr/bin/radosgw --cluster ceph --name client.rgw.`hostname -s` -setuser ceph --setgroup ceph
atm I'm tweaking the settings to setup ssl, and ipv6 support
1: (it wasnt writte... - LN 01:41 PM Task #7632: Setup rados / s3 storage on ceph
- using this manual the radosgw is configured at rgw.llnu.at
https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3/html/installation_guide_for_red_hat_enterprise_linux/manually-installing-ceph-object-gateway
We don't h...
01/22/2020
- NS 03:33 PM Task #7630: Cleanup the DNS64 situation
- No. If somebody does not want that, they should remove the appropriate
script below /etc/one-contex.d
redmine@ungleich.ch writes: - LN 02:00 PM Task #7630: Cleanup the DNS64 situation
- What can we do about that the /etc/resolv.conf is rewritten at every reboot?
Would the best be to advise:
chattr +i /etc/resolv.conf
? - NS 09:11 AM Task #7630 (Closed): Cleanup the DNS64 situation
Thanks!
redmine@ungleich.ch writes:
01/21/2020
- NS 09:42 PM Task #7632: Setup rados / s3 storage on ceph
redmine@ungleich.ch writes:- LN 07:55 PM Task #7632 (Seen): Setup rados / s3 storage on ceph
- LN 07:55 PM Task #7631 (Seen): Report details about pleroma problems to upstream
- TF 02:16 PM Task #7641 (In Progress): create images for uncloud
- SK 12:38 PM Task #7641 (Closed): create images for uncloud
- First images should be the latest alpine, fedora, ubuntu, debian.
How should they be configured:
they should get an ipv6 address from the first network interface,
they should automatically increase the root file system, if the di... - TF 10:57 AM Task #7545 (In Progress): Switch production LDAPs to cdist-managed alpine
- TF 10:32 AM Task #7483 (In Progress): Update the __consul cdist type for alpine
- I got the same issue with `__consul_agent` on Debian, which I am currently fixing upstream.
https://code.ungleich.ch/ungleich-public/cdist/merge_requests/837
01/20/2020
- TF 05:11 PM Task #7630 (Feedback): Cleanup the DNS64 situation
- See https://redmine.ungleich.ch/projects/open-infrastructure/wiki/The_ungleich_DNS_infrastructure.
- TF 12:06 PM Task #7630 (Seen): Cleanup the DNS64 situation
- NS 11:03 AM Task #7630 (Closed): Cleanup the DNS64 situation
- h2. Old situation
* bind nameservers on routers decide based on source IPv6 address whether to give out NAT64 or not
* Overlapping use (dual stack VM vs. IPv6 only) led to problems that the above rule does not apply strictly
* This ... - NS 03:19 PM Task #7636 (Closed): Find out current retention period for monitoring servers and ensure that data is kept for 5 years
- * Looking at monitoring.place6 I see data for less than 90 days.
* My expectation is to be able to zoom out to 5 years so that we can see changes we did over years
* My assumption is that prometheus is configured with some storage siz... - NS 03:06 PM Task #7635 (Closed): Create a simple page explaining DNS64/NAT64 for customers
- * So that we can reference it in support tickets.
* Include exapmles, how to reach github, show the AAAA record resolution, explain how it works if there is already IPv6 for a domain
In simple words (maybe +graphviz/dot image showing... - TF 12:06 PM Task #6694 (Closed): Setup matrix server and bridge matermost into it
- Relevant channels have been bridged. Closing.
- TF 12:05 PM Task #7560 (In Progress): Document DNS64 setup for VMs
- TF 12:05 PM Task #7496 (Closed): Create 2 new IPv6 only unbound based resolving DNS servers providing DNS64
- Unbound DNS(64) servers are now monitored by the prometheus blackbox exporter. Closing.
- NS 11:35 AM Task #7632 (Closed): Setup rados / s3 storage on ceph
- * Including permissions
* Document the setup
* Document how to use it - NS 11:04 AM Task #7631 (Closed): Report details about pleroma problems to upstream
- Follow up with https://git.pleroma.social/pleroma/pleroma-support/issues/10#note_49605
* Create an IPv6 only VM
* Recreate the problem
* Keep the VM running for until the problem has been fixed upstream
Please follow up today so ... - AB 09:27 AM Task #7629 (Rejected): Add referral link system in dynamicweb (DCL, IPv6OnlyHosting etc)
- Referral links are used to reward user/(reviewing website) whenever someone purchase VM (or other service) using their referral link.
This ticket is created to figure out how to implement referral link system.
01/19/2020
- TF 02:48 PM Task #7543 (Closed): Write image definition script for ubuntu 19.10
- The image has been deployed in ONE and configured for the `public-Ubuntu 19.10` and `ipv6only-Ubuntu 19.10` templates. Defined by the "ubuntu-build-opennebula-image.sh script in ungleich-tools":https://code.ungleich.ch/ungleich-public/un...
01/18/2020
- TF 07:39 PM Task #7496: Create 2 new IPv6 only unbound based resolving DNS servers providing DNS64
- It's deployed: there's just monitoring to setup before it can be closed.
- NS 01:35 PM Task #7625: Manually fix consul+node_exporter on new router1.place6
- NS 01:27 PM Task #7625: Manually fix consul+node_exporter on new router1.place6
- NS 01:26 PM Task #7625: Manually fix consul+node_exporter on new router1.place6
- Use alpine's init script:
- NS 01:02 PM Task #7625 (Rejected): Manually fix consul+node_exporter on new router1.place6
01/15/2020
- LN 07:28 PM Task #6671: Setup mastodon/pleroma for ungleich
- could we have a 13373r name?
01/13/2020
- NS 07:30 PM Task #7604 (In Progress): Find out why ciara2 was not automatically detected to be offline
- * ciara2 is half correctly outside of the consul cluster
** It should actually still be inside the cluster, but marked dead
- NS 07:28 PM Task #7604 (Rejected): Find out why ciara2 was not automatically detected to be offline
- * Consul status / prometheus / alert manager should have noticed
- MJ 06:37 PM Task #7186: Add support for general VPN including IPv4
- Errrr what is it with your VPN pricing? Did you go skiing and get altitude sickness?
Market price for VPN services is $5 - $12 per month.
The high end services offer multiple server locations in every continent and dedicated servers fo... - MJ 05:52 PM Task #7544: Write "beginner's guide" for datacenterlight customers
- -IPv6 and IPv4: making the services on my IPv6 VM visible to the IPv4 world
-Guido to VM Management tools: dashboard/django, ungleich-cli, cdist, ucloud
-Reverse DNS PTR entries
-Using my own IPv6 subnet e.g. /64 - NS 12:01 PM Task #7602 (Rejected): Align dynamicweb / opennebula with uncloud
- Stuff that we can & should export from our current setup to etcd in an uncloud alike format:
Prefix for everything is /dynamicweb-opennebula
* user public ssh keys (/dynamicweb-opennebula/user-keys)
* List of VMs (/dynamicweb-open... - NS 11:57 AM Task #7601: Setup an SSH jump host
- Proxycommand w/ windows exists in putty and usually uses plink - more details soon.
- TF 11:54 AM Task #7601 (Seen): Setup an SSH jump host
- * I am familiar with LDAP-backed auth with nslcd.
* ProxyCommand is standard for SSH bastions, it is even available on windows with Putty: https://fedora-infra-docs.readthedocs.io/en/latest/sysadmin-guide/sops/sshaccess.html#putty-ssh-c... - NS 11:12 AM Task #7601 (Rejected): Setup an SSH jump host
- * Authenticated against our ldap
* Allows user to connect to our IPv6 networks
The ways for users to use it:
* via ProxyCommand (some might be able to use that)
I've setup this some time ago and it basically needs a restriction... - AB 08:37 AM Task #7555 (Closed): Setup uncloud at server11 and server12
- AB 03:14 AM Task #7582 (Closed): Add hostname in uncloud file scanning
- AB 03:14 AM Task #7519 (Closed): uncloud test run 2019-12-21
- The above mentioned things were fixed.
01/12/2020
- NS 09:18 PM Task #7580: Preparing for matrix-as-a-service
- channels that can be exported to IRC or matrix:
* ipv6
* foss
* hacking-and-learning
* Town Square
* datacenterlight
* uncloud
More maybe later - TF 09:11 PM Task #7580 (In Progress): Preparing for matrix-as-a-service
- TF 09:10 PM Task #7580: Preparing for matrix-as-a-service
- * Synapse and Matrix Cdist types are (almost) OK.
- I missed one small thing in my __postgres upstream cdist patch, which still have to be fixed.
* Early documentation on https://redmine.ungleich.ch/projects/open-infrastructure/wiki/... - TF 09:12 PM Task #6694: Setup matrix server and bridge matermost into it
- Everything's in place, we just have to choose the channels to be bridged.
01/09/2020
- AB 08:56 PM Task #7596: uncloud-api refactoring & make schemas less horrible
- To see merge request https://code.ungleich.ch/uncloud/uncloud/merge_requests/1
- AB 08:54 PM Task #7596: uncloud-api refactoring & make schemas less horrible
- * Done `uncloud api` refactoring which was due for a long time and is the last refactoring beside https://redmine.ungleich.ch/issues/7590.
* Schemas has been greatly simplified (now approaching to beautiful code :) and are now pleasant ... - AB 08:54 PM Task #7596 (Closed): uncloud-api refactoring & make schemas less horrible
- AB 08:55 PM Task #7585 (Closed): Check whether uncloud-api break if some field is missing
- The behavior is verified and corrected.
- AB 09:32 AM Task #7591: uncloud production checklist 2020-01
- Also, please note
uncloud deployed at server{11, 12}. There are still some issues that would be problem for reliably running uncloud for longer period of time. e.g
1. https://redmine.ungleich.ch/issues/7583 (As, soon as etcd lead... - AB 09:26 AM Task #7591: uncloud production checklist 2020-01
- *Can all required components be deployed (checking on server11) -- document the installation procedures*
Documented
*Is the API secure from outside? I am able to connect without otp at the moment*
I am not sure about what do you... - AB 09:19 AM Task #7591: uncloud production checklist 2020-01
- h1. Installation/Setup
Allow etcd prefix for developer role - NS 09:17 AM Task #7591 (In Progress): uncloud production checklist 2020-01
- NS 09:16 AM Task #7591 (Rejected): uncloud production checklist 2020-01
- h2. Objective
* Migrate internal VMs to uncloud
h2. Checklist
* Can all required components be deployed (checking on server11) -- document the installation procedures
** -api
** -host
** -network
* Is the API secure from out... - AB 08:55 AM Task #7590: Expect everything to fail (uncloud)
- I am little uncertain how to handle failures in etcd. For Example, put every etcd function call in try/except block or do something else.
- AB 08:48 AM Task #7590: Expect everything to fail (uncloud)
- h2. How do we plan to handle failures in etcd
Failures can be temporarily (leadership change) or permanent (etcd cluster dies)
We use four type of functions of etcd
1. get
2. put
3. get_prefix
4. watch_prefix
h3. get
can ... - AB 08:48 AM Task #7590 (Rejected): Expect everything to fail (uncloud)
- Especially external components, such as
1. etcd
2. netbox
3. otp.ungleich.ch
01/08/2020
- AB 08:21 PM Task #7583: Handle etcd leader change or temporary unavailability gracefully in uncloud
- We have to re-evaluate/re-check all the usage of etcd in uncloud to make sure we handle these events correctly/gracefully.
I have modified few things in get_prefix/watch_prefix that would help to correctly/gracefully handle etcd event... - AB 07:34 AM Task #7583: Handle etcd leader change or temporary unavailability gracefully in uncloud
- The later unavailability is due to election for leader.
- TF 12:21 PM Task #7580: Preparing for matrix-as-a-service
- We can do-it in a second-stage, witout a TURN server VoIP might or might not work depending on the situation.
Note that coturn is easy to deploy: https://github.com/matrix-org/synapse/blob/master/docs/turn-howto.md - NS 11:27 AM Task #7580: Preparing for matrix-as-a-service
- Can we do voip in a second stage or will things "look weird" without it?
redmine@ungleich.ch writes: - TF 11:17 AM Task #7580: Preparing for matrix-as-a-service
- We'll also need a TURN server for VoIP.
01/07/2020
- AB 06:38 PM Task #7555: Setup uncloud at server11 and server12
- uncloud deployed at server{11, 12}. There are still some issues that would be problem for reliably running uncloud for longer period of time. e.g
1. https://redmine.ungleich.ch/issues/7583 (As, soon as etcd leader changes or etcd become... - AB 06:33 PM Task #7582: Add hostname in uncloud file scanning
- Nico Schottelius wrote:
> For the example above: the host would probably often more something like "files1.datacenterlight.ch" or "username.datacenterlight.ch" or so.
Yeah, it is just my local machine. So, i put my IPv6 address direc... - NS 04:19 PM Task #7582: Add hostname in uncloud file scanning
- For the example above: the host would probably often more something like "files1.datacenterlight.ch" or "username.datacenterlight.ch" or so.
- NS 04:18 PM Task #7582: Add hostname in uncloud file scanning
- We should have a "created_at" and "deleted_at" for every object.
- AB 02:44 PM Task #7582: Add hostname in uncloud file scanning
- Done. Sample
Before - AB 01:41 PM Task #7582 (Closed): Add hostname in uncloud file scanning
- It is required as nico said that there would be multiple file hosts and not necessarily all files are available on some particular host.
https://chat.ungleich.ch/ungleich/pl/3pf77f1ui7yiupxjyqayzf67ry - AB 04:45 PM Task #7585 (Closed): Check whether uncloud-api break if some field is missing
- The behavior is seen in the past that uncloud-api breaks if we don't pass some fields like *name*, *realm* or *token* etc.
- AB 02:07 PM Task #7583 (Rejected): Handle etcd leader change or temporary unavailability gracefully in uncloud
- Here is leader change.
- TF 12:53 PM Task #7580 (Closed): Preparing for matrix-as-a-service
- Once matrix is deployed at ungleich:
* Build & document MaaS deployment and maintenance pipeline.
- Wiki page.
- A staging environment will be required to test upgrades.
* 1 or 2 blog entries about it? First one maybe a bit mor...