Activity
From 12/14/2019 to 01/12/2020
01/12/2020
- NS 09:18 PM Task #7580: Preparing for matrix-as-a-service
- channels that can be exported to IRC or matrix:
* ipv6
* foss
* hacking-and-learning
* Town Square
* datacenterlight
* uncloud
More maybe later - TF 09:11 PM Task #7580 (In Progress): Preparing for matrix-as-a-service
- TF 09:10 PM Task #7580: Preparing for matrix-as-a-service
- * Synapse and Matrix Cdist types are (almost) OK.
- I missed one small thing in my __postgres upstream cdist patch, which still have to be fixed.
* Early documentation on https://redmine.ungleich.ch/projects/open-infrastructure/wiki/... - TF 09:12 PM Task #6694: Setup matrix server and bridge matermost into it
- Everything's in place, we just have to choose the channels to be bridged.
01/09/2020
- AB 08:56 PM Task #7596: uncloud-api refactoring & make schemas less horrible
- To see merge request https://code.ungleich.ch/uncloud/uncloud/merge_requests/1
- AB 08:54 PM Task #7596: uncloud-api refactoring & make schemas less horrible
- * Done `uncloud api` refactoring which was due for a long time and is the last refactoring beside https://redmine.ungleich.ch/issues/7590.
* Schemas has been greatly simplified (now approaching to beautiful code :) and are now pleasant ... - AB 08:54 PM Task #7596 (Closed): uncloud-api refactoring & make schemas less horrible
- AB 08:55 PM Task #7585 (Closed): Check whether uncloud-api break if some field is missing
- The behavior is verified and corrected.
- AB 09:32 AM Task #7591: uncloud production checklist 2020-01
- Also, please note
uncloud deployed at server{11, 12}. There are still some issues that would be problem for reliably running uncloud for longer period of time. e.g
1. https://redmine.ungleich.ch/issues/7583 (As, soon as etcd lead... - AB 09:26 AM Task #7591: uncloud production checklist 2020-01
- *Can all required components be deployed (checking on server11) -- document the installation procedures*
Documented
*Is the API secure from outside? I am able to connect without otp at the moment*
I am not sure about what do you... - AB 09:19 AM Task #7591: uncloud production checklist 2020-01
- h1. Installation/Setup
Allow etcd prefix for developer role - NS 09:17 AM Task #7591 (In Progress): uncloud production checklist 2020-01
- NS 09:16 AM Task #7591 (Rejected): uncloud production checklist 2020-01
- h2. Objective
* Migrate internal VMs to uncloud
h2. Checklist
* Can all required components be deployed (checking on server11) -- document the installation procedures
** -api
** -host
** -network
* Is the API secure from out... - AB 08:55 AM Task #7590: Expect everything to fail (uncloud)
- I am little uncertain how to handle failures in etcd. For Example, put every etcd function call in try/except block or do something else.
- AB 08:48 AM Task #7590: Expect everything to fail (uncloud)
- h2. How do we plan to handle failures in etcd
Failures can be temporarily (leadership change) or permanent (etcd cluster dies)
We use four type of functions of etcd
1. get
2. put
3. get_prefix
4. watch_prefix
h3. get
can ... - AB 08:48 AM Task #7590 (Rejected): Expect everything to fail (uncloud)
- Especially external components, such as
1. etcd
2. netbox
3. otp.ungleich.ch
01/08/2020
- AB 08:21 PM Task #7583: Handle etcd leader change or temporary unavailability gracefully in uncloud
- We have to re-evaluate/re-check all the usage of etcd in uncloud to make sure we handle these events correctly/gracefully.
I have modified few things in get_prefix/watch_prefix that would help to correctly/gracefully handle etcd event... - AB 07:34 AM Task #7583: Handle etcd leader change or temporary unavailability gracefully in uncloud
- The later unavailability is due to election for leader.
- TF 12:21 PM Task #7580: Preparing for matrix-as-a-service
- We can do-it in a second-stage, witout a TURN server VoIP might or might not work depending on the situation.
Note that coturn is easy to deploy: https://github.com/matrix-org/synapse/blob/master/docs/turn-howto.md - NS 11:27 AM Task #7580: Preparing for matrix-as-a-service
- Can we do voip in a second stage or will things "look weird" without it?
redmine@ungleich.ch writes: - TF 11:17 AM Task #7580: Preparing for matrix-as-a-service
- We'll also need a TURN server for VoIP.
01/07/2020
- AB 06:38 PM Task #7555: Setup uncloud at server11 and server12
- uncloud deployed at server{11, 12}. There are still some issues that would be problem for reliably running uncloud for longer period of time. e.g
1. https://redmine.ungleich.ch/issues/7583 (As, soon as etcd leader changes or etcd become... - AB 06:33 PM Task #7582: Add hostname in uncloud file scanning
- Nico Schottelius wrote:
> For the example above: the host would probably often more something like "files1.datacenterlight.ch" or "username.datacenterlight.ch" or so.
Yeah, it is just my local machine. So, i put my IPv6 address direc... - NS 04:19 PM Task #7582: Add hostname in uncloud file scanning
- For the example above: the host would probably often more something like "files1.datacenterlight.ch" or "username.datacenterlight.ch" or so.
- NS 04:18 PM Task #7582: Add hostname in uncloud file scanning
- We should have a "created_at" and "deleted_at" for every object.
- AB 02:44 PM Task #7582: Add hostname in uncloud file scanning
- Done. Sample
Before - AB 01:41 PM Task #7582 (Closed): Add hostname in uncloud file scanning
- It is required as nico said that there would be multiple file hosts and not necessarily all files are available on some particular host.
https://chat.ungleich.ch/ungleich/pl/3pf77f1ui7yiupxjyqayzf67ry - AB 04:45 PM Task #7585 (Closed): Check whether uncloud-api break if some field is missing
- The behavior is seen in the past that uncloud-api breaks if we don't pass some fields like *name*, *realm* or *token* etc.
- AB 02:07 PM Task #7583 (Rejected): Handle etcd leader change or temporary unavailability gracefully in uncloud
- Here is leader change.
- TF 12:53 PM Task #7580 (Closed): Preparing for matrix-as-a-service
- Once matrix is deployed at ungleich:
* Build & document MaaS deployment and maintenance pipeline.
- Wiki page.
- A staging environment will be required to test upgrades.
* 1 or 2 blog entries about it? First one maybe a bit mor...
01/06/2020
- TF 12:21 PM Task #7543 (In Progress): Write image definition script for ubuntu 19.10
- From Nico:
- TF 12:09 PM Task #7543 (Waiting): Write image definition script for ubuntu 19.10
- There's already a 19.10 image deployed... ???
- TF 11:57 AM Task #7543 (In Progress): Write image definition script for ubuntu 19.10
01/05/2020
- AB 07:09 PM Task #7555: Setup uncloud at server11 and server12
- Remaining things
[ ] IPv6 Prefix on Server 12
[ ] VM with Global IPv6 (2a0a:e5c0:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx)
[ ] Setup on Server 11 - AB 07:02 PM Task #7555: Setup uncloud at server11 and server12
- uncloud filescanner has to be modified as it was using xattrs to track files which does not work on rootfs/tmpfs which is filesystem of netbooted server e.g server{11,12}.
- AB 04:58 PM Task #7555: Setup uncloud at server11 and server12
- Devuan ascii has too old QEMU i.e 2.8 while the latest is 4.2.
It is a problem because it is showing error messages which are not helpful at all and are replaced by good/sensible error messages in newer versions.
??Device needs med... - NS 11:03 AM Task #7565 (Rejected): uncloud run 2020-01-05
- h2. Objective
A test ride to get more nearby prod use
h2. What Nico wants to do us a customer
* register an account
* add an ssh key
* upload an image
* create a VM from that image
* ssh into that VM
h2. What Nico wants...
01/03/2020
- NS 05:35 PM Task #7561: Update mystrom switches to support IPv6 only networks
- NS 05:35 PM Task #7561 (Rejected): Update mystrom switches to support IPv6 only networks
- Using an experimental firmware from mystrom directly:
- NS 04:08 PM Task #7560 (Closed): Document DNS64 setup for VMs
- * After #7496
* Document on how to use it in the [[The_ungleich_DNS_infrastructure]]
* Reference it in [[The_ungleich_VPN_infrastructure]]
* Create a blog entry in the ungleich-staticcms explaining that we now support "full VPNs" - i.... - AB 11:38 AM Task #7555: Setup uncloud at server11 and server12
- Allow etcd prefix for developer role
- AB 10:58 AM Task #7555 (Closed): Setup uncloud at server11 and server12
- Ensure that both server11 and server12 are running with uncloud today and can be used in production? And please note all steps that you took in a redmine ticket. All uncloud scripts should run as user uncloud
01/02/2020
- TF 05:09 PM Task #7496: Create 2 new IPv6 only unbound based resolving DNS servers providing DNS64
- Merge request opened against dot-cdist: https://code.ungleich.ch/ungleich-intern/dot-cdist/merge_requests/65
- NS 03:30 PM Task #7436 (Closed): Hack-a-ucloud-weekend (2019-12-07)
- NS 03:30 PM Task #7438 (Closed): Explore local ucloud setup
- NS 03:30 PM Task #7437 (Closed): Run ucloud with a single authentication token
- NS 03:30 PM Task #7439 (Rejected): Add support for different authentication methods
- Postponed.
- NS 01:35 PM Task #7553: Setup conntrackd to allow active active firewalls
- And config looks like this:
- NS 01:33 PM Task #7553: Setup conntrackd to allow active active firewalls
- Seems like the code is in read_config.yy.c:
- NS 01:08 PM Task #7553: Setup conntrackd to allow active active firewalls
- Added sync section, now getting an IPv6 exception:
- NS 12:59 PM Task #7553: Setup conntrackd to allow active active firewalls
- router2.place6:
- NS 12:50 PM Task #7553 (Rejected): Setup conntrackd to allow active active firewalls
- * So that firewall rules still work with state tracking
Change of objective: get this running on two IPv6 only Alpine VMs first and then we migrate it to the routers - NS 12:47 PM Task #7552 (Closed): Add some non-critical traffic to router1.place6
- * might required conntrackd
Networks first stage:
* internal network
* server network
- NS 12:46 PM Task #7306 (Rejected): Phase in new routers
- Closing in favor of #7284
- NS 12:44 PM Task #7307 (Closed): Update __ungleich_bgp_router for IPv6 based multip bgp
- Not doing this atm, as routers still need IPv4 addresses for VMs
- NS 12:21 PM Task #7520 (Closed): Checkout whether OSPF can be helpful for DCL
- It works!
But not for eBGP routes. - NS 12:07 PM Task #6930: cdist configuration for etcd
- Current state I know of:
* there is etcd{1,2,3}.ungleich.ch
* They have an unknown configuration
* And they have the ROOT key of the certificate deployed
Expected results are:
* etcd fully setup and redoable via cdist
* an ea...
01/01/2020
- MR 05:49 PM Task #7546: VM Security based on LDAP accounts
- We don't have any rate limiting to any of the apis that we have so far.
I think rate limiting would primarily be needed for the user login/signup attempts, but not limited to them only.
We could also add captchas. - NS 05:05 PM Task #7546: VM Security based on LDAP accounts
- * Moris, thanks for reporting.
* Mondi, can you start with your tasks and handover to Timothee when done?
- NS 05:05 PM Task #7546: VM Security based on LDAP accounts
- h2. Clarification 1: "shared login"
We use LDAP servers as a backend to redmine and django (the dashboard). Both systems originally had their own user databases (and passwords), but both have been reconfigured to use the LDAP backend.... - NS 04:53 PM Task #7546 (In Progress): VM Security based on LDAP accounts
12/31/2019
- MJ 07:40 PM Task #7546 (Rejected): VM Security based on LDAP accounts
- Access to VM administration tools should be secured to the same level or higher as root access to the VM itself.
Currently the VM dashboard uses a shared login with redmine.
Admin systems and communication systems should not have a... - TF 06:19 PM Task #6694 (In Progress): Setup matrix server and bridge matermost into it
- After discussion with nico, Matrix get priority over LDAP setup rebuild.
- TF 04:25 PM Task #6694 (Waiting): Setup matrix server and bridge matermost into it
- The matrix deployment works modulo:
* Federating with the IPv4 world (a few lines to add to haproxy's configuration)
* Rebuilding ungleich's production LDAP environment to be able to use custom service accounts in a clean way: http... - TF 04:26 PM Task #7345 (Waiting): Cleanup & upstream matrix-related types
- TF 03:20 PM Task #7545 (Closed): Switch production LDAPs to cdist-managed alpine
- Our production LDAP nodes do not seem to be managed by cdist (anymore?):
* No relevant mention in `grep -R __ungleich_ldap dot-cdist/` or `grep -R ldap1 dot-cdist/`
* Deployed configuration do not exactly match `__ungleich_ldap` ty... - TF 07:36 AM Task #7544 (Rejected): Write "beginner's guide" for datacenterlight customers
- Such a guide should cover:
* What is a VM? How do I choose CPU/Memory/Storage?
* How do I choose a GNU/Linux or *BSD distribution?
* How do I connect to my VM?
- GNU/Linux, *BSD
- MacOS
- Windows
* Managing my ...
12/30/2019
- TF 06:03 PM Task #6694: Setup matrix server and bridge matermost into it
- The matrix deployment is WIP in https://code.ungleich.ch/ungleich-intern/dot-cdist/merge_requests/64/diffs and is starting to look quite decent. I hope to have it usable by tuesday or wednesday depending on the work time I can allocate t...
- TF 07:37 AM Task #6694 (In Progress): Setup matrix server and bridge matermost into it
- TF 02:40 PM Task #7543 (Closed): Write image definition script for ubuntu 19.10
- Similar to what have been done for fedora and CentOS.
- TF 07:37 AM Task #7345: Cleanup & upstream matrix-related types
- The project has been imported under https://code.ungleich.ch/ungleich-public/matrix-cdist-types. Moving to issue #6694 for real-world testing (i.e. 'customer-usable' part).
12/25/2019
12/24/2019
- AB 07:15 PM Task #7427 (Closed): Rough draft to support console on our VMs
- Django part done. LDAP account is created as soon as user login to datacenterlight.
12/21/2019
- NS 02:53 PM Task #7520: Checkout whether OSPF can be helpful for DCL
- parameters mismatch between switch & routers: disable switch for the moment
- NS 02:35 PM Task #7520: Checkout whether OSPF can be helpful for DCL
- Testing on arista:
- NS 02:16 PM Task #7520: Checkout whether OSPF can be helpful for DCL
- Seems like these code blocks are already enough for internal route exchange:
- NS 01:49 PM Task #7520 (Closed): Checkout whether OSPF can be helpful for DCL
- * testing with new routers
* Todo: find out how to limit route imports in arista via ospf - NS 12:25 PM Task #7519: uncloud test run 2019-12-21
- arch
* etcd3 error message might be improve / error out on import error
** fail only when it is required for operations!
* don't error out if ucloud.conf is missing
* don't give traceback if keys are missing
** What is the user ex... - NS 12:22 PM Task #7519 (Closed): uncloud test run 2019-12-21
- * Convince Nico that it works ;-)
* On Alpine and Arch
12/20/2019
- NS 09:47 PM Task #7304: Test NAT64 with distributed routers with joold on alpine
- * Configuring router1.place6 for NAT64
** Prefix 2a0a:e5c0:2:10::/96 - TF 04:10 PM Task #7514 (Closed): Investigate slow sshd start on Fedora/CentOS8 images
- * To investigate
- Likely due to low entropy => havegd should help. - NS 12:56 AM Task #7377: Create an active-active NAT64 gateway
- Session exist on one router, session does not exist on other one -> multicast issue?
- NS 12:28 AM Task #7377: Create an active-active NAT64 gateway
- Using the new NAT64 IPv4 address on both machines:
- NS 12:27 AM Task #7377: Create an active-active NAT64 gateway
- Need to add pool entries for each protocol:
- NS 12:23 AM Task #7377 (In Progress): Create an active-active NAT64 gateway
12/19/2019
- TF 07:41 PM Task #7507 (Rejected): Monitor upstream releases / security advisories
- We deploy some application directly from upstream VCS, which means the underlying distribution does not provide us with (security) updates: we have to do it ourselve.
=> It's 100% manual/human for now. Perhaps we could consume events ... - NS 05:41 PM Task #7345: Cleanup & upstream matrix-related types
- Just added you to ungleich-public, please move there
- TF 04:57 PM Task #7345: Cleanup & upstream matrix-related types
- Both matrix-synapse and matrix-riot should be usable now, although they need some real-world testing which will handled by #6694.
The sources are currently hosted on [git.sr.ht](https://git.sr.ht/~fnux/matrix-cdist-types): shall I mov... - TF 05:29 PM Task #7504 (Rejected): Document OpenNebula image creation
- Peoples ask how to build new images from time to time, it would be easier if the requirements/workflow was documented somewhere.
Relevant wiki page: https://redmine.ungleich.ch/projects/open-infrastructure/wiki/OpenNebula_image_manage... - NS 05:14 PM Task #7478 (Closed): Create script to create centos8 image suitable for opennebula
Thanks!
redmine@ungleich.ch writes:- TF 04:46 PM Task #7478 (Resolved): Create script to create centos8 image suitable for opennebula
- From what I saw on #ungleich-web, the CentOS 8 image has been deployed.
- TF 05:11 PM Task #7496 (In Progress): Create 2 new IPv6 only unbound based resolving DNS servers providing DNS64
- TF 02:13 PM Task #7503: Create script to create new / updated OpenBSD image for OpenNebula
- Note by @kirill on chat.ungleich.ch // datacenterlight:
> regarding Task #7503 - remember to add family inet6 inet4 to /etc/resolv.conf.tail
> ...
See also: https://chat.ungleich.ch/ungleich/pl/1rwad75wu3nzdcerdyb5gk5ocw - NS 09:58 AM Task #7503 (Rejected): Create script to create new / updated OpenBSD image for OpenNebula
- * Upgrading to 6.6
* The original image was made by @reyk
* The logic is basically the same as the other images, however there is no deb/rpm
** Reyk installed / modified "cloud-agent" - this is what we want to install
* Username is "... - NS 09:55 AM Task #7502 (Closed): Create script to create new / updated FreeBSD image for OpenNebula
- * 12.1 is out for a while
* The original image was made by @kamila
* The logic is basically the same as the other images, however there is no deb/rpm for freebsd
** I am not sure where she added the necessary changes (dns, network, ss...
12/17/2019
- TF 06:45 PM Task #7478 (Feedback): Create script to create centos8 image suitable for opennebula
- Related Merge Request: https://code.ungleich.ch/ungleich-public/ungleich-tools/merge_requests/4
The image has been deployed under the ipv6only-centos8 OpenNebula template and seems to be working properly. - NS 05:41 PM Task #7472 (Closed): Fix the fedora image for resize
Nice job!
redmine@ungleich.ch writes:- TF 05:23 PM Task #7472 (Resolved): Fix the fedora image for resize
- Solved by https://code.ungleich.ch/ungleich-public/ungleich-tools/commit/09a05b6a56f55fe27a37d26aaab13c65aa82a9f5.
- NS 12:49 PM Task #7345: Cleanup & upstream matrix-related types
- Moving ticket to Open Infrastructure
- TF 12:37 PM Task #6694: Setup matrix server and bridge matermost into it
- I've been working on matrix cdists types during the last H4G: https://redmine.ungleich.ch/issues/7345
It's not complete yet as it requires testing & tuning for various distributions but the general structure is done: https://git.sr.ht/~... - NS 12:00 PM Task #7496 (Closed): Create 2 new IPv6 only unbound based resolving DNS servers providing DNS64
- h2. Background
We want VPN users or anyone to be able to select if they get NAT64 or not. Currently our bind decides based on the query source IP, whether to provide NAT64 or not and which prefix to use.
We want to have an alternat... - NS 11:14 AM Task #6671: Setup mastodon/pleroma for ungleich
- ack
redmine@ungleich.ch writes: - TF 10:54 AM Task #6671 (Seen): Setup mastodon/pleroma for ungleich
- I just gave a quick look at both the initial Mastodon implementation and Pleroma and will likely go with the later:
* The initial mastodon implementation has a lot of moving parts (Postgres, Redis, ElasticSearch, ...): https://docs.jo...
12/16/2019
- TF 02:02 PM Task #7478 (In Progress): Create script to create centos8 image suitable for opennebula
- TF 02:01 PM Task #7478: Create script to create centos8 image suitable for opennebula
- I bootstraped an initial image but there's still some cleanup to do.
- TF 10:07 AM Task #7482: On Alpine Linux the monit job for node-exporter uses the wrong path
- > Add an "os" check into manifest/dcl in the router section to use a different __ungleich_monit config depending on the OS.
Wouldn't it be better to add this check to the `__ungleich_monit` type, which generate the `/etc/monit/conf.d/...
12/15/2019
- NS 11:03 PM Task #7483 (Closed): Update the __consul cdist type for alpine
- seems like the configuration place is different on current alpine that what we assumed/what is in our normal config file:
- NS 11:01 PM Task #7482 (Closed): On Alpine Linux the monit job for node-exporter uses the wrong path
- Current state is:
- NS 05:31 PM Task #7478 (Closed): Create script to create centos8 image suitable for opennebula
- * centos8-build-opennebula.sh or similar
* goes into ungleich-tools - NS 11:47 AM Task #7472 (Closed): Fix the fedora image for resize
- * In theory the opennebula scripts *should* do that
** I think I have even seen scripts in /etc/one-context.d that *should* do it
** however, it wasn't done...
* it might a missing udev hook???