Activity
From 12/23/2019 to 01/21/2020
01/21/2020
- NS 09:42 PM Task #7632: Setup rados / s3 storage on ceph
redmine@ungleich.ch writes:- LN 07:55 PM Task #7632 (Seen): Setup rados / s3 storage on ceph
- LN 07:55 PM Task #7631 (Seen): Report details about pleroma problems to upstream
- TF 02:16 PM Task #7641 (In Progress): create images for uncloud
- SK 12:38 PM Task #7641 (Closed): create images for uncloud
- First images should be the latest alpine, fedora, ubuntu, debian.
How should they be configured:
they should get an ipv6 address from the first network interface,
they should automatically increase the root file system, if the di... - TF 10:57 AM Task #7545 (In Progress): Switch production LDAPs to cdist-managed alpine
- TF 10:32 AM Task #7483 (In Progress): Update the __consul cdist type for alpine
- I got the same issue with `__consul_agent` on Debian, which I am currently fixing upstream.
https://code.ungleich.ch/ungleich-public/cdist/merge_requests/837
01/20/2020
- TF 05:11 PM Task #7630 (Feedback): Cleanup the DNS64 situation
- See https://redmine.ungleich.ch/projects/open-infrastructure/wiki/The_ungleich_DNS_infrastructure.
- TF 12:06 PM Task #7630 (Seen): Cleanup the DNS64 situation
- NS 11:03 AM Task #7630 (Closed): Cleanup the DNS64 situation
- h2. Old situation
* bind nameservers on routers decide based on source IPv6 address whether to give out NAT64 or not
* Overlapping use (dual stack VM vs. IPv6 only) led to problems that the above rule does not apply strictly
* This ... - NS 03:19 PM Task #7636 (Closed): Find out current retention period for monitoring servers and ensure that data is kept for 5 years
- * Looking at monitoring.place6 I see data for less than 90 days.
* My expectation is to be able to zoom out to 5 years so that we can see changes we did over years
* My assumption is that prometheus is configured with some storage siz... - NS 03:06 PM Task #7635 (Closed): Create a simple page explaining DNS64/NAT64 for customers
- * So that we can reference it in support tickets.
* Include exapmles, how to reach github, show the AAAA record resolution, explain how it works if there is already IPv6 for a domain
In simple words (maybe +graphviz/dot image showing... - TF 12:06 PM Task #6694 (Closed): Setup matrix server and bridge matermost into it
- Relevant channels have been bridged. Closing.
- TF 12:05 PM Task #7560 (In Progress): Document DNS64 setup for VMs
- TF 12:05 PM Task #7496 (Closed): Create 2 new IPv6 only unbound based resolving DNS servers providing DNS64
- Unbound DNS(64) servers are now monitored by the prometheus blackbox exporter. Closing.
- NS 11:35 AM Task #7632 (Closed): Setup rados / s3 storage on ceph
- * Including permissions
* Document the setup
* Document how to use it - NS 11:04 AM Task #7631 (Closed): Report details about pleroma problems to upstream
- Follow up with https://git.pleroma.social/pleroma/pleroma-support/issues/10#note_49605
* Create an IPv6 only VM
* Recreate the problem
* Keep the VM running for until the problem has been fixed upstream
Please follow up today so ... - AB 09:27 AM Task #7629 (Rejected): Add referral link system in dynamicweb (DCL, IPv6OnlyHosting etc)
- Referral links are used to reward user/(reviewing website) whenever someone purchase VM (or other service) using their referral link.
This ticket is created to figure out how to implement referral link system.
01/19/2020
- TF 02:48 PM Task #7543 (Closed): Write image definition script for ubuntu 19.10
- The image has been deployed in ONE and configured for the `public-Ubuntu 19.10` and `ipv6only-Ubuntu 19.10` templates. Defined by the "ubuntu-build-opennebula-image.sh script in ungleich-tools":https://code.ungleich.ch/ungleich-public/un...
01/18/2020
- TF 07:39 PM Task #7496: Create 2 new IPv6 only unbound based resolving DNS servers providing DNS64
- It's deployed: there's just monitoring to setup before it can be closed.
- NS 01:35 PM Task #7625: Manually fix consul+node_exporter on new router1.place6
- NS 01:27 PM Task #7625: Manually fix consul+node_exporter on new router1.place6
- NS 01:26 PM Task #7625: Manually fix consul+node_exporter on new router1.place6
- Use alpine's init script:
- NS 01:02 PM Task #7625 (Rejected): Manually fix consul+node_exporter on new router1.place6
01/15/2020
- LN 07:28 PM Task #6671: Setup mastodon/pleroma for ungleich
- could we have a 13373r name?
01/13/2020
- NS 07:30 PM Task #7604 (In Progress): Find out why ciara2 was not automatically detected to be offline
- * ciara2 is half correctly outside of the consul cluster
** It should actually still be inside the cluster, but marked dead
- NS 07:28 PM Task #7604 (Rejected): Find out why ciara2 was not automatically detected to be offline
- * Consul status / prometheus / alert manager should have noticed
- MJ 06:37 PM Task #7186: Add support for general VPN including IPv4
- Errrr what is it with your VPN pricing? Did you go skiing and get altitude sickness?
Market price for VPN services is $5 - $12 per month.
The high end services offer multiple server locations in every continent and dedicated servers fo... - MJ 05:52 PM Task #7544: Write "beginner's guide" for datacenterlight customers
- -IPv6 and IPv4: making the services on my IPv6 VM visible to the IPv4 world
-Guido to VM Management tools: dashboard/django, ungleich-cli, cdist, ucloud
-Reverse DNS PTR entries
-Using my own IPv6 subnet e.g. /64 - NS 12:01 PM Task #7602 (Rejected): Align dynamicweb / opennebula with uncloud
- Stuff that we can & should export from our current setup to etcd in an uncloud alike format:
Prefix for everything is /dynamicweb-opennebula
* user public ssh keys (/dynamicweb-opennebula/user-keys)
* List of VMs (/dynamicweb-open... - NS 11:57 AM Task #7601: Setup an SSH jump host
- Proxycommand w/ windows exists in putty and usually uses plink - more details soon.
- TF 11:54 AM Task #7601 (Seen): Setup an SSH jump host
- * I am familiar with LDAP-backed auth with nslcd.
* ProxyCommand is standard for SSH bastions, it is even available on windows with Putty: https://fedora-infra-docs.readthedocs.io/en/latest/sysadmin-guide/sops/sshaccess.html#putty-ssh-c... - NS 11:12 AM Task #7601 (Rejected): Setup an SSH jump host
- * Authenticated against our ldap
* Allows user to connect to our IPv6 networks
The ways for users to use it:
* via ProxyCommand (some might be able to use that)
I've setup this some time ago and it basically needs a restriction... - AB 08:37 AM Task #7555 (Closed): Setup uncloud at server11 and server12
- AB 03:14 AM Task #7582 (Closed): Add hostname in uncloud file scanning
- AB 03:14 AM Task #7519 (Closed): uncloud test run 2019-12-21
- The above mentioned things were fixed.
01/12/2020
- NS 09:18 PM Task #7580: Preparing for matrix-as-a-service
- channels that can be exported to IRC or matrix:
* ipv6
* foss
* hacking-and-learning
* Town Square
* datacenterlight
* uncloud
More maybe later - TF 09:11 PM Task #7580 (In Progress): Preparing for matrix-as-a-service
- TF 09:10 PM Task #7580: Preparing for matrix-as-a-service
- * Synapse and Matrix Cdist types are (almost) OK.
- I missed one small thing in my __postgres upstream cdist patch, which still have to be fixed.
* Early documentation on https://redmine.ungleich.ch/projects/open-infrastructure/wiki/... - TF 09:12 PM Task #6694: Setup matrix server and bridge matermost into it
- Everything's in place, we just have to choose the channels to be bridged.
01/09/2020
- AB 08:56 PM Task #7596: uncloud-api refactoring & make schemas less horrible
- To see merge request https://code.ungleich.ch/uncloud/uncloud/merge_requests/1
- AB 08:54 PM Task #7596: uncloud-api refactoring & make schemas less horrible
- * Done `uncloud api` refactoring which was due for a long time and is the last refactoring beside https://redmine.ungleich.ch/issues/7590.
* Schemas has been greatly simplified (now approaching to beautiful code :) and are now pleasant ... - AB 08:54 PM Task #7596 (Closed): uncloud-api refactoring & make schemas less horrible
- AB 08:55 PM Task #7585 (Closed): Check whether uncloud-api break if some field is missing
- The behavior is verified and corrected.
- AB 09:32 AM Task #7591: uncloud production checklist 2020-01
- Also, please note
uncloud deployed at server{11, 12}. There are still some issues that would be problem for reliably running uncloud for longer period of time. e.g
1. https://redmine.ungleich.ch/issues/7583 (As, soon as etcd lead... - AB 09:26 AM Task #7591: uncloud production checklist 2020-01
- *Can all required components be deployed (checking on server11) -- document the installation procedures*
Documented
*Is the API secure from outside? I am able to connect without otp at the moment*
I am not sure about what do you... - AB 09:19 AM Task #7591: uncloud production checklist 2020-01
- h1. Installation/Setup
Allow etcd prefix for developer role - NS 09:17 AM Task #7591 (In Progress): uncloud production checklist 2020-01
- NS 09:16 AM Task #7591 (Rejected): uncloud production checklist 2020-01
- h2. Objective
* Migrate internal VMs to uncloud
h2. Checklist
* Can all required components be deployed (checking on server11) -- document the installation procedures
** -api
** -host
** -network
* Is the API secure from out... - AB 08:55 AM Task #7590: Expect everything to fail (uncloud)
- I am little uncertain how to handle failures in etcd. For Example, put every etcd function call in try/except block or do something else.
- AB 08:48 AM Task #7590: Expect everything to fail (uncloud)
- h2. How do we plan to handle failures in etcd
Failures can be temporarily (leadership change) or permanent (etcd cluster dies)
We use four type of functions of etcd
1. get
2. put
3. get_prefix
4. watch_prefix
h3. get
can ... - AB 08:48 AM Task #7590 (Rejected): Expect everything to fail (uncloud)
- Especially external components, such as
1. etcd
2. netbox
3. otp.ungleich.ch
01/08/2020
- AB 08:21 PM Task #7583: Handle etcd leader change or temporary unavailability gracefully in uncloud
- We have to re-evaluate/re-check all the usage of etcd in uncloud to make sure we handle these events correctly/gracefully.
I have modified few things in get_prefix/watch_prefix that would help to correctly/gracefully handle etcd event... - AB 07:34 AM Task #7583: Handle etcd leader change or temporary unavailability gracefully in uncloud
- The later unavailability is due to election for leader.
- TF 12:21 PM Task #7580: Preparing for matrix-as-a-service
- We can do-it in a second-stage, witout a TURN server VoIP might or might not work depending on the situation.
Note that coturn is easy to deploy: https://github.com/matrix-org/synapse/blob/master/docs/turn-howto.md - NS 11:27 AM Task #7580: Preparing for matrix-as-a-service
- Can we do voip in a second stage or will things "look weird" without it?
redmine@ungleich.ch writes: - TF 11:17 AM Task #7580: Preparing for matrix-as-a-service
- We'll also need a TURN server for VoIP.
01/07/2020
- AB 06:38 PM Task #7555: Setup uncloud at server11 and server12
- uncloud deployed at server{11, 12}. There are still some issues that would be problem for reliably running uncloud for longer period of time. e.g
1. https://redmine.ungleich.ch/issues/7583 (As, soon as etcd leader changes or etcd become... - AB 06:33 PM Task #7582: Add hostname in uncloud file scanning
- Nico Schottelius wrote:
> For the example above: the host would probably often more something like "files1.datacenterlight.ch" or "username.datacenterlight.ch" or so.
Yeah, it is just my local machine. So, i put my IPv6 address direc... - NS 04:19 PM Task #7582: Add hostname in uncloud file scanning
- For the example above: the host would probably often more something like "files1.datacenterlight.ch" or "username.datacenterlight.ch" or so.
- NS 04:18 PM Task #7582: Add hostname in uncloud file scanning
- We should have a "created_at" and "deleted_at" for every object.
- AB 02:44 PM Task #7582: Add hostname in uncloud file scanning
- Done. Sample
Before - AB 01:41 PM Task #7582 (Closed): Add hostname in uncloud file scanning
- It is required as nico said that there would be multiple file hosts and not necessarily all files are available on some particular host.
https://chat.ungleich.ch/ungleich/pl/3pf77f1ui7yiupxjyqayzf67ry - AB 04:45 PM Task #7585 (Closed): Check whether uncloud-api break if some field is missing
- The behavior is seen in the past that uncloud-api breaks if we don't pass some fields like *name*, *realm* or *token* etc.
- AB 02:07 PM Task #7583 (Rejected): Handle etcd leader change or temporary unavailability gracefully in uncloud
- Here is leader change.
- TF 12:53 PM Task #7580 (Closed): Preparing for matrix-as-a-service
- Once matrix is deployed at ungleich:
* Build & document MaaS deployment and maintenance pipeline.
- Wiki page.
- A staging environment will be required to test upgrades.
* 1 or 2 blog entries about it? First one maybe a bit mor...
01/06/2020
- TF 12:21 PM Task #7543 (In Progress): Write image definition script for ubuntu 19.10
- From Nico:
- TF 12:09 PM Task #7543 (Waiting): Write image definition script for ubuntu 19.10
- There's already a 19.10 image deployed... ???
- TF 11:57 AM Task #7543 (In Progress): Write image definition script for ubuntu 19.10
01/05/2020
- AB 07:09 PM Task #7555: Setup uncloud at server11 and server12
- Remaining things
[ ] IPv6 Prefix on Server 12
[ ] VM with Global IPv6 (2a0a:e5c0:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx)
[ ] Setup on Server 11 - AB 07:02 PM Task #7555: Setup uncloud at server11 and server12
- uncloud filescanner has to be modified as it was using xattrs to track files which does not work on rootfs/tmpfs which is filesystem of netbooted server e.g server{11,12}.
- AB 04:58 PM Task #7555: Setup uncloud at server11 and server12
- Devuan ascii has too old QEMU i.e 2.8 while the latest is 4.2.
It is a problem because it is showing error messages which are not helpful at all and are replaced by good/sensible error messages in newer versions.
??Device needs med... - NS 11:03 AM Task #7565 (Rejected): uncloud run 2020-01-05
- h2. Objective
A test ride to get more nearby prod use
h2. What Nico wants to do us a customer
* register an account
* add an ssh key
* upload an image
* create a VM from that image
* ssh into that VM
h2. What Nico wants...
01/03/2020
- NS 05:35 PM Task #7561: Update mystrom switches to support IPv6 only networks
- NS 05:35 PM Task #7561 (Rejected): Update mystrom switches to support IPv6 only networks
- Using an experimental firmware from mystrom directly:
- NS 04:08 PM Task #7560 (Closed): Document DNS64 setup for VMs
- * After #7496
* Document on how to use it in the [[The_ungleich_DNS_infrastructure]]
* Reference it in [[The_ungleich_VPN_infrastructure]]
* Create a blog entry in the ungleich-staticcms explaining that we now support "full VPNs" - i.... - AB 11:38 AM Task #7555: Setup uncloud at server11 and server12
- Allow etcd prefix for developer role
- AB 10:58 AM Task #7555 (Closed): Setup uncloud at server11 and server12
- Ensure that both server11 and server12 are running with uncloud today and can be used in production? And please note all steps that you took in a redmine ticket. All uncloud scripts should run as user uncloud
01/02/2020
- TF 05:09 PM Task #7496: Create 2 new IPv6 only unbound based resolving DNS servers providing DNS64
- Merge request opened against dot-cdist: https://code.ungleich.ch/ungleich-intern/dot-cdist/merge_requests/65
- NS 03:30 PM Task #7436 (Closed): Hack-a-ucloud-weekend (2019-12-07)
- NS 03:30 PM Task #7438 (Closed): Explore local ucloud setup
- NS 03:30 PM Task #7437 (Closed): Run ucloud with a single authentication token
- NS 03:30 PM Task #7439 (Rejected): Add support for different authentication methods
- Postponed.
- NS 01:35 PM Task #7553: Setup conntrackd to allow active active firewalls
- And config looks like this:
- NS 01:33 PM Task #7553: Setup conntrackd to allow active active firewalls
- Seems like the code is in read_config.yy.c:
- NS 01:08 PM Task #7553: Setup conntrackd to allow active active firewalls
- Added sync section, now getting an IPv6 exception:
- NS 12:59 PM Task #7553: Setup conntrackd to allow active active firewalls
- router2.place6:
- NS 12:50 PM Task #7553 (Rejected): Setup conntrackd to allow active active firewalls
- * So that firewall rules still work with state tracking
Change of objective: get this running on two IPv6 only Alpine VMs first and then we migrate it to the routers - NS 12:47 PM Task #7552 (Closed): Add some non-critical traffic to router1.place6
- * might required conntrackd
Networks first stage:
* internal network
* server network
- NS 12:46 PM Task #7306 (Rejected): Phase in new routers
- Closing in favor of #7284
- NS 12:44 PM Task #7307 (Closed): Update __ungleich_bgp_router for IPv6 based multip bgp
- Not doing this atm, as routers still need IPv4 addresses for VMs
- NS 12:21 PM Task #7520 (Closed): Checkout whether OSPF can be helpful for DCL
- It works!
But not for eBGP routes. - NS 12:07 PM Task #6930: cdist configuration for etcd
- Current state I know of:
* there is etcd{1,2,3}.ungleich.ch
* They have an unknown configuration
* And they have the ROOT key of the certificate deployed
Expected results are:
* etcd fully setup and redoable via cdist
* an ea...
01/01/2020
- MR 05:49 PM Task #7546: VM Security based on LDAP accounts
- We don't have any rate limiting to any of the apis that we have so far.
I think rate limiting would primarily be needed for the user login/signup attempts, but not limited to them only.
We could also add captchas. - NS 05:05 PM Task #7546: VM Security based on LDAP accounts
- * Moris, thanks for reporting.
* Mondi, can you start with your tasks and handover to Timothee when done?
- NS 05:05 PM Task #7546: VM Security based on LDAP accounts
- h2. Clarification 1: "shared login"
We use LDAP servers as a backend to redmine and django (the dashboard). Both systems originally had their own user databases (and passwords), but both have been reconfigured to use the LDAP backend.... - NS 04:53 PM Task #7546 (In Progress): VM Security based on LDAP accounts
12/31/2019
- MJ 07:40 PM Task #7546 (Rejected): VM Security based on LDAP accounts
- Access to VM administration tools should be secured to the same level or higher as root access to the VM itself.
Currently the VM dashboard uses a shared login with redmine.
Admin systems and communication systems should not have a... - TF 06:19 PM Task #6694 (In Progress): Setup matrix server and bridge matermost into it
- After discussion with nico, Matrix get priority over LDAP setup rebuild.
- TF 04:25 PM Task #6694 (Waiting): Setup matrix server and bridge matermost into it
- The matrix deployment works modulo:
* Federating with the IPv4 world (a few lines to add to haproxy's configuration)
* Rebuilding ungleich's production LDAP environment to be able to use custom service accounts in a clean way: http... - TF 04:26 PM Task #7345 (Waiting): Cleanup & upstream matrix-related types
- TF 03:20 PM Task #7545 (Closed): Switch production LDAPs to cdist-managed alpine
- Our production LDAP nodes do not seem to be managed by cdist (anymore?):
* No relevant mention in `grep -R __ungleich_ldap dot-cdist/` or `grep -R ldap1 dot-cdist/`
* Deployed configuration do not exactly match `__ungleich_ldap` ty... - TF 07:36 AM Task #7544 (Rejected): Write "beginner's guide" for datacenterlight customers
- Such a guide should cover:
* What is a VM? How do I choose CPU/Memory/Storage?
* How do I choose a GNU/Linux or *BSD distribution?
* How do I connect to my VM?
- GNU/Linux, *BSD
- MacOS
- Windows
* Managing my ...
12/30/2019
- TF 06:03 PM Task #6694: Setup matrix server and bridge matermost into it
- The matrix deployment is WIP in https://code.ungleich.ch/ungleich-intern/dot-cdist/merge_requests/64/diffs and is starting to look quite decent. I hope to have it usable by tuesday or wednesday depending on the work time I can allocate t...
- TF 07:37 AM Task #6694 (In Progress): Setup matrix server and bridge matermost into it
- TF 02:40 PM Task #7543 (Closed): Write image definition script for ubuntu 19.10
- Similar to what have been done for fedora and CentOS.
- TF 07:37 AM Task #7345: Cleanup & upstream matrix-related types
- The project has been imported under https://code.ungleich.ch/ungleich-public/matrix-cdist-types. Moving to issue #6694 for real-world testing (i.e. 'customer-usable' part).
12/25/2019
12/24/2019
- AB 07:15 PM Task #7427 (Closed): Rough draft to support console on our VMs
- Django part done. LDAP account is created as soon as user login to datacenterlight.