Project

General

Profile

Actions

Task #8888

closed

Meta-Issue for Matrix late-winter 2021 cleanup

Added by Timothée Floure almost 4 years ago. Updated about 3 years ago.

Status:
Closed
Priority:
Normal
Target version:
-
Start date:
02/11/2021
Due date:
% Done:

0%

Estimated time:
PM Check date:

Description

Hello there,

We have some performance issues and infrastructure rot on our matrix deployments: I'll work on it here and there over the next few weeks. This meta-issue will make following easier to follow what's going on. I might add things on the fly as I encounter them / link to other issues later on.

  • [x] Cleaning up and upstreaming the __matrix-synapse cdist type. #7345
    - [x] Clean-up.
    - [x] Bring configuration template up-to-date.
    - [x] Add more performance-related flags.
    - [x] Add support for multi-workers (a new __matrix_synapse_worker type might be needed)
    - [x] Upstream to cdist-contrib See https://code.ungleich.ch/ungleich-public/cdist-contrib/-/merge_requests/9
  • [ ] Cleanup and simplify the __ungleich_matrix type
    - [ ] Allow PGSQL tuning / auto-tune from explorer if not provided.
    - [x] Adapt to updated __matrix_synapse type
  • [ ] Revamp matrix monitoring: we need something simpler and more robust.
    - [x] Get back missing instances in monitoring.
    - [ ] Add alerts.
    - [x] Add PGSQL performance monitoring.
  • [x] Update admin UI
  • [ ] Investigate performance issues.
    - [~] Checking out database bottlenecks.
    - [~] Checking out synapse bottlenecks.
    - [ ] Possibily add periodic database cleanup.
  • [ ] Check out the state of the Jitsi integration.
    - [x] Rebuilt with CDIST (small issue with watermark - see https://code.ungleich.ch/ungleich-public/cdist-contrib/-/issues/4)
    - [x] Wire Prometheus to the new Jitsi Exporter
    - [ ] Add simple blackbox monitoring
  • [x] Check state of ext.ungleich.ch homeserver
  • [ ] LOW_PRIO check out if it is useful to deploy our own integration server
  • [ ] Don't forget to document!

Related issues 3 (0 open3 closed)

Blocked by Open Infrastructure - Task #7345: Cleanup & upstream matrix-related typesClosedTimothée Floure11/28/2019

Actions
Blocked by Open Infrastructure - Task #8852: Investigate matrix.ungleich.ch slownessClosedTimothée Floure02/05/2021

Actions
Blocked by Open Infrastructure - Task #8887: Update synapse-adminClosedTimothée Floure02/11/2021

Actions
Actions #1

Updated by Timothée Floure almost 4 years ago

  • Description updated (diff)
Actions #2

Updated by Timothée Floure almost 4 years ago

  • Blocked by Task #7345: Cleanup & upstream matrix-related types added
Actions #3

Updated by Timothée Floure almost 4 years ago

  • Blocked by Task #8852: Investigate matrix.ungleich.ch slowness added
Actions #4

Updated by Timothée Floure almost 4 years ago

  • Blocked by Task #8887: Update synapse-admin added
Actions #5

Updated by Timothée Floure almost 4 years ago

  • Blocks Task #8886: Drop talk.ungleich.ch and rebuild with cdist added
Actions #6

Updated by Timothée Floure almost 4 years ago

  • Blocks Task #8877: Checkout ext.ungleich.ch matrix instance on server1.place4 added
Actions #7

Updated by Timothée Floure almost 4 years ago

  • Description updated (diff)
Actions #8

Updated by Timothée Floure almost 4 years ago

  • Description updated (diff)
Actions #9

Updated by Timothée Floure almost 4 years ago

  • Description updated (diff)
Actions #10

Updated by Timothée Floure almost 4 years ago

  • Description updated (diff)
Actions #11

Updated by Timothée Floure almost 4 years ago

  • Description updated (diff)

The new shiny cdist pipeline seems to work nicely - it's currently deployed for staging and ungleich. We also have metrics exported to monitoring-v3. All of this will be documented and rolled out to customer deployments next week.

Actions #12

Updated by Timothée Floure almost 4 years ago

  • Description updated (diff)
Actions #13

Updated by Nico Schottelius almost 4 years ago

  • Blocks deleted (Task #8877: Checkout ext.ungleich.ch matrix instance on server1.place4)
Actions #14

Updated by Nico Schottelius over 3 years ago

  • Blocks deleted (Task #8886: Drop talk.ungleich.ch and rebuild with cdist)
Actions #15

Updated by Nico Schottelius about 3 years ago

  • Status changed from In Progress to Closed
Actions

Also available in: Atom PDF