Project

General

Profile

Task #8852

Investigate matrix.ungleich.ch slowness

Added by Timothée Floure 4 months ago. Updated 4 months ago.

Status:
In Progress
Priority:
Normal
Target version:
-
Start date:
02/05/2021
Due date:
% Done:

0%

Estimated time:
PM Check date:

Description

matrix.ungleich.ch is so slow it becomes unusable, I'm currently investigating and:

  • Disabling Presence
  • Increase cache sizes
  • Configure Synapse with multiple workers

Related issues

Blocks Open Infrastructure - Task #8888: Meta-Issue for Matrix late-winter 2021 cleanupIn Progress02/11/2021

Actions

History

#1

Updated by Timothée Floure 4 months ago

  • Status changed from New to In Progress
#2

Updated by Timothée Floure 4 months ago

I spend most of my morning playing around with this synapse issues. I moved some processing out of the main process and increased a bit the cache but there's still high latency on registering new events: in-depth investigation including a deeper look at what's going with synapse's internals and potential slowness with PGSQL are required.

I'd also like to:

  • Revamp the whole monitoring pipeline which is utterly broken right now: autodiscovery with consul and a monitoring LAN sounds nice in theory, but implies too many flying components in practice. I want to drop it in favor of good old cdist-managed static configuration files. We also need some alerts (= instead of ranting customers..).
  • Clean up the synapse type (written in my early cdist days ~ 1 year ago...) and merge it into cdist-contrib.
  • Add the multi-worker setup to cdist. It doesn't matter if we add more core with a single synapse process...
  • Make our internal cdist manifest more concise, it's somewhat messy right now.

This means a few work-days. Your thoughts Nico Schottelius?

#3

Updated by Timothée Floure 4 months ago

Note: I commented out the cdist config for our matrix HS, since the changes I applied this morning were all done by hand.

#4

Updated by Timothée Floure 4 months ago

+ write/deploy and run some DB cleanup tools, since it's growing exponentially...

#5

Updated by Timothée Floure 4 months ago

  • Blocks Task #8888: Meta-Issue for Matrix late-winter 2021 cleanup added

Also available in: Atom PDF