Project

General

Profile

Task #7590

Expect everything to fail (uncloud)

Added by Ahmed Bilal over 1 year ago. Updated 8 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Target version:
-
Start date:
01/09/2020
Due date:
% Done:

0%

Estimated time:
PM Check date:

Description

Especially external components, such as
1. etcd
2. netbox
3. otp.ungleich.ch


Related issues

Related to Open Infrastructure - Task #7583: Handle etcd leader change or temporary unavailability gracefully in uncloudWaiting01/07/2020

Actions

History

#1

Updated by Ahmed Bilal over 1 year ago

How do we plan to handle failures in etcd

Failures can be temporarily (leadership change) or permanent (etcd cluster dies)

We use four type of functions of etcd
1. get
2. put
3. get_prefix
4. watch_prefix

get

can output the following:
  • EtcdEntry
  • Exception (KeyError if key not found, and a pleothera of other depending on situations)

TODO: KeyError is recently introduced so would require us to handle it in all stretches of uncloud other errors are also not handled.
TODO: Handle Exception thrown by get.

put

can output the following:
  • Response from etcd library
  • Exception

TODO: See whether the response from etcd library has success/failure indication.
TODO: Handle exception in case something goes wrong.

get_prefix

can output/yield the following:

  • EtcdEntry
  • Exception

TODO: Handle exception in case something goes wrong.

watch_prefix

can output/yield the following:

  • EtcdEntry
  • an empty iter in case of exception
#2

Updated by Ahmed Bilal over 1 year ago

  • Related to Task #7583: Handle etcd leader change or temporary unavailability gracefully in uncloud added
#3

Updated by Ahmed Bilal over 1 year ago

I am little uncertain how to handle failures in etcd. For Example, put every etcd function call in try/except block or do something else.

#4

Updated by Ahmed Bilal 8 months ago

  • Assignee deleted (Ahmed Bilal)

Also available in: Atom PDF