Actions
Task #7583
closedHandle etcd leader change or temporary unavailability gracefully in uncloud
Status:
Rejected
Priority:
Normal
Assignee:
-
Target version:
-
Start date:
01/07/2020
Due date:
% Done:
0%
Estimated time:
PM Check date:
Description
Here is leader change.
uncloud@server12:~$ uncloud scheduler
Traceback (most recent call last):
File "/home/uncloud/.local/lib/python3.5/site-packages/etcd3/client.py", line 46, in handler
return f(*args, **kwargs)
File "/home/uncloud/.local/lib/python3.5/site-packages/etcd3/client.py", line 308, in get_prefix_response
metadata=self.metadata
File "/home/uncloud/.local/lib/python3.5/site-packages/grpc/_channel.py", line 824, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/home/uncloud/.local/lib/python3.5/site-packages/grpc/_channel.py", line 726, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: leader changed"
debug_error_string = "{"created":"@1578303042.125921981","description":"Error received from peer ipv6:[2a0a:e5c0:2:12:0:f0ff:fea9:c43a]:2379","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"etcdserver: leader changed","grpc_status":14}"
and here is temporary unavailability
Cannot connect to etcd: is etcd running as configured in uncloud.conf?
Exception in thread etcd3_watch_7fc862374c18:
Traceback (most recent call last):
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
File "/home/uncloud/.local/lib/python3.5/site-packages/etcd3/watch.py", line 126, in _run
metadata=self._metadata)
File "/home/uncloud/.local/lib/python3.5/site-packages/grpc/_channel.py", line 1084, in __call__
event_handler, self._context)
File "/home/uncloud/.local/lib/python3.5/site-packages/grpc/_channel.py", line 1171, in create
operationses_and_tags, context)
File "src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi", line 478, in grpc._cython.cygrpc.Channel.integrated_call
File "src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi", line 296, in grpc._cython.cygrpc._integrated_call
File "src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi", line 222, in grpc._cython.cygrpc._call
File "src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi", line 262, in grpc._cython.cygrpc._call
ValueError: Cannot invoke RPC: Channel closed!
Process Process-1:
Traceback (most recent call last):
File "/home/uncloud/.local/lib/python3.5/site-packages/etcd3/client.py", line 46, in handler
return f(*args, **kwargs)
File "/home/uncloud/.local/lib/python3.5/site-packages/etcd3/client.py", line 426, in put
metadata=self.metadata
File "/home/uncloud/.local/lib/python3.5/site-packages/grpc/_channel.py", line 824, in__call__
return _end_unary_response_blocking(state, call, False, None)
File "/home/uncloud/.local/lib/python3.5/site-packages/grpc/_channel.py", line 726, in_end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "etcdserver: request timed out"
debug_error_string = "{"created":"@1578304999.737401188","description":"Error received from peer ipv6:[2a0a:e5c0:2:12:0:f0ff:fea9:c43a]:2379","file":"src/core/lib/$
urface/call.cc","file_line":1056,"grpc_message":"etcdserver: request timed out","grpc_status":14}"
Actions