Actions
Task #7122
closed
NS
NS
Setup production etcd cluster in place6
Task #7122:
Setup production etcd cluster in place6
Added by Nico Schottelius over 6 years ago. Updated about 2 years ago.
Start date:
09/09/2019
Due date:
% Done:
80%
Estimated time:
PM Check date:
Description
- 3 nodes
- Ensure that sufficient permissions are used to secure access to etcd
- Include hourly backup to place5
- Check whether we need to make a dump or can backup the data directory directly
- Probably include letsencrypt (?) for CAs / encryption
- Or private CA
AB Updated by Ahmed Bilal over 6 years ago Actions #1
- Status changed from New to Seen
AB Updated by Ahmed Bilal over 6 years ago Actions #2
To Dump ETCD https://www.npmjs.com/package/etcd-dump (Not Working Correctly)- To Create Snapshot https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/recovery.md
- To Create a Certificate Authority to issue certificates https://coreos.com/os/docs/latest/generate-self-signed-certificates.html
- Common Name as username https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/authentication.md#using-tls-common-name
AB Updated by Ahmed Bilal over 6 years ago Actions #3
ca-config.json
{
"signing": {
"default": {
"expiry": "43800h"
},
"profiles": {
"server": {
"expiry": "43800h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
},
"client": {
"expiry": "43800h",
"usages": [
"signing",
"key encipherment",
"client auth"
]
},
"peer": {
"expiry": "43800h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
ca-csr.json
{
"CN": "ungleich",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CH",
"ST": "Glarus"
}
]
}
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
etcd1.json
{
"CN": "etcd1",
"hosts": [
"2a0a:e5c0:0:5:0:78ff:fe11:d761",
"etcd1"
],
"key": {
"algo": "ecdsa",
"size": 256
},
"names": [
{
"C": "CH",
"ST": "Glarus"
}
]
}
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server etcd1.json | cfssljson -bare etcd1
etcd2.json
{
"CN": "etcd2",
"hosts": [
"2a0a:e5c0:0:5:0:78ff:fe11:d762",
"etcd2"
],
"key": {
"algo": "ecdsa",
"size": 256
},
"names": [
{
"C": "CH",
"ST": "Glarus"
}
]
}
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server etcd2.json | cfssljson -bare etcd2
etcd3.json
{
"CN": "etcd3",
"hosts": [
"2a0a:e5c0:0:5:0:78ff:fe11:d763",
"etcd3"
],
"key": {
"algo": "ecdsa",
"size": 256
},
"names": [
{
"C": "CH",
"ST": "Glarus"
}
]
}
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server etcd3.json | cfssljson -bare etcd3
client.json
{
"CN": "client",
"hosts": [""],
"key": {
"algo": "ecdsa",
"size": 256
},
"names": [
{
"C": "CH",
"ST": "Glarus"
}
]
}
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client client.json | cfssljson -bare clientroot.json
{
"CN": "root",
"hosts": [""],
"key": {
"algo": "ecdsa",
"size": 256
},
"names": [
{
"C": "CH",
"ST": "Glarus"
}
]
}
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client root.json | cfssljson -bare rootdeveloper.json
{
"CN": "developer",
"hosts": [""],
"key": {
"algo": "ecdsa",
"size": 256
},
"names": [
{
"C": "CH",
"ST": "Glarus"
}
]
}
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client developer.json | cfssljson -bare developerAB Updated by Ahmed Bilal over 6 years ago Actions #4
To Start Fresh
rm -rf /var/lib/etcd /root/etcd-data/
# Run the following command on first node
etcd --name etcd1 --cert-file=/root/cert/etcd1.pem --key-file=/root/cert/etcd1-key.pem \
--peer-client-cert-auth --peer-trusted-ca-file=/root/cert/ca.pem \
--peer-cert-file=/root/cert/etcd1.pem --peer-key-file=/root/cert/etcd1-key.pem \
--client-cert-auth --trusted-ca-file=/root/cert/ca.pem \
--advertise-client-urls=https://[::]:2379 --listen-client-urls=https://[::]:2379 \
--initial-advertise-peer-urls=https://[::]:2380 --listen-peer-urls=https://[::]:2380 \
--initial-cluster etcd1=https://[::]:2380,etcd2=https://[2a0a:e5c0:0:5:0:78ff:fe11:d762]:2380,etcd3=https://[2a0a:e5c0:0:5:0:78ff:fe11:d763]:2380 \
--initial-cluster-state new --initial-cluster-token etcd-cluster-1 --data-dir etcd-data
# Run the following command on second node
etcd --name etcd2 --cert-file=/root/cert/etcd2.pem --key-file=/root/cert/etcd2-key.pem \
--peer-client-cert-auth --peer-trusted-ca-file=/root/cert/ca.pem \
--peer-cert-file=/root/cert/etcd2.pem --peer-key-file=/root/cert/etcd2-key.pem \
--client-cert-auth --trusted-ca-file=/root/cert/ca.pem \
--advertise-client-urls=https://[::]:2379 --listen-client-urls=https://[::]:2379 \
--initial-advertise-peer-urls=https://[::]:2380 --listen-peer-urls=https://[::]:2380 \
--initial-cluster etcd1=https://[2a0a:e5c0:0:5:0:78ff:fe11:d761]:2380,etcd2=https://[2a0a:e5c0:0:5:0:78ff:fe11:d762]:2380,etcd3=https://[2a0a:e5c0:0:5:0:78ff:fe11:d763]:2380 \
--initial-cluster-state new --initial-cluster-token etcd-cluster-1 --data-dir etcd-data
# Run the following command on third node
etcd --name etcd3 --cert-file=/root/cert/etcd3.pem --key-file=/root/cert/etcd3-key.pem \
--peer-client-cert-auth --peer-trusted-ca-file=/root/cert/ca.pem \
--peer-cert-file=/root/cert/etcd3.pem --peer-key-file=/root/cert/etcd3-key.pem \
--client-cert-auth --trusted-ca-file=/root/cert/ca.pem \
--advertise-client-urls=https://[::]:2379 --listen-client-urls=https://[::]:2379 \
--initial-advertise-peer-urls=https://[2a0a:e5c0:0:5:0:78ff:fe11:d763]:2380 --listen-peer-urls=https://[2a0a:e5c0:0:5:0:78ff:fe11:d763]:2380 \
--initial-cluster etcd1=https://[2a0a:e5c0:0:5:0:78ff:fe11:d761]:2380,etcd2=https://[2a0a:e5c0:0:5:0:78ff:fe11:d762]:2380,etcd3=https://[2a0a:e5c0:0:5:0:78ff:fe11:d763]:2380 \
--initial-cluster-state new --initial-cluster-token etcd-cluster-1 --data-dir etcd-data
AB Updated by Ahmed Bilal over 6 years ago Actions #5
- Status changed from Seen to In Progress
AB Updated by Ahmed Bilal over 6 years ago Actions #6
Correct, permissions
chown -R etcd:etcd /var/lib/etcd/
¶
chown -R etcd:etcd /var/lib/etcd/
Queries to check if things are working correctly¶
Write something
ETCDCTL_API=3 etcdctl --endpoints https://[2a0a:e5c0:0:5:0:78ff:fe11:d761]:2379 --cacert ca.pem --cert root.pem --key root-key.pem put /v1 abc
Read it
ETCDCTL_API=3 etcdctl --endpoints https://[2a0a:e5c0:0:5:0:78ff:fe11:d761]:2379 --cacert ca.pem --cert root.pem --key root-key.pem get /v1
Enable Authentication¶
- Create root user, grant it root role and Enable Authentication**
ETCDCTL_API=3 etcdctl --endpoints https://[2a0a:e5c0:0:5:0:78ff:fe11:d761]:2379 --cacert ca.pem --cert root.pem --key root-key.pem user add root ETCDCTL_API=3 etcdctl --endpoints https://[2a0a:e5c0:0:5:0:78ff:fe11:d761]:2379 --cacert ca.pem --cert root.pem --key root-key.pem user grant-role root root ETCDCTL_API=3 etcdctl --endpoints https://[2a0a:e5c0:0:5:0:78ff:fe11:d761]:2379 --cacert ca.pem --cert root.pem --key root-key.pem auth enable
Create a non-root User e.g developer¶
ETCDCTL_API=3 etcdctl --endpoints https://[2a0a:e5c0:0:5:0:78ff:fe11:d761]:2379 --cacert ca.pem --cert root.pem --key root-key.pem role add developer
ETCDCTL_API=3 etcdctl --endpoints https://[2a0a:e5c0:0:5:0:78ff:fe11:d761]:2379 --cacert ca.pem --cert root.pem --key root-key.pem role grant-permission developer --prefix=true readwrite /v1
ETCDCTL_API=3 etcdctl --endpoints https://[2a0a:e5c0:0:5:0:78ff:fe11:d761]:2379 --cacert ca.pem --cert root.pem --key root-key.pem user add developer
ETCDCTL_API=3 etcdctl --endpoints https://[2a0a:e5c0:0:5:0:78ff:fe11:d761]:2379 --cacert ca.pem --cert root.pem --key root-key.pem user grant-role developer developer
AB Updated by Ahmed Bilal over 6 years ago Actions #7
ETCD is having some issues.
Specifically, it is saying Cluster ID mismatch I file an issue in ETCD's Github repository. https://github.com/etcd-io/etcd/issues/11263
AB Updated by Ahmed Bilal over 6 years ago Actions #8
Issue fixed.
AB Updated by Ahmed Bilal over 6 years ago Actions #9
[meow@meow-pc cert]$ ETCDCTL_API=3 etcdctl --endpoints https://[2a0a:e5c0:2:12:0:f0ff:fea9:c43a]:2379,https://[2a0a:e5c0:2:12:0:f0ff:fea9:c43d]:2379,https://[2a0a:e5c0:2:12:0:f0ff:fea9:c442]:2379 --cacert ~/Desktop/ungleich-issues/7122/cert/ca.pem --cert ~/Desktop/ungleich-issues/7122/cert/client.pem --key ~/Desktop/ungleich-issues/7122/cert/client-key.pem endpoint health
https://[2a0a:e5c0:2:12:0:f0ff:fea9:c442]:2379 is healthy: successfully committed proposal: took = 631.531912ms
https://[2a0a:e5c0:2:12:0:f0ff:fea9:c43d]:2379 is healthy: successfully committed proposal: took = 633.007889ms
https://[2a0a:e5c0:2:12:0:f0ff:fea9:c43a]:2379 is healthy: successfully committed proposal: took = 634.894405ms
AB Updated by Ahmed Bilal over 6 years ago Actions #10
- % Done changed from 0 to 80
cdist type is ready. Testing underway.
AB Updated by Ahmed Bilal over 6 years ago Actions #11
- Deployed at place6
- Authentication enabled. Only clients with valid certificate issued by ungleich's private CA authority can access the etcd.
- Even finer control is employed by setting permissions for individual user to access specific keys or key's prefixes.
Only, backup is remaining.
ETCDCTL_API=3 etcdctl --endpoints https://etcd1.ungleich.ch:2379,https://etcd2.ungleich.ch:2379,https://etcd3.ungleich.ch:2379 --cacert ca.pem --cert developer.pem --key developer-key.pem endpoint health
https://etcd2.ungleich.ch:2379 is healthy: successfully committed proposal: took = 823.064847ms
https://etcd1.ungleich.ch:2379 is healthy: successfully committed proposal: took = 824.459603ms
https://etcd3.ungleich.ch:2379 is healthy: successfully committed proposal: took = 850.761864ms
NS Updated by Nico Schottelius over 6 years ago Actions #12
Is it already in cdist?
redmine@ungleich.ch writes:
AB Updated by Ahmed Bilal over 6 years ago Actions #13
@Nico Schottelius Yes, it is in etcd-cluster branch
AB Updated by Ahmed Bilal over 6 years ago Actions #14
- Assignee changed from Ahmed Bilal to Dominique Roux
Handing it over to rouxdo for review and future maintaining.
AB Updated by Ahmed Bilal over 6 years ago Actions #15
- Status changed from In Progress to Feedback
DR Updated by Dominique Roux about 6 years ago Actions #16
- Status changed from Feedback to Resolved
This is done now
ETCD-Cluster available at:
etcd1.ungleich.ch
etcd2.ungleich.ch
etcd3.ungleich.ch
Currently there are some small problems with nftables (not loaded at boot).
Will contact alpine linux dev team
DR Updated by Dominique Roux about 6 years ago Actions #17
Dominique Roux wrote:
...
Currently there are some small problems with nftables (not loaded at boot).
Will contact alpine linux dev team
nft problem is fixed now.
Problem was: Alpine has it's own init.d script (which works ;-) ). The cdist type was already updated but the submodule was not.
The submodule is now updated too, therefore, this should not happen again in future.
NS Updated by Nico Schottelius about 5 years ago Actions #18
- Assignee changed from Dominique Roux to Nico Schottelius
NS Updated by Nico Schottelius about 2 years ago Actions #19
- Status changed from Resolved to Closed
Actions