Project

General

Profile

Actions
Copy link

Task #7122

closed

Setup production etcd cluster in place6

Added by Nico Schottelius over 4 years ago. Updated 4 months ago.

Status:
Closed
Priority:
Normal
Assignee:
Nico Schottelius
Target version:
-
Start date:
09/09/2019
Due date:
% Done:

80%

Estimated time:
PM Check date:

Description

  • 3 nodes
  • Ensure that sufficient permissions are used to secure access to etcd
  • Include hourly backup to place5
    • Check whether we need to make a dump or can backup the data directory directly
  • Probably include letsencrypt (?) for CAs / encryption
    • Or private CA
Actions
Copy link
 #1

Updated by Ahmed Bilal over 4 years ago

  • Status changed from New to Seen
Actions
Copy link
 #3

Updated by Ahmed Bilal over 4 years ago

ca-config.json

{
    "signing": {
        "default": {
            "expiry": "43800h" 
        },
        "profiles": {
            "server": {
                "expiry": "43800h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth" 
                ]
            },
            "client": {
                "expiry": "43800h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "client auth" 
                ]
            },
            "peer": {
                "expiry": "43800h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth" 
                ]
            }
        }
    }
}

ca-csr.json

{
    "CN": "ungleich",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CH",
            "ST": "Glarus" 
        }
    ]
}

cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

etcd1.json

{
    "CN": "etcd1",
    "hosts": [
        "2a0a:e5c0:0:5:0:78ff:fe11:d761",
        "etcd1" 
    ],
    "key": {
        "algo": "ecdsa",
        "size": 256
    },
    "names": [
        {
            "C": "CH",
            "ST": "Glarus" 
        }
    ]
}


cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server etcd1.json | cfssljson -bare etcd1

etcd2.json

{
    "CN": "etcd2",
    "hosts": [
        "2a0a:e5c0:0:5:0:78ff:fe11:d762",
        "etcd2" 
    ],
    "key": {
        "algo": "ecdsa",
        "size": 256
    },
    "names": [
        {
            "C": "CH",
            "ST": "Glarus" 
        }
    ]
}


cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server etcd2.json | cfssljson -bare etcd2

etcd3.json

{
    "CN": "etcd3",
    "hosts": [
        "2a0a:e5c0:0:5:0:78ff:fe11:d763",
        "etcd3" 
    ],
    "key": {
        "algo": "ecdsa",
        "size": 256
    },
    "names": [
        {
            "C": "CH",
            "ST": "Glarus" 
        }
    ]
}


cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server etcd3.json | cfssljson -bare etcd3

client.json

{
    "CN": "client",
    "hosts": [""],
    "key": {
        "algo": "ecdsa",
        "size": 256
    },
    "names": [
        {
            "C": "CH",
            "ST": "Glarus" 
        }
    ]
}

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client client.json | cfssljson -bare client

root.json

{
    "CN": "root",
    "hosts": [""],
    "key": {
        "algo": "ecdsa",
        "size": 256
    },
    "names": [
        {
            "C": "CH",
            "ST": "Glarus" 
        }
    ]
}

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client root.json | cfssljson -bare root

developer.json

{
    "CN": "developer",
    "hosts": [""],
    "key": {
        "algo": "ecdsa",
        "size": 256
    },
    "names": [
        {
            "C": "CH",
            "ST": "Glarus" 
        }
    ]
}

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client developer.json | cfssljson -bare developer
Actions
Copy link
 #4

Updated by Ahmed Bilal over 4 years ago

To Start Fresh

rm -rf /var/lib/etcd /root/etcd-data/


# Run the following command on first node

etcd --name etcd1 --cert-file=/root/cert/etcd1.pem --key-file=/root/cert/etcd1-key.pem \
--peer-client-cert-auth --peer-trusted-ca-file=/root/cert/ca.pem \
--peer-cert-file=/root/cert/etcd1.pem --peer-key-file=/root/cert/etcd1-key.pem \
--client-cert-auth --trusted-ca-file=/root/cert/ca.pem \
--advertise-client-urls=https://[::]:2379 --listen-client-urls=https://[::]:2379 \
--initial-advertise-peer-urls=https://[::]:2380 --listen-peer-urls=https://[::]:2380 \
--initial-cluster etcd1=https://[::]:2380,etcd2=https://[2a0a:e5c0:0:5:0:78ff:fe11:d762]:2380,etcd3=https://[2a0a:e5c0:0:5:0:78ff:fe11:d763]:2380 \
--initial-cluster-state new --initial-cluster-token etcd-cluster-1 --data-dir etcd-data

# Run the following command on second node

etcd --name etcd2 --cert-file=/root/cert/etcd2.pem --key-file=/root/cert/etcd2-key.pem \
--peer-client-cert-auth --peer-trusted-ca-file=/root/cert/ca.pem \
--peer-cert-file=/root/cert/etcd2.pem --peer-key-file=/root/cert/etcd2-key.pem \
--client-cert-auth --trusted-ca-file=/root/cert/ca.pem \
--advertise-client-urls=https://[::]:2379 --listen-client-urls=https://[::]:2379 \
--initial-advertise-peer-urls=https://[::]:2380 --listen-peer-urls=https://[::]:2380 \
--initial-cluster etcd1=https://[2a0a:e5c0:0:5:0:78ff:fe11:d761]:2380,etcd2=https://[2a0a:e5c0:0:5:0:78ff:fe11:d762]:2380,etcd3=https://[2a0a:e5c0:0:5:0:78ff:fe11:d763]:2380 \
--initial-cluster-state new --initial-cluster-token etcd-cluster-1 --data-dir etcd-data

# Run the following command on third node

etcd --name etcd3 --cert-file=/root/cert/etcd3.pem --key-file=/root/cert/etcd3-key.pem \
--peer-client-cert-auth --peer-trusted-ca-file=/root/cert/ca.pem \
--peer-cert-file=/root/cert/etcd3.pem --peer-key-file=/root/cert/etcd3-key.pem \
--client-cert-auth --trusted-ca-file=/root/cert/ca.pem \
--advertise-client-urls=https://[::]:2379 --listen-client-urls=https://[::]:2379 \
--initial-advertise-peer-urls=https://[2a0a:e5c0:0:5:0:78ff:fe11:d763]:2380 --listen-peer-urls=https://[2a0a:e5c0:0:5:0:78ff:fe11:d763]:2380 \
--initial-cluster etcd1=https://[2a0a:e5c0:0:5:0:78ff:fe11:d761]:2380,etcd2=https://[2a0a:e5c0:0:5:0:78ff:fe11:d762]:2380,etcd3=https://[2a0a:e5c0:0:5:0:78ff:fe11:d763]:2380 \
--initial-cluster-state new --initial-cluster-token etcd-cluster-1 --data-dir etcd-data
Actions
Copy link
 #5

Updated by Ahmed Bilal over 4 years ago

  • Status changed from Seen to In Progress
Actions
Copy link
 #6

Updated by Ahmed Bilal over 4 years ago

Correct, permissions
chown -R etcd:etcd /var/lib/etcd/

Queries to check if things are working correctly

Write something

ETCDCTL_API=3 etcdctl --endpoints https://[2a0a:e5c0:0:5:0:78ff:fe11:d761]:2379 --cacert ca.pem --cert root.pem --key root-key.pem put /v1 abc

Read it

ETCDCTL_API=3 etcdctl --endpoints https://[2a0a:e5c0:0:5:0:78ff:fe11:d761]:2379 --cacert ca.pem --cert root.pem --key root-key.pem get /v1

Enable Authentication

  • Create root user, grant it root role and Enable Authentication**
    ETCDCTL_API=3 etcdctl --endpoints https://[2a0a:e5c0:0:5:0:78ff:fe11:d761]:2379 --cacert ca.pem --cert root.pem --key root-key.pem user add root

ETCDCTL_API=3 etcdctl --endpoints https://[2a0a:e5c0:0:5:0:78ff:fe11:d761]:2379 --cacert ca.pem --cert root.pem --key root-key.pem user grant-role root root

ETCDCTL_API=3 etcdctl --endpoints https://[2a0a:e5c0:0:5:0:78ff:fe11:d761]:2379 --cacert ca.pem --cert root.pem --key root-key.pem auth enable

Create a non-root User e.g developer

ETCDCTL_API=3 etcdctl --endpoints https://[2a0a:e5c0:0:5:0:78ff:fe11:d761]:2379 --cacert ca.pem --cert root.pem --key root-key.pem role add developer

ETCDCTL_API=3 etcdctl --endpoints https://[2a0a:e5c0:0:5:0:78ff:fe11:d761]:2379 --cacert ca.pem --cert root.pem --key root-key.pem role grant-permission developer --prefix=true readwrite /v1

ETCDCTL_API=3 etcdctl --endpoints https://[2a0a:e5c0:0:5:0:78ff:fe11:d761]:2379 --cacert ca.pem --cert root.pem --key root-key.pem user add developer

ETCDCTL_API=3 etcdctl --endpoints https://[2a0a:e5c0:0:5:0:78ff:fe11:d761]:2379 --cacert ca.pem --cert root.pem --key root-key.pem user grant-role developer developer
Actions
Copy link
 #7

Updated by Ahmed Bilal over 4 years ago

ETCD is having some issues.

Specifically, it is saying Cluster ID mismatch I file an issue in ETCD's Github repository. https://github.com/etcd-io/etcd/issues/11263

Actions
Copy link
 #8

Updated by Ahmed Bilal over 4 years ago

Issue fixed.

Actions
Copy link
 #9

Updated by Ahmed Bilal over 4 years ago 

[meow@meow-pc cert]$ ETCDCTL_API=3 etcdctl --endpoints https://[2a0a:e5c0:2:12:0:f0ff:fea9:c43a]:2379,https://[2a0a:e5c0:2:12:0:f0ff:fea9:c43d]:2379,https://[2a0a:e5c0:2:12:0:f0ff:fea9:c442]:2379 --cacert ~/Desktop/ungleich-issues/7122/cert/ca.pem --cert ~/Desktop/ungleich-issues/7122/cert/client.pem --key ~/Desktop/ungleich-issues/7122/cert/client-key.pem endpoint health
https://[2a0a:e5c0:2:12:0:f0ff:fea9:c442]:2379 is healthy: successfully committed proposal: took = 631.531912ms
https://[2a0a:e5c0:2:12:0:f0ff:fea9:c43d]:2379 is healthy: successfully committed proposal: took = 633.007889ms
https://[2a0a:e5c0:2:12:0:f0ff:fea9:c43a]:2379 is healthy: successfully committed proposal: took = 634.894405ms
Actions
Copy link
 #10

Updated by Ahmed Bilal over 4 years ago

  • % Done changed from 0 to 80

cdist type is ready. Testing underway.

Actions
Copy link
 #11

Updated by Ahmed Bilal over 4 years ago

  • Deployed at place6
  • Authentication enabled. Only clients with valid certificate issued by ungleich's private CA authority can access the etcd.
  • Even finer control is employed by setting permissions for individual user to access specific keys or key's prefixes.

Only, backup is remaining.

ETCDCTL_API=3 etcdctl --endpoints https://etcd1.ungleich.ch:2379,https://etcd2.ungleich.ch:2379,https://etcd3.ungleich.ch:2379 --cacert ca.pem --cert developer.pem --key developer-key.pem endpoint health
https://etcd2.ungleich.ch:2379 is healthy: successfully committed proposal: took = 823.064847ms
https://etcd1.ungleich.ch:2379 is healthy: successfully committed proposal: took = 824.459603ms
https://etcd3.ungleich.ch:2379 is healthy: successfully committed proposal: took = 850.761864ms
Actions
Copy link
 #12

Updated by Nico Schottelius over 4 years ago

Is it already in cdist?

writes:

Actions
Copy link
 #13

Updated by Ahmed Bilal over 4 years ago

@Nico Schottelius Yes, it is in etcd-cluster branch

Actions
Copy link
 #14

Updated by Ahmed Bilal over 4 years ago

  • Assignee changed from Ahmed Bilal to Dominique Roux

Handing it over to rouxdo for review and future maintaining.

Actions
Copy link
 #15

Updated by Ahmed Bilal over 4 years ago

  • Status changed from In Progress to Feedback
Actions
Copy link
 #16

Updated by Dominique Roux over 4 years ago

  • Status changed from Feedback to Resolved

This is done now

ETCD-Cluster available at:

etcd1.ungleich.ch
etcd2.ungleich.ch
etcd3.ungleich.ch

Currently there are some small problems with nftables (not loaded at boot).
Will contact alpine linux dev team

Actions
Copy link
 #17

Updated by Dominique Roux over 4 years ago

Dominique Roux wrote:

...

Currently there are some small problems with nftables (not loaded at boot).
Will contact alpine linux dev team

nft problem is fixed now.
Problem was: Alpine has it's own init.d script (which works ;-) ). The cdist type was already updated but the submodule was not.
The submodule is now updated too, therefore, this should not happen again in future.

Actions
Copy link
 #18

Updated by Nico Schottelius over 3 years ago

  • Assignee changed from Dominique Roux to Nico Schottelius
Actions
Copy link
 #19

Updated by Nico Schottelius 4 months ago

  • Status changed from Resolved to Closed
Actions
Copy link

Also available in: Atom PDF