Project

General

Profile

The ungleich ceph handbook » History » Version 9

Nico Schottelius, 11/07/2018 04:59 PM

1 1 Nico Schottelius
h1. The ungleich ceph handbook
2
3 3 Nico Schottelius
{{toc}}
4
5 1 Nico Schottelius
h2. Status
6
7 7 Nico Schottelius
This document is **IN PRODUCTION**.
8 1 Nico Schottelius
9
h2. Introduction
10
11
This article describes the ungleich storage architecture that is based on ceph. It describes our architecture as well maintenance commands. Required for 
12
13
h2. Communication guide
14
15
Usually when disks fails no customer communication is necessary, as it is automatically compensated/rebalanced by ceph. However in case multiple disk failures happen at the same time, I/O speed might be reduced and thus customer experience impacted.
16
17
For this reason communicate whenever I/O recovery settings are temporarily tuned.
18
19 2 Nico Schottelius
h2. Adding a new disk/ssd to the ceph cluster
20
21
h3. For Dell servers
22
23
First find the disk and then add it to the operating system
24
25
<pre>
26
megacli -PDList -aALL  | grep -B16 -i unconfigur
27
28
# Sample output:
29
[19:46:50] server7.place6:~#  megacli -PDList -aALL  | grep -B16 -i unconfigur
30
Enclosure Device ID: N/A
31
Slot Number: 0
32
Enclosure position: N/A
33
Device Id: 0
34
WWN: 0000000000000000
35
Sequence Number: 1
36
Media Error Count: 0
37
Other Error Count: 0
38
Predictive Failure Count: 0
39
Last Predictive Failure Event Seq Number: 0
40
PD Type: SATA
41
42
Raw Size: 894.252 GB [0x6fc81ab0 Sectors]
43
Non Coerced Size: 893.752 GB [0x6fb81ab0 Sectors]
44
Coerced Size: 893.75 GB [0x6fb80000 Sectors]
45
Sector Size:  0
46
Firmware state: Unconfigured(good), Spun Up
47
</pre>
48
49
Then add the disk to the OS:
50
51
<pre>
52
megacli -CfgLdAdd -r0 [enclosure:slot] -aX
53
54
# Sample call, if enclosure and slot are KNOWN (aka not N/A)
55
megacli -CfgLdAdd -r0 [32:0] -a0
56
57
# Sample call, if enclosure is N/A
58
megacli -CfgLdAdd -r0 [:0] -a0
59
</pre>
60
61 1 Nico Schottelius
h2. Moving a disk/ssd to another server
62 4 Nico Schottelius
63
(needs to be described better)
64
65
Generally speaking:
66
67 9 Nico Schottelius
* /opt/ungleich-tools/ceph-osd-stop-disable does the following:
68
** Stop the osd, remove monit on the server you want to take it out
69
** umount the disk
70 1 Nico Schottelius
* Take disk out
71
* Discard preserved cache on the server you took it out 
72 9 Nico Schottelius
** using megacli
73 1 Nico Schottelius
* Insert into new server
74 9 Nico Schottelius
* Clear foreign configuration
75
** using megacli
76
* Disk will now appear in the OS, ceph/udev will automatically start the OSD (!)
77
** No creating of the osd required!
78
* Verify that the disk exists and that the osd is started
79
** using *ps aux*
80
** using *ceph osd tree*
81
* /opt/ungleich-tools/monit-ceph-create-start osd.XX # where osd.XX is the osd + number
82
** Creates the monit configuration file so that monit watches the OSD
83
** Reload monit
84 1 Nico Schottelius
85
h2. Removing a disk/ssd
86 5 Nico Schottelius
87
To permanently remove a failed disk from a cluster, use ***ceph-osd-stop-remove-permanently*** from ungleich-tools repo. Warning: if the disk is still active, the OSD will be shutdown AND removed from the cluster -> all data of that disk will need to be rebalanced.
88 1 Nico Schottelius
89
h2. Handling DOWN osds with filesystem errors
90
91
If an email arrives with the subject "monit alert -- Does not exist osd.XX-whoami", the filesystem of an OSD cannot be read anymore. It is very highly likely that the disk / ssd is broken. Steps that need to be done:
92
93
* Login to any ceph monitor (cephX.placeY.ungleich.ch)
94
* Check **ceph -s**, find host using **ceph osd tree**
95
* Login to the affected host
96
* Run the following commands:
97
** ls /var/lib/ceph/osd/ceph-XX
98
** dmesg
99
* Create a new ticket in the datacenter light project
100
** Subject: "Replace broken OSD.XX on serverX.placeY.ungleich.ch"
101
** Add (partial) output of above commands
102
** Use /opt/ungleich-tools/ceph-osd-stop-remove-permanently XX, where XX is the osd id, to remove the disk from the cluster
103
** Remove the physical disk from the host, checkout if there is warranty on it and if yes
104
*** Create a short letter to the vendor, including technical details a from above
105
*** Record when you sent it in
106
*** Put ticket into status waiting
107
** If there is no warranty, dispose it
108
109
110
111
h2. Change ceph speed for i/o recovery
112
113
By default we want to keep I/O recovery traffic low to not impact customer experience. However when multiple disks fail at the same point, we might want to prioritise recover for data safety over performance.
114
115
The default configuration on our servers contains:
116
117
<pre>
118
[osd]
119
osd max backfills = 1
120
osd recovery max active = 1
121
osd recovery op priority = 2
122
</pre>
123
124
The important settings are *osd max backfills* and *osd recovery max active*, the priority is always kept low so that regular I/O has priority.
125
126
To adjust the number of backfills *per osd* and to change the *number of threads* used for recovery, we can use on any node with the admin keyring:
127
128
<pre>
129
ceph tell osd.* injectargs '--osd-max-backfills Y'
130
ceph tell osd.* injectargs '--osd-recovery-max-active X'
131
</pre>
132
133
where Y and X are the values that we want to use. Experience shows that Y=5 and X=5 doubles to triples the recovery performance, whereas X=10 and Y=10 increases recovery performance 5 times.
134
135
h2. Debug scrub errors / inconsistent pg message
136 6 Nico Schottelius
137 1 Nico Schottelius
From time to time disks don't save what they are told to save. Ceph scrubbing detects these errors and switches to HEALTH_ERR. Use *ceph health detail* to find out which placement groups (*pgs*) are affected. Usually a ***ceph pg repair <number>*** fixes the problem.
138
139
If this does not help, consult https://ceph.com/geen-categorie/ceph-manually-repair-object/.