Project

General

Profile

Actions

Task #7625

closed

Manually fix consul+node_exporter on new router1.place6

Added by Nico Schottelius over 4 years ago. Updated over 2 years ago.

Status:
Rejected
Priority:
Normal
Target version:
-
Start date:
01/18/2020
Due date:
% Done:

0%

Estimated time:
PM Check date:

Description

[13:01] router1.place6:~# /etc/init.d/consul start
/etc/init.d/consul: .: line 39: can't open '/lib/lsb/init-functions': No such file or directory
[13:02] router1.place6:~# /etc/init.d/consul.apk-new  start
 * Checking /etc/consul ...
Config validation failed: Multiple private IPv4 addresses found. Please configure one with 'bind' and/or 'advertise'.                                                                                          [ !! ]
 * checkpath: owner `consul.apk-new:consul.apk-new' not found
 * ERROR: consul.apk-new failed to start
[13:02] router1.place6:~# 

Follow up

  • No init script deployment on alpine
  • Modify data_dir
  • Different configuration directory on alpine
  • Should use the new interface template go syntax, however it did not work on testing
Actions #1

Updated by Nico Schottelius over 4 years ago

  • Description updated (diff)
Actions #2

Updated by Nico Schottelius over 4 years ago

Use alpine's init script:

[13:02] router1.place6:~# mv /etc/init.d/consul.apk-new /etc/init.d/consul
[13:03] router1.place6:~# cat /etc/conf.d/consul 
# Consul startup
consul_opts="agent -config-dir=/etc/consul" 
[13:04] router1.place6:~# 

Changing path to what we used on debian

[13:04] router1.place6:/etc/consul# vi /etc/conf.d/consul
[13:05] router1.place6:/etc/consul# cat /etc/conf.d/consul
# Consul startup
consul_opts="agent -config-dir=/etc/consul/conf.d" 
[13:05] router1.place6:/etc/consul# /etc/init.d/consul start
 * Caching service dependencies ...                                                                                                                                                                            [ ok ]
 * Checking /etc/consul ...
Config validation failed: Multiple private IPv4 addresses found. Please configure one with 'bind' and/or 'advertise'.                                                                                          [ !! ]
 * /var/log/consul.log: creating file
 * /var/log/consul.log: correcting owner
 * Starting consul ...                                                                                                                                                                                         [ ok ]
[13:05] router1.place6:/etc/consul# 
[13:06] router1.place6:/etc/consul# /etc/init.d/consul start
 * Checking /etc/consul ...
Config validation failed: Multiple private IPv4 addresses found. Please configure one with 'bind' and/or 'advertise'.                                                                                          [ !! ]
 * Starting consul ...                                                                                                                                                                                         [ ok ]
[13:06] router1.place6:/etc/consul# cat conf.d/config.json 
{
   "data_dir": "/var/lib/consul" 
   ,"bind_addr": "2a0a:e5c0:2:15::5" 
   ,"client_addr": "2a0a:e5c0:2:15::5 ::1 127.0.0.1" 
   ,"datacenter": "place6" 
   ,"disable_remote_exec": true
   ,"disable_update_check": true
   ,"encrypt": "xx==" 
   ,"retry_join": ["black1.place6.ungleich.ch","black2.place6.ungleich.ch","black3.place6.ungleich.ch"]
}
[13:06] router1.place6:/etc/consul# consul --version
Consul 1.6.2
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)
[13:06] router1.place6:/etc/consul# 

Seem like old bind_addr now supports interfaces - much nicer!

[13:25] router1.place6:/etc/consul#  /etc/init.d/consul restart
 * Stopping consul ...                                                                              [ ok ]
 * Checking /etc/consul ...
Config validation failed: Multiple private IPv4 addresses found. Please configure one with 'bind' and/or 'advertise'.                                                                                         [ !! ]
 * Starting consul ...                                                                              [ ok ]
[13:25] router1.place6:/etc/consul# mv conf.d/*json .
[13:25] router1.place6:/etc/consul#  /etc/init.d/consul restart
 * Stopping consul ...                                                                              [ ok ]
 * Checking /etc/consul ...
Configuration is valid!                                                                             [ ok ]
 * Starting consul ...                                                                              [ ok ]
[13:25] router1.place6:/etc/consul# 

However it does not work -- at the moment hard coded to the routers ipv6 address.
Also the conf.d change does not seem to work.

Actions #3

Updated by Nico Schottelius over 4 years ago

  • Subject changed from Manually fix consul on new router1.place6 to Manually fix consul+node_exporter on new router1.place6
[13:25] router1.place6:/etc/consul# ps aux |grep node
14238 root      0:00 grep node
[13:26] router1.place6:/etc/consul# 
[13:26] router1.place6:/etc/consul# /etc/init.d/node-exporter start
 * Starting node-exporter ...                                                                       [ ok ]
[13:26] router1.place6:/etc/consul# ps aux |grep node
14259 promethe  0:00 /usr/bin/node_exporter
14272 root      0:00 grep node
[13:26] router1.place6:/etc/consul# 

[13:27] router1.place6:/etc/consul# curl -s localhost:9100/metrics | wc -l
1936
[13:27] router1.place6:/etc/consul# 

Actions #4

Updated by Nico Schottelius over 4 years ago

[13:30] router1.place6:/etc/consul# consul members
Error retrieving members: Get http://127.0.0.1:8500/v1/agent/members?segment=_all: dial tcp 127.0.0.1:8500: connect: connection refused
[13:30] router1.place6:/etc/consul# netstat -anp | grep consul

Seems to crash when running under supervise (probably permission error):

Jan 18 13:31:36 router1 daemon.info supervise-daemon[15251]: Child command line: /usr/sbin/consul agent -config-dir=/etc/consul 
Jan 18 13:31:36 router1 daemon.info supervise-daemon[15323]: Supervisor command line: supervise-daemon consul --start --stdout /var/log/consul.log --stderr /var/log/consul.log --pidfile /run/consul.pid --respawn-delay 10 --respawn-max 0 --healthcheck-timer 60 --user cons
Jan 18 13:31:36 router1 daemon.info supervise-daemon[15325]: Child command line: /usr/sbin/consul agent -config-dir=/etc/consul 
Jan 18 13:31:36 router1 daemon.warn supervise-daemon[15324]: /usr/sbin/consul, pid 15325, exited with return code 1
Jan 18 13:31:46 router1 daemon.info supervise-daemon[15353]: Child command line: /usr/sbin/consul agent -config-dir=/etc/consul 
Jan 18 13:31:47 router1 daemon.warn supervise-daemon[15324]: /usr/sbin/consul, pid 15353, exited with return code 1
Jan 18 13:31:49 router1 auth.info sshd[15231]: Connection closed by 123.207.78.83 port 44658 [preauth]
Jan 18 13:31:49 router1 auth.notice sshguard[4977]: Attack from "123.207.78.83" on service SSH with danger 2.
Jan 18 13:31:50 router1 daemon.warn bird: I/O loop cycle took 5291 ms for 1 events
Jan 18 13:31:57 router1 daemon.info supervise-daemon[15375]: Child command line: /usr/sbin/consul agent -config-dir=/etc/consul 
Jan 18 13:31:57 router1 daemon.warn supervise-daemon[15324]: /usr/sbin/consul, pid 15375, exited with return code 1
Actions #5

Updated by Nico Schottelius over 4 years ago

  • Description updated (diff)
Actions #6

Updated by Nico Schottelius over 2 years ago

  • Status changed from In Progress to Rejected
Actions

Also available in: Atom PDF