Sameer Shah created MESOS-7174:
----------------------------------
Summary: Mesos Agent crashes when it is unable to reconnect to
zookeeper
Key: MESOS-7174
URL: https://issues.apache.org/jira/browse/MESOS-7174
Project: Mesos
Issue Type: Bug
Components: agent
Reporter: Sameer Shah
Mesos agent crashed when it was not able to reconnect to zookeeper. Here are
relevant logs. I have removed hostnames and ip's from the logs and replace with
HOSTNAME_1, IP_1, etc.
{quote}
systemd[1]: Starting Mesos Agent...
systemd[1]: Started Mesos Agent.
systemd[1]: mesos-agent.service holdoff time over, scheduling restart.
systemd[1]: mesos-agent.service failed.
systemd[1]: Unit mesos-agent.service entered failed state.
systemd[1]: mesos-agent.service: main process exited, code=killed, status=6/ABRT
mesos-agent[23576]: @ 0x7f933dd9e28d __clone
mesos-agent[23576]: @ 0x7f933e58ddc5 start_thread
mesos-agent[23576]: @ 0x7f933e3341e0 (unknown)
mesos-agent[23576]: @ 0x7f933fe8cf57
_ZNSt6thread5_ImplISt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvEUt_vEEE6_M_runEv
mesos-agent[23576]: @ 0x7f933fe8cca1 process::ProcessManager::resume()
mesos-agent[23576]: @ 0x7f933f98cb88 ZooKeeperProcess::initialize()
mesos-agent[23576]: @ 0x7f933fefd0b2
google::ErrnoLogMessage::~ErrnoLogMessage()
mesos-agent[23576]: @ 0x7f933fefc149 google::LogMessage::~LogMessage()
mesos-agent[23576]: @ 0x7f933fefbf3c google::LogMessage::Flush()
mesos-agent[23576]: @ 0x7f933fefe08c google::LogMessage::SendToLog()
mesos-agent[23576]: @ 0x7f933fefc34d google::LogMessage::Fail()
mesos-agent[23576]: *** Check failure stack trace: ***
mesos-agent[23576]: F0223 05:10:26.906946 23598 zookeeper.cpp:132] Failed to
create ZooKeeper, zookeeper_init: No such file or directory [2]
mesos-agent[23576]: 2017-02-23
05:10:26,906:23576(0x7f933198f700):ZOO_ERROR@getaddrs@613: getaddrinfo: No such
file or directory
mesos-agent[23576]: W0223 05:10:20.030608 23608 slave.cpp:1480] Ignoring run
task message from master@IP:5050 because it is not the expected master: None
mesos-agent[23576]: 2017-02-23
05:09:46,731:23576(0x7f933198f700):ZOO_INFO@zookeeper_init@800: Initiating
client connection,
host=HOSTNAME_1:2181,HOSTNAME_2:2181,HOSTNAME_3:2181,HOSTNAME_4:2181,HOSTNAME_5:2181
sessionTimeout=10000 watcher=0x7f933f98c300 sessionId=0 sessionPasswd=<null>
context=0x7f92d00406a0 flags=0
mesos-agent[23576]: 2017-02-23
05:09:46,731:23576(0x7f933198f700):ZOO_INFO@log_env@767: Client
environment:user.dir=/
mesos-agent[23576]: 2017-02-23
05:09:46,731:23576(0x7f933198f700):ZOO_INFO@log_env@755: Client
environment:user.home=/root
mesos-agent[23576]: 2017-02-23
05:09:46,731:23576(0x7f933198f700):ZOO_INFO@log_env@747: Client
environment:user.name=(null)
mesos-agent[23576]: 2017-02-23
05:09:46,731:23576(0x7f933198f700):ZOO_INFO@log_env@739: Client
environment:os.version=####
mesos-agent[23576]: 2017-02-23
05:09:46,731:23576(0x7f933198f700):ZOO_INFO@log_env@738: Client
environment:os.arch=####
mesos-agent[23576]: 2017-02-23
05:09:46,731:23576(0x7f933198f700):ZOO_INFO@log_env@737: Client
environment:os.name=Linux
mesos-agent[23576]: 2017-02-23
05:09:46,731:23576(0x7f933198f700):ZOO_INFO@log_env@730: Client
environment:host.name=HOSTNAME
mesos-agent[23576]: 2017-02-23
05:09:46,731:23576(0x7f933198f700):ZOO_INFO@log_env@726: Client
environment:zookeeper.version=zookeeper C client 3.4.8
mesos-agent[23576]: I0223 05:09:46.722960 23607 slave.cpp:927] Detecting new
master
mesos-agent[23576]: I0223 05:09:46.722923 23607 slave.cpp:888] Lost leading
master
mesos-agent[23576]: I0223 05:09:46.722776 23589 status_update_manager.cpp:174]
Pausing sending status updates
mesos-agent[23576]: 2017-02-23
05:09:46,722:23576(0x7f932e989700):ZOO_INFO@zookeeper_close@2543: Freeing
zookeeper resources for sessionId=0x300007df99d24f2
mesos-agent[23576]: I0223 05:09:46.722249 23609 detector.cpp:152] Detected a
new leader: None
mesos-agent[23576]: I0223 05:09:46.722100 23588 group.cpp:519] ZooKeeper
session expired
mesos-agent[23576]: W0223 05:09:46.721179 23588 group.cpp:503] Timed out
waiting to connect to ZooKeeper. Forcing ZooKeeper session
(sessionId=300007df99d24f2) expiration
mesos-agent[23576]: 2017-02-23
05:09:43,386:23576(0x7f932ad69700):ZOO_ERROR@handle_socket_error_msg@1666:
Socket [IP2:2181] zk retcode=-7, errno=110(Connection timed out): connection to
IP2:2181 timed out (exceeded timeout by 2ms)
mesos-agent[23576]: I0223 05:09:36.719043 23592 group.cpp:460] Lost connection
to ZooKeeper, attempting to reconnect ...
mesos-agent[23576]: 2017-02-23
05:09:36,718:23576(0x7f932ad69700):ZOO_ERROR@handle_socket_error_msg@1666:
Socket [IP1:2181] zk retcode=-7, errno=110(Connection timed out): connection to
IP1:2181 timed out (exceeded timeout by 4ms)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
