[
https://issues.apache.org/jira/browse/MESOS-8623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16381284#comment-16381284
]
Tomas Barton commented on MESOS-8623:
-------------------------------------
Anything specific I should be looking for? There doesn't appear to be much
useful information. Mesos starts reconciliation
{code}
I0227 03:00:34.023347 23757 master.cpp:7286] Performing implicit task state
reconciliation for framework ecd3a4be-d34c-46f3-b358-c4e26ac0d131-0000 (mar
I0227 03:00:35.845939 23754 master.cpp:7673] Sending 9 offers to framework
911c4b47-2ba7-4959-b59e-c48d896fe210-0005 (kafka) at scheduler-090f04ff-207f
I0227 03:00:35.849963 23757 master.cpp:5120] Processing DECLINE call for
offers: [ 7ad6bb9b-b7f8-4467-8391-51f40c297ac1-O27 ] for framework 911c4b47-2b
I0227 03:00:35.850232 23757 master.cpp:9170] Removing offer
7ad6bb9b-b7f8-4467-8391-51f40c297ac1-O27
I0227 03:00:35.850301 23757 master.cpp:5120] Processing DECLINE call for
offers: [ 7ad6bb9b-b7f8-4467-8391-51f40c297ac1-O28 ] for framework 911c4b47-2b
I0227 03:00:35.850574 23757 master.cpp:9170] Removing offer
7ad6bb9b-b7f8-4467-8391-51f40c297ac1-O28
I0227 03:00:35.850622 23757 master.cpp:5120] Processing DECLINE call for
offers: [ 7ad6bb9b-b7f8-4467-8391-51f40c297ac1-O29 ] for framework 911c4b47-2b
I0227 03:00:35.850850 23757 master.cpp:9170] Removing offer
7ad6bb9b-b7f8-4467-8391-51f40c297ac1-O29
{code}
then hundreds of declined offers follows (over 1000 to be more specific), until
it crashes:
{code}
I0227 02:39:20.596915 31377 master.cpp:5120] Processing DECLINE call for
offers: [ 82c5b27e-5ef9-427e-862a-cc4092b3d8e4-O1031 ] for framework 911c4b47-
I0227 02:39:20.597039 31377 master.cpp:9170] Removing offer
82c5b27e-5ef9-427e-862a-cc4092b3d8e4-O1031
I0227 02:39:20.597054 31377 master.cpp:5120] Processing DECLINE call for
offers: [ 82c5b27e-5ef9-427e-862a-cc4092b3d8e4-O1032 ] for framework 911c4b47-
I0227 02:39:20.597149 31377 master.cpp:9170] Removing offer
82c5b27e-5ef9-427e-862a-cc4092b3d8e4-O1032
Stopping Mesos Master...
Starting Mesos Master...
Started Mesos Master.
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0227 02:39:20.895939 23727 main.cpp:232] Build: 2017-11-18 02:15:41 by admin
I0227 02:39:20.895979 23727 main.cpp:233] Version: 1.4.1
I0227 02:39:20.895983 23727 main.cpp:236] Git tag: 1.4.1
I0227 02:39:20.895985 23727 main.cpp:240] Git SHA:
c844db9ac7c0cef59be87438c6781bfb71adcc42
I0227 02:39:20.896937 23727 main.cpp:340] Using 'HierarchicalDRF' allocator
I0227 02:39:20.934940 23727 replica.cpp:779] Replica recovered with log
positions 7483 -> 7484 with 0 holes and 0 unlearned
2018-02-27 02:39:20,935:23727(0x7fdb3bcea700):ZOO_INFO@log_env@726: Client
environment:zookeeper.version=zookeeper C client 3.4.8
{code}
> Crashed framework brings down the whole Mesos cluster
> -----------------------------------------------------
>
> Key: MESOS-8623
> URL: https://issues.apache.org/jira/browse/MESOS-8623
> Project: Mesos
> Issue Type: Bug
> Components: master
> Affects Versions: 1.4.1
> Environment: Debian 8
> Mesos 1.4.1
> Reporter: Tomas Barton
> Priority: Critical
>
> It might be hard to replicate, but when you do your Mesos cluster is gone.
> The issue was caused by an unresponsive Docker engine on a single agent node.
> Unfortunately even after fixing Docker issues, all Mesos masters repeatedly
> failed to start. In despair I've deleted all {{replicated_log}} data from
> Master and ZooKeeper. Even after that messages agent's {{replicated_log}} got
> replayed and the master crashed again. Average lifetime for Mesos master was
> less than 1min.
> {code}
> mesos-master[3814]: I0228 00:25:55.269835 3828 network.hpp:436] ZooKeeper
> group memberships changed
> mesos-master[3814]: I0228 00:25:55.269979 3832 group.cpp:700] Trying to get
> '/mesos/log_replicas/0000002519' in ZooKeeper
> mesos-master[3814]: I0228 00:25:55.271117 3832 group.cpp:700] Trying to get
> '/mesos/log_replicas/0000002520' in ZooKeeper
> mesos-master[3814]: I0228 00:25:55.277971 3832 group.cpp:700] Trying to get
> '/mesos/log_replicas/0000002521' in ZooKeeper
> mesos-master[3814]: I0228 00:25:55.279296 3827 network.hpp:484] ZooKeeper
> group PIDs: { log-replica(1)
> mesos-master[3814]: W0228 00:26:15.261255 3831 master.hpp:2372] Master
> attempted to send message to disconnected framework
> 911c4b47-2ba7-4959-b59e-c48d896fe210-0005 (kafka)
> mesos-master[3814]: F0228 00:26:15.261318 3831 master.hpp:2382]
> CHECK_SOME(pid): is NONE
> mesos-master[3814]: *** Check failure stack trace: ***
> mesos-master[3814]: @ 0x7f7187ca073d google::LogMessage::Fail()
> mesos-master[3814]: @ 0x7f7187ca23bd google::LogMessage::SendToLog()
> mesos-master[3814]: @ 0x7f7187ca0302 google::LogMessage::Flush()
> mesos-master[3814]: @ 0x7f7187ca2da9
> google::LogMessageFatal::~LogMessageFatal()
> mesos-master[3814]: @ 0x7f7186d6d769 _CheckFatal::~_CheckFatal()
> mesos-master[3814]: @ 0x7f71870465d5
> mesos::internal::master::Framework::send<>()
> mesos-master[3814]: @ 0x7f7186fcfe8a
> mesos::internal::master::Master::executorMessage()
> mesos-master[3814]: @ 0x7f718706b1a1 ProtobufProcess<>::handler4<>()
> mesos-master[3814]: @ 0x7f7187008e36
> std::_Function_handler<>::_M_invoke()
> mesos-master[3814]: @ 0x7f71870293d1 ProtobufProcess<>::visit()
> mesos-master[3814]: @ 0x7f7186fb7ee4
> mesos::internal::master::Master::_visit()
> mesos-master[3814]: @ 0x7f7186fd0d5d
> mesos::internal::master::Master::visit()
> mesos-master[3814]: @ 0x7f7187c02e22 process::ProcessManager::resume()
> mesos-master[3814]: @ 0x7f7187c08d46
> _ZNSt6thread5_ImplISt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvEUlvE_vE
> mesos-master[3814]: @ 0x7f7185babca0 (unknown)
> mesos-master[3814]: @ 0x7f71853c6064 start_thread
> mesos-master[3814]: @ 0x7f71850fb62d (unknown)
> systemd[1]: mesos-master.service: main process exited, code=killed,
> status=6/ABRT
> systemd[1]: Unit mesos-master.service entered failed state.
> systemd[1]: mesos-master.service holdoff time over, scheduling restart.
> systemd[1]: Stopping Mesos Master...
> systemd[1]: Starting Mesos Master...
> systemd[1]: Started Mesos Master.
> mesos-master[27840]: WARNING: Logging before InitGoogleLogging() is written
> to STDERR
> mesos-master[27840]: I0228 01:32:38.294122 27829 main.cpp:232] Build:
> 2017-11-18 02:15:41 by admin
> mesos-master[27840]: I0228 01:32:38.294168 27829 main.cpp:233] Version: 1.4.1
> mesos-master[27840]: I0228 01:32:38.294178 27829 main.cpp:236] Git tag: 1.4.1
> mesos-master[27840]: I0228 01:32:38.294186 27829 main.cpp:240] Git SHA:
> c844db9ac7c0cef59be87438c6781bfb71adcc42
> mesos-master[27840]: I0228 01:32:38.296067 27829 main.cpp:340] Using
> 'HierarchicalDRF' allocator
> mesos-master[27840]: I0228 01:32:38.411576 27829 replica.cpp:779] Replica
> recovered with log positions 13 -> 14 with 0 holes and 0 unlearned
> mesos-master[27840]: 2018-02-28
> 01:32:38,412:27829(0x7fab44755700):ZOO_INFO@log_env@726: Client
> environment:zookeeper.version=zookeeper C client 3.4.8
> mesos-master[27840]: 2018-02-28
> 01:32:38,412:27829(0x7fab44755700):ZOO_INFO@log_env@730: Client
> environment:host.name=svc01
> mesos-master[27840]: 2018-02-28
> 01:32:38,412:27829(0x7fab44755700):ZOO_INFO@log_env@737: Client
> environment:os.name=Linux
> mesos-master[27840]: I0228 01:32:38.412711 27841 log.cpp:107] Attempting to
> join replica to ZooKeeper group
> mesos-master[27840]: 2018-02-28
> 01:32:38,412:27829(0x7fab4775b700):ZOO_INFO@log_env@726: Client
> environment:zookeeper.version=zookeeper C client 3.4.8
> mesos-master[27840]: 2018-02-28
> 01:32:38,412:27829(0x7fab4775b700):ZOO_INFO@log_env@730: Client
> environment:host.name=svc01
> mesos-master[27840]: I0228 01:32:38.412932 27841 recover.cpp:451] Starting
> replica recovery
> mesos-master[27840]: I0228 01:32:38.413024 27846 recover.cpp:477] Replica is
> in VOTING status
> mesos-master[27840]: 2018-02-28
> 01:32:38,412:27829(0x7fab4775b700):ZOO_INFO@log_env@737: Client
> environment:os.name=Linux
> mesos-master[27840]: 2018-02-28
> 01:32:38,413:27829(0x7fab4775b700):ZOO_INFO@log_env@738: Client
> environment:os.arch=3.16.0-5-amd64
> mesos-master[27840]: 2018-02-28
> 01:32:38,413:27829(0x7fab4775b700):ZOO_INFO@log_env@739: Client
> environment:os.version=#1 SMP Debian 3.16.51-3+deb8u1 (2018-01-08)
> mesos-master[27840]: 2018-02-28
> 01:32:38,413:27829(0x7fab47f5c700):ZOO_INFO@log_env@726: Client
> environment:zookeeper.version=zookeeper C client 3.4.8
> {code}
> on slave Mesos is trying many times to kill all instances of kafka framework:
> {code}
> I0227 23:14:57.993875 26049 slave.cpp:2931] Asked to kill task
> service_mesos-kafka_kafka.3bb2ccf6-1c13-11e8-a38c-fef9d3423c7f of framework
> ecd3a4be-d34
> W0227 23:14:57.993914 26049 slave.cpp:3099] Ignoring kill task
> service_mesos-kafka_kafka.3bb2ccf6-1c13-11e8-a38c-fef9d3423c7f because the
> executor 'ser
> I0227 23:15:02.993985 26048 slave.cpp:2931] Asked to kill task
> service_mesos-kafka_kafka.3bb2ccf6-1c13-11e8-a38c-fef9d3423c7f of framework
> ecd3a4be-d34
> W0227 23:15:02.994027 26048 slave.cpp:3099] Ignoring kill task
> service_mesos-kafka_kafka.3bb2ccf6-1c13-11e8-a38c-fef9d3423c7f because the
> executor 'ser
> I0227 23:15:07.992681 26041 slave.cpp:2931] Asked to kill task
> service_mesos-kafka_kafka.3bb2ccf6-1c13-11e8-a38c-fef9d3423c7f of framework
> ecd3a4be-d34
> W0227 23:15:07.992720 26041 slave.cpp:3099] Ignoring kill task
> service_mesos-kafka_kafka.3bb2ccf6-1c13-11e8-a38c-fef9d3423c7f because the
> executor 'ser
> I0227 23:15:12.992703 26039 slave.cpp:2931] Asked to kill task
> service_mesos-kafka_kafka.3bb2ccf6-1c13-11e8-a38c-fef9d3423c7f of framework
> ecd3a4be-d34
> {code}
> the Docker daemon stopped to respond, thus any tasks were failing:
> {code}
> E0227 23:06:20.440865 26044 slave.cpp:5292] Container
> '10763986-e133-4483-b34e-cfe10903a46f' for executor
> 'api_rapi-a.d25fc1e0-1c12-11e8-a38c-fef9d3423
> *** Aborted at 1519772780 (unix time) try "date -d @1519772780" if you are
> using GNU date ***
> PC: @ 0x7f1326961067 (unknown)
> PC: @ 0x7f1326961067 (unknown)
> *** SIGABRT (@0x47c0) received by PID 18368 (TID 0x7f131c27b700) from PID
> 18368; stack trace: ***
> @ 0x7f1326ce6890 (unknown)
> @ 0x7f1326961067 (unknown)
> @ 0x7f1326962448 (unknown)
> @ 0x560218e2f740 (unknown)
> @ 0x560218e2f77c (unknown)
> @ 0x7f1329261ff9 (unknown)
> @ 0x7f1329261c81 (unknown)
> @ 0x7f1329261c41 (unknown)
> @ 0x7f1329263ecf (unknown)
> @ 0x7f1329261561 (unknown)
> @ 0x7f1328422509 (unknown)
> @ 0x7f13284253ff (unknown)
> @ 0x7f1328433134 (unknown)
> @ 0x7f132844ab26 (unknown)
> @ 0x7f1328398a47 (unknown)
> @ 0x7f132839a9ed (unknown)
> @ 0x7f1329260bcc (unknown)
> @ 0x7f1328398a47 (unknown)
> @ 0x7f132839a9ed (unknown)
> @ 0x7f13289ec817 (unknown)
> @ 0x7f13289ec91f (unknown)
> @ 0x7f1329259818 (unknown)
> @ 0x7f1329259ac3 (unknown)
> @ 0x7f1329218e29 (unknown)
> @ 0x7f132921ed16 (unknown)
> @ 0x7f13271c3ca0 (unknown)
> @ 0x7f1326cdf064 start_thread
> @ 0x7f1326a1462d (unknown)
> '
> I0227 23:06:20.441429 26043 slave.cpp:5405] Executor
> 'api_rapi-a.d25fc1e0-1c12-11e8-a38c-fef9d3423c7f' of framework
> ecd3a4be-d34c-46f3-b358-c4e26ac0d13
> I0227 23:06:20.441520 26043 slave.cpp:4399] Handling status update
> TASK_FAILED (UUID: 003e072e-341b-48ce-abb5-79849128ba6f) for task
> api_rapi-a.d25fc1e
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
