[
https://issues.apache.org/jira/browse/MESOS-4899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214665#comment-15214665
]
Greg Mann commented on MESOS-4899:
----------------------------------
[~baraldi86], thanks for your help in figuring out this issue. Has your new
{{work_dir}} configuration solved the problem? I just want to confirm that this
was the root cause so that I can update the docs accordingly.
> Mesos slave crash after killing docker container
> ------------------------------------------------
>
> Key: MESOS-4899
> URL: https://issues.apache.org/jira/browse/MESOS-4899
> Project: Mesos
> Issue Type: Bug
> Affects Versions: 0.27.1
> Reporter: Giulio D'Ippolito
> Priority: Blocker
>
> I have experienced an issue where a Mesos slave crashed because a docker task
> could not be killed properly.
> I'm using marathon to launch the task.
> The setup is the following:
> OS verison: Centos 7.2
> Docker version: 1.8.2
> Mesos-slave: 0.27.1
> Mesos-master: 0.27.1
> Marathon 0.15.3
> The mesos slave crashed (which is not great). This is the log from the mesos
> slave (both mesos slave had the same issue):
> {code}
> Mar 07 15:25:16 marathon_mesos-2 mesos-slave[30866]: I0307 15:25:16.441756
> 30870 slave.cpp:1890] Asked to kill task
> giulio.httpd.test.tag.a47c55be-db01-11e5-a92a-0242eb705eb2 of framework
> ef1354df-7ecc-41ac-82d8
> -d7536e319ea2-0000
> Mar 07 15:25:16 marathon_mesos-2 docker[13019]:
> time="2016-03-07T15:25:16.553925302Z" level=info msg="POST
> /v1.20/containers/mesos-d0f20f55-bc6e-43e8-babb-250b0176f5f6-S145.104860a3-4630-4cbb-8e68-87ecca13fcad/s
> top?t=0"
> Mar 07 15:25:16 marathon_mesos-2 docker[13019]:
> time="2016-03-07T15:25:16.559932661Z" level=info msg="Container
> fe412634ec92bb641a18b4c48d399895f703af29492804b927943646bd81ab8a failed to
> exit within 0 seconds of
> SIGTERM - using the force"
> Mar 07 15:25:16 marathon_mesos-2 systemd[1]: Stopped docker container
> fe412634ec92bb641a18b4c48d399895f703af29492804b927943646bd81ab8a.
> Mar 07 15:25:16 marathon_mesos-2 systemd[1]: Stopping docker container
> fe412634ec92bb641a18b4c48d399895f703af29492804b927943646bd81ab8a.
> Mar 07 15:25:16 marathon_mesos-2 kernel: docker0: port 4(vetha772033) entered
> disabled state
> Mar 07 15:25:16 marathon_mesos-2 NetworkManager[788]: <warn> (vethd55899a):
> failed to find device 106 'vethd55899a' with udev
> Mar 07 15:25:16 marathon_mesos-2 NetworkManager[788]: <info> (vethd55899a):
> new Veth device (carrier: OFF, driver: 'veth', ifindex: 106)
> Mar 07 15:25:16 marathon_mesos-2 NetworkManager[788]: <info> (vetha772033):
> link disconnected
> Mar 07 15:25:16 marathon_mesos-2 kernel: docker0: port 4(vetha772033) entered
> disabled state
> Mar 07 15:25:16 marathon_mesos-2 avahi-daemon[11714]: Withdrawing address
> record for fe80::b06a:2fff:fecc:87f1 on vetha772033.
> Mar 07 15:25:16 marathon_mesos-2 kernel: device vetha772033 left promiscuous
> mode
> Mar 07 15:25:16 marathon_mesos-2 kernel: docker0: port 4(vetha772033) entered
> disabled state
> Mar 07 15:25:16 marathon_mesos-2 avahi-daemon[11714]: Withdrawing workstation
> service for vethd55899a.
> Mar 07 15:25:16 marathon_mesos-2 avahi-daemon[11714]: Withdrawing workstation
> service for vetha772033.
> Mar 07 15:25:16 marathon_mesos-2 NetworkManager[788]: <warn> (vethd55899a):
> failed to disable userspace IPv6LL address handling
> Mar 07 15:25:16 marathon_mesos-2 NetworkManager[788]: <info> (docker0):
> bridge port vetha772033 was detached
> Mar 07 15:25:16 marathon_mesos-2 NetworkManager[788]: <info> (vetha772033):
> released from master docker0
> Mar 07 15:25:16 marathon_mesos-2 NetworkManager[788]: <warn> (vetha772033):
> failed to disable userspace IPv6LL address handling
> Mar 07 15:25:16 marathon_mesos-2 kernel: XFS (dm-8): Unmounting Filesystem
> Mar 07 15:25:16 marathon_mesos-2 mesos-slave[30866]: I0307 15:25:16.879240
> 30868 slave.cpp:3001] Handling status update TASK_KILLED (UUID:
> 1b9446db-dcdc-47a7-aa05-0372e84e1b4d) for task giulio.httpd.test.tag.a47
> c55be-db01-11e5-a92a-0242eb705eb2 of framework
> ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000 from
> executor(1)@XXX.XXX.XXX.XXX:57543
> Mar 07 15:25:16 marathon_mesos-2 mesos-slave[30866]: E0307 15:25:16.880144
> 30868 slave.cpp:3205] Failed to update resources for container
> 104860a3-4630-4cbb-8e68-87ecca13fcad of executor 'giulio.httpd.test.tag.a
> 47c55be-db01-11e5-a92a-0242eb705eb2' running task
> giulio.httpd.test.tag.a47c55be-db01-11e5-a92a-0242eb705eb2 on status update
> for terminal task, destroying container: Failed to determine cgroup for the
> 'cpu' sub
> system: Failed to read /proc/4390/cgroup: Failed to open file
> '/proc/4390/cgroup': No such file or directory
> Mar 07 15:25:16 marathon_mesos-2 mesos-slave[30866]: I0307 15:25:16.880388
> 30868 status_update_manager.cpp:320] Received status update TASK_KILLED
> (UUID: 1b9446db-dcdc-47a7-aa05-0372e84e1b4d) for task giulio.htt
> pd.test.tag.a47c55be-db01-11e5-a92a-0242eb705eb2 of framework
> ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000
> Mar 07 15:25:16 marathon_mesos-2 mesos-slave[30866]: I0307 15:25:16.880416
> 30870 docker.cpp:1455] Destroying container
> '104860a3-4630-4cbb-8e68-87ecca13fcad'
> Mar 07 15:25:16 marathon_mesos-2 mesos-slave[30866]: I0307 15:25:16.880487
> 30870 docker.cpp:1515] Sending SIGTERM to executor with pid: 4318
> Mar 07 15:25:16 marathon_mesos-2 mesos-slave[30866]: I0307 15:25:16.880437
> 30868 status_update_manager.cpp:824] Checkpointing UPDATE for status update
> TASK_KILLED (UUID: 1b9446db-dcdc-47a7-aa05-0372e84e1b4d) for task
> giulio.httpd.test.tag.a47c55be-db01-11e5-a92a-0242eb705eb2 of framework
> ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000
> Mar 07 15:25:16 marathon_mesos-2 mesos-slave[30866]: I0307 15:25:16.899090
> 30868 slave.cpp:3353] Forwarding the update TASK_KILLED (UUID:
> 1b9446db-dcdc-47a7-aa05-0372e84e1b4d) for task
> giulio.httpd.test.tag.a47c55be-db01-11e5-a92a-0242eb705eb2 of framework
> ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000 to mas...@xxx.xxx.xxx.xxx:5051
> Mar 07 15:25:16 marathon_mesos-2 mesos-slave[30866]: I0307 15:25:16.899364
> 30868 slave.cpp:3263] Sending acknowledgement for status update TASK_KILLED
> (UUID: 1b9446db-dcdc-47a7-aa05-0372e84e1b4d) for task
> giulio.httpd.test.tag.a47c55be-db01-11e5-a92a-0242eb705eb2 of framework
> ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000 to executor(1)@XXX.XXX.XXX.XXX:57543
> Mar 07 15:25:16 marathon_mesos-2 mesos-slave[30866]: I0307 15:25:16.923931
> 30867 status_update_manager.cpp:392] Received status update acknowledgement
> (UUID: 1b9446db-dcdc-47a7-aa05-0372e84e1b4d) for task
> giulio.httpd.test.tag.a47c55be-db01-11e5-a92a-0242eb705eb2 of framework
> ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000
> Mar 07 15:25:16 marathon_mesos-2 mesos-slave[30866]: I0307 15:25:16.924121
> 30867 status_update_manager.cpp:824] Checkpointing ACK for status update
> TASK_KILLED (UUID: 1b9446db-dcdc-47a7-aa05-0372e84e1b4d) for task
> giulio.httpd.test.tag.a47c55be-db01-11e5-a92a-0242eb705eb2 of framework
> ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000
> Mar 07 15:25:16 marathon_mesos-2 mesos-slave[30866]: I0307 15:25:16.950709
> 30870 docker.cpp:1557] Running docker stop on container
> '104860a3-4630-4cbb-8e68-87ecca13fcad'
> Mar 07 15:25:16 marathon_mesos-2 mesos-slave[30866]: I0307 15:25:16.955036
> 30870 slave.cpp:3481] executor(1)@XXX.XXX.XXX.XXX:57543 exited
> Mar 07 15:25:17 marathon_mesos-2 docker[13019]:
> time="2016-03-07T15:25:17.003800525Z" level=info msg="POST
> /v1.20/containers/mesos-d0f20f55-bc6e-43e8-babb-250b0176f5f6-S145.104860a3-4630-4cbb-8e68-87ecca13fcad/stop?t=0"
> Mar 07 15:25:17 marathon_mesos-2 mesos-slave[30866]: I0307 15:25:17.042575
> 30872 docker.cpp:1654] Executor for container
> '104860a3-4630-4cbb-8e68-87ecca13fcad' has exited
> Mar 07 15:25:17 marathon_mesos-2 mesos-slave[30866]: I0307 15:25:17.043073
> 30868 slave.cpp:3816] Executor
> 'giulio.httpd.test.tag.a47c55be-db01-11e5-a92a-0242eb705eb2' of framework
> ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000 terminated with signal Terminated
> Mar 07 15:25:17 marathon_mesos-2 mesos-slave[30866]: I0307 15:25:17.043175
> 30868 slave.cpp:3920] Cleaning up executor
> 'giulio.httpd.test.tag.a47c55be-db01-11e5-a92a-0242eb705eb2' of framework
> ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000 at executor(1)@XXX.XXX.XXX.XXX:57543
> Mar 07 15:25:17 marathon_mesos-2 mesos-slave[30866]: F0307 15:25:17.043442
> 30868 slave.cpp:3945] CHECK_SOME(os::touch(path)): Failed to open file: No
> such file or directory
> Mar 07 15:25:17 marathon_mesos-2 mesos-slave[30866]: *** Check failure stack
> trace: ***
> Mar 07 15:25:17 marathon_mesos-2 mesos-slave[30866]: @ 0x7f4a5f73133d
> google::LogMessage::Fail()
> Mar 07 15:25:17 marathon_mesos-2 mesos-slave[30866]: @ 0x7f4a5f73307c
> google::LogMessage::SendToLog()
> Mar 07 15:25:17 marathon_mesos-2 mesos-slave[30866]: @ 0x7f4a5f730f2c
> google::LogMessage::Flush()
> Mar 07 15:25:17 marathon_mesos-2 mesos-slave[30866]: @ 0x7f4a5f733979
> google::LogMessageFatal::~LogMessageFatal()
> Mar 07 15:25:17 marathon_mesos-2 mesos-slave[30866]: @ 0x7f4a5f0faf1c
> mesos::internal::slave::Slave::removeExecutor()
> Mar 07 15:25:17 marathon_mesos-2 mesos-slave[30866]: @ 0x7f4a5f112970
> mesos::internal::slave::Slave::executorTerminated()
> Mar 07 15:25:17 marathon_mesos-2 mesos-slave[30866]: @ 0x7f4a5f6dee21
> process::ProcessManager::resume()
> Mar 07 15:25:17 marathon_mesos-2 mesos-slave[30866]: @ 0x7f4a5f6df127
>
> _ZNSt6thread5_ImplISt12_Bind_simpleIFSt5_BindIFZN7process14ProcessManager12init_threadsEvEUlRKSt11atomic_boolE_St17reference_wrapperIS6_EEEvEEE6_M_runEv
> Mar 07 15:25:17 marathon_mesos-2 mesos-slave[30866]: @ 0x7f4a5e023220
> (unknown)
> Mar 07 15:25:17 marathon_mesos-2 mesos-slave[30866]: @ 0x7f4a5e27ddc5
> start_thread
> Mar 07 15:25:17 marathon_mesos-2 mesos-slave[30866]: @ 0x7f4a5da8d28d
> __clone
> Mar 07 15:25:17 marathon_mesos-2 systemd[1]: mesos-slave.service: main
> process exited, code=killed, status=6/ABRT
> Mar 07 15:25:17 marathon_mesos-2 systemd[1]: Unit mesos-slave.service entered
> failed state.
> Mar 07 15:25:17 marathon_mesos-2 systemd[1]: mesos-slave.service failed.
> Mar 07 15:25:17 marathon_mesos-2 ntpd[728]: Deleting interface #42
> vetha772033, fe80::b06a:2fff:fecc:87f1#123, interface stats: received=0,
> sent=0, dropped=0, active_time=1040732 secs
> Mar 07 15:25:37 marathon_mesos-2 systemd[1]: mesos-slave.service holdoff time
> over, scheduling restart.
> Mar 07 15:25:37 marathon_mesos-2 systemd[1]: Started Mesos Slave.
> Mar 07 15:25:37 marathon_mesos-2 systemd[1]: Starting Mesos Slave...
> {code}
> Mesos master log
> {code}
> I0307 15:25:16.443166 8 master.cpp:3855] Telling slave
> d0f20f55-bc6e-43e8-babb-250b0176f5f6-S145 at slave(1)@XXX.XXX.XXX.XXX:5051
> (marathon_mesos-2.domain.com) to kill task
> giulio.httpd.test.tag.a47c55be-db01-11e5-a92a-0242eb705eb2 of framework
> ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000 (marathon) at
> scheduler-2582f6fa-94eb-4044-9b0b-fa8988a01...@xxx.xxx.xxx.xxx:55663
> I0307 15:25:16.443385 8 master.cpp:3855] Telling slave
> d0f20f55-bc6e-43e8-babb-250b0176f5f6-S146 at slave(1)@XXX.XXX.XXX.XXX:5051
> (marathon_mesos-1.domain.com) to kill task
> giulio.httpd.test.tag.a47c079d-db01-11e5-a92a-0242eb705eb2 of framework
> ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000 (marathon) at
> scheduler-2582f6fa-94eb-4044-9b0b-fa8988a01...@xxx.xxx.xxx.xxx:55663
> I0307 15:25:16.443476 8 master.cpp:3855] Telling slave
> d0f20f55-bc6e-43e8-babb-250b0176f5f6-S146 at slave(1)@XXX.XXX.XXX.XXX:5051
> (marathon_mesos-1.domain.com) to kill task
> giulio.httpd.test.tag.a47d401f-db01-11e5-a92a-0242eb705eb2 of framework
> ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000 (marathon) at
> scheduler-2582f6fa-94eb-4044-9b0b-fa8988a01...@xxx.xxx.xxx.xxx:55663
> I0307 15:25:16.903249 8 master.cpp:4789] Status update TASK_KILLED (UUID:
> 1b9446db-dcdc-47a7-aa05-0372e84e1b4d) for task
> giulio.httpd.test.tag.a47c55be-db01-11e5-a92a-0242eb705eb2 of framework
> ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000 from slave
> d0f20f55-bc6e-43e8-babb-250b0176f5f6-S145 at slave(1)@XXX.XXX.XXX.XXX:5051
> (marathon_mesos-2.domain.com)
> I0307 15:25:16.903378 8 master.cpp:4837] Forwarding status update
> TASK_KILLED (UUID: 1b9446db-dcdc-47a7-aa05-0372e84e1b4d) for task
> giulio.httpd.test.tag.a47c55be-db01-11e5-a92a-0242eb705eb2 of framework
> ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000
> I0307 15:25:16.903561 8 master.cpp:6445] Updating the state of task
> giulio.httpd.test.tag.a47c55be-db01-11e5-a92a-0242eb705eb2 of framework
> ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000 (latest state: TASK_KILLED, status
> update state: TASK_KILLED)
> I0307 15:25:16.924135 8 master.cpp:3947] Processing ACKNOWLEDGE call
> 1b9446db-dcdc-47a7-aa05-0372e84e1b4d for task
> giulio.httpd.test.tag.a47c55be-db01-11e5-a92a-0242eb705eb2 of framework
> ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000 (marathon) at
> scheduler-2582f6fa-94eb-4044-9b0b-fa8988a01...@xxx.xxx.xxx.xxx:55663 on slave
> d0f20f55-bc6e-43e8-babb-250b0176f5f6-S145
> I0307 15:25:16.924257 8 master.cpp:6511] Removing task
> giulio.httpd.test.tag.a47c55be-db01-11e5-a92a-0242eb705eb2 with resources
> cpus(*):0.04; mem(*):32; ports(*):[31511-31511] of framework
> ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000 on slave
> d0f20f55-bc6e-43e8-babb-250b0176f5f6-S145 at slave(1)@XXX.XXX.XXX.XXX:5051
> (marathon_mesos-2.domain.com)
> I0307 15:25:17.059509 8 master.cpp:4789] Status update TASK_KILLED (UUID:
> 6ab5a70a-9690-43df-8d05-f4ea08803ac2) for task
> giulio.httpd.test.tag.a47d401f-db01-11e5-a92a-0242eb705eb2 of framework
> ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000 from slave
> d0f20f55-bc6e-43e8-babb-250b0176f5f6-S146 at slave(1)@XXX.XXX.XXX.XXX:5051
> (marathon_mesos-1.domain.com)
> I0307 15:25:17.059617 8 master.cpp:4837] Forwarding status update
> TASK_KILLED (UUID: 6ab5a70a-9690-43df-8d05-f4ea08803ac2) for task
> giulio.httpd.test.tag.a47d401f-db01-11e5-a92a-0242eb705eb2 of framework
> ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000
> I0307 15:25:17.059797 8 master.cpp:6445] Updating the state of task
> giulio.httpd.test.tag.a47d401f-db01-11e5-a92a-0242eb705eb2 of framework
> ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000 (latest state: TASK_KILLED, status
> update state: TASK_KILLED)
> I0307 15:25:17.073132 8 master.cpp:3947] Processing ACKNOWLEDGE call
> 6ab5a70a-9690-43df-8d05-f4ea08803ac2 for task
> giulio.httpd.test.tag.a47d401f-db01-11e5-a92a-0242eb705eb2 of framework
> ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000 (marathon) at
> scheduler-2582f6fa-94eb-4044-9b0b-fa8988a01...@xxx.xxx.xxx.xxx:55663 on slave
> d0f20f55-bc6e-43e8-babb-250b0176f5f6-S146
> I0307 15:25:17.073266 8 master.cpp:6511] Removing task
> giulio.httpd.test.tag.a47d401f-db01-11e5-a92a-0242eb705eb2 with resources
> cpus(*):0.04; mem(*):32; ports(*):[31833-31833] of framework
> ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000 on slave
> d0f20f55-bc6e-43e8-babb-250b0176f5f6-S146 at slave(1)@XXX.XXX.XXX.XXX:5051
> (marathon_mesos-1.domain.com)
> I0307 15:25:17.088891 8 master.cpp:1172] Slave
> d0f20f55-bc6e-43e8-babb-250b0176f5f6-S145 at slave(1)@XXX.XXX.XXX.XXX:5051
> (marathon_mesos-2.domain.com) disconnected
> I0307 15:25:17.088999 8 master.cpp:2633] Disconnecting slave
> d0f20f55-bc6e-43e8-babb-250b0176f5f6-S145 at slave(1)@XXX.XXX.XXX.XXX:5051
> (marathon_mesos-2.domain.com)
> I0307 15:25:17.089064 8 master.cpp:2652] Deactivating slave
> d0f20f55-bc6e-43e8-babb-250b0176f5f6-S145 at slave(1)@XXX.XXX.XXX.XXX:5051
> (marathon_mesos-2.domain.com)
> I0307 15:25:17.089191 8 hierarchical.cpp:560] Slave
> d0f20f55-bc6e-43e8-babb-250b0176f5f6-S145 deactivated
> I0307 15:25:17.121345 8 master.cpp:4789] Status update TASK_KILLED (UUID:
> 582f00d2-037c-43a1-88c1-2f610717a5d5) for task
> giulio.httpd.test.tag.a47c079d-db01-11e5-a92a-0242eb705eb2 of framework
> ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000 from slave
> d0f20f55-bc6e-43e8-babb-250b0176f5f6-S146 at slave(1)@XXX.XXX.XXX.XXX:5051
> (marathon_mesos-1.domain.com)
> I0307 15:25:17.121451 8 master.cpp:4837] Forwarding status update
> TASK_KILLED (UUID: 582f00d2-037c-43a1-88c1-2f610717a5d5) for task
> giulio.httpd.test.tag.a47c079d-db01-11e5-a92a-0242eb705eb2 of framework
> ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000
> I0307 15:25:17.121590 8 master.cpp:6445] Updating the state of task
> giulio.httpd.test.tag.a47c079d-db01-11e5-a92a-0242eb705eb2 of framework
> ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000 (latest state: TASK_KILLED, status
> update state: TASK_KILLED)
> I0307 15:25:17.145149 8 master.cpp:3947] Processing ACKNOWLEDGE call
> 582f00d2-037c-43a1-88c1-2f610717a5d5 for task
> giulio.httpd.test.tag.a47c079d-db01-11e5-a92a-0242eb705eb2 of framework
> ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000 (marathon) at
> scheduler-2582f6fa-94eb-4044-9b0b-fa8988a01...@xxx.xxx.xxx.xxx:55663 on slave
> d0f20f55-bc6e-43e8-babb-250b0176f5f6-S146
> I0307 15:25:17.145256 8 master.cpp:6511] Removing task
> giulio.httpd.test.tag.a47c079d-db01-11e5-a92a-0242eb705eb2 with resources
> cpus(*):0.04; mem(*):32; ports(*):[31167-31167] of framework
> ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000 on slave
> d0f20f55-bc6e-43e8-babb-250b0176f5f6-S146 at slave(1)@XXX.XXX.XXX.XXX:5051
> (marathon_mesos-1.domain.com)
> I0307 15:25:17.287336 8 master.cpp:1172] Slave
> d0f20f55-bc6e-43e8-babb-250b0176f5f6-S146 at slave(1)@XXX.XXX.XXX.XXX:5051
> (marathon_mesos-1.domain.com) disconnected
> I0307 15:25:17.287485 8 master.cpp:2633] Disconnecting slave
> d0f20f55-bc6e-43e8-babb-250b0176f5f6-S146 at slave(1)@XXX.XXX.XXX.XXX:5051
> (marathon_mesos-1.domain.com)
> I0307 15:25:17.287531 8 master.cpp:2652] Deactivating slave
> d0f20f55-bc6e-43e8-babb-250b0176f5f6-S146 at slave(1)@XXX.XXX.XXX.XXX:5051
> (marathon_mesos-1.domain.com)
> I0307 15:25:17.287618 8 hierarchical.cpp:560] Slave
> d0f20f55-bc6e-43e8-babb-250b0176f5f6-S146 deactivated
> {code}
> Marathon framework log:
> {code}
> [2016-03-07 15:25:16,366] INFO Upgrade group id:/
> version:2016-03-07T15:25:16.366Z with force:true
> (mesosphere.marathon.state.GroupManager:marathon-akka.actor.default-dispatcher-46207)
> [2016-03-07 15:25:16,368] INFO Computed new deployment plan:
> [2016-03-07 15:25:16,368] INFO Deploy plan with force=true:
> [2016-03-07 15:25:16,374] INFO [/giulio.httpd.test.tag] expunging app
> (mesosphere.marathon.state.GroupManager:ForkJoinPool-2-worker-5)
> [2016-03-07 15:25:16,458] INFO Updated groups/apps according to deployment
> plan bc0e6fa3-f1cc-4658-9787-efe85e957869
> (mesosphere.marathon.state.GroupManager:ForkJoinPool-2-worker-5)
> [2016-03-07 15:25:16,459] INFO Deployment acknowledged. Waiting to get
> processed:
> [2016-03-07 15:25:16,690] INFO XXX.XXX.XXX.XXX - - [07/Mar/2016:15:25:16
> +0000] "DELETE //XXX.XXX.XXX.XXX:8080/v2/apps//giulio.httpd.test.tag
> HTTP/1.1" 200 92 "http://XXX.XXX.XXX.XXX:8080/ui/"; "Mozilla/5.0 (X11; Fedora;
> Linux x86_64; rv:44.0) Gecko/20100101 Firefox/44.0" 32
> (mesosphere.chaos.http.ChaosRequestLog$$EnhancerByGuice$$562bf4ff:qtp1361931872-249367)
> [2016-03-07 15:25:16,863] INFO XXX.XXX.XXX.XXX - - [07/Mar/2016:15:25:16
> +0000] "GET //XXX.XXX.XXX.XXX:8080/ui/img/marathon-favicon.ico HTTP/1.1" 200
> 32038 "-" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:44.0) Gecko/20100101
> Firefox/44.0"
> (mesosphere.chaos.http.ChaosRequestLog$$EnhancerByGuice$$562bf4ff:qtp1361931872-143659)
> [2016-03-07 15:25:16,904] INFO Received status update for task
> giulio.httpd.test.tag.a47c55be-db01-11e5-a92a-0242eb705eb2: TASK_KILLED ()
> (mesosphere.marathon.MarathonScheduler$$EnhancerByGuice$$4304ca77:Thread-63362)
> [2016-03-07 15:25:16,910] INFO XXX.XXX.XXX.XXX - - [07/Mar/2016:15:25:16
> +0000] "GET //XXX.XXX.XXX.XXX:8080/v2/deployments HTTP/1.1" 200 308
> "http://XXX.XXX.XXX.XXX:8080/ui/"; "Mozilla/5.0 (X11; Fedora; Linux x86_64;
> rv:44.0) Gecko/20100101 Firefox/44.0"
> (mesosphere.chaos.http.ChaosRequestLog$$EnhancerByGuice$$562bf4ff:qtp1361931872-123617)
> [2016-03-07 15:25:16,912] INFO XXX.XXX.XXX.XXX - - [07/Mar/2016:15:25:16
> +0000] "GET //XXX.XXX.XXX.XXX:8080/v2/apps HTTP/1.1" 200 4360
> "http://XXX.XXX.XXX.XXX:8080/ui/"; "Mozilla/5.0 (X11; Fedora; Linux x86_64;
> rv:44.0) Gecko/20100101 Firefox/44.0"
> (mesosphere.chaos.http.ChaosRequestLog$$EnhancerByGuice$$562bf4ff:qtp1361931872-116139)
> [2016-03-07 15:25:16,912] INFO XXX.XXX.XXX.XXX - - [07/Mar/2016:15:25:16
> +0000] "GET //XXX.XXX.XXX.XXX:8080/v2/queue HTTP/1.1" 200 12
> "http://XXX.XXX.XXX.XXX:8080/ui/"; "Mozilla/5.0 (X11; Fedora; Linux x86_64;
> rv:44.0) Gecko/20100101 Firefox/44.0"
> (mesosphere.chaos.http.ChaosRequestLog$$EnhancerByGuice$$562bf4ff:qtp1361931872-143659)
> [2016-03-07 15:25:16,921] INFO Sending event notification for task
> [giulio.httpd.test.tag.a47c55be-db01-11e5-a92a-0242eb705eb2] of app
> [/giulio.httpd.test.tag]: TASK_KILLED
> (mesosphere.marathon.core.task.update.impl.steps.PostToEventStreamStepImpl$$EnhancerByGuice$$8fcb1905:ForkJoinPool-2-worker-25)
> [2016-03-07 15:25:16,921] INFO initiating a scale check for app
> [/giulio.httpd.test.tag] after task
> [giulio.httpd.test.tag.a47c55be-db01-11e5-a92a-0242eb705eb2}] terminated
> (mesosphere.marathon.core.task.update.impl.steps.ScaleAppUpdateStepImpl$$EnhancerByGuice$$fc6d4908:ForkJoinPool-2-worker-25)
> [2016-03-07 15:25:16,922] INFO Task
> giulio.httpd.test.tag.a47c55be-db01-11e5-a92a-0242eb705eb2 has been killed.
> Waiting for 2 more tasks to be killed.
> (mesosphere.marathon.upgrade.AppStopActor:marathon-akka.actor.default-dispatcher-46201)
> [2016-03-07 15:25:17,060] INFO Received status update for task
> giulio.httpd.test.tag.a47d401f-db01-11e5-a92a-0242eb705eb2: TASK_KILLED ()
> (mesosphere.marathon.MarathonScheduler$$EnhancerByGuice$$4304ca77:Thread-63363)
> [2016-03-07 15:25:17,070] INFO Sending event notification for task
> [giulio.httpd.test.tag.a47d401f-db01-11e5-a92a-0242eb705eb2] of app
> [/giulio.httpd.test.tag]: TASK_KILLED
> (mesosphere.marathon.core.task.update.impl.steps.PostToEventStreamStepImpl$$EnhancerByGuice$$8fcb1905:ForkJoinPool-2-worker-5)
> [2016-03-07 15:25:17,071] INFO initiating a scale check for app
> [/giulio.httpd.test.tag] after task
> [giulio.httpd.test.tag.a47d401f-db01-11e5-a92a-0242eb705eb2}] terminated
> (mesosphere.marathon.core.task.update.impl.steps.ScaleAppUpdateStepImpl$$EnhancerByGuice$$fc6d4908:ForkJoinPool-2-worker-5)
> [2016-03-07 15:25:17,071] INFO Task
> giulio.httpd.test.tag.a47d401f-db01-11e5-a92a-0242eb705eb2 has been killed.
> Waiting for 1 more tasks to be killed.
> (mesosphere.marathon.upgrade.AppStopActor:marathon-akka.actor.default-dispatcher-46207)
> [2016-03-07 15:25:17,122] INFO Received status update for task
> giulio.httpd.test.tag.a47c079d-db01-11e5-a92a-0242eb705eb2: TASK_KILLED ()
> (mesosphere.marathon.MarathonScheduler$$EnhancerByGuice$$4304ca77:Thread-63364)
> [2016-03-07 15:25:17,132] INFO Removed app [/giulio.httpd.test.tag] from
> tracker
> (mesosphere.marathon.core.task.tracker.TaskTracker$TasksByApp$:marathon-akka.actor.default-dispatcher-46161)
> [2016-03-07 15:25:17,133] INFO Sending event notification for task
> [giulio.httpd.test.tag.a47c079d-db01-11e5-a92a-0242eb705eb2] of app
> [/giulio.httpd.test.tag]: TASK_KILLED
> (mesosphere.marathon.core.task.update.impl.steps.PostToEventStreamStepImpl$$EnhancerByGuice$$8fcb1905:ForkJoinPool-2-worker-11)
> [2016-03-07 15:25:17,133] INFO initiating a scale check for app
> [/giulio.httpd.test.tag] after task
> [giulio.httpd.test.tag.a47c079d-db01-11e5-a92a-0242eb705eb2}] terminated
> (mesosphere.marathon.core.task.update.impl.steps.ScaleAppUpdateStepImpl$$EnhancerByGuice$$fc6d4908:ForkJoinPool-2-worker-11)
> [2016-03-07 15:25:17,134] INFO Task
> giulio.httpd.test.tag.a47c079d-db01-11e5-a92a-0242eb705eb2 has been killed.
> Waiting for 0 more tasks to be killed.
> (mesosphere.marathon.upgrade.AppStopActor:marathon-akka.actor.default-dispatcher-46195)
> [2016-03-07 15:25:17,134] INFO Successfully killed all the tasks
> (mesosphere.marathon.upgrade.AppStopActor:marathon-akka.actor.default-dispatcher-46195)
> [2016-03-07 15:25:17,184] INFO Stopping app /giulio.httpd.test.tag
> (mesosphere.marathon.SchedulerActions:marathon-akka.actor.default-dispatcher-46195)
> [2016-03-07 15:25:17,188] INFO Deployment of / successful
> (mesosphere.marathon.MarathonSchedulerActor:marathon-akka.actor.default-dispatcher-46161)
> [2016-03-07 15:25:17,188] INFO Removing bc0e6fa3-f1cc-4658-9787-efe85e957869
> from list of running deployments
> (mesosphere.marathon.upgrade.DeploymentManager:marathon-akka.actor.default-dispat
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
