[
https://issues.apache.org/jira/browse/MESOS-6656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16113839#comment-16113839
]
Benjamin Mahler commented on MESOS-6656:
----------------------------------------
Bug filed here: MESOS-7858
> Nested containers can become unkillable
> ---------------------------------------
>
> Key: MESOS-6656
> URL: https://issues.apache.org/jira/browse/MESOS-6656
> Project: Mesos
> Issue Type: Bug
> Components: agent, containerization
> Reporter: Greg Mann
> Labels: nested
>
> An incident occurred recently in a cluster running a build of Mesos based on
> commit {{757319357471227c0a1e906076eae8f9aa2fdbd6}} from master. A task group
> of five tasks was launched via Marathon. After the tasks were launched, one
> of the containers quickly exited and was successfully destroyed. A couple
> minutes later, the task group was killed manually via Marathon, and the agent
> can then be seen repeatedly attempting to kill the tasks for hours. No calls
> to {{WAIT_NESTED_CONTAINER}} are visible in the agent logs, and the executor
> logs do not indicate at any point that the nested containers were launched
> successfully.
> Agent logs:
> {code}
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.890911
> 6406 slave.cpp:1539] Got assigned task group containing tasks [
> dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server1,
> dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server2,
> dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server3,
> dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server4,
> dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server5 ] for
> framework ce4bd8be-1198-4819-81d4-9a8439439741-0000
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.892299
> 6406 gc.cpp:83] Unscheduling
> '/var/lib/mesos/slave/slaves/ce4bd8be-1198-4819-81d4-9a8439439741-S1/frameworks/ce4bd8be-1198-4819-81d4-9a8439439741-0000'
> from gc
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.892379
> 6406 gc.cpp:83] Unscheduling
> '/var/lib/mesos/slave/meta/slaves/ce4bd8be-1198-4819-81d4-9a8439439741-S1/frameworks/ce4bd8be-1198-4819-81d4-9a8439439741-0000'
> from gc
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.893131
> 6405 slave.cpp:1701] Launching task group containing tasks [
> dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server1,
> dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server2,
> dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server3,
> dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server4,
> dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server5 ] for
> framework ce4bd8be-1198-4819-81d4-9a8439439741-0000
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.893435
> 6405 paths.cpp:536] Trying to chown
> '/var/lib/mesos/slave/slaves/ce4bd8be-1198-4819-81d4-9a8439439741-S1/frameworks/ce4bd8be-1198-4819-81d4-9a8439439741-0000/executors/instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001/runs/8750c2a7-8bef-4a69-8ef2-b873f884bf91'
> to user 'root'
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.898026
> 6405 slave.cpp:6179] Launching executor
> 'instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001' of framework
> ce4bd8be-1198-4819-81d4-9a8439439741-0000 with resources cpus(*):0.1;
> mem(*):32; disk(*):10; ports(*):[21421-21425] in work directory
> '/var/lib/mesos/slave/slaves/ce4bd8be-1198-4819-81d4-9a8439439741-S1/frameworks/ce4bd8be-1198-4819-81d4-9a8439439741-0000/executors/instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001/runs/8750c2a7-8bef-4a69-8ef2-b873f884bf91'
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.898731
> 6407 docker.cpp:1000] Skipping non-docker container
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.899050
> 6407 containerizer.cpp:938] Starting container
> 8750c2a7-8bef-4a69-8ef2-b873f884bf91 for executor
> 'instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001' of framework
> ce4bd8be-1198-4819-81d4-9a8439439741-0000
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.899909
> 6405 slave.cpp:1987] Queued task group containing tasks [
> dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server1,
> dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server2,
> dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server3,
> dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server4,
> dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server5 ] for
> executor 'instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001' of
> framework ce4bd8be-1198-4819-81d4-9a8439439741-0000
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.907033
> 6405 memory.cpp:451] Started listening for OOM events for container
> 8750c2a7-8bef-4a69-8ef2-b873f884bf91
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.908336
> 6405 memory.cpp:562] Started listening on 'low' memory pressure events for
> container 8750c2a7-8bef-4a69-8ef2-b873f884bf91
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.909638
> 6405 memory.cpp:562] Started listening on 'medium' memory pressure events for
> container 8750c2a7-8bef-4a69-8ef2-b873f884bf91
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.910923
> 6405 memory.cpp:562] Started listening on 'critical' memory pressure events
> for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.912464
> 6405 memory.cpp:199] Updated 'memory.soft_limit_in_bytes' to 32MB for
> container 8750c2a7-8bef-4a69-8ef2-b873f884bf91
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.914640
> 6405 memory.cpp:251] Updated 'memory.limit_in_bytes' to 32MB for container
> 8750c2a7-8bef-4a69-8ef2-b873f884bf91
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.915769
> 6405 cpu.cpp:101] Updated 'cpu.shares' to 102 (cpus 0.1) for container
> 8750c2a7-8bef-4a69-8ef2-b873f884bf91
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.917944
> 6405 cpu.cpp:121] Updated 'cpu.cfs_period_us' to 100ms and 'cpu.cfs_quota_us'
> to 10ms (cpus 0.1) for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.918515
> 6404 isolator_module.cpp:55] Container prepare: container_id[value:
> "8750c2a7-8bef-4a69-8ef2-b873f884bf91"] container_config[directory:
> "/var/lib/mesos/slave/slaves/ce4bd8be-1198-4819-81d4-9a8439439741-S1/frameworks/ce4bd8be-1198-4819-81d4-9a8439439741-0000/executors/instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001/runs/8750c2a7-8bef-4a69-8ef2-b873f884bf91"
> user: "root" executor_info { executor_id { value:
> "instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001" } resources { name:
> "cpus" type: SCALAR scalar { value: 0.1 } } resources { name: "mem" type:
> SCALAR scalar { value: 32 } } resources { name: "disk" type: SCALAR scalar {
> value: 10 } } resources { name: "ports" type: RANGES ranges { range { begin:
> 21421 end: 21425 } } role: "*" } command { value:
> "/opt/mesosphere/packages/mesos--a00089c894f82b6455d1db7c4267a742458020b3/libexec/mesos/mesos-default-executor"
> user: "root" shell: false arguments: "mesos-default-executor" } framework_id
> { value: "ce4bd8be-1198-4819-81d4-9a8439439741-0000" } container { type:
> MESOS network_infos { labels { labels { key: "rcluster.location" value:
> "globalqa.las2.test0" } labels { key: "rcluster.group" value: "dat" } labels
> { key: "rcluster.application" value: "scout" } } name: "rcluster-cni"
> port_mappings { host_port: 21421 container_port: 8080 protocol: "tcp" }
> port_mappings { host_port: 21422 container_port: 8090 protocol: "tcp" }
> port_mappings { host_port: 21423 container_port: 8093 protocol: "tcp" }
> port_mappings { host_port: 21424 container_port: 8090 protocol: "tcp" }
> port_mappings { host_port: 21425 container_port: 8090 protocol: "tcp" } } }
> labels { } type: DEFAULT } command_info { value:
> "/opt/mesosphere/packages/mesos--a00089c894f82b6455d1db7c4267a742458020b3/libexec/mesos/mesos-default-executor"
> user: "root" shell: false arguments: "mesos-default-executor" }
> container_info { type: MESOS network_infos { labels { labels { key:
> "rcluster.location" value: "globalqa.las2.test0" } labels { key:
> "rcluster.group" value: "dat" } labels { key: "rc
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: luster.application"
> value: "scout" } } name: "rcluster-cni" port_mappings { host_port: 21421
> container_port: 8080 protocol: "tcp" } port_mappings { host_port: 21422
> container_port: 8090 protocol: "tcp" } port_mappings { host_port: 21423
> container_port: 8093 protocol: "tcp" } port_mappings { host_port: 21424
> container_port: 8090 protocol: "tcp" } port_mappings { host_port: 21425
> container_port: 8090 protocol: "tcp" } } } resources { name: "cpus" type:
> SCALAR scalar { value: 0.1 } } resources { name: "mem" type: SCALAR scalar {
> value: 32 } } resources { name: "disk" type: SCALAR scalar { value: 10 } }
> resources { name: "ports" type: RANGES ranges { range { begin: 21421 end:
> 21425 } } role: "*" }]
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.918647
> 6404 container_assigner.cpp:64] Registering and retrieving endpoint for
> container_id[value: "8750c2a7-8bef-4a69-8ef2-b873f884bf91"]
> executor_info[executor_id { value:
> "instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001" } resources { name:
> "cpus" type: SCALAR scalar { value: 0.1 } } resources { name: "mem" type:
> SCALAR scalar { value: 32 } } resources { name: "disk" type: SCALAR scalar {
> value: 10 } } resources { name: "ports" type: RANGES ranges { range { begin:
> 21421 end: 21425 } } role: "*" } command { value:
> "/opt/mesosphere/packages/mesos--a00089c894f82b6455d1db7c4267a742458020b3/libexec/mesos/mesos-default-executor"
> user: "root" shell: false arguments: "mesos-default-executor" } framework_id
> { value: "ce4bd8be-1198-4819-81d4-9a8439439741-0000" } container { type:
> MESOS network_infos { labels { labels { key: "rcluster.location" value:
> "globalqa.las2.test0" } labels { key: "rcluster.group" value: "dat" } labels
> { key: "rcluster.application" value: "scout" } } name: "rcluster-cni"
> port_mappings { host_port: 21421 container_port: 8080 protocol: "tcp" }
> port_mappings { host_port: 21422 container_port: 8090 protocol: "tcp" }
> port_mappings { host_port: 21423 container_port: 8093 protocol: "tcp" }
> port_mappings { host_port: 21424 container_port: 8090 protocol: "tcp" }
> port_mappings { host_port: 21425 container_port: 8090 protocol: "tcp" } } }
> labels { } type: DEFAULT].
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.918715
> 6404 sync_util.hpp:39] Dispatching and waiting <=5s for ticket 431:
> register_and_update_cache
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.918865
> 6488 container_reader_impl.cpp:34] Reader constructed for 198.51.100.1:0
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.919052
> 6488 container_reader_impl.cpp:103] Reader listening on 198.51.100.1:51243
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.919108
> 6488 container_assigner_strategy.cpp:156] New ephemeral-port reader for
> container[value: "8750c2a7-8bef-4a69-8ef2-b873f884bf91"] created at
> endpoint[198.51.100.1:51243].
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.919144
> 6488 container_state_cache_impl.cpp:134] Writing container
> file[/var/run/mesos/isolators/com_mesosphere_MetricsIsolatorModule/containers/8750c2a7-8bef-4a69-8ef2-b873f884bf91]
> with endpoint[198.51.100.1:51243]
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.919220
> 6488 sync_util.hpp:136] Result for ticket 431 complete, returning value.
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.919275
> 6404 sync_util.hpp:83] Dispatch result obtained for ticket 431 after waiting
> <=5s: register_and_update_cache
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.924211
> 6404 systemd.cpp:96] Assigned child process '2701' to 'mesos_executors.slice'
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.927724
> 6404 systemd.cpp:96] Assigned child process '2702' to 'mesos_executors.slice'
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.929244
> 6403 linux_launcher.cpp:421] Launching container
> 8750c2a7-8bef-4a69-8ef2-b873f884bf91 and cloning with namespaces CLONE_NEWNS
> | CLONE_NEWUTS | CLONE_NEWNET
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.935976
> 6403 systemd.cpp:96] Assigned child process '2705' to 'mesos_executors.slice'
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.943640
> 6409 containerizer.cpp:1489] Checkpointing container's forked pid 2705 to
> '/var/lib/mesos/slave/meta/slaves/ce4bd8be-1198-4819-81d4-9a8439439741-S1/frameworks/ce4bd8be-1198-4819-81d4-9a8439439741-0000/executors/instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001/runs/8750c2a7-8bef-4a69-8ef2-b873f884bf91/pids/forked.pid'
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.945247
> 6410 cni.cpp:814] Bind mounted '/proc/2705/ns/net' to
> '/run/mesos/isolators/network/cni/8750c2a7-8bef-4a69-8ef2-b873f884bf91/ns'
> for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: WARNING: Logging before
> InitGoogleLogging() is written to STDERR
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: W1129 04:04:16.961654
> 2701 process.cpp:882] Failed SSL connections will be downgraded to a non-SSL
> socket
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: WARNING: Logging before
> InitGoogleLogging() is written to STDERR
> Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: W1129 04:04:16.963691
> 2702 process.cpp:882] Failed SSL connections will be downgraded to a non-SSL
> socket
> Nov 29 04:04:17 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:17.250937
> 6410 cni.cpp:1230] Got assigned IPv4 address '10.47.11.252/32' from CNI
> network 'rcluster-cni' for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91
> Nov 29 04:04:17 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:17.251171
> 6410 cni.cpp:939] DNS nameservers for container
> 8750c2a7-8bef-4a69-8ef2-b873f884bf91 are:
> Nov 29 04:04:17 ip-10-190-112-199 mesos-agent[6397]: nameserver 10.190.4.22
> Nov 29 04:04:17 ip-10-190-112-199 mesos-agent[6397]: nameserver 10.190.4.23
> Nov 29 04:04:17 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:17.942303
> 6404 http.cpp:277] HTTP POST for /slave(1)/api/v1/executor from
> 10.47.11.252:38416
> Nov 29 04:04:17 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:17.942446
> 6404 slave.cpp:3022] Received Subscribe request for HTTP executor
> 'instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001' of framework
> ce4bd8be-1198-4819-81d4-9a8439439741-0000
> Nov 29 04:04:17 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:17.942533
> 6404 slave.cpp:3085] Creating a marker file for HTTP based executor
> 'instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001' of framework
> ce4bd8be-1198-4819-81d4-9a8439439741-0000 (via HTTP) at path
> '/var/lib/mesos/slave/meta/slaves/ce4bd8be-1198-4819-81d4-9a8439439741-S1/frameworks/ce4bd8be-1198-4819-81d4-9a8439439741-0000/executors/instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001/runs/8750c2a7-8bef-4a69-8ef2-b873f884bf91/http.marker'
> Nov 29 04:04:17 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:17.944046
> 6404 disk.cpp:207] Updating the disk resources for container
> 8750c2a7-8bef-4a69-8ef2-b873f884bf91 to cpus(*):0.6; mem(*):192; disk(*):170;
> ports(*):[21421-21425]
> Nov 29 04:04:17 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:17.945669
> 6407 memory.cpp:199] Updated 'memory.soft_limit_in_bytes' to 192MB for
> container 8750c2a7-8bef-4a69-8ef2-b873f884bf91
> Nov 29 04:04:17 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:17.947779
> 6407 memory.cpp:251] Updated 'memory.limit_in_bytes' to 192MB for container
> 8750c2a7-8bef-4a69-8ef2-b873f884bf91
> Nov 29 04:04:17 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:17.948951
> 6407 cpu.cpp:101] Updated 'cpu.shares' to 614 (cpus 0.6) for container
> 8750c2a7-8bef-4a69-8ef2-b873f884bf91
> Nov 29 04:04:17 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:17.951117
> 6407 cpu.cpp:121] Updated 'cpu.cfs_period_us' to 100ms and 'cpu.cfs_quota_us'
> to 60ms (cpus 0.6) for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91
> Nov 29 04:04:17 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:17.952219
> 6404 slave.cpp:2220] Sending queued task group task group containing tasks [
> dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server1,
> dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server2,
> dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server3,
> dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server4,
> dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server5 ] to
> executor 'instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001' of
> framework ce4bd8be-1198-4819-81d4-9a8439439741-0000 (via HTTP)
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.002063
> 6408 http.cpp:277] HTTP POST for /slave(1)/api/v1 from 10.47.11.252:38420
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.002223
> 6408 http.cpp:353] Processing call LAUNCH_NESTED_CONTAINER
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.003235
> 6409 containerizer.cpp:1657] Starting nested container
> 8750c2a7-8bef-4a69-8ef2-b873f884bf91.063dbcd6-1a0a-4058-a615-7e6deaf17707
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.003327
> 6407 http.cpp:277] HTTP POST for /slave(1)/api/v1 from 10.47.11.252:38420
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.003432
> 6409 containerizer.cpp:1681] Trying to chown
> '/var/lib/mesos/slave/slaves/ce4bd8be-1198-4819-81d4-9a8439439741-S1/frameworks/ce4bd8be-1198-4819-81d4-9a8439439741-0000/executors/instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001/runs/8750c2a7-8bef-4a69-8ef2-b873f884bf91/containers/063dbcd6-1a0a-4058-a615-7e6deaf17707'
> to user 'nobody'
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.003465
> 6407 http.cpp:353] Processing call LAUNCH_NESTED_CONTAINER
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.005939
> 6407 http.cpp:277] HTTP POST for /slave(1)/api/v1 from 10.47.11.252:38420
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.006057
> 6407 http.cpp:353] Processing call LAUNCH_NESTED_CONTAINER
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.006265
> 6407 http.cpp:277] HTTP POST for /slave(1)/api/v1 from 10.47.11.252:38420
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.006384
> 6407 http.cpp:353] Processing call LAUNCH_NESTED_CONTAINER
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.008982
> 6409 containerizer.cpp:1657] Starting nested container
> 8750c2a7-8bef-4a69-8ef2-b873f884bf91.a3b19108-d56c-44ea-b20a-912d539dde7b
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.009096
> 6409 containerizer.cpp:1681] Trying to chown
> '/var/lib/mesos/slave/slaves/ce4bd8be-1198-4819-81d4-9a8439439741-S1/frameworks/ce4bd8be-1198-4819-81d4-9a8439439741-0000/executors/instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001/runs/8750c2a7-8bef-4a69-8ef2-b873f884bf91/containers/a3b19108-d56c-44ea-b20a-912d539dde7b'
> to user 'nobody'
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.013618
> 6409 containerizer.cpp:1657] Starting nested container
> 8750c2a7-8bef-4a69-8ef2-b873f884bf91.02c64ae4-ccdd-415d-946a-3f665acbb668
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.013708
> 6409 containerizer.cpp:1681] Trying to chown
> '/var/lib/mesos/slave/slaves/ce4bd8be-1198-4819-81d4-9a8439439741-S1/frameworks/ce4bd8be-1198-4819-81d4-9a8439439741-0000/executors/instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001/runs/8750c2a7-8bef-4a69-8ef2-b873f884bf91/containers/02c64ae4-ccdd-415d-946a-3f665acbb668'
> to user 'nobody'
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.018230
> 6409 containerizer.cpp:1657] Starting nested container
> 8750c2a7-8bef-4a69-8ef2-b873f884bf91.c5c10cff-a6af-4e51-911e-61d04e5c1d47
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.018399
> 6409 containerizer.cpp:1681] Trying to chown
> '/var/lib/mesos/slave/slaves/ce4bd8be-1198-4819-81d4-9a8439439741-S1/frameworks/ce4bd8be-1198-4819-81d4-9a8439439741-0000/executors/instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001/runs/8750c2a7-8bef-4a69-8ef2-b873f884bf91/containers/c5c10cff-a6af-4e51-911e-61d04e5c1d47'
> to user 'nobody'
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.022992
> 6403 http.cpp:277] HTTP POST for /slave(1)/api/v1 from 10.47.11.252:38420
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.023133
> 6403 http.cpp:353] Processing call LAUNCH_NESTED_CONTAINER
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.023598
> 6406 containerizer.cpp:1657] Starting nested container
> 8750c2a7-8bef-4a69-8ef2-b873f884bf91.c8f6dc94-baba-44be-ab65-49538645a31c
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.023705
> 6406 containerizer.cpp:1681] Trying to chown
> '/var/lib/mesos/slave/slaves/ce4bd8be-1198-4819-81d4-9a8439439741-S1/frameworks/ce4bd8be-1198-4819-81d4-9a8439439741-0000/executors/instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001/runs/8750c2a7-8bef-4a69-8ef2-b873f884bf91/containers/c8f6dc94-baba-44be-ab65-49538645a31c'
> to user 'nobody'
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.462172
> 6405 provisioner.cpp:294] Provisioning image rootfs
> '/var/lib/mesos/slave/provisioner/containers/8750c2a7-8bef-4a69-8ef2-b873f884bf91/containers/063dbcd6-1a0a-4058-a615-7e6deaf17707/backends/copy/rootfses/97f1a1df-1eec-46e3-8a93-8a2b80d48dd2'
> for container
> 8750c2a7-8bef-4a69-8ef2-b873f884bf91.063dbcd6-1a0a-4058-a615-7e6deaf17707
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.462401
> 6405 provisioner.cpp:294] Provisioning image rootfs
> '/var/lib/mesos/slave/provisioner/containers/8750c2a7-8bef-4a69-8ef2-b873f884bf91/containers/a3b19108-d56c-44ea-b20a-912d539dde7b/backends/copy/rootfses/b93ffe07-6f64-42e1-b8c4-77bb224fc9e0'
> for container
> 8750c2a7-8bef-4a69-8ef2-b873f884bf91.a3b19108-d56c-44ea-b20a-912d539dde7b
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.465819
> 6410 provisioner.cpp:294] Provisioning image rootfs
> '/var/lib/mesos/slave/provisioner/containers/8750c2a7-8bef-4a69-8ef2-b873f884bf91/containers/02c64ae4-ccdd-415d-946a-3f665acbb668/backends/copy/rootfses/5d388325-8b4d-496f-b520-4a96584a5971'
> for container
> 8750c2a7-8bef-4a69-8ef2-b873f884bf91.02c64ae4-ccdd-415d-946a-3f665acbb668
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.470854
> 6409 provisioner.cpp:294] Provisioning image rootfs
> '/var/lib/mesos/slave/provisioner/containers/8750c2a7-8bef-4a69-8ef2-b873f884bf91/containers/c5c10cff-a6af-4e51-911e-61d04e5c1d47/backends/copy/rootfses/5b4d32da-7d82-45d2-ad28-85a48456a18b'
> for container
> 8750c2a7-8bef-4a69-8ef2-b873f884bf91.c5c10cff-a6af-4e51-911e-61d04e5c1d47
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.471132
> 6409 provisioner.cpp:294] Provisioning image rootfs
> '/var/lib/mesos/slave/provisioner/containers/8750c2a7-8bef-4a69-8ef2-b873f884bf91/containers/c8f6dc94-baba-44be-ab65-49538645a31c/backends/copy/rootfses/244e36f5-ecca-444f-9bf6-05aa77cb7263'
> for container
> 8750c2a7-8bef-4a69-8ef2-b873f884bf91.c8f6dc94-baba-44be-ab65-49538645a31c
> Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.893782
> 6403 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: W1129 04:04:20.164958
> 6405 runtime.cpp:109] Container user 'nobody' is not supported yet for
> container
> 8750c2a7-8bef-4a69-8ef2-b873f884bf91.063dbcd6-1a0a-4058-a615-7e6deaf17707
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.171018
> 6405 systemd.cpp:96] Assigned child process '2880' to 'mesos_executors.slice'
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.176822
> 6405 systemd.cpp:96] Assigned child process '2881' to 'mesos_executors.slice'
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.179283
> 6405 linux_launcher.cpp:421] Launching nested container
> 8750c2a7-8bef-4a69-8ef2-b873f884bf91.063dbcd6-1a0a-4058-a615-7e6deaf17707 and
> cloning with namespaces CLONE_NEWNS
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: W1129 04:04:20.185796
> 6409 runtime.cpp:109] Container user 'nobody' is not supported yet for
> container
> 8750c2a7-8bef-4a69-8ef2-b873f884bf91.c5c10cff-a6af-4e51-911e-61d04e5c1d47
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.200559
> 6405 systemd.cpp:96] Assigned child process '2884' to 'mesos_executors.slice'
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: WARNING: Logging before
> InitGoogleLogging() is written to STDERR
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: W1129 04:04:20.209010
> 2881 process.cpp:882] Failed SSL connections will be downgraded to a non-SSL
> socket
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: WARNING: Logging before
> InitGoogleLogging() is written to STDERR
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: W1129 04:04:20.211642
> 2880 process.cpp:882] Failed SSL connections will be downgraded to a non-SSL
> socket
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: W1129 04:04:20.215493
> 6405 runtime.cpp:109] Container user 'nobody' is not supported yet for
> container
> 8750c2a7-8bef-4a69-8ef2-b873f884bf91.a3b19108-d56c-44ea-b20a-912d539dde7b
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.228160
> 6407 systemd.cpp:96] Assigned child process '2896' to 'mesos_executors.slice'
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.234012
> 6407 systemd.cpp:96] Assigned child process '2897' to 'mesos_executors.slice'
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.239496
> 6409 linux_launcher.cpp:421] Launching nested container
> 8750c2a7-8bef-4a69-8ef2-b873f884bf91.c5c10cff-a6af-4e51-911e-61d04e5c1d47 and
> cloning with namespaces CLONE_NEWNS
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: W1129 04:04:20.242473
> 6405 runtime.cpp:109] Container user 'nobody' is not supported yet for
> container
> 8750c2a7-8bef-4a69-8ef2-b873f884bf91.02c64ae4-ccdd-415d-946a-3f665acbb668
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: W1129 04:04:20.243351
> 6405 runtime.cpp:109] Container user 'nobody' is not supported yet for
> container
> 8750c2a7-8bef-4a69-8ef2-b873f884bf91.c8f6dc94-baba-44be-ab65-49538645a31c
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.254703
> 6409 systemd.cpp:96] Assigned child process '2900' to 'mesos_executors.slice'
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: WARNING: Logging before
> InitGoogleLogging() is written to STDERR
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: W1129 04:04:20.268317
> 2896 process.cpp:882] Failed SSL connections will be downgraded to a non-SSL
> socket
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.274961
> 6405 systemd.cpp:96] Assigned child process '2902' to 'mesos_executors.slice'
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.279268
> 6405 systemd.cpp:96] Assigned child process '2911' to 'mesos_executors.slice'
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.284328
> 6405 systemd.cpp:96] Assigned child process '2912' to 'mesos_executors.slice'
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.286672
> 6404 linux_launcher.cpp:421] Launching nested container
> 8750c2a7-8bef-4a69-8ef2-b873f884bf91.a3b19108-d56c-44ea-b20a-912d539dde7b and
> cloning with namespaces CLONE_NEWNS
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.299939
> 6405 systemd.cpp:96] Assigned child process '2914' to 'mesos_executors.slice'
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.304103
> 6405 systemd.cpp:96] Assigned child process '2917' to 'mesos_executors.slice'
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.307452
> 6405 systemd.cpp:96] Assigned child process '2918' to 'mesos_executors.slice'
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: WARNING: Logging before
> InitGoogleLogging() is written to STDERR
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: W1129 04:04:20.309020
> 2897 process.cpp:882] Failed SSL connections will be downgraded to a non-SSL
> socket
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.311029
> 6404 systemd.cpp:96] Assigned child process '2916' to 'mesos_executors.slice'
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: WARNING: Logging before
> InitGoogleLogging() is written to STDERR
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: W1129 04:04:20.331065
> 2911 process.cpp:882] Failed SSL connections will be downgraded to a non-SSL
> socket
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.333595
> 6405 linux_launcher.cpp:421] Launching nested container
> 8750c2a7-8bef-4a69-8ef2-b873f884bf91.02c64ae4-ccdd-415d-946a-3f665acbb668 and
> cloning with namespaces CLONE_NEWNS
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: WARNING: Logging before
> InitGoogleLogging() is written to STDERR
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: W1129 04:04:20.342017
> 2902 process.cpp:882] Failed SSL connections will be downgraded to a non-SSL
> socket
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.372247
> 6405 systemd.cpp:96] Assigned child process '2937' to 'mesos_executors.slice'
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: WARNING: Logging before
> InitGoogleLogging() is written to STDERR
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: W1129 04:04:20.372742
> 2912 process.cpp:882] Failed SSL connections will be downgraded to a non-SSL
> socket
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: WARNING: Logging before
> InitGoogleLogging() is written to STDERR
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: W1129 04:04:20.378247
> 2914 process.cpp:882] Failed SSL connections will be downgraded to a non-SSL
> socket
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: WARNING: Logging before
> InitGoogleLogging() is written to STDERR
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: W1129 04:04:20.388391
> 2917 process.cpp:882] Failed SSL connections will be downgraded to a non-SSL
> socket
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.388517
> 6410 linux_launcher.cpp:421] Launching nested container
> 8750c2a7-8bef-4a69-8ef2-b873f884bf91.c8f6dc94-baba-44be-ab65-49538645a31c and
> cloning with namespaces CLONE_NEWNS
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: WARNING: Logging before
> InitGoogleLogging() is written to STDERR
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: W1129 04:04:20.396700
> 2918 process.cpp:882] Failed SSL connections will be downgraded to a non-SSL
> socket
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.400269
> 6410 systemd.cpp:96] Assigned child process '2969' to 'mesos_executors.slice'
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.772186
> 6403 containerizer.cpp:2313] Container
> 8750c2a7-8bef-4a69-8ef2-b873f884bf91.c8f6dc94-baba-44be-ab65-49538645a31c has
> exited
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.772228
> 6403 containerizer.cpp:1950] Destroying container
> 8750c2a7-8bef-4a69-8ef2-b873f884bf91.c8f6dc94-baba-44be-ab65-49538645a31c in
> RUNNING state
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.772330
> 6403 linux_launcher.cpp:498] Asked to destroy container
> 8750c2a7-8bef-4a69-8ef2-b873f884bf91.c8f6dc94-baba-44be-ab65-49538645a31c
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.773825
> 6403 linux_launcher.cpp:541] Using freezer to destroy cgroup
> mesos/8750c2a7-8bef-4a69-8ef2-b873f884bf91/mesos/c8f6dc94-baba-44be-ab65-49538645a31c
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.776217
> 6407 cgroups.cpp:2705] Freezing cgroup
> /sys/fs/cgroup/freezer/mesos/8750c2a7-8bef-4a69-8ef2-b873f884bf91/mesos/c8f6dc94-baba-44be-ab65-49538645a31c
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.780053
> 6410 cgroups.cpp:1439] Successfully froze cgroup
> /sys/fs/cgroup/freezer/mesos/8750c2a7-8bef-4a69-8ef2-b873f884bf91/mesos/c8f6dc94-baba-44be-ab65-49538645a31c
> after 3.798016ms
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.783502
> 6407 cgroups.cpp:2723] Thawing cgroup
> /sys/fs/cgroup/freezer/mesos/8750c2a7-8bef-4a69-8ef2-b873f884bf91/mesos/c8f6dc94-baba-44be-ab65-49538645a31c
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.786717
> 6407 cgroups.cpp:1468] Successfully thawed cgroup
> /sys/fs/cgroup/freezer/mesos/8750c2a7-8bef-4a69-8ef2-b873f884bf91/mesos/c8f6dc94-baba-44be-ab65-49538645a31c
> after 3.186176ms
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.791746
> 6404 provisioner.cpp:488] Destroying container rootfs at
> '/var/lib/mesos/slave/provisioner/containers/8750c2a7-8bef-4a69-8ef2-b873f884bf91/containers/c8f6dc94-baba-44be-ab65-49538645a31c/backends/copy/rootfses/244e36f5-ecca-444f-9bf6-05aa77cb7263'
> for container
> 8750c2a7-8bef-4a69-8ef2-b873f884bf91.c8f6dc94-baba-44be-ab65-49538645a31c
> Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.873241
> 6405 containerizer.cpp:2229] Checkpointing termination state to nested
> container's runtime directory
> '/var/run/mesos/containers/8750c2a7-8bef-4a69-8ef2-b873f884bf91/containers/c8f6dc94-baba-44be-ab65-49538645a31c/termination'
> Nov 29 04:04:21 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:21.153594
> 6408 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:04:23 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:23.410384
> 6404 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:04:25 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:25.667546
> 6410 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:04:27 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:27.925597
> 6408 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:04:30 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:30.198536
> 6410 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:04:32 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:32.456645
> 6407 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:04:34 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:34.716507
> 6405 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:04:36 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:36.974459
> 6409 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:04:39 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:39.229652
> 6406 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:04:41 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:41.482465
> 6404 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:04:43 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:43.736668
> 6405 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:04:45 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:45.994588
> 6407 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:04:45 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:45.999001
> 6488 metrics_tcp_sender.cpp:252] TCP Throughput (bytes): sent=0, dropped=0,
> failed=0, pending=0 (state CONNECT_PENDING)
> Nov 29 04:04:46 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:46.187460
> 6408 slave.cpp:5044] Current disk usage 10.89%. Max allowed age:
> 1.582110178819861days
> Nov 29 04:04:48 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:48.161314
> 6405 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.74:57362 with
> User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36
> (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' with
> X-Forwarded-For='10.192.128.26'
> Nov 29 04:04:48 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:48.250435
> 6409 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:04:49 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:49.033789
> 6488 metrics_tcp_sender.cpp:127] Timed out when opening metrics connection to
> 127.0.0.1:8124. This is expected if no Metrics Collector is running on this
> agent. (state CONNECT_PENDING)
> Nov 29 04:04:49 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:49.033946
> 6488 metrics_tcp_sender.cpp:112] Attempting to open connection to
> 127.0.0.1:8124
> Nov 29 04:04:49 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:49.034059
> 6488 metrics_tcp_sender.cpp:140] Got error 'Connection refused'(system:111)
> when connecting to metrics service at 127.0.0.1:8124. This is expected if no
> Metrics Collector is running on this agent. (state CONNECT_IN_PROGRESS)
> Nov 29 04:04:49 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:49.034081
> 6488 metrics_tcp_sender.cpp:94] Scheduling reconnect to 127.0.0.1:8124 in
> 60s...
> Nov 29 04:04:50 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:50.161448
> 6403 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.74:57426 with
> User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36
> (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' with
> X-Forwarded-For='10.192.128.26'
> Nov 29 04:04:50 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:50.503459
> 6404 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:04:52 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:52.161912
> 6406 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.74:57486 with
> User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36
> (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' with
> X-Forwarded-For='10.192.128.26'
> Nov 29 04:04:52 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:52.765455
> 6407 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:04:54 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:54.161412
> 6405 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.74:57546 with
> User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36
> (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' with
> X-Forwarded-For='10.192.128.26'
> Nov 29 04:04:55 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:55.022529
> 6410 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:04:56 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:56.163820
> 6410 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.74:57606 with
> User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36
> (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' with
> X-Forwarded-For='10.192.128.26'
> Nov 29 04:04:57 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:57.279559
> 6405 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:04:58 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:58.162578
> 6405 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.74:57666 with
> User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36
> (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' with
> X-Forwarded-For='10.192.128.26'
> Nov 29 04:04:59 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:59.538532
> 6408 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:00 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:00.161715
> 6403 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.74:57726 with
> User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36
> (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' with
> X-Forwarded-For='10.192.128.26'
> Nov 29 04:05:01 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:01.801764
> 6403 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:02 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:02.160838
> 6408 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.74:57790 with
> User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36
> (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' with
> X-Forwarded-For='10.192.128.26'
> Nov 29 04:05:04 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:04.060504
> 6409 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:04 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:04.162878
> 6409 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.74:57908 with
> User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36
> (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' with
> X-Forwarded-For='10.192.128.26'
> Nov 29 04:05:06 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:06.161875
> 6407 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.74:57962 with
> User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36
> (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' with
> X-Forwarded-For='10.192.128.26'
> Nov 29 04:05:06 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:06.318395
> 6409 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:08 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:08.160797
> 6408 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.74:58016 with
> User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36
> (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' with
> X-Forwarded-For='10.192.128.26'
> Nov 29 04:05:08 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:08.572476
> 6404 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:10 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:10.162160
> 6408 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.74:58070 with
> User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36
> (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' with
> X-Forwarded-For='10.192.128.26'
> Nov 29 04:05:10 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:10.828313
> 6408 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:12 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:12.161572
> 6407 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.74:58124 with
> User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36
> (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' with
> X-Forwarded-For='10.192.128.26'
> Nov 29 04:05:13 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:13.084312
> 6406 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:14 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:14.164269
> 6403 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.74:58182 with
> User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36
> (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' with
> X-Forwarded-For='10.192.128.26'
> Nov 29 04:05:15 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:15.339344
> 6406 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:16.165223
> 6405 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.74:58238 with
> User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36
> (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' with
> X-Forwarded-For='10.192.128.26'
> Nov 29 04:05:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:16.919145
> 6488 container_reader_impl.cpp:146] Throughput from container at port 51243
> (bytes): received=0, throttled=0
> Nov 29 04:05:17 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:17.476977
> 6403 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.74:58298 with
> User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36
> (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' with
> X-Forwarded-For='10.192.128.26'
> Nov 29 04:05:17 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:17.594532
> 6407 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:19 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:19.856494
> 6403 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:22 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:22.112598
> 6404 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:24 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:24.371556
> 6406 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:26 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:26.628479
> 6410 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:28 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:28.885504
> 6403 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:31 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:31.143674
> 6408 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:33 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:33.434650
> 6410 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:35 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:35.694718
> 6408 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:37 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:37.952450
> 6403 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:40 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:40.209468
> 6403 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:42 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:42.466473
> 6403 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:44 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:44.724468
> 6406 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:45 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:45.999135
> 6488 metrics_tcp_sender.cpp:252] TCP Throughput (bytes): sent=0, dropped=373,
> failed=0, pending=0 (state CONNECT_PENDING)
> Nov 29 04:05:46 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:46.188647
> 6408 slave.cpp:5044] Current disk usage 10.83%. Max allowed age:
> 1.583336138877014days
> Nov 29 04:05:46 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:46.986548
> 6403 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:49 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:49.034080
> 6488 metrics_tcp_sender.cpp:127] Timed out when opening metrics connection to
> 127.0.0.1:8124. This is expected if no Metrics Collector is running on this
> agent. (state CONNECT_PENDING)
> Nov 29 04:05:49 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:49.034231
> 6488 metrics_tcp_sender.cpp:112] Attempting to open connection to
> 127.0.0.1:8124
> Nov 29 04:05:49 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:49.034332
> 6488 metrics_tcp_sender.cpp:140] Got error 'Connection refused'(system:111)
> when connecting to metrics service at 127.0.0.1:8124. This is expected if no
> Metrics Collector is running on this agent. (state CONNECT_IN_PROGRESS)
> Nov 29 04:05:49 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:49.034358
> 6488 metrics_tcp_sender.cpp:94] Scheduling reconnect to 127.0.0.1:8124 in
> 60s...
> Nov 29 04:05:49 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:49.243544
> 6406 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:51 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:51.500578
> 6410 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:53 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:53.755667
> 6408 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:56 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:56.010568
> 6409 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:05:58 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:58.269441
> 6408 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:06:00 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:00.526623
> 6408 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:06:02 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:02.784615
> 6406 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:06:05 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:05.043455
> 6407 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:06:07 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:07.299536
> 6410 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:06:09 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:09.556404
> 6405 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:06:11 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:11.813442
> 6403 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:06:14 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:14.068482
> 6403 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:06:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:16.325479
> 6410 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:06:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:16.919404
> 6488 container_reader_impl.cpp:146] Throughput from container at port 51243
> (bytes): received=0, throttled=0
> Nov 29 04:06:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:18.581571
> 6403 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:06:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:20.841563
> 6407 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:06:23 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:23.098562
> 6403 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:06:25 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:25.358496
> 6404 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:06:26 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:26.886557
> 6406 slave.cpp:2288] Asked to kill task
> dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server5 of
> framework ce4bd8be-1198-4819-81d4-9a8439439741-0000
> Nov 29 04:06:26 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:26.886762
> 6405 slave.cpp:2288] Asked to kill task
> dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server1 of
> framework ce4bd8be-1198-4819-81d4-9a8439439741-0000
> Nov 29 04:06:26 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:26.886831
> 6405 slave.cpp:2288] Asked to kill task
> dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server3 of
> framework ce4bd8be-1198-4819-81d4-9a8439439741-0000
> Nov 29 04:06:26 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:26.886884
> 6405 slave.cpp:2288] Asked to kill task
> dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server4 of
> framework ce4bd8be-1198-4819-81d4-9a8439439741-0000
> Nov 29 04:06:26 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:26.886924
> 6405 slave.cpp:2288] Asked to kill task
> dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server2 of
> framework ce4bd8be-1198-4819-81d4-9a8439439741-0000
> Nov 29 04:06:27 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:27.615567
> 6404 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:06:29 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:29.877619
> 6405 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:06:32 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:32.135524
> 6408 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:06:34 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:34.402644
> 6409 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:06:36 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:36.664479
> 6407 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
> Nov 29 04:06:36 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:36.903115
> 6405 slave.cpp:2288] Asked to kill task
> dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server5 of
> framework ce4bd8be-1198-4819-81d4-9a8439439741-0000
> Nov 29 04:06:36 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:36.903259
> 6405 slave.cpp:2288] Asked to kill task
> dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server1 of
> framework ce4bd8be-1198-4819-81d4-9a8439439741-0000
> Nov 29 04:06:36 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:36.903317
> 6405 slave.cpp:2288] Asked to kill task
> dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server3 of
> framework ce4bd8be-1198-4819-81d4-9a8439439741-0000
> Nov 29 04:06:36 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:36.903436
> 6408 slave.cpp:2288] Asked to kill task
> dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server4 of
> framework ce4bd8be-1198-4819-81d4-9a8439439741-0000
> Nov 29 04:06:36 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:36.903501
> 6408 slave.cpp:2288] Asked to kill task
> dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server2 of
> framework ce4bd8be-1198-4819-81d4-9a8439439741-0000
> {code}
> Executor log:
> {code}
> I1129 04:04:17.832391 2734 executor.cpp:189] Version: 1.1.0
> I1129 04:04:17.956600 2738 default_executor.cpp:123] Received SUBSCRIBED
> event
> I1129 04:04:17.956832 2738 default_executor.cpp:127] Subscribed executor on
> 10.190.112.199
> I1129 04:04:17.968544 2735 default_executor.cpp:123] Received LAUNCH_GROUP
> event
> I1129 04:06:26.899070 2735 default_executor.cpp:123] Received KILL event
> I1129 04:06:26.899101 2735 default_executor.cpp:802] Received kill for task
> 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server5'
> W1129 04:06:26.899122 2735 default_executor.cpp:813] Ignoring kill for task
> 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server5' as it
> is no longer active
> I1129 04:06:26.899235 2742 default_executor.cpp:123] Received KILL event
> I1129 04:06:26.899251 2742 default_executor.cpp:802] Received kill for task
> 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server1'
> W1129 04:06:26.899260 2742 default_executor.cpp:813] Ignoring kill for task
> 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server1' as it
> is no longer active
> I1129 04:06:26.899425 2741 default_executor.cpp:123] Received KILL event
> I1129 04:06:26.899443 2741 default_executor.cpp:802] Received kill for task
> 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server3'
> W1129 04:06:26.899448 2741 default_executor.cpp:813] Ignoring kill for task
> 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server3' as it
> is no longer active
> I1129 04:06:26.899591 2740 default_executor.cpp:123] Received KILL event
> I1129 04:06:26.899616 2740 default_executor.cpp:802] Received kill for task
> 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server4'
> W1129 04:06:26.899627 2740 default_executor.cpp:813] Ignoring kill for task
> 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server4' as it
> is no longer active
> I1129 04:06:26.899699 2735 default_executor.cpp:123] Received KILL event
> I1129 04:06:26.899713 2735 default_executor.cpp:802] Received kill for task
> 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server2'
> W1129 04:06:26.899721 2735 default_executor.cpp:813] Ignoring kill for task
> 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server2' as it
> is no longer active
> I1129 04:06:36.915755 2740 default_executor.cpp:123] Received KILL event
> I1129 04:06:36.915786 2740 default_executor.cpp:802] Received kill for task
> 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server5'
> W1129 04:06:36.915796 2740 default_executor.cpp:813] Ignoring kill for task
> 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server5' as it
> is no longer active
> I1129 04:06:36.915915 2736 default_executor.cpp:123] Received KILL event
> I1129 04:06:36.915931 2736 default_executor.cpp:802] Received kill for task
> 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server1'
> W1129 04:06:36.915940 2736 default_executor.cpp:813] Ignoring kill for task
> 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server1' as it
> is no longer active
> I1129 04:06:36.915962 2736 default_executor.cpp:123] Received KILL event
> I1129 04:06:36.915972 2736 default_executor.cpp:802] Received kill for task
> 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server3'
> W1129 04:06:36.915979 2736 default_executor.cpp:813] Ignoring kill for task
> 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server3' as it
> is no longer active
> I1129 04:06:36.915997 2736 default_executor.cpp:123] Received KILL event
> I1129 04:06:36.916007 2736 default_executor.cpp:802] Received kill for task
> 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server4'
> W1129 04:06:36.916013 2736 default_executor.cpp:813] Ignoring kill for task
> 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server4' as it
> is no longer active
> I1129 04:06:36.916031 2736 default_executor.cpp:123] Received KILL event
> I1129 04:06:36.916040 2736 default_executor.cpp:802] Received kill for task
> 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server2'
> W1129 04:06:36.916048 2736 default_executor.cpp:813] Ignoring kill for task
> 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server2' as it
> is no longer active
> continuing for hours ...
> {code}
> Meanwhile, the tasks show up as {{STAGING}} in the Mesos web UI, and their
> sandboxes are empty.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
