[
https://issues.apache.org/jira/browse/MESOS-7601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Rukletsov reassigned MESOS-7601:
------------------------------------------
Assignee: Alexander Rukletsov
https://reviews.apache.org/r/59746/
> Some container launch failures are mistakenly treated as errors.
> ----------------------------------------------------------------
>
> Key: MESOS-7601
> URL: https://issues.apache.org/jira/browse/MESOS-7601
> Project: Mesos
> Issue Type: Bug
> Components: containerization
> Affects Versions: 1.3.0
> Reporter: Alexander Rukletsov
> Assignee: Alexander Rukletsov
> Labels: containerizer, mesosphere
>
> I've observed a case when a scheduler stops (i.e. calls TEARDOWN) while some
> of its tasks are being launched. While this is a valid behaviour, the agent
> prints an error and increased container launch errors metrics.
> Below are log excerpts for such framework,
> {{6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092}}.
> *Master log*
> {noformat}
> [centos@ip-172-31-6-200 ~]$ journalctl _PID=29716 --since "2 hours ago"
> --no-pager | grep
> "6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092"
> Jun 01 11:32:58 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:32:58.226218 29724 master.cpp:6072] Updating
> info for framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> Jun 01 11:32:58 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:32:58.226356 29728 hierarchical.cpp:274] Added
> framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> Jun 01 11:32:58 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:32:58.226405 29728 hierarchical.cpp:379]
> Deactivated framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> Jun 01 11:32:58 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:32:58.228570 29728 hierarchical.cpp:343]
> Activated framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> Jun 01 11:32:58 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:32:58.246068 29721 master.cpp:7105] Sending 1
> offers to framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> (TeraValidate) at
> scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:32:58 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:32:58.247851 29721 master.cpp:7194] Sending 1
> inverse offers to framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> (TeraValidate) at
> scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:32:58 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:32:58.912937 29728 master.cpp:4806] Processing
> DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509464 ] for
> framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> (TeraValidate) at
> scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:32:59 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:32:59.804184 29727 master.cpp:7105] Sending 2
> offers to framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> (TeraValidate) at
> scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:32:59 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:32:59.804411 29727 master.cpp:7194] Sending 2
> inverse offers to framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> (TeraValidate) at
> scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:01 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:01.248924 29721 master.cpp:7105] Sending 2
> offers to framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> (TeraValidate) at
> scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:01 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:01.249289 29721 master.cpp:7194] Sending 2
> inverse offers to framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> (TeraValidate) at
> scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:01 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:01.249724 29721 master.cpp:3851] Processing
> ACCEPT call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509469 ] on
> agent 36a25adb-4ea2-49d3-a195-448cff1dc146-S35 at slave(1)@172.31.13.122:5051
> (172.31.13.122) for framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> (TeraValidate) at
> scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:01 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:01.250141 29721 master.cpp:3851] Processing
> ACCEPT call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509470 ] on
> agent 36a25adb-4ea2-49d3-a195-448cff1dc146-S2 at slave(1)@172.31.7.202:5051
> (172.31.7.202) for framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> (TeraValidate) at
> scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:01 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:01.252516 29721 master.cpp:4501] Launching
> task 0 of framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> (TeraValidate) at
> scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531 with
> resources cpus(*)(allocated: *):1; mem(*)(allocated: *):1408 on agent
> 36a25adb-4ea2-49d3-a195-448cff1dc146-S35 at slave(1)@172.31.13.122:5051
> (172.31.13.122)
> Jun 01 11:33:01 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:01.254794 29721 master.cpp:4501] Launching
> task 1 of framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> (TeraValidate) at
> scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531 with
> resources cpus(*)(allocated: *):1; mem(*)(allocated: *):1408 on agent
> 36a25adb-4ea2-49d3-a195-448cff1dc146-S2 at slave(1)@172.31.7.202:5051
> (172.31.7.202)
> Jun 01 11:33:01 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:01.255506 29725 hierarchical.cpp:855]
> Updated allocation of framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 on agent
> 36a25adb-4ea2-49d3-a195-448cff1dc146-S35 from ports(*)(allocated:
> *):[1025-2180, 2182-3887, 3889-5049, 5052-6999, 7002-7076, 7078-7079,
> 7082-7198, 7200-8079, 8082-8180, 8183-8608, 8610-8982, 8984-9002, 9006-9041,
> 9043-9159, 9161-10012, 10016-10666, 10669-11682, 11685-11872, 11874-21620,
> 21622-30643, 30646-32000]; cpus(*)(allocated: *):1.1; mem(*)(allocated:
> *):6456; disk(*)(allocated: *):46032; disk(*)(allocated:
> *)[MOUNT:/dcos/volume3]:47540; disk(*)(allocated:
> *)[MOUNT:/dcos/volume4]:47540 to ports(*)(allocated: *):[1025-2180,
> 2182-3887, 3889-5049, 5052-6999, 7002-7076, 7078-7079, 7082-7198, 7200-8079,
> 8082-8180, 8183-8608, 8610-8982, 8984-9002, 9006-9041, 9043-9159, 9161-10012,
> 10016-10666, 10669-11682, 11685-11872, 11874-21620, 21622-30643,
> 30646-32000]; cpus(*)(allocated: *):1.1; mem(*)(allocated: *):6456;
> disk(*)(allocated: *):46032; disk(*)(allocated:
> *)[MOUNT:/dcos/volume3]:47540; disk(*)(allocated:
> *)[MOUNT:/dcos/volume4]:47540
> Jun 01 11:33:01 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:01.258015 29725 hierarchical.cpp:855]
> Updated allocation of framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 on agent
> 36a25adb-4ea2-49d3-a195-448cff1dc146-S2 from ports(*)(allocated:
> *):[1027-2180, 2182-3887, 3889-5049, 5052-6875, 6877-8079, 8082-8180,
> 8182-9299, 9301-9543, 9545-14041, 14043-15028, 15030-24844, 24846-32000];
> disk(*)(allocated: *)[MOUNT:/dcos/volume0]:51042; disk(*)(allocated:
> *):43530; cpus(*)(allocated: *):1.08; mem(*)(allocated: *):9357 to
> ports(*)(allocated: *):[1027-2180, 2182-3887, 3889-5049, 5052-6875,
> 6877-8079, 8082-8180, 8182-9299, 9301-9543, 9545-14041, 14043-15028,
> 15030-24844, 24846-32000]; disk(*)(allocated: *)[MOUNT:/dcos/volume0]:51042;
> disk(*)(allocated: *):43530; cpus(*)(allocated: *):1.08; mem(*)(allocated:
> *):9357
> Jun 01 11:33:01 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:01.322147 29724 master.cpp:4806] Processing
> DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509473 ] for
> framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> (TeraValidate) at
> scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:01 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:01.322619 29724 master.cpp:4806] Processing
> DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509474 ] for
> framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> (TeraValidate) at
> scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:03 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:03.113775 29722 master.cpp:6269] Status
> update TASK_RUNNING (UUID: 646de179-526f-48e4-8fe9-4deda3a09179) for task 0
> of framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 from
> agent 36a25adb-4ea2-49d3-a195-448cff1dc146-S35 at slave(1)@172.31.13.122:5051
> (172.31.13.122)
> Jun 01 11:33:03 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:03.113813 29722 master.cpp:6337] Forwarding
> status update TASK_RUNNING (UUID: 646de179-526f-48e4-8fe9-4deda3a09179) for
> task 0 of framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> Jun 01 11:33:03 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:03.117269 29722 master.cpp:8368] Updating
> the state of task 0 of framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (latest
> state: TASK_RUNNING, status update state: TASK_RUNNING)
> Jun 01 11:33:03 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:03.216639 29723 master.cpp:5163] Processing
> ACKNOWLEDGE call 646de179-526f-48e4-8fe9-4deda3a09179 for task 0 of framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> (TeraValidate) at
> scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531 on agent
> 36a25adb-4ea2-49d3-a195-448cff1dc146-S35
> Jun 01 11:33:05 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:05.410168 29722 master.cpp:7105] Sending 1
> offers to framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> (TeraValidate) at
> scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:05 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:05.410367 29722 master.cpp:7194] Sending 1
> inverse offers to framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> (TeraValidate) at
> scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:05 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:05.413863 29722 master.cpp:4806] Processing
> DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509489 ] for
> framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> (TeraValidate) at
> scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:06 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:06.643015 29723 master.cpp:7105] Sending 1
> offers to framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> (TeraValidate) at
> scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:06 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:06.645283 29722 master.cpp:4806] Processing
> DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509492 ] for
> framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> (TeraValidate) at
> scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:08 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:08.385871 29728 master.cpp:7105] Sending 1
> offers to framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> (TeraValidate) at
> scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:08 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:08.388234 29726 master.cpp:4806] Processing
> DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509495 ] for
> framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> (TeraValidate) at
> scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:11 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:11.465273 29723 master.cpp:7105] Sending 1
> offers to framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> (TeraValidate) at
> scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:11 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:11.467978 29725 master.cpp:4806] Processing
> DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509499 ] for
> framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> (TeraValidate) at
> scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:12 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:12.481941 29726 master.cpp:7105] Sending 1
> offers to framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> (TeraValidate) at
> scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:12 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:12.484498 29721 master.cpp:4806] Processing
> DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509500 ] for
> framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> (TeraValidate) at
> scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:16 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:16.552039 29724 master.cpp:6269] Status
> update TASK_FINISHED (UUID: f49ba849-90cc-4110-b897-0d5d16a17588) for task 0
> of framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 from
> agent 36a25adb-4ea2-49d3-a195-448cff1dc146-S35 at slave(1)@172.31.13.122:5051
> (172.31.13.122)
> Jun 01 11:33:16 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:16.552119 29724 master.cpp:6337] Forwarding
> status update TASK_FINISHED (UUID: f49ba849-90cc-4110-b897-0d5d16a17588) for
> task 0 of framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> Jun 01 11:33:16 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:16.553474 29724 master.cpp:8368] Updating
> the state of task 0 of framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (latest
> state: TASK_FINISHED, status update state: TASK_FINISHED)
> Jun 01 11:33:16 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:16.556002 29724 master.cpp:5163] Processing
> ACKNOWLEDGE call f49ba849-90cc-4110-b897-0d5d16a17588 for task 0 of framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> (TeraValidate) at
> scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531 on agent
> 36a25adb-4ea2-49d3-a195-448cff1dc146-S35
> Jun 01 11:33:16 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:16.556046 29724 master.cpp:8462] Removing
> task 0 with resources cpus(*)(allocated: *):1; mem(*)(allocated: *):1408 of
> framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 on agent
> 36a25adb-4ea2-49d3-a195-448cff1dc146-S35 at slave(1)@172.31.13.122:5051
> (172.31.13.122)
> Jun 01 11:33:16 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:16.556318 29727 master.cpp:4911] Processing
> REVIVE call for framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> (TeraValidate) at
> scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:16 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:16.556380 29727 hierarchical.cpp:1260]
> Revived offers for roles { * } of framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> Jun 01 11:33:16 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:16.607833 29724 master.cpp:7105] Sending 1
> offers to framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> (TeraValidate) at
> scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:16 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:16.611508 29728 master.cpp:4806] Processing
> DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509503 ] for
> framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> (TeraValidate) at
> scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:17 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:17.590775 29725 master.cpp:7105] Sending 1
> offers to framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> (TeraValidate) at
> scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:17 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:17.592618 29723 master.cpp:4806] Processing
> DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509504 ] for
> framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> (TeraValidate) at
> scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:18 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:18.153723 29725 master.cpp:7105] Sending 1
> offers to framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> (TeraValidate) at
> scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:18 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:18.155370 29723 master.cpp:4806] Processing
> DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509505 ] for
> framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> (TeraValidate) at
> scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:22 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:22.695742 29722 master.cpp:7105] Sending 1
> offers to framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> (TeraValidate) at
> scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:22 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:22.697412 29724 master.cpp:4806] Processing
> DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509512 ] for
> framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> (TeraValidate) at
> scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:23 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:23.714365 29722 master.cpp:7105] Sending 1
> offers to framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> (TeraValidate) at
> scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:23 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:23.716039 29726 master.cpp:4806] Processing
> DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509514 ] for
> framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> (TeraValidate) at
> scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:24 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:24.728322 29727 master.cpp:7105] Sending 1
> offers to framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> (TeraValidate) at
> scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:24 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:24.728845 29727 master.cpp:7194] Sending 1
> inverse offers to framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> (TeraValidate) at
> scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:24 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:24.729948 29728 master.cpp:4806] Processing
> DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509515 ] for
> framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> (TeraValidate) at
> scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:25 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:25.295658 29723 master.cpp:7788] Processing
> TEARDOWN call for framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> (TeraValidate) at
> scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:25 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:25.295702 29723 master.cpp:7800] Removing
> framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> (TeraValidate) at
> scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:25 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:25.295724 29723 master.cpp:3160]
> Deactivating framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> (TeraValidate) at
> scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:25 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:25.296236 29724 hierarchical.cpp:379]
> Deactivated framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> Jun 01 11:33:25 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:25.298550 29723 master.cpp:8368] Updating
> the state of task 1 of framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (latest
> state: TASK_KILLED, status update state: TASK_KILLED)
> Jun 01 11:33:25 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:25.298677 29723 master.cpp:8462] Removing
> task 1 with resources cpus(*)(allocated: *):1; mem(*)(allocated: *):1408 of
> framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 on agent
> 36a25adb-4ea2-49d3-a195-448cff1dc146-S2 at slave(1)@172.31.7.202:5051
> (172.31.7.202)
> Jun 01 11:33:25 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:25.298863 29726 hierarchical.cpp:326]
> Removed framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> Jun 01 11:33:25 ip-172-31-6-200.us-west-2.compute.internal
> mesos-master[29716]: I0601 11:33:25.299028 29723 master.cpp:7118] Master
> ignoring inverse offers to framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 because
> the framework has terminated or is inactive
> {noformat}
> *Agent log*
> {noformat}
> [centos@ip-172-31-7-202 ~]$ journalctl _PID=12073 --since "1 hour ago"
> --no-pager | grep -C 10 "failed to start:"
> Jun 01 11:33:28 ip-172-31-7-202.us-west-2.compute.internal
> mesos-agent[12073]: I0601 11:33:28.785028 12081 http.cpp:1115] HTTP GET for
> /slave(1)/state from 172.31.7.202:52855
> Jun 01 11:33:29 ip-172-31-7-202.us-west-2.compute.internal
> mesos-agent[12073]: I0601 11:33:29.109624 12080 logfmt.cpp:177]
> dstip=172.31.7.202 type=audit timestamp=2017-06-01 11:33:29.109526016+00:00
> reason="Valid authorization token" uid="dcos_metrics_agent"
> object="/slave(1)/state" agent="dcos-metrics/1.1.0-64-g62702c3"
> authorizer="mesos-agent" action="GET" result=allow srcip=172.31.7.202
> dstport=5051 srcport=49102
> Jun 01 11:33:29 ip-172-31-7-202.us-west-2.compute.internal
> mesos-agent[12073]: I0601 11:33:29.109864 12081 http.cpp:1115] HTTP GET for
> /slave(1)/state from 172.31.7.202:49102 with
> User-Agent='dcos-metrics/1.1.0-64-g62702c3'
> Jun 01 11:33:29 ip-172-31-7-202.us-west-2.compute.internal
> mesos-agent[12073]: I0601 11:33:29.347921 12084 logfmt.cpp:177]
> dstip=172.31.7.202 type=audit timestamp=2017-06-01 11:33:29.347860992+00:00
> reason="Valid authorization token" uid="dcos_metrics_agent"
> object="/slave(1)/containers" agent="dcos-metrics/1.1.0-64-g62702c3"
> authorizer="mesos-agent" action="GET" result=allow srcip=172.31.7.202
> dstport=5051 srcport=49102
> Jun 01 11:33:29 ip-172-31-7-202.us-west-2.compute.internal
> mesos-agent[12073]: I0601 11:33:29.348116 12077 http.cpp:1115] HTTP GET for
> /slave(1)/containers from 172.31.7.202:49102 with
> User-Agent='dcos-metrics/1.1.0-64-g62702c3'
> Jun 01 11:33:29 ip-172-31-7-202.us-west-2.compute.internal
> mesos-agent[12073]: W0601 11:33:29.712091 12079 http.cpp:2160] Failed to get
> resource statistics for executor '"1"' of framework
> "6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092":
> Failed to run 'docker -H unix:///var/run/docker.sock inspect
> mesos-36a25adb-4ea2-49d3-a195-448cff1dc146-S2.5c1f53e4-fffe-4f3a-8847-c5a252a25d3b':
> exited with status 1; stderr='Error: No such object:
> mesos-36a25adb-4ea2-49d3-a195-448cff1dc146-S2.5c1f53e4-fffe-4f3a-8847-c5a252a25d3b
> Jun 01 11:33:29 ip-172-31-7-202.us-west-2.compute.internal
> mesos-agent[12073]: '
> Jun 01 11:33:30 ip-172-31-7-202.us-west-2.compute.internal
> mesos-agent[12073]: I0601 11:33:30.298966 12083 slave.cpp:5548] Killing
> executor '1' of framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> Jun 01 11:33:30 ip-172-31-7-202.us-west-2.compute.internal
> mesos-agent[12073]: I0601 11:33:30.299078 12083 docker.cpp:2123] Destroying
> container 5c1f53e4-fffe-4f3a-8847-c5a252a25d3b
> Jun 01 11:33:30 ip-172-31-7-202.us-west-2.compute.internal
> mesos-agent[12073]: I0601 11:33:30.299108 12083 docker.cpp:2165] Destroying
> container 5c1f53e4-fffe-4f3a-8847-c5a252a25d3b in PULLING state
> Jun 01 11:33:30 ip-172-31-7-202.us-west-2.compute.internal
> mesos-agent[12073]: E0601 11:33:30.415652 12082 slave.cpp:5041] Container
> '5c1f53e4-fffe-4f3a-8847-c5a252a25d3b' for executor '1' of framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 failed
> to start: future discarded
> Jun 01 11:33:30 ip-172-31-7-202.us-west-2.compute.internal
> mesos-agent[12073]: E0601 11:33:30.415765 12082 slave.cpp:5148] Termination
> of executor '1' of framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 failed:
> unknown container
> Jun 01 11:33:30 ip-172-31-7-202.us-west-2.compute.internal
> mesos-agent[12073]: I0601 11:33:30.415794 12082 slave.cpp:5261] Cleaning up
> executor '1' of framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> Jun 01 11:33:30 ip-172-31-7-202.us-west-2.compute.internal
> mesos-agent[12073]: W0601 11:33:30.415937 12077 composing.cpp:638] Attempted
> to destroy unknown container 5c1f53e4-fffe-4f3a-8847-c5a252a25d3b
> Jun 01 11:33:30 ip-172-31-7-202.us-west-2.compute.internal
> mesos-agent[12073]: I0601 11:33:30.415966 12082 slave.cpp:5349] Cleaning up
> framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> Jun 01 11:33:30 ip-172-31-7-202.us-west-2.compute.internal
> mesos-agent[12073]: I0601 11:33:30.415992 12077 gc.cpp:55] Scheduling
> '/var/lib/mesos/slave/slaves/36a25adb-4ea2-49d3-a195-448cff1dc146-S2/frameworks/6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092/executors/1/runs/5c1f53e4-fffe-4f3a-8847-c5a252a25d3b'
> for gc 1.99999518647111days in the future
> Jun 01 11:33:30 ip-172-31-7-202.us-west-2.compute.internal
> mesos-agent[12073]: I0601 11:33:30.416050 12082
> status_update_manager.cpp:285] Closing status update streams for framework
> 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> Jun 01 11:33:30 ip-172-31-7-202.us-west-2.compute.internal
> mesos-agent[12073]: I0601 11:33:30.416061 12077 gc.cpp:55] Scheduling
> '/var/lib/mesos/slave/slaves/36a25adb-4ea2-49d3-a195-448cff1dc146-S2/frameworks/6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092/executors/1'
> for gc 1.99999518583407days in the future
> Jun 01 11:33:30 ip-172-31-7-202.us-west-2.compute.internal
> mesos-agent[12073]: I0601 11:33:30.416138 12077 gc.cpp:55] Scheduling
> '/var/lib/mesos/slave/slaves/36a25adb-4ea2-49d3-a195-448cff1dc146-S2/frameworks/6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092'
> for gc 1.99999518486222days in the future
> Jun 01 11:33:31 ip-172-31-7-202.us-west-2.compute.internal
> mesos-agent[12073]: I0601 11:33:31.574026 12079 logfmt.cpp:177]
> dstip=172.31.7.202 type=audit timestamp=2017-06-01 11:33:31.573729024+00:00
> reason="Valid authorization token" uid="dcos_navstar_agent"
> object="/slave(1)/state" authorizer="mesos-agent" action="GET" result=allow
> srcip=172.31.7.202 dstport=5051 srcport=52855
> Jun 01 11:33:31 ip-172-31-7-202.us-west-2.compute.internal
> mesos-agent[12073]: I0601 11:33:31.574285 12079 http.cpp:1115] HTTP GET for
> /slave(1)/state from 172.31.7.202:52855
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
