See <https://builds.apache.org/job/Mesos-Buildbot/BUILDTOOL=cmake,COMPILER=gcc,CONFIGURATION=--verbose%20--disable-libtool-wrappers%20--enable-libevent%20--enable-ssl,ENVIRONMENT=GLOG_v=1%20MESOS_VERBOSE=1,OS=centos%3A7,label_exp=(ubuntu)&&(!ubuntu-us1)&&(!ubuntu-eu2)&&(!qnode3)&&(!H23)/4491/display/redirect?page=changes>
Changes: [alexr] Added a comment about PythonFramework test system requirements. ------------------------------------------ [...truncated 3.46 MB...] 3: I1122 16:48:16.498090 17556 slave.cpp:1806] Got assigned task '1' for framework 83d06bbb-5575-4b50-99b7-bcdc230fc387-0000 3: I1122 16:48:16.498714 17556 slave.cpp:2074] Authorizing task '1' for framework 83d06bbb-5575-4b50-99b7-bcdc230fc387-0000 3: I1122 16:48:16.498766 17556 slave.cpp:7270] Authorizing framework principal 'test-principal' to launch task 1 3: I1122 16:48:16.499374 17565 slave.cpp:2242] Launching task '1' for framework 83d06bbb-5575-4b50-99b7-bcdc230fc387-0000 3: I1122 16:48:16.499982 17565 paths.cpp:621] Trying to chown '/tmp/FaultToleranceTest_SchedulerFailoverExecutorToFrameworkMessage_qZcazY/slaves/83d06bbb-5575-4b50-99b7-bcdc230fc387-S0/frameworks/83d06bbb-5575-4b50-99b7-bcdc230fc387-0000/executors/default/runs/4cc82b19-82ee-4cb9-9e4e-b3b00958d947' to user 'mesos' 3: I1122 16:48:16.500236 17565 slave.cpp:7744] Launching executor 'default' of framework 83d06bbb-5575-4b50-99b7-bcdc230fc387-0000 with resources [] in work directory '/tmp/FaultToleranceTest_SchedulerFailoverExecutorToFrameworkMessage_qZcazY/slaves/83d06bbb-5575-4b50-99b7-bcdc230fc387-S0/frameworks/83d06bbb-5575-4b50-99b7-bcdc230fc387-0000/executors/default/runs/4cc82b19-82ee-4cb9-9e4e-b3b00958d947' 3: I1122 16:48:16.501106 17565 slave.cpp:2470] Queued task '1' for executor 'default' of framework 83d06bbb-5575-4b50-99b7-bcdc230fc387-0000 3: I1122 16:48:16.501195 17565 slave.cpp:958] Successfully attached '/tmp/FaultToleranceTest_SchedulerFailoverExecutorToFrameworkMessage_qZcazY/slaves/83d06bbb-5575-4b50-99b7-bcdc230fc387-S0/frameworks/83d06bbb-5575-4b50-99b7-bcdc230fc387-0000/executors/default/runs/4cc82b19-82ee-4cb9-9e4e-b3b00958d947' to virtual path '/tmp/FaultToleranceTest_SchedulerFailoverExecutorToFrameworkMessage_qZcazY/slaves/83d06bbb-5575-4b50-99b7-bcdc230fc387-S0/frameworks/83d06bbb-5575-4b50-99b7-bcdc230fc387-0000/executors/default/runs/latest' 3: I1122 16:48:16.501240 17565 slave.cpp:958] Successfully attached '/tmp/FaultToleranceTest_SchedulerFailoverExecutorToFrameworkMessage_qZcazY/slaves/83d06bbb-5575-4b50-99b7-bcdc230fc387-S0/frameworks/83d06bbb-5575-4b50-99b7-bcdc230fc387-0000/executors/default/runs/4cc82b19-82ee-4cb9-9e4e-b3b00958d947' to virtual path '/frameworks/83d06bbb-5575-4b50-99b7-bcdc230fc387-0000/executors/default/runs/latest' 3: I1122 16:48:16.501287 17565 slave.cpp:958] Successfully attached '/tmp/FaultToleranceTest_SchedulerFailoverExecutorToFrameworkMessage_qZcazY/slaves/83d06bbb-5575-4b50-99b7-bcdc230fc387-S0/frameworks/83d06bbb-5575-4b50-99b7-bcdc230fc387-0000/executors/default/runs/4cc82b19-82ee-4cb9-9e4e-b3b00958d947' to virtual path '/tmp/FaultToleranceTest_SchedulerFailoverExecutorToFrameworkMessage_qZcazY/slaves/83d06bbb-5575-4b50-99b7-bcdc230fc387-S0/frameworks/83d06bbb-5575-4b50-99b7-bcdc230fc387-0000/executors/default/runs/4cc82b19-82ee-4cb9-9e4e-b3b00958d947' 3: I1122 16:48:16.501572 17565 slave.cpp:2933] Launching container 4cc82b19-82ee-4cb9-9e4e-b3b00958d947 for executor 'default' of framework 83d06bbb-5575-4b50-99b7-bcdc230fc387-0000 3: W1122 16:48:16.504072 17557 process.cpp:2756] Attempted to spawn already running process [email protected]:56733 3: I1122 16:48:16.504268 17557 exec.cpp:162] Version: 1.5.0 3: I1122 16:48:16.504583 17560 exec.cpp:212] Executor started at: executor(6)@172.17.0.2:56733 with pid 17551 3: I1122 16:48:16.504828 17555 slave.cpp:4066] Got registration for executor 'default' of framework 83d06bbb-5575-4b50-99b7-bcdc230fc387-0000 from executor(6)@172.17.0.2:56733 3: I1122 16:48:16.505214 17562 exec.cpp:237] Executor registered on agent 83d06bbb-5575-4b50-99b7-bcdc230fc387-S0 3: I1122 16:48:16.505259 17562 exec.cpp:249] Executor::registered took 26688ns 3: I1122 16:48:16.505720 17555 slave.cpp:2672] Sending queued task '1' to executor 'default' of framework 83d06bbb-5575-4b50-99b7-bcdc230fc387-0000 at executor(6)@172.17.0.2:56733 3: I1122 16:48:16.506219 17554 exec.cpp:331] Executor asked to run task '1' 3: I1122 16:48:16.506306 17554 exec.cpp:340] Executor::launchTask took 63602ns 3: I1122 16:48:16.506408 17554 exec.cpp:571] Executor sending status update TASK_RUNNING (UUID: eaa8bf85-75f7-4799-bb1c-5f77a19f0c80) for task 1 of framework 83d06bbb-5575-4b50-99b7-bcdc230fc387-0000 3: I1122 16:48:16.506716 17552 slave.cpp:4520] Handling status update TASK_RUNNING (UUID: eaa8bf85-75f7-4799-bb1c-5f77a19f0c80) for task 1 of framework 83d06bbb-5575-4b50-99b7-bcdc230fc387-0000 from executor(6)@172.17.0.2:56733 3: I1122 16:48:16.507316 17567 task_status_update_manager.cpp:328] Received task status update TASK_RUNNING (UUID: eaa8bf85-75f7-4799-bb1c-5f77a19f0c80) for task 1 of framework 83d06bbb-5575-4b50-99b7-bcdc230fc387-0000 3: I1122 16:48:16.507352 17567 task_status_update_manager.cpp:507] Creating StatusUpdate stream for task 1 of framework 83d06bbb-5575-4b50-99b7-bcdc230fc387-0000 3: I1122 16:48:16.507841 17567 task_status_update_manager.cpp:383] Forwarding task status update TASK_RUNNING (UUID: eaa8bf85-75f7-4799-bb1c-5f77a19f0c80) for task 1 of framework 83d06bbb-5575-4b50-99b7-bcdc230fc387-0000 to the agent 3: I1122 16:48:16.508045 17561 slave.cpp:5003] Forwarding the update TASK_RUNNING (UUID: eaa8bf85-75f7-4799-bb1c-5f77a19f0c80) for task 1 of framework 83d06bbb-5575-4b50-99b7-bcdc230fc387-0000 to [email protected]:56733 3: I1122 16:48:16.508273 17561 slave.cpp:4896] Task status update manager successfully handled status update TASK_RUNNING (UUID: eaa8bf85-75f7-4799-bb1c-5f77a19f0c80) for task 1 of framework 83d06bbb-5575-4b50-99b7-bcdc230fc387-0000 3: I1122 16:48:16.508327 17561 slave.cpp:4912] Sending acknowledgement for status update TASK_RUNNING (UUID: eaa8bf85-75f7-4799-bb1c-5f77a19f0c80) for task 1 of framework 83d06bbb-5575-4b50-99b7-bcdc230fc387-0000 to executor(6)@172.17.0.2:56733 3: I1122 16:48:16.508400 17556 master.cpp:7268] Status update TASK_RUNNING (UUID: eaa8bf85-75f7-4799-bb1c-5f77a19f0c80) for task 1 of framework 83d06bbb-5575-4b50-99b7-bcdc230fc387-0000 from agent 83d06bbb-5575-4b50-99b7-bcdc230fc387-S0 at slave(50)@172.17.0.2:56733 (a79443d495cc) 3: I1122 16:48:16.508491 17556 master.cpp:7330] Forwarding status update TASK_RUNNING (UUID: eaa8bf85-75f7-4799-bb1c-5f77a19f0c80) for task 1 of framework 83d06bbb-5575-4b50-99b7-bcdc230fc387-0000 3: I1122 16:48:16.508554 17559 exec.cpp:388] Executor received status update acknowledgement eaa8bf85-75f7-4799-bb1c-5f77a19f0c80 for task 1 of framework 83d06bbb-5575-4b50-99b7-bcdc230fc387-0000 3: I1122 16:48:16.508698 17556 master.cpp:9523] Updating the state of task 1 of framework 83d06bbb-5575-4b50-99b7-bcdc230fc387-0000 (latest state: TASK_RUNNING, status update state: TASK_RUNNING) 3: I1122 16:48:16.508916 17559 sched.cpp:1029] Scheduler::statusUpdate took 83013ns 3: I1122 16:48:16.509356 17560 master.cpp:5793] Processing ACKNOWLEDGE call eaa8bf85-75f7-4799-bb1c-5f77a19f0c80 for task 1 of framework 83d06bbb-5575-4b50-99b7-bcdc230fc387-0000 (default) at [email protected]:56733 on agent 83d06bbb-5575-4b50-99b7-bcdc230fc387-S0 3: W1122 16:48:16.509661 17551 process.cpp:2756] Attempted to spawn already running process [email protected]:56733 3: I1122 16:48:16.509699 17555 task_status_update_manager.cpp:401] Received task status update acknowledgement (UUID: eaa8bf85-75f7-4799-bb1c-5f77a19f0c80) for task 1 of framework 83d06bbb-5575-4b50-99b7-bcdc230fc387-0000 3: I1122 16:48:16.510059 17552 slave.cpp:3804] Task status update manager successfully handled status update acknowledgement (UUID: eaa8bf85-75f7-4799-bb1c-5f77a19f0c80) for task 1 of framework 83d06bbb-5575-4b50-99b7-bcdc230fc387-0000 3: I1122 16:48:16.510412 17551 sched.cpp:232] Version: 1.5.0 3: I1122 16:48:16.511055 17558 sched.cpp:336] New master detected at [email protected]:56733 3: I1122 16:48:16.511163 17558 sched.cpp:396] Authenticating with master [email protected]:56733 3: I1122 16:48:16.511176 17558 sched.cpp:403] Using default CRAM-MD5 authenticatee 3: I1122 16:48:16.511540 17558 authenticatee.cpp:121] Creating new client SASL connection 3: I1122 16:48:16.512054 17558 master.cpp:8306] Authenticating [email protected]:56733 3: I1122 16:48:16.512199 17559 authenticator.cpp:414] Starting authentication session for crammd5-authenticatee(112)@172.17.0.2:56733 3: I1122 16:48:16.512467 17557 authenticator.cpp:98] Creating new server SASL connection 3: I1122 16:48:16.512717 17560 authenticatee.cpp:213] Received SASL authentication mechanisms: CRAM-MD5 3: I1122 16:48:16.512743 17560 authenticatee.cpp:239] Attempting to authenticate with mechanism 'CRAM-MD5' 3: I1122 16:48:16.512867 17553 authenticator.cpp:204] Received SASL authentication start 3: I1122 16:48:16.512935 17553 authenticator.cpp:326] Authentication requires more steps 3: I1122 16:48:16.513054 17554 authenticatee.cpp:259] Received SASL authentication step 3: I1122 16:48:16.513200 17555 authenticator.cpp:232] Received SASL authentication step 3: I1122 16:48:16.513238 17555 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'a79443d495cc' server FQDN: 'a79443d495cc' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false 3: I1122 16:48:16.513252 17555 auxprop.cpp:181] Looking up auxiliary property '*userPassword' 3: I1122 16:48:16.513300 17555 auxprop.cpp:181] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' 3: I1122 16:48:16.513321 17555 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'a79443d495cc' server FQDN: 'a79443d495cc' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true 3: I1122 16:48:16.513334 17555 auxprop.cpp:131] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true 3: I1122 16:48:16.513340 17555 auxprop.cpp:131] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true 3: I1122 16:48:16.513360 17555 authenticator.cpp:318] Authentication success 3: I1122 16:48:16.513435 17552 authenticatee.cpp:299] Authentication success 3: I1122 16:48:16.513572 17566 master.cpp:8336] Successfully authenticated principal 'test-principal' at [email protected]:56733 3: I1122 16:48:16.513643 17556 authenticator.cpp:432] Authentication session cleanup for crammd5-authenticatee(112)@172.17.0.2:56733 3: I1122 16:48:16.513696 17552 sched.cpp:502] Successfully authenticated with master [email protected]:56733 3: I1122 16:48:16.513715 17552 sched.cpp:824] Sending SUBSCRIBE call to [email protected]:56733 3: I1122 16:48:16.513857 17552 sched.cpp:857] Will retry registration in 724.603362ms if necessary 3: I1122 16:48:16.514075 17565 master.cpp:2963] Received SUBSCRIBE call for framework 'default' at [email protected]:56733 3: I1122 16:48:16.514145 17565 master.cpp:2280] Authorizing framework principal 'test-principal' to receive offers for roles '{ * }' 3: I1122 16:48:16.514601 17558 master.cpp:3043] Subscribing framework default with checkpointing disabled and capabilities [ RESERVATION_REFINEMENT ] 3: I1122 16:48:16.514655 17558 master.cpp:6988] Updating info for framework 83d06bbb-5575-4b50-99b7-bcdc230fc387-0000 3: I1122 16:48:16.514756 17558 master.cpp:3152] Framework 83d06bbb-5575-4b50-99b7-bcdc230fc387-0000 (default) at [email protected]:56733 failed over 3: I1122 16:48:16.514888 17560 sched.cpp:1175] Got error 'Framework failed over' 3: I1122 16:48:16.514910 17560 sched.cpp:2043] Asked to abort the driver 3: I1122 16:48:16.515017 17560 sched.cpp:1186] Scheduler::error took 77950ns 3: I1122 16:48:16.515125 17553 sched.cpp:751] Framework registered with 83d06bbb-5575-4b50-99b7-bcdc230fc387-0000 3: I1122 16:48:16.515178 17553 sched.cpp:765] Scheduler::registered took 31524ns 3: I1122 16:48:16.515245 17560 sched.cpp:1221] Aborting framework 83d06bbb-5575-4b50-99b7-bcdc230fc387-0000 3: I1122 16:48:16.515298 17567 slave.cpp:3441] Updating info for framework 83d06bbb-5575-4b50-99b7-bcdc230fc387-0000 with pid updated to [email protected]:56733 3: I1122 16:48:16.515403 17561 task_status_update_manager.cpp:188] Resuming sending task status updates 3: W1122 16:48:16.515475 17563 master.cpp:3282] Ignoring deactivate framework message for framework 83d06bbb-5575-4b50-99b7-bcdc230fc387-0000 (default) at [email protected]:56733 because it is not expected from [email protected]:56733 3: I1122 16:48:16.515851 17566 slave.cpp:5071] Sending message for framework 83d06bbb-5575-4b50-99b7-bcdc230fc387-0000 to [email protected]:56733 3: I1122 16:48:16.516072 17556 sched.cpp:1165] Scheduler::frameworkMessage took 51674ns 3: I1122 16:48:16.516335 17551 sched.cpp:2009] Asked to stop the driver 3: I1122 16:48:16.516376 17551 sched.cpp:2009] Asked to stop the driver 3: I1122 16:48:16.516415 17564 sched.cpp:1191] Stopping framework 83d06bbb-5575-4b50-99b7-bcdc230fc387-0000 3: I1122 16:48:16.516485 17560 sched.cpp:1191] Stopping framework 83d06bbb-5575-4b50-99b7-bcdc230fc387-0000 3: W1122 16:48:16.516644 17562 master.cpp:2346] Dropping TEARDOWN call from framework 83d06bbb-5575-4b50-99b7-bcdc230fc387-0000 at [email protected]:56733: Call is not from registered framework 3: I1122 16:48:16.516824 17562 master.cpp:8813] Processing TEARDOWN call for framework 83d06bbb-5575-4b50-99b7-bcdc230fc387-0000 (default) at [email protected]:56733 3: I1122 16:48:16.516849 17562 master.cpp:8825] Removing framework 83d06bbb-5575-4b50-99b7-bcdc230fc387-0000 (default) at [email protected]:56733 3: I1122 16:48:16.516860 17562 master.cpp:3333] Deactivating framework 83d06bbb-5575-4b50-99b7-bcdc230fc387-0000 (default) at [email protected]:56733 3: I1122 16:48:16.517002 17553 hierarchical.cpp:419] Deactivated framework 83d06bbb-5575-4b50-99b7-bcdc230fc387-0000 3: I1122 16:48:16.517024 17559 slave.cpp:3270] Asked to shut down framework 83d06bbb-5575-4b50-99b7-bcdc230fc387-0000 by [email protected]:56733 3: I1122 16:48:16.517033 17562 master.cpp:9523] Updating the state of task 1 of framework 83d06bbb-5575-4b50-99b7-bcdc230fc387-0000 (latest state: TASK_KILLED, status update state: TASK_KILLED) 3: I1122 16:48:16.517057 17559 slave.cpp:3295] Shutting down framework 83d06bbb-5575-4b50-99b7-bcdc230fc387-0000 3: I1122 16:48:16.517104 17559 slave.cpp:5879] Shutting down executor 'default' of framework 83d06bbb-5575-4b50-99b7-bcdc230fc387-0000 at executor(6)@172.17.0.2:56733 3: I1122 16:48:16.517220 17567 exec.cpp:435] Executor asked to shutdown 3: I1122 16:48:16.517258 17567 exec.cpp:450] Executor::shutdown took 23011ns 3: I1122 16:48:16.517457 17567 slave.cpp:5135] Got exited event for executor(6)@172.17.0.2:56733 3: I1122 16:48:16.517690 17562 master.cpp:9617] Removing task 1 with resources [{"allocation_info":{"role":"*"},"name":"cpus","scalar":{"value":2.0},"type":"SCALAR"},{"allocation_info":{"role":"*"},"name":"mem","scalar":{"value":1024.0},"type":"SCALAR"},{"allocation_info":{"role":"*"},"name":"disk","scalar":{"value":1024.0},"type":"SCALAR"},{"allocation_info":{"role":"*"},"name":"ports","ranges":{"range":[{"begin":31000,"end":32000}]},"type":"RANGES"}] of framework 83d06bbb-5575-4b50-99b7-bcdc230fc387-0000 on agent 83d06bbb-5575-4b50-99b7-bcdc230fc387-S0 at slave(50)@172.17.0.2:56733 (a79443d495cc) 3: I1122 16:48:16.518296 17556 hierarchical.cpp:1132] Recovered cpus(allocated: *):2; mem(allocated: *):1024; disk(allocated: *):1024; ports(allocated: *):[31000-32000] (total: cpus:2; mem:1024; disk:1024; ports:[31000-32000], allocated: {}) on agent 83d06bbb-5575-4b50-99b7-bcdc230fc387-S0 from framework 83d06bbb-5575-4b50-99b7-bcdc230fc387-0000 3: I1122 16:48:16.518330 17562 master.cpp:9646] Removing executor 'default' with resources [] of framework 83d06bbb-5575-4b50-99b7-bcdc230fc387-0000 on agent 83d06bbb-5575-4b50-99b7-bcdc230fc387-S0 at slave(50)@172.17.0.2:56733 (a79443d495cc) 3: I1122 16:48:16.518743 17567 slave.cpp:5544] Executor 'default' of framework 83d06bbb-5575-4b50-99b7-bcdc230fc387-0000 exited with status 0 3: I1122 16:48:16.518817 17567 slave.cpp:5648] Cleaning up executor 'default' of framework 83d06bbb-5575-4b50-99b7-bcdc230fc387-0000 at executor(6)@172.17.0.2:56733 3: W1122 16:48:16.518965 17562 master.cpp:7474] Ignoring unknown exited executor 'default' of framework 83d06bbb-5575-4b50-99b7-bcdc230fc387-0000 on agent 83d06bbb-5575-4b50-99b7-bcdc230fc387-S0 at slave(50)@172.17.0.2:56733 (a79443d495cc) 3: I1122 16:48:16.519119 17559 hierarchical.cpp:358] Removed framework 83d06bbb-5575-4b50-99b7-bcdc230fc387-0000 3: I1122 16:48:16.519142 17554 gc.cpp:90] Scheduling '/tmp/FaultToleranceTest_SchedulerFailoverExecutorToFrameworkMessage_qZcazY/slaves/83d06bbb-5575-4b50-99b7-bcdc230fc387-S0/frameworks/83d06bbb-5575-4b50-99b7-bcdc230fc387-0000/executors/default/runs/4cc82b19-82ee-4cb9-9e4e-b3b00958d947' for gc 6.99999399340148days in the future 3: I1122 16:48:16.519323 17557 gc.cpp:90] Scheduling '/tmp/FaultToleranceTest_SchedulerFailoverExecutorToFrameworkMessage_qZcazY/slaves/83d06bbb-5575-4b50-99b7-bcdc230fc387-S0/frameworks/83d06bbb-5575-4b50-99b7-bcdc230fc387-0000/executors/default' for gc 6.99999399019556days in the future 3: I1122 16:48:16.519330 17567 slave.cpp:5755] Cleaning up framework 83d06bbb-5575-4b50-99b7-bcdc230fc387-0000 3: I1122 16:48:16.519435 17563 task_status_update_manager.cpp:289] Closing task status update streams for framework 83d06bbb-5575-4b50-99b7-bcdc230fc387-0000 3: I1122 16:48:16.519471 17563 task_status_update_manager.cpp:538] Cleaning up status update stream for task 1 of framework 83d06bbb-5575-4b50-99b7-bcdc230fc387-0000 3: I1122 16:48:16.519534 17556 gc.cpp:90] Scheduling '/tmp/FaultToleranceTest_SchedulerFailoverExecutorToFrameworkMessage_qZcazY/slaves/83d06bbb-5575-4b50-99b7-bcdc230fc387-S0/frameworks/83d06bbb-5575-4b50-99b7-bcdc230fc387-0000' for gc 6.99999398775407days in the future 3: I1122 16:48:16.519678 17551 slave.cpp:883] Agent terminating 3: I1122 16:48:16.519820 17554 master.cpp:1311] Agent 83d06bbb-5575-4b50-99b7-bcdc230fc387-S0 at slave(50)@172.17.0.2:56733 (a79443d495cc) disconnected 3: I1122 16:48:16.519841 17554 master.cpp:3370] Disconnecting agent 83d06bbb-5575-4b50-99b7-bcdc230fc387-S0 at slave(50)@172.17.0.2:56733 (a79443d495cc) 3: I1122 16:48:16.519913 17554 master.cpp:3389] Deactivating agent 83d06bbb-5575-4b50-99b7-bcdc230fc387-S0 at slave(50)@172.17.0.2:56733 (a79443d495cc) 3: I1122 16:48:16.520004 17557 hierarchical.cpp:697] Agent 83d06bbb-5575-4b50-99b7-bcdc230fc387-S0 deactivated 3: I1122 16:48:16.523160 17551 master.cpp:1153] Master terminating 3: I1122 16:48:16.523674 17557 hierarchical.cpp:633] Removed agent 83d06bbb-5575-4b50-99b7-bcdc230fc387-S0 3: [ OK ] FaultToleranceTest.SchedulerFailoverExecutorToFrameworkMessage (72 ms) 3: [ RUN ] FaultToleranceTest.SchedulerFailoverFrameworkToExecutorMessage 3: I1122 16:48:16.529587 17551 cluster.cpp:162] Creating default 'local' authorizer 3: I1122 16:48:16.532505 17552 master.cpp:448] Master 91174f41-74b4-4798-9ac0-3abdab06c1c2 (a79443d495cc) started on 172.17.0.2:56733 3: I1122 16:48:16.532529 17552 master.cpp:450] Flags at startup: --acls="" --agent_ping_timeout="15secs" --agent_reregister_timeout="10mins" --allocation_interval="1secs" --allocator="HierarchicalDRF" --authenticate_agents="true" --authenticate_frameworks="true" --authenticate_http_frameworks="true" --authenticate_http_readonly="true" --authenticate_http_readwrite="true" --authenticators="crammd5" --authorizers="local" --credentials="/tmp/u39l0M/credentials" --filter_gpu_resources="true" --framework_sorter="drf" --help="false" --hostname_lookup="true" --http_authenticators="basic" --http_framework_authenticators="basic" --initialize_driver_logging="true" --log_auto_initialize="true" --logbufsecs="0" --logging_level="INFO" --max_agent_ping_timeouts="5" --max_completed_frameworks="50" --max_completed_tasks_per_framework="1000" --max_unreachable_tasks_per_framework="1000" --port="5050" --quiet="false" --recovery_agent_removal_limit="100%" --registry="in_memory" --registry_fetch_timeout="1mins" --registry_gc_interval="15mins" --registry_max_agent_age="2weeks" --registry_max_agent_count="102400" --registry_store_timeout="100secs" --registry_strict="false" --root_submissions="true" --user_sorter="drf" --version="false" --webui_dir="/usr/local/share/mesos/webui" --work_dir="/tmp/u39l0M/master" --zk_session_timeout="10secs" 3: I1122 16:48:16.532842 17552 master.cpp:499] Master only allowing authenticated frameworks to register 3: I1122 16:48:16.532850 17552 master.cpp:505] Master only allowing authenticated agents to register 3: I1122 16:48:16.532853 17552 master.cpp:511] Master only allowing authenticated HTTP frameworks to register 3: I1122 16:48:16.532857 17552 credentials.hpp:37] Loading credentials for authentication from '/tmp/u39l0M/credentials' 3: I1122 16:48:16.533097 17552 master.cpp:555] Using default 'crammd5' authenticator 3: I1122 16:48:16.533216 17552 http.cpp:1045] Creating default 'basic' HTTP authenticator for realm 'mesos-master-readonly' 3: I1122 16:48:16.533361 17552 http.cpp:1045] Creating default 'basic' HTTP authenticator for realm 'mesos-master-readwrite' 3: I1122 16:48:16.533483 17552 http.cpp:1045] Creating default 'basic' HTTP authenticator for realm 'mesos-master-scheduler' 3: I1122 16:48:16.533601 17552 master.cpp:634] Authorization enabled 3: I1122 16:48:16.533746 17555 whitelist_watcher.cpp:77] No whitelist given 3: I1122 16:48:16.533831 17560 hierarchical.cpp:173] Initialized hierarchical allocator process 3: I1122 16:48:16.536335 17554 master.cpp:2215] Elected as the leading master! 3: I1122 16:48:16.536370 17554 master.cpp:1695] Recovering from registrar 3: I1122 16:48:16.536522 17557 registrar.cpp:347] Recovering registrar 3: I1122 16:48:16.537070 17557 registrar.cpp:391] Successfully fetched the registry (0B) in 509952ns 3: I1122 16:48:16.537158 17557 registrar.cpp:495] Applied 1 operations in 24516ns; attempting to update the registry 3: I1122 16:48:16.537644 17557 registrar.cpp:552] Successfully updated the registry in 440832ns 3: I1122 16:48:16.537741 17557 registrar.cpp:424] Successfully recovered registrar 3: I1122 16:48:16.538086 17557 master.cpp:1808] Recovered 0 agents from the registry (129B); allowing 10mins for agents to re-register 3: I1122 16:48:16.538138 17559 hierarchical.cpp:211] Skipping recovery of hierarchical allocator: nothing to recover 3: W1122 16:48:16.543503 17551 process.cpp:2756] Attempted to spawn already running process [email protected]:56733 3: I1122 16:48:16.543987 17551 cluster.cpp:448] Creating default 'local' authorizer 3: I1122 16:48:16.546305 17560 slave.cpp:262] Mesos agent started on (51)@172.17.0.2:56733 3: I1122 16:48:16.546337 17560 slave.cpp:263] Flags at startup: --acls="" --appc_simple_discovery_uri_prefix="http://"; --appc_store_dir="/tmp/FaultToleranceTest_SchedulerFailoverFrameworkToExecutorMessage_Nn22tH/store/appc" --authenticate_http_executors="true" --authenticate_http_readonly="true" --authenticate_http_readwrite="true" --authenticatee="crammd5" --authentication_backoff_factor="1secs" --authorizer="local" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --credential="/tmp/FaultToleranceTest_SchedulerFailoverFrameworkToExecutorMessage_Nn22tH/credential" --default_role="*" --disallow_sharing_agent_pid_namespace="false" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io"; --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/FaultToleranceTest_SchedulerFailoverFrameworkToExecutorMessage_Nn22tH/store/docker" --docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_reregistration_timeout="2secs" --executor_secret_key="/tmp/FaultToleranceTest_SchedulerFailoverFrameworkToExecutorMessage_Nn22tH/executor_secret_key" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/FaultToleranceTest_SchedulerFailoverFrameworkToExecutorMessage_Nn22tH/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname_lookup="true" --http_command_executor="false" --http_credentials="/tmp/FaultToleranceTest_SchedulerFailoverFrameworkToExecutorMessage_Nn22tH/http_credentials" --http_heartbeat_interval="30secs" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher="posix" --launcher_dir="/mesos/build/src" --logbufsecs="0" --logging_level="INFO" --max_completed_executors_per_framework="150" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --port="5051" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="10ms" --resources="cpus:2;gpus:0;mem:1024;disk:1024;ports:[31000-32000]" --revocable_cpu_low_priority="true" --runtime_dir="/tmp/FaultToleranceTest_SchedulerFailoverFrameworkToExecutorMessage_Nn22tH" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_enable_support="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/tmp/FaultToleranceTest_SchedulerFailoverFrameworkToExecutorMessage_053bff" --zk_session_timeout="10secs" 3: I1122 16:48:16.546844 17560 credentials.hpp:86] Loading credential for authentication from '/tmp/FaultToleranceTest_SchedulerFailoverFrameworkToExecutorMessage_Nn22tH/credential' 3: W1122 16:48:16.546887 17551 process.cpp:2756] Attempted to spawn already running process [email protected]:56733 3: I1122 16:48:16.547070 17560 slave.cpp:295] Agent using credential for: test-principal 3: I1122 16:48:16.547087 17560 credentials.hpp:37] Loading credentials for authentication from '/tmp/FaultToleranceTest_SchedulerFailoverFrameworkToExecutorMessage_Nn22tH/http_credentials' 3: I1122 16:48:16.547382 17560 http.cpp:1045] Creating default 'basic' HTTP authenticator for realm 'mesos-agent-executor' 3: I1122 16:48:16.547595 17560 http.cpp:1066] Creating default 'jwt' HTTP authenticator for realm 'mesos-agent-executor' 3: I1122 16:48:16.547828 17560 http.cpp:1045] Creating default 'basic' HTTP authenticator for realm 'mesos-agent-readonly' 3: I1122 16:48:16.547952 17560 http.cpp:1066] Creating default 'jwt' HTTP authenticator for realm 'mesos-agent-readonly' 3: I1122 16:48:16.547951 17551 sched.cpp:232] Version: 1.5.0 3: I1122 16:48:16.548147 17560 http.cpp:1045] Creating default 'basic' HTTP authenticator for realm 'mesos-agent-readwrite' 3: I1122 16:48:16.548254 17560 http.cpp:1066] Creating default 'jwt' HTTP authenticator for realm 'mesos-agent-readwrite' 3: I1122 16:48:16.548594 17558 sched.cpp:336] New master detected at [email protected]:56733 3: I1122 16:48:16.548784 17558 sched.cpp:396] Authenticating with master [email protected]:56733 3: I1122 16:48:16.548801 17558 sched.cpp:403] Using default CRAM-MD5 authenticatee 3: I1122 16:48:16.549146 17567 authenticatee.cpp:121] Creating new client SASL connection 3: I1122 16:48:16.549484 17565 master.cpp:8306] Authenticating [email protected]:56733 3: I1122 16:48:16.549630 17564 authenticator.cpp:414] Starting authentication session for crammd5-authenticatee(113)@172.17.0.2:56733 3: I1122 16:48:16.549911 17552 authenticator.cpp:98] Creating new server SASL connection 3: I1122 16:48:16.549931 17560 slave.cpp:593] Agent resources: [{"name":"cpus","scalar":{"value":2.0},"type":"SCALAR"},{"name":"mem","scalar":{"value":1024.0},"type":"SCALAR"},{"name":"disk","scalar":{"value":1024.0},"type":"SCALAR"},{"name":"ports","ranges":{"range":[{"begin":31000,"end":32000}]},"type":"RANGES"}] 3: I1122 16:48:16.550169 17560 slave.cpp:601] Agent attributes: [ ] 3: I1122 16:48:16.550184 17560 slave.cpp:610] Agent hostname: a79443d495cc 3: I1122 16:48:16.550196 17553 authenticatee.cpp:213] Received SASL authentication mechanisms: CRAM-MD5 3: I1122 16:48:16.550217 17553 authenticatee.cpp:239] Attempting to authenticate with mechanism 'CRAM-MD5' 3: I1122 16:48:16.550328 17553 authenticator.cpp:204] Received SASL authentication start 3: I1122 16:48:16.550384 17553 authenticator.cpp:326] Authentication requires more steps 3: I1122 16:48:16.550391 17557 task_status_update_manager.cpp:181] Pausing sending task status updates 3: I1122 16:48:16.550501 17554 authenticatee.cpp:259] Received SASL authentication step 3: I1122 16:48:16.550618 17559 authenticator.cpp:232] Received SASL authentication step 3: I1122 16:48:16.550645 17559 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'a79443d495cc' server FQDN: 'a79443d495cc' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false 3: I1122 16:48:16.550657 17559 auxprop.cpp:181] Looking up auxiliary property '*userPassword' 3: I1122 16:48:16.550704 17559 auxprop.cpp:181] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' 3: I1122 16:48:16.550719 17559 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'a79443d495cc' server FQDN: 'a79443d495cc' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true 3: I1122 16:48:16.550725 17559 auxprop.cpp:131] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true 3: I1122 16:48:16.550730 17559 auxprop.cpp:131] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true 3: I1122 16:48:16.550745 17559 authenticator.cpp:318] Authentication success 3: I1122 16:48:16.550869 17562 authenticatee.cpp:299] Authentication success 3: I1122 16:48:16.550961 17555 master.cpp:8336] Successfully authenticated principal 'test-principal' at [email protected]:56733 3: I1122 16:48:16.551035 17563 authenticator.cpp:432] Authentication session cleanup for crammd5-authenticatee(113)@172.17.0.2:56733 3: I1122 16:48:16.551208 17561 sched.cpp:502] Successfully authenticated with master [email protected]:56733 3: I1122 16:48:16.551236 17561 sched.cpp:824] Sending SUBSCRIBE call to [email protected]:56733 3: I1122 16:48:16.551378 17561 sched.cpp:857] Will retry registration in 1.646693394secs if necessary 3: I1122 16:48:16.551656 17565 master.cpp:2963] Received SUBSCRIBE call for framework 'default' at [email protected]:56733 3: I1122 16:48:16.551743 17565 master.cpp:2280] Authorizing framework principal 'test-principal' to receive offers for roles '{ * }' 3: I1122 16:48:16.552314 17566 master.cpp:3043] Subscribing framework default with checkpointing disabled and capabilities [ RESERVATION_REFINEMENT ] 3: I1122 16:48:16.552356 17557 state.cpp:64] Recovering state from '/tmp/FaultToleranceTest_SchedulerFailoverFrameworkToExecutorMessage_053bff/meta' 3: I1122 16:48:16.552708 17559 task_status_update_manager.cpp:207] Recovering task status update manager 3: I1122 16:48:16.552958 17559 sched.cpp:751] Framework registered with 91174f41-74b4-4798-9ac0-3abdab06c1c2-0000 3: I1122 16:48:16.553037 17559 sched.cpp:765] Scheduler::registered took 54169ns 3: I1122 16:48:16.553068 17565 slave.cpp:6449] Finished recovery 3: I1122 16:48:16.553092 17562 hierarchical.cpp:306] Added framework 91174f41-74b4-4798-9ac0-3abdab06c1c2-0000 3: I1122 16:48:16.553264 17562 hierarchical.cpp:1457] Performed allocation for 0 agents in 64872ns 3: I1122 16:48:16.553665 17554 task_status_update_manager.cpp:181] Pausing sending task status updates 3: I1122 16:48:16.553665 17560 slave.cpp:1007] New master detected at [email protected]:56733 3: I1122 16:48:16.553761 17560 slave.cpp:1042] Detecting new master 3: I1122 16:48:16.566058 17553 slave.cpp:1069] Authenticating with master [email protected]:56733 3: I1122 16:48:16.566207 17553 slave.cpp:1078] Using default CRAM-MD5 authenticatee 3: I1122 16:48:16.566495 17566 authenticatee.cpp:121] Creating new client SASL connection 3: I1122 16:48:16.566758 17561 master.cpp:8306] Authenticating slave(51)@172.17.0.2:56733 3: I1122 16:48:16.566889 17555 authenticator.cpp:414] Starting authentication session for crammd5-authenticatee(114)@172.17.0.2:56733 3: I1122 16:48:16.567169 17556 authenticator.cpp:98] Creating new server SASL connection 3: I1122 16:48:16.567466 17563 authenticatee.cpp:213] Received SASL authentication mechanisms: CRAM-MD5 3: I1122 16:48:16.567490 17563 authenticatee.cpp:239] Attempting to authenticate with mechanism 'CRAM-MD5' 3: I1122 16:48:16.567577 17563 authenticator.cpp:204] Received SASL authentication start 3: I1122 16:48:16.567621 17563 authenticator.cpp:326] Authentication requires more steps 3: I1122 16:48:16.567713 17552 authenticatee.cpp:259] Received SASL authentication step 3: I1122 16:48:16.567849 17562 authenticator.cpp:232] Received SASL authentication step 3: I1122 16:48:16.567881 17562 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'a79443d495cc' server FQDN: 'a79443d495cc' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false 3: I1122 16:48:16.567930 17562 auxprop.cpp:181] Looking up auxiliary property '*userPassword' 3: I1122 16:48:16.567983 17562 auxprop.cpp:181] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' 3: I1122 16:48:16.568027 17562 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'a79443d495cc' server FQDN: 'a79443d495cc' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true 3: I1122 16:48:16.568049 17562 auxprop.cpp:131] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true 3: I1122 16:48:16.568061 17562 auxprop.cpp:131] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true 3: I1122 16:48:16.568085 17562 authenticator.cpp:318] Authentication success 3: I1122 16:48:16.568200 17565 authenticatee.cpp:299] Authentication success 3: I1122 16:48:16.568275 17564 master.cpp:8336] Successfully authenticated principal 'test-principal' at slave(51)@172.17.0.2:56733 3: I1122 16:48:16.568337 17554 authenticator.cpp:432] Authentication session cleanup for crammd5-authenticatee(114)@172.17.0.2:56733 3: I1122 16:48:16.568493 17567 slave.cpp:1161] Successfully authenticated with master [email protected]:56733 3: I1122 16:48:16.568778 17567 slave.cpp:1685] Will retry registration in 516171ns if necessary 3: I1122 16:48:16.569031 17566 master.cpp:6036] Received register agent message from slave(51)@172.17.0.2:56733 (a79443d495cc) 3: I1122 16:48:16.569170 17566 master.cpp:3872] Authorizing agent with principal 'test-principal' 3: I1122 16:48:16.569591 17555 master.cpp:6098] Authorized registration of agent at slave(51)@172.17.0.2:56733 (a79443d495cc) 3: I1122 16:48:16.569707 17555 master.cpp:6191] Registering agent at slave(51)@172.17.0.2:56733 (a79443d495cc) with id 91174f41-74b4-4798-9ac0-3abdab06c1c2-S0 3: I1122 16:48:16.570274 17556 registrar.cpp:495] Applied 1 operations in 104315ns; attempting to update the registry 3: I1122 16:48:16.570950 17556 registrar.cpp:552] Successfully updated the registry in 606976ns 3: I1122 16:48:16.571192 17562 master.cpp:6240] Admitted agent 91174f41-74b4-4798-9ac0-3abdab06c1c2-S0 at slave(51)@172.17.0.2:56733 (a79443d495cc) 3: I1122 16:48:16.572129 17564 slave.cpp:1207] Registered with master [email protected]:56733; given agent ID 91174f41-74b4-4798-9ac0-3abdab06c1c2-S0 3: I1122 16:48:16.571869 17562 master.cpp:6273] Registered agent 91174f41-74b4-4798-9ac0-3abdab06c1c2-S0 at slave(51)@172.17.0.2:56733 (a79443d495cc) with [{"name":"cpus","scalar":{"value":2.0},"type":"SCALAR"},{"name":"mem","scalar":{"value":1024.0},"type":"SCALAR"},{"name":"disk","scalar":{"value":1024.0},"type":"SCALAR"},{"name":"ports","ranges":{"range":[{"begin":31000,"end":32000}]},"type":"RANGES"}] 3: I1122 16:48:16.572278 17562 task_status_update_manager.cpp:188] Resuming sending task status updates 3: I1122 16:48:16.572371 17560 hierarchical.cpp:600] Added agent 91174f41-74b4-4798-9ac0-3abdab06c1c2-S0 (a79443d495cc) with cpus:2; mem:1024; disk:1024; ports:[31000-32000] (allocated: {}) 3: I1122 16:48:16.572563 17564 slave.cpp:1227] Checkpointing SlaveInfo to '/tmp/FaultToleranceTest_SchedulerFailoverFrameworkToExecutorMessage_053bff/meta/slaves/91174f41-74b4-4798-9ac0-3abdab06c1c2-S0/slave.info' 3: I1122 16:48:16.573056 17564 slave.cpp:1298] Forwarding total oversubscribed resources {} 3: I1122 16:48:16.573294 17567 master.cpp:7085] Received update of agent 91174f41-74b4-4798-9ac0-3abdab06c1c2-S0 at slave(51)@172.17.0.2:56733 (a79443d495cc) with total oversubscribed resources {} 3: I1122 16:48:16.573483 17567 master.cpp:7103] Ignoring update on agent 91174f41-74b4-4798-9ac0-3abdab06c1c2-S0 at slave(51)@172.17.0.2:56733 (a79443d495cc) as it reports no changes 3: I1122 16:48:16.573921 17560 hierarchical.cpp:1457] Performed allocation for 1 agents in 1.339203ms 3: I1122 16:48:16.574389 17566 master.cpp:8136] Sending 1 offers to framework 91174f41-74b4-4798-9ac0-3abdab06c1c2-0000 (default) at [email protected]:56733 3: I1122 16:48:16.574980 17561 sched.cpp:921] Scheduler::resourceOffers took 122887ns 3: I1122 16:48:16.577209 17557 master.cpp:10045] Removing offer 91174f41-74b4-4798-9ac0-3abdab06c1c2-O0 3: I1122 16:48:16.577381 17557 master.cpp:4230] Processing ACCEPT call for offers: [ 91174f41-74b4-4798-9ac0-3abdab06c1c2-O0 ] on agent 91174f41-74b4-4798-9ac0-3abdab06c1c2-S0 at slave(51)@172.17.0.2:56733 (a79443d495cc) for framework 91174f41-74b4-4798-9ac0-3abdab06c1c2-0000 (default) at [email protected]:56733 3: I1122 16:48:16.577499 17557 master.cpp:3599] Authorizing framework principal 'test-principal' to launch task a91259b5-31be-4187-ad50-5a87f7d014d6 3: W1122 16:48:16.579553 17554 validation.cpp:1391] Executor 'default' for task 'a91259b5-31be-4187-ad50-5a87f7d014d6' uses less CPUs (None) than the minimum required (0.01). Please update your executor, as this will be mandatory in future releases. 3: W1122 16:48:16.579587 17554 validation.cpp:1403] Executor 'default' for task 'a91259b5-31be-4187-ad50-5a87f7d014d6' uses less memory (None) than the minimum required (32MB). Please update your executor, as this will be mandatory in future releases. write /dev/stdout: resource temporarily unavailable + docker rmi mesos-1511367263-27105 Build step 'Execute shell' marked build as failure
