[ https://issues.apache.org/jira/browse/MESOS-4044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bernd Mathiske updated MESOS-4044: ---------------------------------- Labels: cgroups flaky-test mesosphere (was: mesosphere) > SlaveRecoveryTest/0.Reboot is flaky > ----------------------------------- > > Key: MESOS-4044 > URL: https://issues.apache.org/jira/browse/MESOS-4044 > Project: Mesos > Issue Type: Bug > Components: slave > Environment: Debian 8 on VirtualBox > {{configure --enable-debug --enable-ssl --enable-libevent}} > Reporter: Alexander Rojas > Labels: cgroups, flaky-test, mesosphere > > Running the test program as: > {code} > sudo src/mesos-tests --gtest_filter="SlaveRecoveryTest/0.Reboot" > --gtest_repeat=100 --verbose --gtest_break_on_failure > {code} > ends up every time at some point with the failure: > {noformat} > [ RUN ] SlaveRecoveryTest/0.Reboot > I1202 15:18:00.036594 26328 leveldb.cpp:176] Opened db in 12.924775ms > I1202 15:18:00.037643 26328 leveldb.cpp:183] Compacted db in 980477ns > I1202 15:18:00.037693 26328 leveldb.cpp:198] Created db iterator in 15079ns > I1202 15:18:00.037706 26328 leveldb.cpp:204] Seeked to beginning of db in > 1356ns > I1202 15:18:00.037716 26328 leveldb.cpp:273] Iterated through 0 keys in the > db in 313ns > I1202 15:18:00.037753 26328 replica.cpp:780] Replica recovered with log > positions 0 -> 0 with 1 holes and 0 unlearned > I1202 15:18:00.038360 26346 recover.cpp:449] Starting replica recovery > I1202 15:18:00.040987 26346 master.cpp:367] Master > baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8 (debian-vm.localdomain) started on > 127.0.1.1:33625 > I1202 15:18:00.040998 26346 master.cpp:369] Flags at startup: --acls="" > --allocation_interval="1secs" --allocator="HierarchicalDRF" > --authenticate="true" --authenticate_slaves="true" --authenticators="crammd5" > --authorizers="local" --credentials="/tmp/xt1N2F/credentials" > --framework_sorter="drf" --help="false" --hostname_lookup="true" > --initialize_driver_logging="true" --log_auto_initialize="true" > --logbufsecs="0" --logging_level="INFO" --max_slave_ping_timeouts="5" > --quiet="false" --recovery_slave_removal_limit="100%" > --registry="replicated_log" --registry_fetch_timeout="1mins" > --registry_store_timeout="25secs" --registry_strict="true" > --root_submissions="true" --slave_ping_timeout="15secs" > --slave_reregister_timeout="10mins" --user_sorter="drf" --version="false" > --webui_dir="/usr/local/share/mesos/webui" --work_dir="/tmp/xt1N2F/master" > --zk_session_timeout="10secs" > I1202 15:18:00.041157 26346 master.cpp:414] Master only allowing > authenticated frameworks to register > I1202 15:18:00.041163 26346 master.cpp:419] Master only allowing > authenticated slaves to register > I1202 15:18:00.041168 26346 credentials.hpp:37] Loading credentials for > authentication from '/tmp/xt1N2F/credentials' > I1202 15:18:00.041410 26346 master.cpp:458] Using default 'crammd5' > authenticator > I1202 15:18:00.041524 26346 master.cpp:495] Authorization enabled > I1202 15:18:00.042917 26343 recover.cpp:475] Replica is in EMPTY status > I1202 15:18:00.043557 26343 master.cpp:1606] The newly elected leader is > master@127.0.1.1:33625 with id baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8 > I1202 15:18:00.043577 26343 master.cpp:1619] Elected as the leading master! > I1202 15:18:00.043589 26343 master.cpp:1379] Recovering from registrar > I1202 15:18:00.043766 26343 registrar.cpp:309] Recovering registrar > I1202 15:18:00.044668 26344 replica.cpp:676] Replica in EMPTY status received > a broadcasted recover request from (21064)@127.0.1.1:33625 > I1202 15:18:00.045027 26349 recover.cpp:195] Received a recover response from > a replica in EMPTY status > I1202 15:18:00.045497 26349 recover.cpp:566] Updating replica status to > STARTING > I1202 15:18:00.055539 26349 leveldb.cpp:306] Persisting metadata (8 bytes) to > leveldb took 9.859161ms > I1202 15:18:00.055599 26349 replica.cpp:323] Persisted replica status to > STARTING > I1202 15:18:00.055958 26346 recover.cpp:475] Replica is in STARTING status > I1202 15:18:00.057106 26342 replica.cpp:676] Replica in STARTING status > received a broadcasted recover request from (21065)@127.0.1.1:33625 > I1202 15:18:00.057462 26343 recover.cpp:195] Received a recover response from > a replica in STARTING status > I1202 15:18:00.057886 26347 recover.cpp:566] Updating replica status to VOTING > I1202 15:18:00.058706 26345 leveldb.cpp:306] Persisting metadata (8 bytes) to > leveldb took 634303ns > I1202 15:18:00.058724 26345 replica.cpp:323] Persisted replica status to > VOTING > I1202 15:18:00.058821 26345 recover.cpp:580] Successfully joined the Paxos > group > I1202 15:18:00.058980 26345 recover.cpp:464] Recover process terminated > I1202 15:18:00.059288 26348 log.cpp:661] Attempting to start the writer > I1202 15:18:00.060330 26342 replica.cpp:496] Replica received implicit > promise request from (21066)@127.0.1.1:33625 with proposal 1 > I1202 15:18:00.061751 26342 leveldb.cpp:306] Persisting metadata (8 bytes) to > leveldb took 1.395961ms > I1202 15:18:00.061774 26342 replica.cpp:345] Persisted promised to 1 > I1202 15:18:00.062237 26342 coordinator.cpp:240] Coordinator attempting to > fill missing positions > I1202 15:18:00.063148 26342 replica.cpp:391] Replica received explicit > promise request from (21067)@127.0.1.1:33625 for position 0 with proposal 2 > I1202 15:18:00.064757 26342 leveldb.cpp:343] Persisting action (8 bytes) to > leveldb took 1.581382ms > I1202 15:18:00.064785 26342 replica.cpp:715] Persisted action at 0 > I1202 15:18:00.065717 26342 replica.cpp:540] Replica received write request > for position 0 from (21068)@127.0.1.1:33625 > I1202 15:18:00.065758 26342 leveldb.cpp:438] Reading position from leveldb > took 21294ns > I1202 15:18:00.066664 26342 leveldb.cpp:343] Persisting action (14 bytes) to > leveldb took 875354ns > I1202 15:18:00.066699 26342 replica.cpp:715] Persisted action at 0 > I1202 15:18:00.067416 26349 replica.cpp:694] Replica received learned notice > for position 0 from @0.0.0.0:0 > I1202 15:18:00.068152 26349 leveldb.cpp:343] Persisting action (16 bytes) to > leveldb took 682342ns > I1202 15:18:00.068188 26349 replica.cpp:715] Persisted action at 0 > I1202 15:18:00.068208 26349 replica.cpp:700] Replica learned NOP action at > position 0 > I1202 15:18:00.068622 26345 log.cpp:677] Writer started with ending position 0 > I1202 15:18:00.069576 26345 leveldb.cpp:438] Reading position from leveldb > took 79910ns > I1202 15:18:00.070322 26349 registrar.cpp:342] Successfully fetched the > registry (0B) in 26528us > I1202 15:18:00.070417 26349 registrar.cpp:441] Applied 1 operations in > 27033ns; attempting to update the 'registry' > I1202 15:18:00.071035 26349 log.cpp:685] Attempting to append 187 bytes to > the log > I1202 15:18:00.071144 26347 coordinator.cpp:350] Coordinator attempting to > write APPEND action at position 1 > I1202 15:18:00.071885 26347 replica.cpp:540] Replica received write request > for position 1 from (21069)@127.0.1.1:33625 > I1202 15:18:00.072844 26347 leveldb.cpp:343] Persisting action (206 bytes) to > leveldb took 929311ns > I1202 15:18:00.072862 26347 replica.cpp:715] Persisted action at 1 > I1202 15:18:00.073323 26344 replica.cpp:694] Replica received learned notice > for position 1 from @0.0.0.0:0 > I1202 15:18:00.073979 26344 leveldb.cpp:343] Persisting action (208 bytes) to > leveldb took 637468ns > I1202 15:18:00.073995 26344 replica.cpp:715] Persisted action at 1 > I1202 15:18:00.074007 26344 replica.cpp:700] Replica learned APPEND action at > position 1 > I1202 15:18:00.075078 26344 registrar.cpp:486] Successfully updated the > 'registry' in 4.587008ms > I1202 15:18:00.075166 26344 registrar.cpp:372] Successfully recovered > registrar > I1202 15:18:00.075309 26344 log.cpp:704] Attempting to truncate the log to 1 > I1202 15:18:00.075595 26344 master.cpp:1416] Recovered 0 slaves from the > Registry (148B) ; allowing 10mins for slaves to re-register > I1202 15:18:00.075649 26344 coordinator.cpp:350] Coordinator attempting to > write TRUNCATE action at position 2 > I1202 15:18:00.076445 26344 replica.cpp:540] Replica received write request > for position 2 from (21070)@127.0.1.1:33625 > I1202 15:18:00.077129 26344 leveldb.cpp:343] Persisting action (16 bytes) to > leveldb took 660682ns > I1202 15:18:00.077177 26344 replica.cpp:715] Persisted action at 2 > I1202 15:18:00.077822 26344 replica.cpp:694] Replica received learned notice > for position 2 from @0.0.0.0:0 > I1202 15:18:00.078547 26344 leveldb.cpp:343] Persisting action (18 bytes) to > leveldb took 527711ns > I1202 15:18:00.078614 26344 leveldb.cpp:401] Deleting ~1 keys from leveldb > took 21673ns > I1202 15:18:00.078631 26344 replica.cpp:715] Persisted action at 2 > I1202 15:18:00.078650 26344 replica.cpp:700] Replica learned TRUNCATE action > at position 2 > I1202 15:18:00.087874 26328 containerizer.cpp:142] Using isolation: > cgroups/cpu,cgroups/mem,filesystem/posix > I1202 15:18:00.891749 26328 linux_launcher.cpp:103] Using > /sys/fs/cgroup/freezer as the freezer hierarchy for the Linux launcher > I1202 15:18:00.897735 26328 systemd.cpp:210] Started systemd slice > `mesos_executors.slice` > I1202 15:18:00.917435 26343 slave.cpp:191] Slave started on > 655)@127.0.1.1:33625 > I1202 15:18:00.917466 26343 slave.cpp:192] Flags at startup: > --appc_store_dir="/tmp/mesos/store/appc" --authenticatee="crammd5" > --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" > --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" > --cgroups_root="mesos_test_d5b6bb71-dab3-4457-ae03-84a046ed9c62" > --container_disk_watch_interval="15secs" --containerizers="mesos" > --credential="/tmp/SlaveRecoveryTest_0_Reboot_qT6DBx/credential" > --default_role="*" --disk_watch_interval="1mins" --docker="docker" > --docker_auth_server="auth.docker.io" --docker_auth_server_port="443" > --docker_kill_orphans="true" > --docker_local_archives_dir="/tmp/mesos/images/docker" > --docker_puller="local" --docker_puller_timeout="60" > --docker_registry="registry-1.docker.io" --docker_registry_port="443" > --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" > --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" > --enforce_container_disk_quota="false" > --executor_registration_timeout="1mins" > --executor_shutdown_grace_period="5secs" > --fetcher_cache_dir="/tmp/SlaveRecoveryTest_0_Reboot_qT6DBx/fetch" > --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" > --gc_disk_headroom="0.1" --hadoop_home="" --help="false" > --hostname_lookup="true" --image_provisioner_backend="copy" > --initialize_driver_logging="true" --isolation="cgroups/cpu,cgroups/mem" > --launcher_dir="/home/alexander/Documents/workspace/mesos/build/src" > --logbufsecs="0" --logging_level="INFO" > --oversubscribed_resources_interval="15secs" --perf_duration="10secs" > --perf_interval="1mins" --qos_correction_interval_min="0ns" --quiet="false" > --recover="reconnect" --recovery_timeout="15mins" > --registration_backoff_factor="10ms" > --resources="cpus:2;mem:1024;disk:1024;ports:[31000-32000]" > --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" > --slave_subsystems="memory,cpuacct" --strict="false" --switch_user="true" > --systemd_runtime_directory="/run/systemd/system" --version="false" > --work_dir="/tmp/SlaveRecoveryTest_0_Reboot_qT6DBx" > I1202 15:18:00.917687 26343 slave.cpp:212] Moving slave process into its own > cgroup for subsystem: memory > I1202 15:18:00.919559 26328 sched.cpp:166] Version: 0.26.0 > I1202 15:18:00.921807 26344 sched.cpp:264] New master detected at > master@127.0.1.1:33625 > I1202 15:18:00.921869 26344 sched.cpp:320] Authenticating with master > master@127.0.1.1:33625 > I1202 15:18:00.921880 26344 sched.cpp:327] Using default CRAM-MD5 > authenticatee > I1202 15:18:00.922087 26344 authenticatee.cpp:123] Creating new client SASL > connection > I1202 15:18:00.922412 26348 master.cpp:5150] Authenticating > scheduler-82a736f1-e818-4a76-8126-93ac22dcfde0@127.0.1.1:33625 > I1202 15:18:00.922798 26348 authenticator.cpp:100] Creating new server SASL > connection > I1202 15:18:00.922977 26344 authenticatee.cpp:214] Received SASL > authentication mechanisms: CRAM-MD5 > I1202 15:18:00.922999 26344 authenticatee.cpp:240] Attempting to authenticate > with mechanism 'CRAM-MD5' > I1202 15:18:00.923074 26344 authenticator.cpp:205] Received SASL > authentication start > I1202 15:18:00.923105 26344 authenticator.cpp:327] Authentication requires > more steps > I1202 15:18:00.923151 26344 authenticatee.cpp:260] Received SASL > authentication step > I1202 15:18:00.923216 26344 authenticator.cpp:233] Received SASL > authentication step > I1202 15:18:00.923282 26344 authenticator.cpp:319] Authentication success > I1202 15:18:00.923379 26344 authenticatee.cpp:300] Authentication success > I1202 15:18:00.923432 26344 master.cpp:5180] Successfully authenticated > principal 'test-principal' at > scheduler-82a736f1-e818-4a76-8126-93ac22dcfde0@127.0.1.1:33625 > I1202 15:18:00.923672 26344 sched.cpp:409] Successfully authenticated with > master master@127.0.1.1:33625 > I1202 15:18:00.923964 26349 master.cpp:2176] Received SUBSCRIBE call for > framework 'default' at > scheduler-82a736f1-e818-4a76-8126-93ac22dcfde0@127.0.1.1:33625 > I1202 15:18:00.924010 26349 master.cpp:1645] Authorizing framework principal > 'test-principal' to receive offers for role '*' > I1202 15:18:00.924242 26349 master.cpp:2247] Subscribing framework default > with checkpointing enabled and capabilities [ ] > I1202 15:18:00.924561 26344 hierarchical.cpp:195] Added framework > baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-0000 > I1202 15:18:00.924584 26349 sched.cpp:643] Framework registered with > baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-0000 > I1202 15:18:01.209614 26343 slave.cpp:212] Moving slave process into its own > cgroup for subsystem: cpuacct > I1202 15:18:01.409137 26343 credentials.hpp:85] Loading credential for > authentication from '/tmp/SlaveRecoveryTest_0_Reboot_qT6DBx/credential' > I1202 15:18:01.409307 26343 slave.cpp:322] Slave using credential for: > test-principal > I1202 15:18:01.409860 26343 slave.cpp:392] Slave resources: cpus(*):2; > mem(*):1024; disk(*):1024; ports(*):[31000-32000] > I1202 15:18:01.409906 26343 slave.cpp:400] Slave attributes: [ ] > I1202 15:18:01.409927 26343 slave.cpp:405] Slave hostname: > debian-vm.localdomain > I1202 15:18:01.409932 26343 slave.cpp:410] Slave checkpoint: true > I1202 15:18:01.410773 26346 state.cpp:54] Recovering state from > '/tmp/SlaveRecoveryTest_0_Reboot_qT6DBx/meta' > I1202 15:18:01.411038 26346 status_update_manager.cpp:202] Recovering status > update manager > I1202 15:18:01.411185 26346 containerizer.cpp:384] Recovering containerizer > I1202 15:18:01.473423 26349 slave.cpp:4230] Finished recovery > I1202 15:18:01.474261 26344 slave.cpp:729] New master detected at > master@127.0.1.1:33625 > I1202 15:18:01.474325 26342 status_update_manager.cpp:176] Pausing sending > status updates > I1202 15:18:01.474383 26344 slave.cpp:792] Authenticating with master > master@127.0.1.1:33625 > I1202 15:18:01.474417 26344 slave.cpp:797] Using default CRAM-MD5 > authenticatee > I1202 15:18:01.474566 26344 slave.cpp:765] Detecting new master > I1202 15:18:01.474706 26345 authenticatee.cpp:123] Creating new client SASL > connection > I1202 15:18:01.475159 26345 master.cpp:5150] Authenticating > slave(655)@127.0.1.1:33625 > I1202 15:18:01.475553 26345 authenticator.cpp:100] Creating new server SASL > connection > I1202 15:18:01.475754 26342 authenticatee.cpp:214] Received SASL > authentication mechanisms: CRAM-MD5 > I1202 15:18:01.475793 26342 authenticatee.cpp:240] Attempting to authenticate > with mechanism 'CRAM-MD5' > I1202 15:18:01.475867 26342 authenticator.cpp:205] Received SASL > authentication start > I1202 15:18:01.475903 26342 authenticator.cpp:327] Authentication requires > more steps > I1202 15:18:01.475989 26342 authenticatee.cpp:260] Received SASL > authentication step > I1202 15:18:01.476095 26342 authenticator.cpp:233] Received SASL > authentication step > I1202 15:18:01.476172 26342 authenticator.cpp:319] Authentication success > I1202 15:18:01.476294 26343 authenticatee.cpp:300] Authentication success > I1202 15:18:01.476307 26349 master.cpp:5180] Successfully authenticated > principal 'test-principal' at slave(655)@127.0.1.1:33625 > I1202 15:18:01.476681 26345 slave.cpp:860] Successfully authenticated with > master master@127.0.1.1:33625 > I1202 15:18:01.476958 26343 master.cpp:3859] Registering slave at > slave(655)@127.0.1.1:33625 (debian-vm.localdomain) with id > baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-S0 > I1202 15:18:01.477411 26345 registrar.cpp:441] Applied 1 operations in > 62621ns; attempting to update the 'registry' > I1202 15:18:01.478070 26345 log.cpp:685] Attempting to append 365 bytes to > the log > I1202 15:18:01.478272 26345 coordinator.cpp:350] Coordinator attempting to > write APPEND action at position 3 > I1202 15:18:01.479032 26345 replica.cpp:540] Replica received write request > for position 3 from (21089)@127.0.1.1:33625 > I1202 15:18:01.479346 26344 master.cpp:3847] Ignoring register slave message > from slave(655)@127.0.1.1:33625 (debian-vm.localdomain) as admission is > already in progress > I1202 15:18:01.488145 26345 leveldb.cpp:343] Persisting action (384 bytes) to > leveldb took 8.718277ms > I1202 15:18:01.488211 26345 replica.cpp:715] Persisted action at 3 > I1202 15:18:01.489114 26345 replica.cpp:694] Replica received learned notice > for position 3 from @0.0.0.0:0 > I1202 15:18:01.489850 26345 leveldb.cpp:343] Persisting action (386 bytes) to > leveldb took 620665ns > I1202 15:18:01.489914 26345 replica.cpp:715] Persisted action at 3 > I1202 15:18:01.489971 26345 replica.cpp:700] Replica learned APPEND action at > position 3 > I1202 15:18:01.491174 26347 registrar.cpp:486] Successfully updated the > 'registry' in 13.647104ms > I1202 15:18:01.491349 26345 log.cpp:704] Attempting to truncate the log to 3 > I1202 15:18:01.491489 26345 coordinator.cpp:350] Coordinator attempting to > write TRUNCATE action at position 4 > I1202 15:18:01.491860 26347 master.cpp:3927] Registered slave > baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-S0 at slave(655)@127.0.1.1:33625 > (debian-vm.localdomain) with cpus(*):2; mem(*):1024; disk(*):1024; > ports(*):[31000-32000] > I1202 15:18:01.492015 26345 hierarchical.cpp:344] Added slave > baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-S0 (debian-vm.localdomain) with > cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] (allocated: ) > I1202 15:18:01.492398 26347 replica.cpp:540] Replica received write request > for position 4 from (21090)@127.0.1.1:33625 > I1202 15:18:01.492027 26348 slave.cpp:904] Registered with master > master@127.0.1.1:33625; given slave ID baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-S0 > I1202 15:18:01.492795 26345 master.cpp:4979] Sending 1 offers to framework > baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-0000 (default) at > scheduler-82a736f1-e818-4a76-8126-93ac22dcfde0@127.0.1.1:33625 > I1202 15:18:01.492897 26345 status_update_manager.cpp:183] Resuming sending > status updates > I1202 15:18:01.493070 26348 slave.cpp:963] Forwarding total oversubscribed > resources > I1202 15:18:01.493188 26345 master.cpp:4269] Received update of slave > baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-S0 at slave(655)@127.0.1.1:33625 > (debian-vm.localdomain) with total oversubscribed resources > I1202 15:18:01.493386 26348 hierarchical.cpp:400] Slave > baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-S0 (debian-vm.localdomain) updated with > oversubscribed resources (total: cpus(*):2; mem(*):1024; disk(*):1024; > ports(*):[31000-32000], allocated: cpus(*):2; mem(*):1024; disk(*):1024; > ports(*):[31000-32000]) > I1202 15:18:01.494815 26344 master.cpp:2915] Processing ACCEPT call for > offers: [ baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-O0 ] on slave > baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-S0 at slave(655)@127.0.1.1:33625 > (debian-vm.localdomain) for framework > baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-0000 (default) at > scheduler-82a736f1-e818-4a76-8126-93ac22dcfde0@127.0.1.1:33625 > I1202 15:18:01.494904 26344 master.cpp:2711] Authorizing framework principal > 'test-principal' to launch task b2102462-a9c1-45bf-94f2-9a59abb36e73 as user > 'root' > I1202 15:18:01.495087 26347 leveldb.cpp:343] Persisting action (16 bytes) to > leveldb took 2.635152ms > I1202 15:18:01.495126 26347 replica.cpp:715] Persisted action at 4 > I1202 15:18:01.495736 26342 replica.cpp:694] Replica received learned notice > for position 4 from @0.0.0.0:0 > I1202 15:18:01.496106 26347 master.hpp:176] Adding task > b2102462-a9c1-45bf-94f2-9a59abb36e73 with resources cpus(*):2; mem(*):1024; > disk(*):1024; ports(*):[31000-32000] on slave > baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-S0 (debian-vm.localdomain) > I1202 15:18:01.496330 26347 master.cpp:3245] Launching task > b2102462-a9c1-45bf-94f2-9a59abb36e73 of framework > baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-0000 (default) at > scheduler-82a736f1-e818-4a76-8126-93ac22dcfde0@127.0.1.1:33625 with resources > cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] on slave > baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-S0 at slave(655)@127.0.1.1:33625 > (debian-vm.localdomain) > I1202 15:18:01.496820 26344 slave.cpp:1294] Got assigned task > b2102462-a9c1-45bf-94f2-9a59abb36e73 for framework > baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-0000 > I1202 15:18:01.497508 26344 slave.cpp:1410] Launching task > b2102462-a9c1-45bf-94f2-9a59abb36e73 for framework > baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-0000 > I1202 15:18:01.498034 26344 paths.cpp:436] Trying to chown > '/tmp/SlaveRecoveryTest_0_Reboot_qT6DBx/slaves/baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-S0/frameworks/baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-0000/executors/b2102462-a9c1-45bf-94f2-9a59abb36e73/runs/ceb8eefc-8de6-461c-add8-2b22666a1617' > to user 'root' > I1202 15:18:01.497663 26342 leveldb.cpp:343] Persisting action (18 bytes) to > leveldb took 1.863299ms > I1202 15:18:01.505702 26342 leveldb.cpp:401] Deleting ~2 keys from leveldb > took 86618ns > I1202 15:18:01.505772 26342 replica.cpp:715] Persisted action at 4 > I1202 15:18:01.505803 26342 replica.cpp:700] Replica learned TRUNCATE action > at position 4 > I1202 15:18:01.508184 26344 slave.cpp:4999] Launching executor > b2102462-a9c1-45bf-94f2-9a59abb36e73 of framework > baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-0000 with resources cpus(*):0.1; > mem(*):32 in work directory > '/tmp/SlaveRecoveryTest_0_Reboot_qT6DBx/slaves/baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-S0/frameworks/baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-0000/executors/b2102462-a9c1-45bf-94f2-9a59abb36e73/runs/ceb8eefc-8de6-461c-add8-2b22666a1617' > I1202 15:18:01.508643 26347 containerizer.cpp:618] Starting container > 'ceb8eefc-8de6-461c-add8-2b22666a1617' for executor > 'b2102462-a9c1-45bf-94f2-9a59abb36e73' of framework > 'baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-0000' > I1202 15:18:01.508885 26344 slave.cpp:1628] Queuing task > 'b2102462-a9c1-45bf-94f2-9a59abb36e73' for executor > 'b2102462-a9c1-45bf-94f2-9a59abb36e73' of framework > baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-0000 > I1202 15:18:01.575203 26349 cpushare.cpp:392] Updated 'cpu.shares' to 2150 > (cpus 2.1) for container ceb8eefc-8de6-461c-add8-2b22666a1617 > I1202 15:18:01.639154 26347 mem.cpp:605] Started listening for OOM events for > container ceb8eefc-8de6-461c-add8-2b22666a1617 > 2015-12-02 > 15:18:01,650:26328(0x7f9841de9700):ZOO_ERROR@handle_socket_error_msg@1697: > Socket [127.0.0.1:52030] zk retcode=-4, errno=111(Connection refused): server > refused to accept the client > I1202 15:18:01.656420 26347 mem.cpp:725] Started listening on low memory > pressure events for container ceb8eefc-8de6-461c-add8-2b22666a1617 > I1202 15:18:01.678865 26347 mem.cpp:725] Started listening on medium memory > pressure events for container ceb8eefc-8de6-461c-add8-2b22666a1617 > I1202 15:18:01.713045 26347 mem.cpp:725] Started listening on critical memory > pressure events for container ceb8eefc-8de6-461c-add8-2b22666a1617 > I1202 15:18:01.729801 26347 mem.cpp:356] Updated 'memory.soft_limit_in_bytes' > to 1056MB for container ceb8eefc-8de6-461c-add8-2b22666a1617 > I1202 15:18:01.766522 26347 mem.cpp:391] Updated 'memory.limit_in_bytes' to > 1056MB for container ceb8eefc-8de6-461c-add8-2b22666a1617 > 2015-12-02 > 15:18:01,786:26328(0x7f983adf7700):ZOO_ERROR@handle_socket_error_msg@1697: > Socket [127.0.0.1:56378] zk retcode=-4, errno=111(Connection refused): server > refused to accept the client > I1202 15:18:01.811028 26345 linux_launcher.cpp:365] Cloning child process > with flags = > I1202 15:18:01.850016 26345 linux_launcher.cpp:422] Assigned child process > '14143' to 'mesos_executors.slice' > I1202 15:18:01.850262 26345 containerizer.cpp:851] Checkpointing executor's > forked pid 14143 to > '/tmp/SlaveRecoveryTest_0_Reboot_qT6DBx/meta/slaves/baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-S0/frameworks/baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-0000/executors/b2102462-a9c1-45bf-94f2-9a59abb36e73/runs/ceb8eefc-8de6-461c-add8-2b22666a1617/pids/forked.pid' > I1202 15:18:01.944136 14157 exec.cpp:136] Version: 0.26.0 > I1202 15:18:01.946939 26343 slave.cpp:2405] Got registration for executor > 'b2102462-a9c1-45bf-94f2-9a59abb36e73' of framework > baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-0000 from executor(1)@127.0.1.1:57954 > I1202 15:18:01.948669 14177 exec.cpp:210] Executor registered on slave > baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-S0 > Registered executor on debian-vm.localdomain > I1202 15:18:01.967314 26347 mem.cpp:356] Updated 'memory.soft_limit_in_bytes' > to 1056MB for container ceb8eefc-8de6-461c-add8-2b22666a1617 > I1202 15:18:01.971539 26344 cpushare.cpp:392] Updated 'cpu.shares' to 2150 > (cpus 2.1) for container ceb8eefc-8de6-461c-add8-2b22666a1617 > I1202 15:18:01.985469 26344 slave.cpp:1793] Sending queued task > 'b2102462-a9c1-45bf-94f2-9a59abb36e73' to executor > 'b2102462-a9c1-45bf-94f2-9a59abb36e73' of framework > baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-0000 at executor(1)@127.0.1.1:57954 > Starting task b2102462-a9c1-45bf-94f2-9a59abb36e73 > Forked command at 14180 > sh -c 'sleep 1000' > I1202 15:18:02.001322 26347 slave.cpp:2762] Handling status update > TASK_RUNNING (UUID: 444a54f5-32d6-49e5-84c6-c2729395428e) for task > b2102462-a9c1-45bf-94f2-9a59abb36e73 of framework > baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-0000 from executor(1)@127.0.1.1:57954 > I1202 15:18:02.001744 26347 status_update_manager.cpp:322] Received status > update TASK_RUNNING (UUID: 444a54f5-32d6-49e5-84c6-c2729395428e) for task > b2102462-a9c1-45bf-94f2-9a59abb36e73 of framework > baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-0000 > I1202 15:18:02.002167 26347 status_update_manager.cpp:826] Checkpointing > UPDATE for status update TASK_RUNNING (UUID: > 444a54f5-32d6-49e5-84c6-c2729395428e) for task > b2102462-a9c1-45bf-94f2-9a59abb36e73 of framework > baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-0000 > I1202 15:18:02.013846 26349 slave.cpp:3087] Forwarding the update > TASK_RUNNING (UUID: 444a54f5-32d6-49e5-84c6-c2729395428e) for task > b2102462-a9c1-45bf-94f2-9a59abb36e73 of framework > baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-0000 to master@127.0.1.1:33625 > I1202 15:18:02.014194 26349 slave.cpp:3011] Sending acknowledgement for > status update TASK_RUNNING (UUID: 444a54f5-32d6-49e5-84c6-c2729395428e) for > task b2102462-a9c1-45bf-94f2-9a59abb36e73 of framework > baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-0000 to executor(1)@127.0.1.1:57954 > I1202 15:18:02.014359 26347 master.cpp:4414] Status update TASK_RUNNING > (UUID: 444a54f5-32d6-49e5-84c6-c2729395428e) for task > b2102462-a9c1-45bf-94f2-9a59abb36e73 of framework > baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-0000 from slave > baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-S0 at slave(655)@127.0.1.1:33625 > (debian-vm.localdomain) > I1202 15:18:02.014411 26347 master.cpp:4462] Forwarding status update > TASK_RUNNING (UUID: 444a54f5-32d6-49e5-84c6-c2729395428e) for task > b2102462-a9c1-45bf-94f2-9a59abb36e73 of framework > baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-0000 > I1202 15:18:02.014533 26347 master.cpp:6066] Updating the state of task > b2102462-a9c1-45bf-94f2-9a59abb36e73 of framework > baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-0000 (latest state: TASK_RUNNING, status > update state: TASK_RUNNING) > I1202 15:18:02.015163 26347 master.cpp:3571] Processing ACKNOWLEDGE call > 444a54f5-32d6-49e5-84c6-c2729395428e for task > b2102462-a9c1-45bf-94f2-9a59abb36e73 of framework > baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-0000 (default) at > scheduler-82a736f1-e818-4a76-8126-93ac22dcfde0@127.0.1.1:33625 on slave > baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-S0 > I1202 15:18:02.015419 26347 status_update_manager.cpp:394] Received status > update acknowledgement (UUID: 444a54f5-32d6-49e5-84c6-c2729395428e) for task > b2102462-a9c1-45bf-94f2-9a59abb36e73 of framework > baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-0000 > I1202 15:18:02.015508 26347 status_update_manager.cpp:826] Checkpointing ACK > for status update TASK_RUNNING (UUID: 444a54f5-32d6-49e5-84c6-c2729395428e) > for task b2102462-a9c1-45bf-94f2-9a59abb36e73 of framework > baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-0000 > I1202 15:18:02.016258 26328 slave.cpp:601] Slave terminating > I1202 15:18:02.016489 26345 master.cpp:1083] Slave > baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-S0 at slave(655)@127.0.1.1:33625 > (debian-vm.localdomain) disconnected > I1202 15:18:02.016530 26345 master.cpp:2531] Disconnecting slave > baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-S0 at slave(655)@127.0.1.1:33625 > (debian-vm.localdomain) > I1202 15:18:02.017040 26345 master.cpp:2550] Deactivating slave > baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-S0 at slave(655)@127.0.1.1:33625 > (debian-vm.localdomain) > I1202 15:18:02.017151 26344 hierarchical.cpp:429] Slave > baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-S0 deactivated > I1202 15:18:02.089155 14174 exec.cpp:383] Executor asked to shutdown > Shutting down > Sending SIGTERM to process tree at pid 14180 > Killing the following process trees: > [ > -+- 14180 sh -c sleep 1000 > \--- 14181 sleep 1000 > ] > Command terminated with signal Terminated (pid: 14180) > I1202 15:18:03.298529 26328 containerizer.cpp:142] Using isolation: > cgroups/cpu,cgroups/mem,filesystem/posix > I1202 15:18:04.043941 26328 linux_launcher.cpp:103] Using > /sys/fs/cgroup/freezer as the freezer hierarchy for the Linux launcher > I1202 15:18:04.050012 26328 systemd.cpp:210] Started systemd slice > `mesos_executors.slice` > I1202 15:18:04.072232 26344 slave.cpp:191] Slave started on > 656)@127.0.1.1:33625 > I1202 15:18:04.072262 26344 slave.cpp:192] Flags at startup: > --appc_store_dir="/tmp/mesos/store/appc" --authenticatee="crammd5" > --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" > --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" > --cgroups_root="mesos_test_d5b6bb71-dab3-4457-ae03-84a046ed9c62" > --container_disk_watch_interval="15secs" --containerizers="mesos" > --credential="/tmp/SlaveRecoveryTest_0_Reboot_qT6DBx/credential" > --default_role="*" --disk_watch_interval="1mins" --docker="docker" > --docker_auth_server="auth.docker.io" --docker_auth_server_port="443" > --docker_kill_orphans="true" > --docker_local_archives_dir="/tmp/mesos/images/docker" > --docker_puller="local" --docker_puller_timeout="60" > --docker_registry="registry-1.docker.io" --docker_registry_port="443" > --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" > --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" > --enforce_container_disk_quota="false" > --executor_registration_timeout="1mins" > --executor_shutdown_grace_period="5secs" > --fetcher_cache_dir="/tmp/SlaveRecoveryTest_0_Reboot_qT6DBx/fetch" > --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" > --gc_disk_headroom="0.1" --hadoop_home="" --help="false" > --hostname_lookup="true" --image_provisioner_backend="copy" > --initialize_driver_logging="true" --isolation="cgroups/cpu,cgroups/mem" > --launcher_dir="/home/alexander/Documents/workspace/mesos/build/src" > --logbufsecs="0" --logging_level="INFO" > --oversubscribed_resources_interval="15secs" --perf_duration="10secs" > --perf_interval="1mins" --qos_correction_interval_min="0ns" --quiet="false" > --recover="reconnect" --recovery_timeout="15mins" > --registration_backoff_factor="10ms" > --resources="cpus:2;mem:1024;disk:1024;ports:[31000-32000]" > --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" > --slave_subsystems="memory,cpuacct" --strict="false" --switch_user="true" > --systemd_runtime_directory="/run/systemd/system" --version="false" > --work_dir="/tmp/SlaveRecoveryTest_0_Reboot_qT6DBx" > I1202 15:18:04.072510 26344 slave.cpp:212] Moving slave process into its own > cgroup for subsystem: memory > I1202 15:18:04.334131 26344 slave.cpp:212] Moving slave process into its own > cgroup for subsystem: cpuacct > I1202 15:18:04.516194 26344 credentials.hpp:85] Loading credential for > authentication from '/tmp/SlaveRecoveryTest_0_Reboot_qT6DBx/credential' > I1202 15:18:04.516338 26344 slave.cpp:322] Slave using credential for: > test-principal > I1202 15:18:04.516819 26344 slave.cpp:392] Slave resources: cpus(*):2; > mem(*):1024; disk(*):1024; ports(*):[31000-32000] > I1202 15:18:04.516865 26344 slave.cpp:400] Slave attributes: [ ] > I1202 15:18:04.516873 26344 slave.cpp:405] Slave hostname: > debian-vm.localdomain > I1202 15:18:04.516878 26344 slave.cpp:410] Slave checkpoint: true > I1202 15:18:04.517696 26346 state.cpp:54] Recovering state from > '/tmp/SlaveRecoveryTest_0_Reboot_qT6DBx/meta' > I1202 15:18:04.517777 26346 state.cpp:681] No checkpointed resources found at > '/tmp/SlaveRecoveryTest_0_Reboot_qT6DBx/meta/resources/resources.info' > I1202 15:18:04.517849 26346 state.cpp:85] Slave host rebooted > I1202 15:18:04.518209 26346 status_update_manager.cpp:202] Recovering status > update manager > I1202 15:18:04.518307 26346 containerizer.cpp:384] Recovering containerizer > I1202 15:18:04.592492 26345 containerizer.cpp:522] Removing orphan container > ceb8eefc-8de6-461c-add8-2b22666a1617 > I1202 15:18:04.651180 26345 slave.cpp:4230] Finished recovery > I1202 15:18:04.651376 26349 cgroups.cpp:2429] Freezing cgroup > /sys/fs/cgroup/freezer/mesos_test_d5b6bb71-dab3-4457-ae03-84a046ed9c62/ceb8eefc-8de6-461c-add8-2b22666a1617 > I1202 15:18:04.651582 26345 slave.cpp:4263] Garbage collecting old slave > baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-S0 > I1202 15:18:04.651935 26345 gc.cpp:56] Scheduling > '/tmp/SlaveRecoveryTest_0_Reboot_qT6DBx/slaves/baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-S0' > for gc 6.99999245687407days in the future > I1202 15:18:04.652065 26345 gc.cpp:56] Scheduling > '/tmp/SlaveRecoveryTest_0_Reboot_qT6DBx/meta/slaves/baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-S0' > for gc 6.99999245635556days in the future > I1202 15:18:04.652354 26345 slave.cpp:729] New master detected at > master@127.0.1.1:33625 > I1202 15:18:04.652434 26345 slave.cpp:792] Authenticating with master > master@127.0.1.1:33625 > I1202 15:18:04.652451 26345 slave.cpp:797] Using default CRAM-MD5 > authenticatee > I1202 15:18:04.652549 26345 slave.cpp:765] Detecting new master > I1202 15:18:04.652704 26345 status_update_manager.cpp:176] Pausing sending > status updates > I1202 15:18:04.652803 26345 authenticatee.cpp:123] Creating new client SASL > connection > I1202 15:18:04.653151 26345 master.cpp:5150] Authenticating > slave(656)@127.0.1.1:33625 > I1202 15:18:04.653491 26345 authenticator.cpp:100] Creating new server SASL > connection > I1202 15:18:04.654045 26345 authenticatee.cpp:214] Received SASL > authentication mechanisms: CRAM-MD5 > I1202 15:18:04.654069 26345 authenticatee.cpp:240] Attempting to authenticate > with mechanism 'CRAM-MD5' > I1202 15:18:04.654127 26345 authenticator.cpp:205] Received SASL > authentication start > I1202 15:18:04.654168 26345 authenticator.cpp:327] Authentication requires > more steps > I1202 15:18:04.654232 26345 authenticatee.cpp:260] Received SASL > authentication step > I1202 15:18:04.654295 26345 authenticator.cpp:233] Received SASL > authentication step > I1202 15:18:04.654358 26345 authenticator.cpp:319] Authentication success > I1202 15:18:04.654491 26345 authenticatee.cpp:300] Authentication success > I1202 15:18:04.654752 26344 slave.cpp:860] Successfully authenticated with > master master@127.0.1.1:33625 > I1202 15:18:04.654968 26345 master.cpp:5180] Successfully authenticated > principal 'test-principal' at slave(656)@127.0.1.1:33625 > I1202 15:18:04.655432 26345 master.cpp:3859] Registering slave at > slave(656)@127.0.1.1:33625 (debian-vm.localdomain) with id > baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-S1 > I1202 15:18:04.656077 26343 registrar.cpp:441] Applied 1 operations in > 76820ns; attempting to update the 'registry' > I1202 15:18:04.657047 26343 log.cpp:685] Attempting to append 540 bytes to > the log > I1202 15:18:04.657225 26343 coordinator.cpp:350] Coordinator attempting to > write APPEND action at position 5 > I1202 15:18:04.658035 26343 replica.cpp:540] Replica received write request > for position 5 from (21123)@127.0.1.1:33625 > I1202 15:18:04.665920 26343 leveldb.cpp:343] Persisting action (559 bytes) to > leveldb took 7.814853ms > I1202 15:18:04.665997 26343 replica.cpp:715] Persisted action at 5 > I1202 15:18:04.666776 26343 replica.cpp:694] Replica received learned notice > for position 5 from @0.0.0.0:0 > I1202 15:18:04.667973 26343 leveldb.cpp:343] Persisting action (561 bytes) to > leveldb took 1.08753ms > I1202 15:18:04.668018 26343 replica.cpp:715] Persisted action at 5 > I1202 15:18:04.668038 26343 replica.cpp:700] Replica learned APPEND action at > position 5 > I1202 15:18:04.672534 26346 registrar.cpp:486] Successfully updated the > 'registry' in 16.38784ms > I1202 15:18:04.672734 26343 log.cpp:704] Attempting to truncate the log to 5 > I1202 15:18:04.672901 26342 master.cpp:3847] Ignoring register slave message > from slave(656)@127.0.1.1:33625 (debian-vm.localdomain) as admission is > already in progress > I1202 15:18:04.672914 26343 coordinator.cpp:350] Coordinator attempting to > write TRUNCATE action at position 6 > I1202 15:18:04.673462 26342 master.cpp:3927] Registered slave > baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-S1 at slave(656)@127.0.1.1:33625 > (debian-vm.localdomain) with cpus(*):2; mem(*):1024; disk(*):1024; > ports(*):[31000-32000] > I1202 15:18:04.673705 26343 hierarchical.cpp:344] Added slave > baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-S1 (debian-vm.localdomain) with > cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] (allocated: ) > I1202 15:18:04.673727 26346 replica.cpp:540] Replica received write request > for position 6 from (21124)@127.0.1.1:33625 > I1202 15:18:04.674177 26342 slave.cpp:904] Registered with master > master@127.0.1.1:33625; given slave ID baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-S1 > I1202 15:18:04.674335 26347 status_update_manager.cpp:183] Resuming sending > status updates > I1202 15:18:04.674424 26348 master.cpp:4979] Sending 1 offers to framework > baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-0000 (default) at > scheduler-82a736f1-e818-4a76-8126-93ac22dcfde0@127.0.1.1:33625 > I1202 15:18:04.674509 26342 slave.cpp:963] Forwarding total oversubscribed > resources > I1202 15:18:04.674677 26348 master.cpp:4269] Received update of slave > baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-S1 at slave(656)@127.0.1.1:33625 > (debian-vm.localdomain) with total oversubscribed resources > I1202 15:18:04.674923 26343 hierarchical.cpp:400] Slave > baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-S1 (debian-vm.localdomain) updated with > oversubscribed resources (total: cpus(*):2; mem(*):1024; disk(*):1024; > ports(*):[31000-32000], allocated: cpus(*):2; mem(*):1024; disk(*):1024; > ports(*):[31000-32000]) > I1202 15:18:04.675171 26346 leveldb.cpp:343] Persisting action (16 bytes) to > leveldb took 1.412868ms > I1202 15:18:04.675211 26346 replica.cpp:715] Persisted action at 6 > I1202 15:18:04.675493 26328 sched.cpp:1805] Asked to stop the driver > I1202 15:18:04.675585 26328 master.cpp:922] Master terminating > I1202 15:18:04.675717 26346 sched.cpp:1043] Stopping framework > 'baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-0000' > I1202 15:18:04.675988 26346 hierarchical.cpp:373] Removed slave > baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-S1 > W1202 15:18:04.676062 26328 master.cpp:6118] Removing task > b2102462-a9c1-45bf-94f2-9a59abb36e73 with resources cpus(*):2; mem(*):1024; > disk(*):1024; ports(*):[31000-32000] of framework > baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-0000 on slave > baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-S0 at slave(655)@127.0.1.1:33625 > (debian-vm.localdomain) in non-terminal state TASK_RUNNING > I1202 15:18:04.676136 26346 hierarchical.cpp:373] Removed slave > baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-S0 > I1202 15:18:04.676923 26347 hierarchical.cpp:230] Removed framework > baeb70c6-c960-4d0d-9dc7-48b9a54ef8a8-0000 > I1202 15:18:04.677057 26346 slave.cpp:3215] master@127.0.1.1:33625 exited > W1202 15:18:04.677093 26346 slave.cpp:3218] Master disconnected! Waiting for > a new master to be elected > I1202 15:18:04.678817 26343 replica.cpp:694] Replica received learned notice > for position 6 from @0.0.0.0:0 > I1202 15:18:04.679985 26343 leveldb.cpp:343] Persisting action (18 bytes) to > leveldb took 1.113234ms > I1202 15:18:04.680058 26343 leveldb.cpp:401] Deleting ~2 keys from leveldb > took 25679ns > I1202 15:18:04.680094 26343 replica.cpp:715] Persisted action at 6 > I1202 15:18:04.680116 26343 replica.cpp:700] Replica learned TRUNCATE action > at position 6 > I1202 15:18:04.681684 26348 slave.cpp:601] Slave terminating > I1202 15:18:04.721125 26349 cgroups.cpp:1411] Successfully froze cgroup > /sys/fs/cgroup/freezer/mesos_test_d5b6bb71-dab3-4457-ae03-84a046ed9c62/ceb8eefc-8de6-461c-add8-2b22666a1617 > after 69.67808ms > I1202 15:18:04.778825 26347 cgroups.cpp:2447] Thawing cgroup > /sys/fs/cgroup/freezer/mesos_test_d5b6bb71-dab3-4457-ae03-84a046ed9c62/ceb8eefc-8de6-461c-add8-2b22666a1617 > I1202 15:18:04.843261 26349 cgroups.cpp:1440] Successfullly thawed cgroup > /sys/fs/cgroup/freezer/mesos_test_d5b6bb71-dab3-4457-ae03-84a046ed9c62/ceb8eefc-8de6-461c-add8-2b22666a1617 > after 64.342016ms > I1202 15:18:04.854326 26345 cgroups.cpp:2429] Freezing cgroup > /sys/fs/cgroup/freezer/mesos_test_d5b6bb71-dab3-4457-ae03-84a046ed9c62/ceb8eefc-8de6-461c-add8-2b22666a1617 > ../../src/tests/mesos.cpp:781: Failure > (cgroups::destroy(hierarchy, cgroup)).failure(): Failed to kill tasks in > nested cgroups: Collect failed: Invalid freezer cgroup: > 'mesos_test_d5b6bb71-dab3-4457-ae03-84a046ed9c62/ceb8eefc-8de6-461c-add8-2b22666a1617' > is not a valid cgroup > *** Aborted at 1449065884 (unix time) try "date -d @1449065884" if you are > using GNU date *** > PC: @ 0x14b07ae testing::UnitTest::AddTestPartResult() > *** SIGSEGV (@0x0) received by PID 26328 (TID 0x7f9891edb7c0) from PID 0; > stack trace: *** > @ 0x7f9879fc366c os::Linux::chained_handler() > @ 0x7f9879fc7a0a JVM_handle_linux_signal > @ 0x7f988b7f88d0 (unknown) > @ 0x14b07ae testing::UnitTest::AddTestPartResult() > @ 0x14a51e7 testing::internal::AssertHelper::operator=() > @ 0xf564d1 > mesos::internal::tests::ContainerizerTest<>::TearDown() > @ 0x14ce2d0 > testing::internal::HandleSehExceptionsInMethodIfSupported<>() > @ 0x14c9248 > testing::internal::HandleExceptionsInMethodIfSupported<>() > @ 0x14aa5d0 testing::Test::Run() > @ 0x14aad15 testing::TestInfo::Run() > @ 0x14ab350 testing::TestCase::Run() > @ 0x14b1c9f testing::internal::UnitTestImpl::RunAllTests() > @ 0x14cef5f > testing::internal::HandleSehExceptionsInMethodIfSupported<>() > @ 0x14c9d9e > testing::internal::HandleExceptionsInMethodIfSupported<>() > @ 0x14b09cf testing::UnitTest::Run() > @ 0xd63e02 RUN_ALL_TESTS() > @ 0xd639e0 main > @ 0x7f988b461b45 (unknown) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)