[ https://issues.apache.org/jira/browse/MESOS-2331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15220477#comment-15220477 ]
Vinod Kone commented on MESOS-2331: ----------------------------------- Saw this on ASF CI when I committed the upgrade to protobuf 2.6.1 (not sure if it's just coincidental). {code} [ RUN ] MasterSlaveReconciliationTest.ReconcileRace I0331 18:41:04.018868 1701 cluster.cpp:139] Creating default 'local' authorizer I0331 18:41:04.025219 1701 leveldb.cpp:174] Opened db in 6.145168ms I0331 18:41:04.027966 1701 leveldb.cpp:181] Compacted db in 2.725315ms I0331 18:41:04.028023 1701 leveldb.cpp:196] Created db iterator in 32098ns I0331 18:41:04.028041 1701 leveldb.cpp:202] Seeked to beginning of db in 9050ns I0331 18:41:04.028053 1701 leveldb.cpp:271] Iterated through 0 keys in the db in 7186ns I0331 18:41:04.028097 1701 replica.cpp:779] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned I0331 18:41:04.029259 1731 recover.cpp:447] Starting replica recovery I0331 18:41:04.029541 1719 recover.cpp:473] Replica is in EMPTY status I0331 18:41:04.030627 1720 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (5660)@172.17.0.2:47086 I0331 18:41:04.031069 1723 recover.cpp:193] Received a recover response from a replica in EMPTY status I0331 18:41:04.036355 1723 recover.cpp:564] Updating replica status to STARTING I0331 18:41:04.038835 1729 master.cpp:376] Master 2cabf4ec-31e0-4769-9e38-5581d3d8fdfd (ff176ecfc889) started on 172.17.0.2:47086 I0331 18:41:04.038995 1723 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 2.398241ms I0331 18:41:04.039036 1723 replica.cpp:320] Persisted replica status to STARTING I0331 18:41:04.038871 1729 master.cpp:378] Flags at startup: --acls="" --allocation_interval="1secs" --allocator="HierarchicalDRF" --authenticate="true" --authenticate_http="true" --authenticate_slaves="true" --authenticators="crammd5" --authorizers="local" --credentials="/tmp/3c2EEo/credentials" --framework_sorter="drf" --help="false" --hostname_lookup="true" --http_authenticators="basic" --initialize_driver_logging="true" --log_auto_initialize="true" --logbufsecs="0" --logging_level="INFO" --max_completed_frameworks="50" --max_completed_tasks_per_framework="1000" --max_slave_ping_timeouts="5" --quiet="false" --recovery_slave_removal_limit="100%" --registry="replicated_log" --registry_fetch_timeout="1mins" --registry_store_timeout="100secs" --registry_strict="true" --root_submissions="true" --slave_ping_timeout="15secs" --slave_reregister_timeout="10mins" --user_sorter="drf" --version="false" --webui_dir="/mesos/mesos-0.29.0/_inst/share/mesos/webui" --work_dir="/tmp/3c2EEo/master" --zk_session_timeout="10secs" I0331 18:41:04.040617 1729 master.cpp:427] Master only allowing authenticated frameworks to register I0331 18:41:04.040630 1729 master.cpp:432] Master only allowing authenticated slaves to register I0331 18:41:04.040640 1729 credentials.hpp:37] Loading credentials for authentication from '/tmp/3c2EEo/credentials' I0331 18:41:04.040983 1729 master.cpp:474] Using default 'crammd5' authenticator I0331 18:41:04.041123 1729 master.cpp:545] Using default 'basic' HTTP authenticator I0331 18:41:04.041169 1731 recover.cpp:473] Replica is in STARTING status I0331 18:41:04.041245 1729 master.cpp:583] Authorization enabled I0331 18:41:04.041767 1731 hierarchical.cpp:144] Initialized hierarchical allocator process I0331 18:41:04.041832 1731 whitelist_watcher.cpp:77] No whitelist given I0331 18:41:04.043683 1729 master.cpp:1826] The newly elected leader is master@172.17.0.2:47086 with id 2cabf4ec-31e0-4769-9e38-5581d3d8fdfd I0331 18:41:04.043718 1729 master.cpp:1839] Elected as the leading master! I0331 18:41:04.043731 1729 master.cpp:1526] Recovering from registrar I0331 18:41:04.043979 1729 registrar.cpp:307] Recovering registrar I0331 18:41:04.048002 1729 replica.cpp:673] Replica in STARTING status received a broadcasted recover request from (5662)@172.17.0.2:47086 I0331 18:41:04.048368 1729 recover.cpp:193] Received a recover response from a replica in STARTING status I0331 18:41:04.048904 1729 recover.cpp:564] Updating replica status to VOTING I0331 18:41:04.052000 1722 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 2.020256ms I0331 18:41:04.052044 1722 replica.cpp:320] Persisted replica status to VOTING I0331 18:41:04.052170 1722 recover.cpp:578] Successfully joined the Paxos group I0331 18:41:04.052336 1722 recover.cpp:462] Recover process terminated I0331 18:41:04.052865 1722 log.cpp:659] Attempting to start the writer I0331 18:41:04.055016 1730 replica.cpp:493] Replica received implicit promise request from (5663)@172.17.0.2:47086 with proposal 1 I0331 18:41:04.056974 1730 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 1.935964ms I0331 18:41:04.057008 1730 replica.cpp:342] Persisted promised to 1 I0331 18:41:04.058007 1730 coordinator.cpp:238] Coordinator attempting to fill missing positions I0331 18:41:04.059571 1726 replica.cpp:388] Replica received explicit promise request from (5664)@172.17.0.2:47086 for position 0 with proposal 2 I0331 18:41:04.060089 1726 leveldb.cpp:341] Persisting action (8 bytes) to leveldb took 430415ns I0331 18:41:04.060119 1726 replica.cpp:712] Persisted action at 0 I0331 18:41:04.061415 1726 replica.cpp:537] Replica received write request for position 0 from (5665)@172.17.0.2:47086 I0331 18:41:04.061477 1726 leveldb.cpp:436] Reading position from leveldb took 43631ns I0331 18:41:04.061990 1726 leveldb.cpp:341] Persisting action (14 bytes) to leveldb took 486638ns I0331 18:41:04.062021 1726 replica.cpp:712] Persisted action at 0 I0331 18:41:04.062736 1726 replica.cpp:691] Replica received learned notice for position 0 from @0.0.0.0:0 I0331 18:41:04.063989 1726 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 1.235406ms I0331 18:41:04.064026 1726 replica.cpp:712] Persisted action at 0 I0331 18:41:04.064050 1726 replica.cpp:697] Replica learned NOP action at position 0 I0331 18:41:04.064846 1726 log.cpp:675] Writer started with ending position 0 I0331 18:41:04.066151 1726 leveldb.cpp:436] Reading position from leveldb took 41291ns I0331 18:41:04.082175 1732 registrar.cpp:340] Successfully fetched the registry (0B) in 38.127872ms I0331 18:41:04.082320 1732 registrar.cpp:439] Applied 1 operations in 49229ns; attempting to update the 'registry' I0331 18:41:04.083664 1726 log.cpp:683] Attempting to append 170 bytes to the log I0331 18:41:04.083865 1726 coordinator.cpp:348] Coordinator attempting to write APPEND action at position 1 I0331 18:41:04.084794 1728 replica.cpp:537] Replica received write request for position 1 from (5666)@172.17.0.2:47086 I0331 18:41:04.086985 1728 leveldb.cpp:341] Persisting action (189 bytes) to leveldb took 2.159231ms I0331 18:41:04.087025 1728 replica.cpp:712] Persisted action at 1 I0331 18:41:04.101534 1720 replica.cpp:691] Replica received learned notice for position 1 from @0.0.0.0:0 I0331 18:41:04.102982 1720 leveldb.cpp:341] Persisting action (191 bytes) to leveldb took 1.426768ms I0331 18:41:04.103018 1720 replica.cpp:712] Persisted action at 1 I0331 18:41:04.103045 1720 replica.cpp:697] Replica learned APPEND action at position 1 I0331 18:41:04.104568 1720 registrar.cpp:484] Successfully updated the 'registry' in 22.184192ms I0331 18:41:04.104714 1720 registrar.cpp:370] Successfully recovered registrar I0331 18:41:04.105062 1731 log.cpp:702] Attempting to truncate the log to 1 I0331 18:41:04.110205 1721 master.cpp:1634] Recovered 0 slaves from the Registry (131B) ; allowing 10mins for slaves to re-register I0331 18:41:04.110333 1721 hierarchical.cpp:171] Skipping recovery of hierarchical allocator: nothing to recover I0331 18:41:04.110416 1723 coordinator.cpp:348] Coordinator attempting to write TRUNCATE action at position 2 I0331 18:41:04.111536 1729 replica.cpp:537] Replica received write request for position 2 from (5667)@172.17.0.2:47086 I0331 18:41:04.111982 1729 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 419481ns I0331 18:41:04.112012 1729 replica.cpp:712] Persisted action at 2 I0331 18:41:04.113961 1732 replica.cpp:691] Replica received learned notice for position 2 from @0.0.0.0:0 I0331 18:41:04.114981 1732 leveldb.cpp:341] Persisting action (18 bytes) to leveldb took 1.002446ms I0331 18:41:04.115056 1732 leveldb.cpp:399] Deleting ~1 keys from leveldb took 49334ns I0331 18:41:04.115077 1732 replica.cpp:712] Persisted action at 2 I0331 18:41:04.115099 1732 replica.cpp:697] Replica learned TRUNCATE action at position 2 I0331 18:41:04.135689 1724 slave.cpp:200] Slave started on 136)@172.17.0.2:47086 I0331 18:41:04.135746 1724 slave.cpp:201] Flags at startup: --appc_simple_discovery_uri_prefix="http://" --appc_store_dir="/tmp/mesos/store/appc" --authenticate_http="true" --authenticatee="crammd5" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --credential="/tmp/MasterSlaveReconciliationTest_ReconcileRace_Kkms7H/credential" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/MasterSlaveReconciliationTest_ReconcileRace_Kkms7H/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname_lookup="true" --http_authenticators="basic" --http_credentials="/tmp/MasterSlaveReconciliationTest_ReconcileRace_Kkms7H/http_credentials" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher_dir="/mesos/mesos-0.29.0/_build/src" --logbufsecs="0" --logging_level="INFO" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="10ms" --resources="cpus:2;mem:1024;disk:1024;ports:[31000-32000]" --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_enable_support="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/tmp/MasterSlaveReconciliationTest_ReconcileRace_Kkms7H" I0331 18:41:04.136354 1724 credentials.hpp:86] Loading credential for authentication from '/tmp/MasterSlaveReconciliationTest_ReconcileRace_Kkms7H/credential' I0331 18:41:04.136620 1724 slave.cpp:338] Slave using credential for: test-principal I0331 18:41:04.136657 1724 credentials.hpp:37] Loading credentials for authentication from '/tmp/MasterSlaveReconciliationTest_ReconcileRace_Kkms7H/http_credentials' I0331 18:41:04.136956 1724 slave.cpp:390] Using default 'basic' HTTP authenticator I0331 18:41:04.137255 1724 resources.cpp:572] Parsing resources as JSON failed: cpus:2;mem:1024;disk:1024;ports:[31000-32000] Trying semicolon-delimited string format instead I0331 18:41:04.137820 1724 slave.cpp:589] Slave resources: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] I0331 18:41:04.137903 1724 slave.cpp:597] Slave attributes: [ ] I0331 18:41:04.137918 1724 slave.cpp:602] Slave hostname: ff176ecfc889 I0331 18:41:04.144213 1724 state.cpp:57] Recovering state from '/tmp/MasterSlaveReconciliationTest_ReconcileRace_Kkms7H/meta' I0331 18:41:04.144577 1724 status_update_manager.cpp:200] Recovering status update manager I0331 18:41:04.144851 1724 slave.cpp:4773] Finished recovery I0331 18:41:04.145391 1724 slave.cpp:4945] Querying resource estimator for oversubscribable resources I0331 18:41:04.145653 1724 status_update_manager.cpp:174] Pausing sending status updates I0331 18:41:04.145663 1723 slave.cpp:928] New master detected at master@172.17.0.2:47086 I0331 18:41:04.145766 1723 slave.cpp:991] Authenticating with master master@172.17.0.2:47086 I0331 18:41:04.145787 1723 slave.cpp:996] Using default CRAM-MD5 authenticatee I0331 18:41:04.145933 1723 slave.cpp:964] Detecting new master I0331 18:41:04.146148 1726 authenticatee.cpp:121] Creating new client SASL connection I0331 18:41:04.146497 1721 master.cpp:5679] Authenticating slave(136)@172.17.0.2:47086 I0331 18:41:04.146170 1723 slave.cpp:4959] Received oversubscribable resources from the resource estimator I0331 18:41:04.146739 1719 authenticator.cpp:413] Starting authentication session for crammd5_authenticatee(328)@172.17.0.2:47086 I0331 18:41:04.147104 1720 authenticator.cpp:98] Creating new server SASL connection I0331 18:41:04.147367 1720 authenticatee.cpp:212] Received SASL authentication mechanisms: CRAM-MD5 I0331 18:41:04.147408 1720 authenticatee.cpp:238] Attempting to authenticate with mechanism 'CRAM-MD5' I0331 18:41:04.147510 1720 authenticator.cpp:203] Received SASL authentication start I0331 18:41:04.147579 1720 authenticator.cpp:325] Authentication requires more steps I0331 18:41:04.147676 1720 authenticatee.cpp:258] Received SASL authentication step I0331 18:41:04.147789 1720 authenticator.cpp:231] Received SASL authentication step I0331 18:41:04.147828 1720 auxprop.cpp:107] Request to lookup properties for user: 'test-principal' realm: 'ff176ecfc889' server FQDN: 'ff176ecfc889' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I0331 18:41:04.147848 1720 auxprop.cpp:179] Looking up auxiliary property '*userPassword' I0331 18:41:04.147910 1720 auxprop.cpp:179] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I0331 18:41:04.147965 1720 auxprop.cpp:107] Request to lookup properties for user: 'test-principal' realm: 'ff176ecfc889' server FQDN: 'ff176ecfc889' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I0331 18:41:04.147985 1720 auxprop.cpp:129] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I0331 18:41:04.147996 1720 auxprop.cpp:129] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I0331 18:41:04.148020 1720 authenticator.cpp:317] Authentication success I0331 18:41:04.148200 1720 authenticatee.cpp:298] Authentication success I0331 18:41:04.148305 1720 master.cpp:5709] Successfully authenticated principal 'test-principal' at slave(136)@172.17.0.2:47086 I0331 18:41:04.148555 1720 slave.cpp:1061] Successfully authenticated with master master@172.17.0.2:47086 I0331 18:41:04.148659 1719 authenticator.cpp:431] Authentication session cleanup for crammd5_authenticatee(328)@172.17.0.2:47086 I0331 18:41:04.148704 1720 slave.cpp:1457] Will retry registration in 17.097919ms if necessary I0331 18:41:04.149015 1720 master.cpp:4390] Registering slave at slave(136)@172.17.0.2:47086 (ff176ecfc889) with id 2cabf4ec-31e0-4769-9e38-5581d3d8fdfd-S0 I0331 18:41:04.149469 1733 registrar.cpp:439] Applied 1 operations in 118008ns; attempting to update the 'registry' I0331 18:41:04.150331 1720 log.cpp:683] Attempting to append 339 bytes to the log I0331 18:41:04.150454 1719 coordinator.cpp:348] Coordinator attempting to write APPEND action at position 3 I0331 18:41:04.152674 1734 replica.cpp:537] Replica received write request for position 3 from (5673)@172.17.0.2:47086 I0331 18:41:04.153719 1734 leveldb.cpp:341] Persisting action (358 bytes) to leveldb took 1.015497ms I0331 18:41:04.153755 1734 replica.cpp:712] Persisted action at 3 I0331 18:41:04.158766 1734 replica.cpp:691] Replica received learned notice for position 3 from @0.0.0.0:0 I0331 18:41:04.159459 1734 leveldb.cpp:341] Persisting action (360 bytes) to leveldb took 655938ns I0331 18:41:04.159507 1734 replica.cpp:712] Persisted action at 3 I0331 18:41:04.159600 1734 replica.cpp:697] Replica learned APPEND action at position 3 I0331 18:41:04.162052 1723 registrar.cpp:484] Successfully updated the 'registry' in 12.511232ms I0331 18:41:04.162322 1734 log.cpp:702] Attempting to truncate the log to 3 I0331 18:41:04.167841 1726 master.cpp:4458] Registered slave 2cabf4ec-31e0-4769-9e38-5581d3d8fdfd-S0 at slave(136)@172.17.0.2:47086 (ff176ecfc889) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] I0331 18:41:04.168000 1723 coordinator.cpp:348] Coordinator attempting to write TRUNCATE action at position 4 I0331 18:41:04.168165 1722 slave.cpp:1457] Will retry registration in 26.342301ms if necessary I0331 18:41:04.168268 1726 hierarchical.cpp:476] Added slave 2cabf4ec-31e0-4769-9e38-5581d3d8fdfd-S0 (ff176ecfc889) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] (allocated: ) I0331 18:41:04.168419 1722 slave.cpp:1105] Registered with master master@172.17.0.2:47086; given slave ID 2cabf4ec-31e0-4769-9e38-5581d3d8fdfd-S0 I0331 18:41:04.168447 1722 fetcher.cpp:81] Clearing fetcher cache I0331 18:41:04.168460 1726 hierarchical.cpp:1490] No resources available to allocate! I0331 18:41:04.168524 1726 hierarchical.cpp:1164] Performed allocation for slave 2cabf4ec-31e0-4769-9e38-5581d3d8fdfd-S0 in 215436ns I0331 18:41:04.168815 1726 status_update_manager.cpp:181] Resuming sending status updates I0331 18:41:04.168823 1722 slave.cpp:1128] Checkpointing SlaveInfo to '/tmp/MasterSlaveReconciliationTest_ReconcileRace_Kkms7H/meta/slaves/2cabf4ec-31e0-4769-9e38-5581d3d8fdfd-S0/slave.info' I0331 18:41:04.169225 1722 slave.cpp:1165] Forwarding total oversubscribed resources I0331 18:41:04.169271 1728 replica.cpp:537] Replica received write request for position 4 from (5674)@172.17.0.2:47086 I0331 18:41:04.169319 1722 slave.cpp:3664] Received ping from slave-observer(139)@172.17.0.2:47086 I0331 18:41:04.171452 1701 sched.cpp:222] Version: 0.29.0 I0331 18:41:04.171977 1728 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 2.679636ms I0331 18:41:04.172013 1728 replica.cpp:712] Persisted action at 4 I0331 18:41:04.172127 1725 master.cpp:4360] Slave 2cabf4ec-31e0-4769-9e38-5581d3d8fdfd-S0 at slave(136)@172.17.0.2:47086 (ff176ecfc889) already registered, resending acknowledgement I0331 18:41:04.172195 1730 status_update_manager.cpp:174] Pausing sending status updates I0331 18:41:04.172194 1721 slave.cpp:928] New master detected at master@172.17.0.2:47086 I0331 18:41:04.172260 1721 slave.cpp:991] Authenticating with master master@172.17.0.2:47086 I0331 18:41:04.172279 1721 slave.cpp:996] Using default CRAM-MD5 authenticatee I0331 18:41:04.172277 1725 master.cpp:4802] Received update of slave 2cabf4ec-31e0-4769-9e38-5581d3d8fdfd-S0 at slave(136)@172.17.0.2:47086 (ff176ecfc889) with total oversubscribed resources I0331 18:41:04.172406 1721 slave.cpp:964] Detecting new master I0331 18:41:04.172539 1721 slave.cpp:1105] Registered with master master@172.17.0.2:47086; given slave ID 2cabf4ec-31e0-4769-9e38-5581d3d8fdfd-S0 I0331 18:41:04.172565 1721 fetcher.cpp:81] Clearing fetcher cache I0331 18:41:04.172850 1721 slave.cpp:1128] Checkpointing SlaveInfo to '/tmp/MasterSlaveReconciliationTest_ReconcileRace_Kkms7H/meta/slaves/2cabf4ec-31e0-4769-9e38-5581d3d8fdfd-S0/slave.info' I0331 18:41:04.172998 1725 authenticatee.cpp:121] Creating new client SASL connection I0331 18:41:04.173025 1719 status_update_manager.cpp:181] Resuming sending status updates I0331 18:41:04.173079 1722 hierarchical.cpp:534] Slave 2cabf4ec-31e0-4769-9e38-5581d3d8fdfd-S0 (ff176ecfc889) updated with oversubscribed resources (total: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000], allocated: ) I0331 18:41:04.173269 1721 slave.cpp:1165] Forwarding total oversubscribed resources I0331 18:41:04.173291 1725 master.cpp:5679] Authenticating slave(136)@172.17.0.2:47086 I0331 18:41:04.173270 1722 hierarchical.cpp:1490] No resources available to allocate! I0331 18:41:04.173389 1722 hierarchical.cpp:1164] Performed allocation for slave 2cabf4ec-31e0-4769-9e38-5581d3d8fdfd-S0 in 265667ns I0331 18:41:04.173470 1722 sched.cpp:326] New master detected at master@172.17.0.2:47086 I0331 18:41:04.173542 1722 sched.cpp:382] Authenticating with master master@172.17.0.2:47086 I0331 18:41:04.173563 1722 sched.cpp:389] Using default CRAM-MD5 authenticatee I0331 18:41:04.173826 1722 authenticatee.cpp:121] Creating new client SASL connection I0331 18:41:04.174032 1725 master.cpp:4802] Received update of slave 2cabf4ec-31e0-4769-9e38-5581d3d8fdfd-S0 at slave(136)@172.17.0.2:47086 (ff176ecfc889) with total oversubscribed resources I0331 18:41:04.174213 1727 authenticator.cpp:413] Starting authentication session for crammd5_authenticatee(329)@172.17.0.2:47086 I0331 18:41:04.174413 1725 master.cpp:5679] Authenticating scheduler-5008e686-24e2-4daf-8f86-00b83211de8f@172.17.0.2:47086 I0331 18:41:04.174626 1725 authenticator.cpp:98] Creating new server SASL connection I0331 18:41:04.174813 1725 authenticator.cpp:413] Starting authentication session for crammd5_authenticatee(330)@172.17.0.2:47086 I0331 18:41:04.174892 1722 hierarchical.cpp:534] Slave 2cabf4ec-31e0-4769-9e38-5581d3d8fdfd-S0 (ff176ecfc889) updated with oversubscribed resources (total: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000], allocated: ) I0331 18:41:04.175007 1727 authenticatee.cpp:212] Received SASL authentication mechanisms: CRAM-MD5 I0331 18:41:04.175047 1727 authenticatee.cpp:238] Attempting to authenticate with mechanism 'CRAM-MD5' I0331 18:41:04.175068 1720 authenticator.cpp:98] Creating new server SASL connection I0331 18:41:04.175143 1727 authenticator.cpp:203] Received SASL authentication start I0331 18:41:04.175202 1727 authenticator.cpp:325] Authentication requires more steps I0331 18:41:04.175278 1720 authenticatee.cpp:212] Received SASL authentication mechanisms: CRAM-MD5 I0331 18:41:04.175302 1731 authenticatee.cpp:258] Received SASL authentication step I0331 18:41:04.175314 1720 authenticatee.cpp:238] Attempting to authenticate with mechanism 'CRAM-MD5' I0331 18:41:04.175346 1722 hierarchical.cpp:1490] No resources available to allocate! I0331 18:41:04.175395 1722 hierarchical.cpp:1164] Performed allocation for slave 2cabf4ec-31e0-4769-9e38-5581d3d8fdfd-S0 in 462409ns I0331 18:41:04.175690 1722 replica.cpp:691] Replica received learned notice for position 4 from @0.0.0.0:0 I0331 18:41:04.176033 1720 authenticator.cpp:203] Received SASL authentication start I0331 18:41:04.176082 1720 authenticator.cpp:325] Authentication requires more steps I0331 18:41:04.176134 1731 authenticator.cpp:231] Received SASL authentication step I0331 18:41:04.176158 1720 authenticatee.cpp:258] Received SASL authentication step I0331 18:41:04.176167 1731 auxprop.cpp:107] Request to lookup properties for user: 'test-principal' realm: 'ff176ecfc889' server FQDN: 'ff176ecfc889' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I0331 18:41:04.176185 1731 auxprop.cpp:179] Looking up auxiliary property '*userPassword' I0331 18:41:04.176234 1731 auxprop.cpp:179] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I0331 18:41:04.176241 1720 authenticator.cpp:231] Received SASL authentication step I0331 18:41:04.176265 1731 auxprop.cpp:107] Request to lookup properties for user: 'test-principal' realm: 'ff176ecfc889' server FQDN: 'ff176ecfc889' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I0331 18:41:04.176286 1731 auxprop.cpp:129] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I0331 18:41:04.176301 1731 auxprop.cpp:129] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I0331 18:41:04.176321 1731 authenticator.cpp:317] Authentication success I0331 18:41:04.176267 1720 auxprop.cpp:107] Request to lookup properties for user: 'test-principal' realm: 'ff176ecfc889' server FQDN: 'ff176ecfc889' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I0331 18:41:04.176455 1720 auxprop.cpp:179] Looking up auxiliary property '*userPassword' I0331 18:41:04.176468 1731 authenticatee.cpp:298] Authentication success I0331 18:41:04.176494 1720 auxprop.cpp:179] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I0331 18:41:04.176522 1720 auxprop.cpp:107] Request to lookup properties for user: 'test-principal' realm: 'ff176ecfc889' server FQDN: 'ff176ecfc889' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I0331 18:41:04.176537 1720 auxprop.cpp:129] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I0331 18:41:04.176548 1720 auxprop.cpp:129] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I0331 18:41:04.176545 1731 master.cpp:5709] Successfully authenticated principal 'test-principal' at slave(136)@172.17.0.2:47086 I0331 18:41:04.176565 1720 authenticator.cpp:317] Authentication success I0331 18:41:04.176623 1731 authenticator.cpp:431] Authentication session cleanup for crammd5_authenticatee(329)@172.17.0.2:47086 I0331 18:41:04.176724 1727 authenticatee.cpp:298] Authentication success I0331 18:41:04.176776 1722 leveldb.cpp:341] Persisting action (18 bytes) to leveldb took 1.065537ms I0331 18:41:04.176846 1722 leveldb.cpp:399] Deleting ~2 keys from leveldb took 55804ns I0331 18:41:04.176856 1727 master.cpp:5709] Successfully authenticated principal 'test-principal' at scheduler-5008e686-24e2-4daf-8f86-00b83211de8f@172.17.0.2:47086 I0331 18:41:04.176867 1722 replica.cpp:712] Persisted action at 4 I0331 18:41:04.176982 1722 replica.cpp:697] Replica learned TRUNCATE action at position 4 I0331 18:41:04.177363 1727 sched.cpp:472] Successfully authenticated with master master@172.17.0.2:47086 I0331 18:41:04.177572 1727 sched.cpp:777] Sending SUBSCRIBE call to master@172.17.0.2:47086 I0331 18:41:04.177673 1727 sched.cpp:810] Will retry registration in 630.59737ms if necessary I0331 18:41:04.177528 1731 authenticator.cpp:431] Authentication session cleanup for crammd5_authenticatee(330)@172.17.0.2:47086 I0331 18:41:04.178020 1727 master.cpp:2346] Received SUBSCRIBE call for framework 'default' at scheduler-5008e686-24e2-4daf-8f86-00b83211de8f@172.17.0.2:47086 I0331 18:41:04.178175 1727 master.cpp:1865] Authorizing framework principal 'test-principal' to receive offers for role '*' I0331 18:41:04.178434 1720 slave.cpp:1061] Successfully authenticated with master master@172.17.0.2:47086 I0331 18:41:04.178589 1722 master.cpp:2417] Subscribing framework default with checkpointing disabled and capabilities [ ] I0331 18:41:04.179615 1722 hierarchical.cpp:266] Added framework 2cabf4ec-31e0-4769-9e38-5581d3d8fdfd-0000 I0331 18:41:04.179891 1731 sched.cpp:704] Framework registered with 2cabf4ec-31e0-4769-9e38-5581d3d8fdfd-0000 I0331 18:41:04.179942 1731 sched.cpp:718] Scheduler::registered took 41883ns I0331 18:41:04.181011 1722 hierarchical.cpp:1585] No inverse offers to send out! I0331 18:41:04.181149 1722 hierarchical.cpp:1141] Performed allocation for 1 slaves in 1.518786ms I0331 18:41:04.181681 1722 master.cpp:5508] Sending 1 offers to framework 2cabf4ec-31e0-4769-9e38-5581d3d8fdfd-0000 (default) at scheduler-5008e686-24e2-4daf-8f86-00b83211de8f@172.17.0.2:47086 I0331 18:41:04.182219 1722 sched.cpp:874] Scheduler::resourceOffers took 123860ns I0331 18:41:05.043104 1731 hierarchical.cpp:1490] No resources available to allocate! I0331 18:41:05.043179 1731 hierarchical.cpp:1585] No inverse offers to send out! I0331 18:41:05.043257 1731 hierarchical.cpp:1141] Performed allocation for 1 slaves in 404153ns I0331 18:41:06.043798 1720 hierarchical.cpp:1490] No resources available to allocate! I0331 18:41:06.043874 1720 hierarchical.cpp:1585] No inverse offers to send out! I0331 18:41:06.043961 1720 hierarchical.cpp:1141] Performed allocation for 1 slaves in 400426ns I0331 18:41:07.044714 1720 hierarchical.cpp:1490] No resources available to allocate! I0331 18:41:07.044787 1720 hierarchical.cpp:1585] No inverse offers to send out! I0331 18:41:07.044864 1720 hierarchical.cpp:1141] Performed allocation for 1 slaves in 389076ns I0331 18:41:08.045873 1728 hierarchical.cpp:1490] No resources available to allocate! I0331 18:41:08.045958 1728 hierarchical.cpp:1585] No inverse offers to send out! I0331 18:41:08.046043 1728 hierarchical.cpp:1141] Performed allocation for 1 slaves in 411035ns I0331 18:41:09.047009 1726 hierarchical.cpp:1490] No resources available to allocate! I0331 18:41:09.047085 1726 hierarchical.cpp:1585] No inverse offers to send out! I0331 18:41:09.047169 1726 hierarchical.cpp:1141] Performed allocation for 1 slaves in 414015ns I0331 18:41:10.048630 1728 hierarchical.cpp:1490] No resources available to allocate! I0331 18:41:10.048703 1728 hierarchical.cpp:1585] No inverse offers to send out! I0331 18:41:10.048781 1728 hierarchical.cpp:1141] Performed allocation for 1 slaves in 391902ns I0331 18:41:11.049744 1731 hierarchical.cpp:1490] No resources available to allocate! I0331 18:41:11.049816 1731 hierarchical.cpp:1585] No inverse offers to send out! I0331 18:41:11.049895 1731 hierarchical.cpp:1141] Performed allocation for 1 slaves in 391125ns I0331 18:41:12.050537 1734 hierarchical.cpp:1490] No resources available to allocate! I0331 18:41:12.050611 1734 hierarchical.cpp:1585] No inverse offers to send out! I0331 18:41:12.050689 1734 hierarchical.cpp:1141] Performed allocation for 1 slaves in 395495ns I0331 18:41:13.051849 1724 hierarchical.cpp:1490] No resources available to allocate! I0331 18:41:13.051923 1724 hierarchical.cpp:1585] No inverse offers to send out! I0331 18:41:13.052011 1724 hierarchical.cpp:1141] Performed allocation for 1 slaves in 402378ns I0331 18:41:14.053644 1731 hierarchical.cpp:1490] No resources available to allocate! I0331 18:41:14.053721 1731 hierarchical.cpp:1585] No inverse offers to send out! I0331 18:41:14.053802 1731 hierarchical.cpp:1141] Performed allocation for 1 slaves in 392842ns I0331 18:41:15.055392 1734 hierarchical.cpp:1490] No resources available to allocate! I0331 18:41:15.055465 1734 hierarchical.cpp:1585] No inverse offers to send out! I0331 18:41:15.055544 1734 hierarchical.cpp:1141] Performed allocation for 1 slaves in 392132ns I0331 18:41:16.057188 1724 hierarchical.cpp:1490] No resources available to allocate! I0331 18:41:16.057263 1724 hierarchical.cpp:1585] No inverse offers to send out! I0331 18:41:16.057343 1724 hierarchical.cpp:1141] Performed allocation for 1 slaves in 412204ns I0331 18:41:17.058703 1727 hierarchical.cpp:1490] No resources available to allocate! I0331 18:41:17.058779 1727 hierarchical.cpp:1585] No inverse offers to send out! I0331 18:41:17.058861 1727 hierarchical.cpp:1141] Performed allocation for 1 slaves in 422727ns I0331 18:41:18.059716 1726 hierarchical.cpp:1490] No resources available to allocate! I0331 18:41:18.059792 1726 hierarchical.cpp:1585] No inverse offers to send out! I0331 18:41:18.059872 1726 hierarchical.cpp:1141] Performed allocation for 1 slaves in 394115ns I0331 18:41:19.060812 1725 hierarchical.cpp:1490] No resources available to allocate! I0331 18:41:19.060889 1725 hierarchical.cpp:1585] No inverse offers to send out! I0331 18:41:19.060981 1725 hierarchical.cpp:1141] Performed allocation for 1 slaves in 433161ns I0331 18:41:19.147261 1727 slave.cpp:4945] Querying resource estimator for oversubscribable resources I0331 18:41:19.147531 1727 slave.cpp:4959] Received oversubscribable resources from the resource estimator I0331 18:41:19.168444 1723 slave.cpp:3664] Received ping from slave-observer(139)@172.17.0.2:47086 ../../src/tests/master_slave_reconciliation_tests.cpp:306: Failure Failed to wait 15secs for reregisterSlaveMessage I0331 18:41:19.198300 1732 master.cpp:1269] Framework 2cabf4ec-31e0-4769-9e38-5581d3d8fdfd-0000 (default) at scheduler-5008e686-24e2-4daf-8f86-00b83211de8f@172.17.0.2:47086 disconnected I0331 18:41:19.198372 1732 master.cpp:2642] Disconnecting framework 2cabf4ec-31e0-4769-9e38-5581d3d8fdfd-0000 (default) at scheduler-5008e686-24e2-4daf-8f86-00b83211de8f@172.17.0.2:47086 I0331 18:41:19.198410 1732 master.cpp:2666] Deactivating framework 2cabf4ec-31e0-4769-9e38-5581d3d8fdfd-0000 (default) at scheduler-5008e686-24e2-4daf-8f86-00b83211de8f@172.17.0.2:47086 I0331 18:41:19.199043 1726 hierarchical.cpp:378] Deactivated framework 2cabf4ec-31e0-4769-9e38-5581d3d8fdfd-0000 I0331 18:41:19.199724 1726 hierarchical.cpp:893] Recovered cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] (total: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000], allocated: ) on slave 2cabf4ec-31e0-4769-9e38-5581d3d8fdfd-S0 from framework 2cabf4ec-31e0-4769-9e38-5581d3d8fdfd-0000 I0331 18:41:19.199826 1701 slave.cpp:800] Slave terminating W0331 18:41:19.200469 1732 master.hpp:1822] Master attempted to send message to disconnected framework 2cabf4ec-31e0-4769-9e38-5581d3d8fdfd-0000 (default) at scheduler-5008e686-24e2-4daf-8f86-00b83211de8f@172.17.0.2:47086 I0331 18:41:19.201146 1732 master.cpp:1293] Giving framework 2cabf4ec-31e0-4769-9e38-5581d3d8fdfd-0000 (default) at scheduler-5008e686-24e2-4daf-8f86-00b83211de8f@172.17.0.2:47086 0ns to failover I0331 18:41:19.201282 1732 master.cpp:1230] Slave 2cabf4ec-31e0-4769-9e38-5581d3d8fdfd-S0 at slave(136)@172.17.0.2:47086 (ff176ecfc889) disconnected I0331 18:41:19.201313 1732 master.cpp:2701] Disconnecting slave 2cabf4ec-31e0-4769-9e38-5581d3d8fdfd-S0 at slave(136)@172.17.0.2:47086 (ff176ecfc889) I0331 18:41:19.201364 1732 master.cpp:2720] Deactivating slave 2cabf4ec-31e0-4769-9e38-5581d3d8fdfd-S0 at slave(136)@172.17.0.2:47086 (ff176ecfc889) I0331 18:41:19.201483 1732 master.cpp:5360] Framework failover timeout, removing framework 2cabf4ec-31e0-4769-9e38-5581d3d8fdfd-0000 (default) at scheduler-5008e686-24e2-4daf-8f86-00b83211de8f@172.17.0.2:47086 I0331 18:41:19.201514 1732 master.cpp:6093] Removing framework 2cabf4ec-31e0-4769-9e38-5581d3d8fdfd-0000 (default) at scheduler-5008e686-24e2-4daf-8f86-00b83211de8f@172.17.0.2:47086 I0331 18:41:19.201807 1732 hierarchical.cpp:563] Slave 2cabf4ec-31e0-4769-9e38-5581d3d8fdfd-S0 deactivated I0331 18:41:19.202538 1732 hierarchical.cpp:329] Removed framework 2cabf4ec-31e0-4769-9e38-5581d3d8fdfd-0000 I0331 18:41:19.205302 1701 master.cpp:1083] Master terminating I0331 18:41:19.205778 1730 hierarchical.cpp:508] Removed slave 2cabf4ec-31e0-4769-9e38-5581d3d8fdfd-S0 ../../3rdparty/libprocess/include/process/gmock.hpp:446: Failure Actual function call count doesn't match EXPECT_CALL(filter->mock, filter(testing::A<const MessageEvent&>()))... Expected args: message matcher (8-byte object <C8-70 04-FC 80-2B 00-00>, 1, 1) Expected: to be called once Actual: never called - unsatisfied and active [ FAILED ] MasterSlaveReconciliationTest.ReconcileRace (15193 ms) {code} > MasterSlaveReconciliationTest.ReconcileRace is flaky > ---------------------------------------------------- > > Key: MESOS-2331 > URL: https://issues.apache.org/jira/browse/MESOS-2331 > Project: Mesos > Issue Type: Bug > Components: test > Affects Versions: 0.22.0 > Reporter: Yan Xu > Assignee: Qian Zhang > Labels: flaky > > {noformat:title=} > [ RUN ] MasterSlaveReconciliationTest.ReconcileRace > Using temporary directory > '/tmp/MasterSlaveReconciliationTest_ReconcileRace_NE9nhV' > I0206 19:09:44.196542 32362 leveldb.cpp:175] Opened db in 38.230192ms > I0206 19:09:44.206826 32362 leveldb.cpp:182] Compacted db in 9.988493ms > I0206 19:09:44.207164 32362 leveldb.cpp:197] Created db iterator in 29979ns > I0206 19:09:44.207641 32362 leveldb.cpp:203] Seeked to beginning of db in > 4478ns > I0206 19:09:44.207929 32362 leveldb.cpp:272] Iterated through 0 keys in the > db in 737ns > I0206 19:09:44.208222 32362 replica.cpp:743] Replica recovered with log > positions 0 -> 0 with 1 holes and 0 unlearned > I0206 19:09:44.209132 32384 recover.cpp:448] Starting replica recovery > I0206 19:09:44.209524 32384 recover.cpp:474] Replica is in EMPTY status > I0206 19:09:44.211094 32384 replica.cpp:640] Replica in EMPTY status received > a broadcasted recover request > I0206 19:09:44.211385 32384 recover.cpp:194] Received a recover response from > a replica in EMPTY status > I0206 19:09:44.211902 32384 recover.cpp:565] Updating replica status to > STARTING > I0206 19:09:44.236177 32381 master.cpp:344] Master > 20150206-190944-16842879-36452-32362 (lucid) started on 127.0.1.1:36452 > I0206 19:09:44.236291 32381 master.cpp:390] Master only allowing > authenticated frameworks to register > I0206 19:09:44.236305 32381 master.cpp:395] Master only allowing > authenticated slaves to register > I0206 19:09:44.236327 32381 credentials.hpp:35] Loading credentials for > authentication from > '/tmp/MasterSlaveReconciliationTest_ReconcileRace_NE9nhV/credentials' > I0206 19:09:44.236601 32381 master.cpp:439] Authorization enabled > I0206 19:09:44.238539 32381 hierarchical_allocator_process.hpp:284] > Initialized hierarchical allocator process > I0206 19:09:44.238662 32381 whitelist_watcher.cpp:64] No whitelist given > I0206 19:09:44.239364 32381 master.cpp:1350] The newly elected leader is > master@127.0.1.1:36452 with id 20150206-190944-16842879-36452-32362 > I0206 19:09:44.239392 32381 master.cpp:1363] Elected as the leading master! > I0206 19:09:44.239413 32381 master.cpp:1181] Recovering from registrar > I0206 19:09:44.239645 32381 registrar.cpp:312] Recovering registrar > I0206 19:09:44.241142 32384 leveldb.cpp:305] Persisting metadata (8 bytes) to > leveldb took 29.029117ms > I0206 19:09:44.241189 32384 replica.cpp:322] Persisted replica status to > STARTING > I0206 19:09:44.241478 32384 recover.cpp:474] Replica is in STARTING status > I0206 19:09:44.243075 32384 replica.cpp:640] Replica in STARTING status > received a broadcasted recover request > I0206 19:09:44.243398 32384 recover.cpp:194] Received a recover response from > a replica in STARTING status > I0206 19:09:44.243964 32384 recover.cpp:565] Updating replica status to VOTING > I0206 19:09:44.255692 32384 leveldb.cpp:305] Persisting metadata (8 bytes) to > leveldb took 11.502759ms > I0206 19:09:44.255765 32384 replica.cpp:322] Persisted replica status to > VOTING > I0206 19:09:44.256009 32384 recover.cpp:579] Successfully joined the Paxos > group > I0206 19:09:44.256253 32384 recover.cpp:463] Recover process terminated > I0206 19:09:44.257669 32384 log.cpp:659] Attempting to start the writer > I0206 19:09:44.259944 32377 replica.cpp:476] Replica received implicit > promise request with proposal 1 > I0206 19:09:44.268805 32377 leveldb.cpp:305] Persisting metadata (8 bytes) to > leveldb took 8.45858ms > I0206 19:09:44.269067 32377 replica.cpp:344] Persisted promised to 1 > I0206 19:09:44.277974 32383 coordinator.cpp:229] Coordinator attemping to > fill missing position > I0206 19:09:44.279767 32383 replica.cpp:377] Replica received explicit > promise request for position 0 with proposal 2 > I0206 19:09:44.288940 32383 leveldb.cpp:342] Persisting action (8 bytes) to > leveldb took 9.128603ms > I0206 19:09:44.289294 32383 replica.cpp:678] Persisted action at 0 > I0206 19:09:44.296417 32377 replica.cpp:510] Replica received write request > for position 0 > I0206 19:09:44.296944 32377 leveldb.cpp:437] Reading position from leveldb > took 48457ns > I0206 19:09:44.305337 32377 leveldb.cpp:342] Persisting action (14 bytes) to > leveldb took 8.141689ms > I0206 19:09:44.305662 32377 replica.cpp:678] Persisted action at 0 > I0206 19:09:44.318168 32378 replica.cpp:657] Replica received learned notice > for position 0 > I0206 19:09:44.326036 32378 leveldb.cpp:342] Persisting action (16 bytes) to > leveldb took 6.910907ms > I0206 19:09:44.326654 32378 replica.cpp:678] Persisted action at 0 > I0206 19:09:44.326942 32378 replica.cpp:663] Replica learned NOP action at > position 0 > I0206 19:09:44.338552 32377 log.cpp:675] Writer started with ending position 0 > I0206 19:09:44.340262 32377 leveldb.cpp:437] Reading position from leveldb > took 58420ns > I0206 19:09:44.342963 32377 registrar.cpp:345] Successfully fetched the > registry (0B) in 103.275776ms > I0206 19:09:44.343070 32377 registrar.cpp:444] Applied 1 operations in > 26702ns; attempting to update the 'registry' > I0206 19:09:44.345824 32377 log.cpp:683] Attempting to append 118 bytes to > the log > I0206 19:09:44.346019 32377 coordinator.cpp:339] Coordinator attempting to > write APPEND action at position 1 > I0206 19:09:44.347017 32382 replica.cpp:510] Replica received write request > for position 1 > I0206 19:09:44.354070 32382 leveldb.cpp:342] Persisting action (135 bytes) to > leveldb took 7.011319ms > I0206 19:09:44.354118 32382 replica.cpp:678] Persisted action at 1 > I0206 19:09:44.355094 32382 replica.cpp:657] Replica received learned notice > for position 1 > I0206 19:09:44.358392 32382 leveldb.cpp:342] Persisting action (137 bytes) to > leveldb took 3.267785ms > I0206 19:09:44.358621 32382 replica.cpp:678] Persisted action at 1 > I0206 19:09:44.358949 32382 replica.cpp:663] Replica learned APPEND action at > position 1 > I0206 19:09:44.367099 32382 registrar.cpp:489] Successfully updated the > 'registry' in 23.9488ms > I0206 19:09:44.367259 32382 registrar.cpp:375] Successfully recovered > registrar > I0206 19:09:44.367698 32382 master.cpp:1208] Recovered 0 slaves from the > Registry (82B) ; allowing 10mins for slaves to re-register > I0206 19:09:44.367790 32382 log.cpp:702] Attempting to truncate the log to 1 > I0206 19:09:44.368067 32382 coordinator.cpp:339] Coordinator attempting to > write TRUNCATE action at position 2 > I0206 19:09:44.369544 32381 replica.cpp:510] Replica received write request > for position 2 > I0206 19:09:44.378540 32381 leveldb.cpp:342] Persisting action (16 bytes) to > leveldb took 8.940568ms > I0206 19:09:44.378811 32381 replica.cpp:678] Persisted action at 2 > I0206 19:09:44.387471 32377 replica.cpp:657] Replica received learned notice > for position 2 > I0206 19:09:44.394693 32377 leveldb.cpp:342] Persisting action (18 bytes) to > leveldb took 6.892787ms > I0206 19:09:44.394815 32377 leveldb.cpp:400] Deleting ~1 keys from leveldb > took 57353ns > I0206 19:09:44.394840 32377 replica.cpp:678] Persisted action at 2 > I0206 19:09:44.394875 32377 replica.cpp:663] Replica learned TRUNCATE action > at position 2 > I0206 19:09:44.423720 32380 slave.cpp:172] Slave started on > 14)@127.0.1.1:36452 > I0206 19:09:44.426247 32380 credentials.hpp:83] Loading credential for > authentication from > '/tmp/MasterSlaveReconciliationTest_ReconcileRace_nT15cx/credential' > I0206 19:09:44.426745 32380 slave.cpp:281] Slave using credential for: > test-principal > I0206 19:09:44.427234 32380 slave.cpp:299] Slave resources: cpus(*):2; > mem(*):1024; disk(*):1024; ports(*):[31000-32000] > I0206 19:09:44.427616 32380 slave.cpp:328] Slave hostname: lucid > I0206 19:09:44.427829 32380 slave.cpp:329] Slave checkpoint: false > W0206 19:09:44.428047 32380 slave.cpp:331] Disabling checkpointing is > deprecated and the --checkpoint flag will be removed in a future release. > Please avoid using this flag > I0206 19:09:44.429821 32379 state.cpp:32] Recovering state from > '/tmp/MasterSlaveReconciliationTest_ReconcileRace_nT15cx/meta' > I0206 19:09:44.438045 32380 status_update_manager.cpp:196] Recovering status > update manager > I0206 19:09:44.438377 32380 slave.cpp:3526] Finished recovery > I0206 19:09:44.439144 32380 slave.cpp:620] New master detected at > master@127.0.1.1:36452 > I0206 19:09:44.439275 32380 slave.cpp:683] Authenticating with master > master@127.0.1.1:36452 > I0206 19:09:44.439297 32380 slave.cpp:688] Using default CRAM-MD5 > authenticatee > I0206 19:09:44.439472 32380 slave.cpp:656] Detecting new master > I0206 19:09:44.439589 32380 status_update_manager.cpp:170] Pausing sending > status updates > I0206 19:09:44.439757 32380 authenticatee.hpp:137] Creating new client SASL > connection > I0206 19:09:44.440255 32380 master.cpp:3786] Authenticating > slave(14)@127.0.1.1:36452 > I0206 19:09:44.440287 32380 master.cpp:3797] Using default CRAM-MD5 > authenticator > I0206 19:09:44.440712 32380 authenticator.hpp:169] Creating new server SASL > connection > I0206 19:09:44.441010 32380 authenticatee.hpp:228] Received SASL > authentication mechanisms: CRAM-MD5 > I0206 19:09:44.441036 32380 authenticatee.hpp:254] Attempting to authenticate > with mechanism 'CRAM-MD5' > I0206 19:09:44.441136 32380 authenticator.hpp:275] Received SASL > authentication start > I0206 19:09:44.441216 32380 authenticator.hpp:397] Authentication requires > more steps > I0206 19:09:44.441303 32380 authenticatee.hpp:274] Received SASL > authentication step > I0206 19:09:44.441395 32380 authenticator.hpp:303] Received SASL > authentication step > I0206 19:09:44.441431 32380 auxprop.cpp:98] Request to lookup properties for > user: 'test-principal' realm: 'lucid' server FQDN: 'lucid' > SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false > I0206 19:09:44.441442 32380 auxprop.cpp:170] Looking up auxiliary property > '*userPassword' > I0206 19:09:44.441493 32380 auxprop.cpp:170] Looking up auxiliary property > '*cmusaslsecretCRAM-MD5' > I0206 19:09:44.441526 32380 auxprop.cpp:98] Request to lookup properties for > user: 'test-principal' realm: 'lucid' server FQDN: 'lucid' > SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true > I0206 19:09:44.441540 32380 auxprop.cpp:120] Skipping auxiliary property > '*userPassword' since SASL_AUXPROP_AUTHZID == true > I0206 19:09:44.441546 32380 auxprop.cpp:120] Skipping auxiliary property > '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true > I0206 19:09:44.441561 32380 authenticator.hpp:389] Authentication success > I0206 19:09:44.441668 32380 authenticatee.hpp:314] Authentication success > I0206 19:09:44.441756 32380 master.cpp:3844] Successfully authenticated > principal 'test-principal' at slave(14)@127.0.1.1:36452 > I0206 19:09:44.442070 32380 slave.cpp:754] Successfully authenticated with > master master@127.0.1.1:36452 > I0206 19:09:44.442211 32380 slave.cpp:1082] Will retry registration in > 19.488037ms if necessary > I0206 19:09:44.442474 32380 master.cpp:2911] Registering slave at > slave(14)@127.0.1.1:36452 (lucid) with id > 20150206-190944-16842879-36452-32362-S0 > I0206 19:09:44.442922 32380 registrar.cpp:444] Applied 1 operations in > 72989ns; attempting to update the 'registry' > I0206 19:09:44.445914 32380 log.cpp:683] Attempting to append 283 bytes to > the log > I0206 19:09:44.446125 32380 coordinator.cpp:339] Coordinator attempting to > write APPEND action at position 3 > I0206 19:09:44.448060 32384 replica.cpp:510] Replica received write request > for position 3 > I0206 19:09:44.450963 32384 leveldb.cpp:342] Persisting action (302 bytes) to > leveldb took 2.859704ms > I0206 19:09:44.451252 32384 replica.cpp:678] Persisted action at 3 > I0206 19:09:44.452453 32382 replica.cpp:657] Replica received learned notice > for position 3 > I0206 19:09:44.463752 32382 leveldb.cpp:342] Persisting action (304 bytes) to > leveldb took 11.254934ms > I0206 19:09:44.463817 32382 replica.cpp:678] Persisted action at 3 > I0206 19:09:44.463848 32382 replica.cpp:663] Replica learned APPEND action at > position 3 > I0206 19:09:44.465450 32382 registrar.cpp:489] Successfully updated the > 'registry' in 22.45888ms > I0206 19:09:44.465725 32382 log.cpp:702] Attempting to truncate the log to 3 > I0206 19:09:44.466614 32379 coordinator.cpp:339] Coordinator attempting to > write TRUNCATE action at position 4 > I0206 19:09:44.466958 32379 slave.cpp:1082] Will retry registration in > 34.738916ms if necessary > I0206 19:09:44.467807 32379 replica.cpp:510] Replica received write request > for position 4 > I0206 19:09:44.469007 32381 slave.cpp:2595] Received ping from > slave-observer(14)@127.0.1.1:36452 > I0206 19:09:44.469740 32377 hierarchical_allocator_process.hpp:450] Added > slave 20150206-190944-16842879-36452-32362-S0 (lucid) with cpus(*):2; > mem(*):1024; disk(*):1024; ports(*):[31000-32000] (and cpus(*):2; > mem(*):1024; disk(*):1024; ports(*):[31000-32000] available) > I0206 19:09:44.469871 32377 hierarchical_allocator_process.hpp:828] No > resources available to allocate! > I0206 19:09:44.469892 32377 hierarchical_allocator_process.hpp:753] Performed > allocation for slave 20150206-190944-16842879-36452-32362-S0 in 107661ns > I0206 19:09:44.470643 32378 slave.cpp:788] Registered with master > master@127.0.1.1:36452; given slave ID 20150206-190944-16842879-36452-32362-S0 > I0206 19:09:44.473881 32383 status_update_manager.cpp:177] Resuming sending > status updates > I0206 19:09:44.476253 32379 leveldb.cpp:342] Persisting action (16 bytes) to > leveldb took 8.4136ms > I0206 19:09:44.476297 32379 replica.cpp:678] Persisted action at 4 > I0206 19:09:44.488441 32382 master.cpp:2968] Registered slave > 20150206-190944-16842879-36452-32362-S0 at slave(14)@127.0.1.1:36452 (lucid) > with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] > I0206 19:09:44.489220 32382 master.cpp:2887] Slave > 20150206-190944-16842879-36452-32362-S0 at slave(14)@127.0.1.1:36452 (lucid) > already registered, resending acknowledgement > I0206 19:09:44.490684 32379 replica.cpp:657] Replica received learned notice > for position 4 > I0206 19:09:44.491462 32362 sched.cpp:149] Version: 0.22.0 > I0206 19:09:44.494812 32378 slave.cpp:620] New master detected at > master@127.0.1.1:36452 > I0206 19:09:44.494870 32378 slave.cpp:683] Authenticating with master > master@127.0.1.1:36452 > I0206 19:09:44.494889 32378 slave.cpp:688] Using default CRAM-MD5 > authenticatee > I0206 19:09:44.495014 32378 slave.cpp:656] Detecting new master > I0206 19:09:44.495160 32378 sched.cpp:246] New master detected at > master@127.0.1.1:36452 > I0206 19:09:44.495232 32378 sched.cpp:302] Authenticating with master > master@127.0.1.1:36452 > I0206 19:09:44.495252 32378 sched.cpp:309] Using default CRAM-MD5 > authenticatee > I0206 19:09:44.495458 32378 status_update_manager.cpp:170] Pausing sending > status updates > I0206 19:09:44.495589 32378 authenticatee.hpp:137] Creating new client SASL > connection > I0206 19:09:44.495846 32378 authenticatee.hpp:137] Creating new client SASL > connection > I0206 19:09:44.498770 32382 master.cpp:3786] Authenticating > slave(14)@127.0.1.1:36452 > I0206 19:09:44.498812 32382 master.cpp:3797] Using default CRAM-MD5 > authenticator > I0206 19:09:44.499124 32382 master.cpp:3786] Authenticating > scheduler-167ce2f8-8268-41f7-9c72-a4e29dc3316c@127.0.1.1:36452 > I0206 19:09:44.499151 32382 master.cpp:3797] Using default CRAM-MD5 > authenticator > I0206 19:09:44.499363 32382 slave.cpp:788] Registered with master > master@127.0.1.1:36452; given slave ID 20150206-190944-16842879-36452-32362-S0 > I0206 19:09:44.499675 32382 authenticator.hpp:169] Creating new server SASL > connection > I0206 19:09:44.499925 32382 authenticator.hpp:169] Creating new server SASL > connection > I0206 19:09:44.500035 32382 status_update_manager.cpp:177] Resuming sending > status updates > I0206 19:09:44.500330 32382 authenticatee.hpp:228] Received SASL > authentication mechanisms: CRAM-MD5 > I0206 19:09:44.500356 32382 authenticatee.hpp:254] Attempting to authenticate > with mechanism 'CRAM-MD5' > I0206 19:09:44.500501 32382 authenticatee.hpp:228] Received SASL > authentication mechanisms: CRAM-MD5 > I0206 19:09:44.500526 32382 authenticatee.hpp:254] Attempting to authenticate > with mechanism 'CRAM-MD5' > I0206 19:09:44.500691 32382 authenticator.hpp:275] Received SASL > authentication start > I0206 19:09:44.500758 32382 authenticator.hpp:397] Authentication requires > more steps > I0206 19:09:44.500836 32382 authenticator.hpp:275] Received SASL > authentication start > I0206 19:09:44.500874 32382 authenticator.hpp:397] Authentication requires > more steps > I0206 19:09:44.500943 32382 authenticatee.hpp:274] Received SASL > authentication step > I0206 19:09:44.501026 32382 authenticatee.hpp:274] Received SASL > authentication step > I0206 19:09:44.501104 32382 authenticator.hpp:303] Received SASL > authentication step > I0206 19:09:44.501130 32382 auxprop.cpp:98] Request to lookup properties for > user: 'test-principal' realm: 'lucid' server FQDN: 'lucid' > SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false > I0206 19:09:44.501142 32382 auxprop.cpp:170] Looking up auxiliary property > '*userPassword' > I0206 19:09:44.501188 32382 auxprop.cpp:170] Looking up auxiliary property > '*cmusaslsecretCRAM-MD5' > I0206 19:09:44.501212 32382 auxprop.cpp:98] Request to lookup properties for > user: 'test-principal' realm: 'lucid' server FQDN: 'lucid' > SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true > I0206 19:09:44.501224 32382 auxprop.cpp:120] Skipping auxiliary property > '*userPassword' since SASL_AUXPROP_AUTHZID == true > I0206 19:09:44.501230 32382 auxprop.cpp:120] Skipping auxiliary property > '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true > I0206 19:09:44.501245 32382 authenticator.hpp:389] Authentication success > I0206 19:09:44.501368 32382 authenticator.hpp:303] Received SASL > authentication step > I0206 19:09:44.501394 32382 auxprop.cpp:98] Request to lookup properties for > user: 'test-principal' realm: 'lucid' server FQDN: 'lucid' > SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false > I0206 19:09:44.501405 32382 auxprop.cpp:170] Looking up auxiliary property > '*userPassword' > I0206 19:09:44.501435 32382 auxprop.cpp:170] Looking up auxiliary property > '*cmusaslsecretCRAM-MD5' > I0206 19:09:44.501458 32382 auxprop.cpp:98] Request to lookup properties for > user: 'test-principal' realm: 'lucid' server FQDN: 'lucid' > SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true > I0206 19:09:44.501469 32382 auxprop.cpp:120] Skipping auxiliary property > '*userPassword' since SASL_AUXPROP_AUTHZID == true > I0206 19:09:44.501477 32382 auxprop.cpp:120] Skipping auxiliary property > '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true > I0206 19:09:44.501489 32382 authenticator.hpp:389] Authentication success > I0206 19:09:44.501582 32382 authenticatee.hpp:314] Authentication success > I0206 19:09:44.501672 32382 master.cpp:3844] Successfully authenticated > principal 'test-principal' at slave(14)@127.0.1.1:36452 > I0206 19:09:44.501898 32382 master.cpp:3844] Successfully authenticated > principal 'test-principal' at > scheduler-167ce2f8-8268-41f7-9c72-a4e29dc3316c@127.0.1.1:36452 > I0206 19:09:44.502104 32382 authenticatee.hpp:314] Authentication success > I0206 19:09:44.502272 32382 slave.cpp:754] Successfully authenticated with > master master@127.0.1.1:36452 > I0206 19:09:44.502421 32382 sched.cpp:390] Successfully authenticated with > master master@127.0.1.1:36452 > I0206 19:09:44.502447 32382 sched.cpp:513] Sending registration request to > master@127.0.1.1:36452 > I0206 19:09:44.502512 32382 sched.cpp:546] Will retry registration in > 1.990458655secs if necessary > I0206 19:09:44.502679 32382 master.cpp:1568] Received registration request > for framework 'default' at > scheduler-167ce2f8-8268-41f7-9c72-a4e29dc3316c@127.0.1.1:36452 > I0206 19:09:44.502760 32382 master.cpp:1429] Authorizing framework principal > 'test-principal' to receive offers for role '*' > I0206 19:09:44.503207 32379 leveldb.cpp:342] Persisting action (18 bytes) to > leveldb took 12.488335ms > I0206 19:09:44.503340 32379 leveldb.cpp:400] Deleting ~2 keys from leveldb > took 95587ns > I0206 19:09:44.503365 32379 replica.cpp:678] Persisted action at 4 > I0206 19:09:44.503394 32379 replica.cpp:663] Replica learned TRUNCATE action > at position 4 > I0206 19:09:44.504719 32382 master.cpp:1632] Registering framework > 20150206-190944-16842879-36452-32362-0000 (default) at > scheduler-167ce2f8-8268-41f7-9c72-a4e29dc3316c@127.0.1.1:36452 > I0206 19:09:44.505327 32380 hierarchical_allocator_process.hpp:318] Added > framework 20150206-190944-16842879-36452-32362-0000 > I0206 19:09:44.505712 32380 hierarchical_allocator_process.hpp:735] Performed > allocation for 1 slaves in 353338ns > I0206 19:09:44.517038 32382 master.cpp:3728] Sending 1 offers to framework > 20150206-190944-16842879-36452-32362-0000 (default) at > scheduler-167ce2f8-8268-41f7-9c72-a4e29dc3316c@127.0.1.1:36452 > I0206 19:09:44.517523 32382 sched.cpp:440] Framework registered with > 20150206-190944-16842879-36452-32362-0000 > I0206 19:09:44.517575 32382 sched.cpp:454] Scheduler::registered took 26978ns > I0206 19:09:44.517899 32382 sched.cpp:603] Scheduler::resourceOffers took > 48926ns > I0206 19:09:45.246789 32381 hierarchical_allocator_process.hpp:828] No > resources available to allocate! > I0206 19:09:45.246841 32381 hierarchical_allocator_process.hpp:735] Performed > allocation for 1 slaves in 239355ns > I0206 19:09:46.257019 32383 hierarchical_allocator_process.hpp:828] No > resources available to allocate! > I0206 19:09:46.257325 32383 hierarchical_allocator_process.hpp:735] Performed > allocation for 1 slaves in 642985ns > I0206 19:09:47.259109 32382 hierarchical_allocator_process.hpp:828] No > resources available to allocate! > I0206 19:09:47.259160 32382 hierarchical_allocator_process.hpp:735] Performed > allocation for 1 slaves in 247172ns > I0206 19:09:48.267207 32378 hierarchical_allocator_process.hpp:828] No > resources available to allocate! > I0206 19:09:48.267590 32378 hierarchical_allocator_process.hpp:735] Performed > allocation for 1 slaves in 737879ns > I0206 19:09:49.276871 32379 hierarchical_allocator_process.hpp:828] No > resources available to allocate! > I0206 19:09:49.276924 32379 hierarchical_allocator_process.hpp:735] Performed > allocation for 1 slaves in 243974ns > I0206 19:09:50.287142 32382 hierarchical_allocator_process.hpp:828] No > resources available to allocate! > I0206 19:09:50.287504 32382 hierarchical_allocator_process.hpp:735] Performed > allocation for 1 slaves in 784304ns > I0206 19:09:51.296839 32379 hierarchical_allocator_process.hpp:828] No > resources available to allocate! > I0206 19:09:51.296893 32379 hierarchical_allocator_process.hpp:735] Performed > allocation for 1 slaves in 244814ns > I0206 19:09:52.297719 32378 hierarchical_allocator_process.hpp:828] No > resources available to allocate! > I0206 19:09:52.299626 32378 hierarchical_allocator_process.hpp:735] Performed > allocation for 1 slaves in 2.277105ms > I0206 19:09:53.306870 32384 hierarchical_allocator_process.hpp:828] No > resources available to allocate! > I0206 19:09:53.306924 32384 hierarchical_allocator_process.hpp:735] Performed > allocation for 1 slaves in 262416ns > I0206 19:09:54.317059 32382 hierarchical_allocator_process.hpp:828] No > resources available to allocate! > I0206 19:09:54.320051 32382 hierarchical_allocator_process.hpp:735] Performed > allocation for 1 slaves in 3.270301ms > I0206 19:09:55.326977 32379 hierarchical_allocator_process.hpp:828] No > resources available to allocate! > I0206 19:09:55.330344 32379 hierarchical_allocator_process.hpp:735] Performed > allocation for 1 slaves in 3.569401ms > I0206 19:09:56.339260 32381 hierarchical_allocator_process.hpp:828] No > resources available to allocate! > I0206 19:09:56.343905 32381 hierarchical_allocator_process.hpp:735] Performed > allocation for 1 slaves in 5.144928ms > I0206 19:09:57.346943 32377 hierarchical_allocator_process.hpp:828] No > resources available to allocate! > I0206 19:09:57.347012 32377 hierarchical_allocator_process.hpp:735] Performed > allocation for 1 slaves in 326655ns > I0206 19:09:58.348335 32380 hierarchical_allocator_process.hpp:828] No > resources available to allocate! > I0206 19:09:58.351338 32380 hierarchical_allocator_process.hpp:735] Performed > allocation for 1 slaves in 3.622285ms > I0206 19:09:59.356886 32383 hierarchical_allocator_process.hpp:828] No > resources available to allocate! > I0206 19:09:59.356941 32383 hierarchical_allocator_process.hpp:735] Performed > allocation for 1 slaves in 271092ns > I0206 19:09:59.477521 32381 slave.cpp:2595] Received ping from > slave-observer(14)@127.0.1.1:36452 > tests/master_slave_reconciliation_tests.cpp:283: Failure > Failed to wait 15secs for reregisterSlaveMessage > I0206 19:09:59.510076 32377 master.cpp:872] Framework > 20150206-190944-16842879-36452-32362-0000 (default) at > scheduler-167ce2f8-8268-41f7-9c72-a4e29dc3316c@127.0.1.1:36452 disconnected > I0206 19:09:59.510154 32377 master.cpp:1937] Disconnecting framework > 20150206-190944-16842879-36452-32362-0000 (default) at > scheduler-167ce2f8-8268-41f7-9c72-a4e29dc3316c@127.0.1.1:36452 > I0206 19:09:59.510216 32377 master.cpp:1953] Deactivating framework > 20150206-190944-16842879-36452-32362-0000 (default) at > scheduler-167ce2f8-8268-41f7-9c72-a4e29dc3316c@127.0.1.1:36452 > I0206 19:09:59.511065 32377 master.cpp:894] Giving framework > 20150206-190944-16842879-36452-32362-0000 (default) at > scheduler-167ce2f8-8268-41f7-9c72-a4e29dc3316c@127.0.1.1:36452 0ns to failover > I0206 19:09:59.512804 32383 hierarchical_allocator_process.hpp:397] > Deactivated framework 20150206-190944-16842879-36452-32362-0000 > I0206 19:09:59.512990 32383 hierarchical_allocator_process.hpp:642] Recovered > cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] (total > allocatable: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000]) on > slave 20150206-190944-16842879-36452-32362-S0 from framework > 20150206-190944-16842879-36452-32362-0000 > I0206 19:09:59.513653 32382 slave.cpp:609] Re-detecting master > I0206 19:09:59.513687 32382 slave.cpp:656] Detecting new master > *** Aborted at 1423278599 (unix time) try "date -d @1423278599" if you are > using GNU date *** > I0206 19:09:59.514680 32380 status_update_manager.cpp:170] Pausing sending > status updates > I0206 19:09:59.516027 32379 master.cpp:781] Master terminating > PC: @ 0x2b2b91358b79 mesos::slave::Slave::detected() > *** SIGSEGV (@0x10) received by PID 32362 (TID 0x2b2b98dad700) from PID 16; > stack trace: *** > @ 0x2b2b9b6ff3b4 os::Linux::chained_handler() > @ 0x2b2b9b70459a JVM_handle_linux_signal > @ 0x2b2b8f0a98f0 (unknown) > @ 0x2b2b91358b79 mesos::slave::Slave::detected() > @ 0x2b2b913d31b4 std::tr1::_Mem_fn<>::operator()() > @ 0x2b2b913cd54f > _ZNSt3tr15_BindIFNS_7_Mem_fnIMN5mesos5slave5SlaveEFvRKN7process6FutureI6OptionINS2_10MasterInfoEEEEEEENS_12_PlaceholderILi1EEESA_EE6__callIIRPS4_EILi0ELi1EEEENS_9result_ofIFSF_NSN_IFNS_3_MuISH_Lb0ELb1EEESH_NS_5tupleIIDpT_EEEEE4typeENSN_IFNSO_ISA_Lb0ELb0EEESA_ST_EE4typeEEE4typeERKST_NS_12_Index_tupleIIXspT0_EEEE > @ 0x2b2b913c3216 std::tr1::_Bind<>::operator()<>() > @ 0x2b2b913b6a7e std::tr1::_Function_handler<>::_M_invoke() > @ 0x7f6b9a std::tr1::function<>::operator()() > @ 0x7e765c process::internal::vdispatcher<>() > @ 0x80e823 > _ZNSt3tr15_BindIFPFvPN7process11ProcessBaseENS_10shared_ptrINS_8functionIFvPN5mesos5slave5SlaveEEEEEEENS_12_PlaceholderILi1EEESC_EE6__callIIRS3_EILi0ELi1EEEENS_9result_ofIFSE_NSL_IFNS_3_MuISG_Lb0ELb1EEESG_NS_5tupleIIDpT_EEEEE4typeENSL_IFNSM_ISC_Lb0ELb0EEESC_SR_EE4typeEEE4typeERKSR_NS_12_Index_tupleIIXspT0_EEEE > @ 0x803b6c std::tr1::_Bind<>::operator()<>() > @ 0x7f6c86 std::tr1::_Function_handler<>::_M_invoke() > @ 0x2b2b91a25714 std::tr1::function<>::operator()() > @ 0x2b2b91a0ae4f process::ProcessBase::visit() > @ 0x2b2b91a0f3a0 process::DispatchEvent::visit() > @ 0x8a56f2 process::ProcessBase::serve() > @ 0x2b2b91a06fc2 process::ProcessManager::resume() > @ 0x2b2b919fabfc process::schedule() > @ 0x2b2b8f0a09ca start_thread > @ 0x2b2b94ab745d (unknown) > make[3]: *** [check-local] Segmentation fault > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)