[ https://issues.apache.org/jira/browse/MESOS-5844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Neil Conway updated MESOS-5844: ------------------------------- Shepherd: Alexander Rukletsov > PersistentVolumeEndpointsTest.OfferCreateThenEndpointRemove test is flaky > ------------------------------------------------------------------------- > > Key: MESOS-5844 > URL: https://issues.apache.org/jira/browse/MESOS-5844 > Project: Mesos > Issue Type: Bug > Affects Versions: 1.0.0 > Environment: ASF CI > Reporter: Vinod Kone > Assignee: Neil Conway > > Observed on ASF CI: > https://builds.apache.org/job/Mesos/BUILDTOOL=autotools,COMPILER=gcc,CONFIGURATION=--verbose,ENVIRONMENT=GLOG_v=1%20MESOS_VERBOSE=1,OS=ubuntu%3A14.04,label_exp=(docker%7C%7CHadoop)&&(!ubuntu-us1)&&(!ubuntu-6)/2497/changes > {code} > [ RUN ] PersistentVolumeEndpointsTest.OfferCreateThenEndpointRemove > I0713 18:43:55.968503 28220 cluster.cpp:155] Creating default 'local' > authorizer > I0713 18:43:56.082345 28220 leveldb.cpp:174] Opened db in 113.403661ms > I0713 18:43:56.131445 28220 leveldb.cpp:181] Compacted db in 49.034774ms > I0713 18:43:56.131533 28220 leveldb.cpp:196] Created db iterator in 28012ns > I0713 18:43:56.131552 28220 leveldb.cpp:202] Seeked to beginning of db in > 3046ns > I0713 18:43:56.131564 28220 leveldb.cpp:271] Iterated through 0 keys in the > db in 255ns > I0713 18:43:56.131614 28220 replica.cpp:779] Replica recovered with log > positions 0 -> 0 with 1 holes and 0 unlearned > I0713 18:43:56.134064 28246 recover.cpp:451] Starting replica recovery > I0713 18:43:56.134627 28246 recover.cpp:477] Replica is in EMPTY status > I0713 18:43:56.136396 28252 replica.cpp:673] Replica in EMPTY status received > a broadcasted recover request from (9553)@172.17.0.8:35418 > I0713 18:43:56.136759 28252 recover.cpp:197] Received a recover response from > a replica in EMPTY status > I0713 18:43:56.137676 28246 recover.cpp:568] Updating replica status to > STARTING > I0713 18:43:56.148720 28242 master.cpp:382] Master > 2258d072-b0c9-4c40-874c-6cf933ee345a (500c3e866abe) started on > 172.17.0.8:35418 > I0713 18:43:56.148759 28242 master.cpp:384] Flags at startup: --acls="" > --agent_ping_timeout="15secs" --agent_reregister_timeout="10mins" > --allocation_interval="50ms" --allocator="HierarchicalDRF" > --authenticate_agents="true" --authenticate_frameworks="true" > --authenticate_http="true" --authenticate_http_frameworks="true" > --authenticators="crammd5" --authorizers="local" > --credentials="/tmp/LrwRl4/credentials" --framework_sorter="drf" > --help="false" --hostname_lookup="true" --http_authenticators="basic" > --http_framework_authenticators="basic" --initialize_driver_logging="true" > --log_auto_initialize="true" --logbufsecs="0" --logging_level="INFO" > --max_agent_ping_timeouts="5" --max_completed_frameworks="50" > --max_completed_tasks_per_framework="1000" --quiet="false" > --recovery_agent_removal_limit="100%" --registry="replicated_log" > --registry_fetch_timeout="1mins" --registry_store_timeout="100secs" > --registry_strict="true" --roles="role1" --root_submissions="true" > --user_sorter="drf" --version="false" > --webui_dir="/mesos/mesos-1.1.0/_inst/share/mesos/webui" > --work_dir="/tmp/LrwRl4/master" --zk_session_timeout="10secs" > I0713 18:43:56.149247 28242 master.cpp:434] Master only allowing > authenticated frameworks to register > I0713 18:43:56.149265 28242 master.cpp:448] Master only allowing > authenticated agents to register > I0713 18:43:56.149273 28242 master.cpp:461] Master only allowing > authenticated HTTP frameworks to register > I0713 18:43:56.149283 28242 credentials.hpp:37] Loading credentials for > authentication from '/tmp/LrwRl4/credentials' > I0713 18:43:56.149780 28242 master.cpp:506] Using default 'crammd5' > authenticator > I0713 18:43:56.149940 28242 master.cpp:578] Using default 'basic' HTTP > authenticator > I0713 18:43:56.150091 28242 master.cpp:658] Using default 'basic' HTTP > framework authenticator > I0713 18:43:56.150209 28242 master.cpp:705] Authorization enabled > W0713 18:43:56.150233 28242 master.cpp:768] The '--roles' flag is deprecated. > This flag will be removed in the future. See the Mesos 0.27 upgrade notes for > more information > I0713 18:43:56.150760 28240 hierarchical.cpp:151] Initialized hierarchical > allocator process > I0713 18:43:56.151018 28249 whitelist_watcher.cpp:77] No whitelist given > I0713 18:43:56.155668 28242 master.cpp:1973] The newly elected leader is > master@172.17.0.8:35418 with id 2258d072-b0c9-4c40-874c-6cf933ee345a > I0713 18:43:56.155781 28242 master.cpp:1986] Elected as the leading master! > I0713 18:43:56.155848 28242 master.cpp:1673] Recovering from registrar > I0713 18:43:56.156065 28254 registrar.cpp:332] Recovering registrar > I0713 18:43:56.201568 28245 hierarchical.cpp:1537] No allocations performed > I0713 18:43:56.201666 28245 hierarchical.cpp:1172] Performed allocation for 0 > agents in 167962ns > I0713 18:43:56.218626 28246 leveldb.cpp:304] Persisting metadata (8 bytes) to > leveldb took 80.746657ms > I0713 18:43:56.218705 28246 replica.cpp:320] Persisted replica status to > STARTING > I0713 18:43:56.219219 28246 recover.cpp:477] Replica is in STARTING status > I0713 18:43:56.221391 28246 replica.cpp:673] Replica in STARTING status > received a broadcasted recover request from (9556)@172.17.0.8:35418 > I0713 18:43:56.221869 28253 recover.cpp:197] Received a recover response from > a replica in STARTING status > I0713 18:43:56.222760 28249 recover.cpp:568] Updating replica status to VOTING > I0713 18:43:56.252303 28254 hierarchical.cpp:1537] No allocations performed > I0713 18:43:56.252404 28254 hierarchical.cpp:1172] Performed allocation for 0 > agents in 167038ns > I0713 18:43:56.270256 28249 leveldb.cpp:304] Persisting metadata (8 bytes) to > leveldb took 47.316392ms > I0713 18:43:56.270387 28249 replica.cpp:320] Persisted replica status to > VOTING > I0713 18:43:56.270700 28250 recover.cpp:582] Successfully joined the Paxos > group > I0713 18:43:56.271121 28250 recover.cpp:466] Recover process terminated > I0713 18:43:56.271503 28248 log.cpp:553] Attempting to start the writer > I0713 18:43:56.273140 28240 replica.cpp:493] Replica received implicit > promise request from (9557)@172.17.0.8:35418 with proposal 1 > I0713 18:43:56.303086 28254 hierarchical.cpp:1537] No allocations performed > I0713 18:43:56.303175 28254 hierarchical.cpp:1172] Performed allocation for 0 > agents in 155905ns > I0713 18:43:56.312978 28240 leveldb.cpp:304] Persisting metadata (8 bytes) to > leveldb took 39.718643ms > I0713 18:43:56.313405 28240 replica.cpp:342] Persisted promised to 1 > I0713 18:43:56.314775 28245 coordinator.cpp:238] Coordinator attempting to > fill missing positions > I0713 18:43:56.316547 28250 replica.cpp:388] Replica received explicit > promise request from (9558)@172.17.0.8:35418 for position 0 with proposal 2 > I0713 18:43:56.354794 28239 hierarchical.cpp:1537] No allocations performed > I0713 18:43:56.354898 28239 hierarchical.cpp:1172] Performed allocation for 0 > agents in 178033ns > I0713 18:43:56.363484 28250 leveldb.cpp:341] Persisting action (8 bytes) to > leveldb took 46.846904ms > I0713 18:43:56.363585 28250 replica.cpp:712] Persisted action at 0 > I0713 18:43:56.365622 28250 replica.cpp:537] Replica received write request > for position 0 from (9559)@172.17.0.8:35418 > I0713 18:43:56.365727 28250 leveldb.cpp:436] Reading position from leveldb > took 45172ns > I0713 18:43:56.406314 28252 hierarchical.cpp:1537] No allocations performed > I0713 18:43:56.406421 28252 hierarchical.cpp:1172] Performed allocation for 0 > agents in 177001ns > I0713 18:43:56.421867 28250 leveldb.cpp:341] Persisting action (14 bytes) to > leveldb took 56.06514ms > I0713 18:43:56.421968 28250 replica.cpp:712] Persisted action at 0 > I0713 18:43:56.423286 28254 replica.cpp:691] Replica received learned notice > for position 0 from @0.0.0.0:0 > I0713 18:43:56.458665 28250 hierarchical.cpp:1537] No allocations performed > I0713 18:43:56.458799 28250 hierarchical.cpp:1172] Performed allocation for 0 > agents in 250863ns > I0713 18:43:56.470486 28254 leveldb.cpp:341] Persisting action (16 bytes) to > leveldb took 47.13918ms > I0713 18:43:56.470552 28254 replica.cpp:712] Persisted action at 0 > I0713 18:43:56.470584 28254 replica.cpp:697] Replica learned NOP action at > position 0 > I0713 18:43:56.471782 28247 log.cpp:569] Writer started with ending position 0 > I0713 18:43:56.475908 28253 leveldb.cpp:436] Reading position from leveldb > took 79764ns > I0713 18:43:56.479058 28247 registrar.cpp:365] Successfully fetched the > registry (0B) in 322.939904ms > I0713 18:43:56.479388 28247 registrar.cpp:464] Applied 1 operations in > 50643ns; attempting to update the 'registry' > I0713 18:43:56.483093 28247 log.cpp:577] Attempting to append 168 bytes to > the log > I0713 18:43:56.483269 28249 coordinator.cpp:348] Coordinator attempting to > write APPEND action at position 1 > I0713 18:43:56.484673 28245 replica.cpp:537] Replica received write request > for position 1 from (9560)@172.17.0.8:35418 > I0713 18:43:56.509866 28239 hierarchical.cpp:1537] No allocations performed > I0713 18:43:56.509959 28239 hierarchical.cpp:1172] Performed allocation for 0 > agents in 157789ns > I0713 18:43:56.512147 28245 leveldb.cpp:341] Persisting action (187 bytes) to > leveldb took 27.358967ms > I0713 18:43:56.512193 28245 replica.cpp:712] Persisted action at 1 > I0713 18:43:56.513278 28250 replica.cpp:691] Replica received learned notice > for position 1 from @0.0.0.0:0 > I0713 18:43:56.537894 28250 leveldb.cpp:341] Persisting action (189 bytes) to > leveldb took 24.568093ms > I0713 18:43:56.537973 28250 replica.cpp:712] Persisted action at 1 > I0713 18:43:56.538008 28250 replica.cpp:697] Replica learned APPEND action at > position 1 > I0713 18:43:56.539737 28252 registrar.cpp:509] Successfully updated the > 'registry' in 60.26496ms > I0713 18:43:56.539949 28252 registrar.cpp:395] Successfully recovered > registrar > I0713 18:43:56.540544 28252 master.cpp:1781] Recovered 0 agents from the > Registry (129B) ; allowing 10mins for agents to re-register > I0713 18:43:56.540832 28250 hierarchical.cpp:178] Skipping recovery of > hierarchical allocator: nothing to recover > I0713 18:43:56.541285 28251 log.cpp:596] Attempting to truncate the log to 1 > I0713 18:43:56.541637 28248 coordinator.cpp:348] Coordinator attempting to > write TRUNCATE action at position 2 > I0713 18:43:56.542763 28240 replica.cpp:537] Replica received write request > for position 2 from (9561)@172.17.0.8:35418 > I0713 18:43:56.571691 28240 leveldb.cpp:341] Persisting action (16 bytes) to > leveldb took 28.798341ms > I0713 18:43:56.571889 28240 replica.cpp:712] Persisted action at 2 > I0713 18:43:56.573218 28240 replica.cpp:691] Replica received learned notice > for position 2 from @0.0.0.0:0 > I0713 18:43:56.620200 28240 leveldb.cpp:341] Persisting action (18 bytes) to > leveldb took 46.927607ms > I0713 18:43:56.620338 28240 leveldb.cpp:399] Deleting ~1 keys from leveldb > took 59898ns > I0713 18:43:56.620512 28240 replica.cpp:712] Persisted action at 2 > I0713 18:43:56.620630 28240 replica.cpp:697] Replica learned TRUNCATE action > at position 2 > I0713 18:43:56.624091 28249 hierarchical.cpp:1537] No allocations performed > I0713 18:43:56.624169 28249 hierarchical.cpp:1172] Performed allocation for 0 > agents in 140818ns > I0713 18:43:56.628180 28220 containerizer.cpp:196] Using isolation: > posix/cpu,posix/mem,filesystem/posix,network/cni > W0713 18:43:56.629341 28220 backend.cpp:75] Failed to create 'aufs' backend: > AufsBackend requires root privileges, but is running as user mesos > W0713 18:43:56.629616 28220 backend.cpp:75] Failed to create 'bind' backend: > BindBackend requires root privileges > I0713 18:43:56.631988 28220 cluster.cpp:432] Creating default 'local' > authorizer > I0713 18:43:56.635001 28243 slave.cpp:205] Agent started on > 251)@172.17.0.8:35418 > I0713 18:43:56.635308 28220 resources.cpp:572] Parsing resources as JSON > failed: disk:512 > Trying semicolon-delimited string format instead > I0713 18:43:56.635026 28243 slave.cpp:206] Flags at startup: --acls="" > --appc_simple_discovery_uri_prefix="http://" > --appc_store_dir="/tmp/mesos/store/appc" --authenticate_http="true" > --authenticatee="crammd5" --authentication_backoff_factor="1secs" > --authorizer="local" --cgroups_cpu_enable_pids_and_tids_count="false" > --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" > --cgroups_limit_swap="false" --cgroups_root="mesos" > --container_disk_watch_interval="15secs" --containerizers="mesos" > --credential="/tmp/PersistentVolumeEndpointsTest_OfferCreateThenEndpointRemove_gqStXQ/credential" > --default_role="*" --disk_watch_interval="1mins" --docker="docker" > --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io" > --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" > --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" > --docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume" > --enforce_container_disk_quota="false" > --executor_registration_timeout="1mins" > --executor_shutdown_grace_period="5secs" > --fetcher_cache_dir="/tmp/PersistentVolumeEndpointsTest_OfferCreateThenEndpointRemove_gqStXQ/fetch" > --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" > --gc_disk_headroom="0.1" --hadoop_home="" --help="false" > --hostname_lookup="true" --http_authenticators="basic" > --http_command_executor="false" > --http_credentials="/tmp/PersistentVolumeEndpointsTest_OfferCreateThenEndpointRemove_gqStXQ/http_credentials" > --image_provisioner_backend="copy" --initialize_driver_logging="true" > --isolation="posix/cpu,posix/mem" > --launcher_dir="/mesos/mesos-1.1.0/_build/src" --logbufsecs="0" > --logging_level="INFO" --oversubscribed_resources_interval="15secs" > --perf_duration="10secs" --perf_interval="1mins" > --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" > --recovery_timeout="15mins" --registration_backoff_factor="10ms" > --resources="disk(*):1024" --revocable_cpu_low_priority="true" > --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" > --systemd_enable_support="true" > --systemd_runtime_directory="/run/systemd/system" --version="false" > --work_dir="/tmp/PersistentVolumeEndpointsTest_OfferCreateThenEndpointRemove_gqStXQ" > I0713 18:43:56.635709 28243 credentials.hpp:86] Loading credential for > authentication from > '/tmp/PersistentVolumeEndpointsTest_OfferCreateThenEndpointRemove_gqStXQ/credential' > I0713 18:43:56.635892 28243 slave.cpp:343] Agent using credential for: > test-principal > I0713 18:43:56.635924 28243 credentials.hpp:37] Loading credentials for > authentication from > '/tmp/PersistentVolumeEndpointsTest_OfferCreateThenEndpointRemove_gqStXQ/http_credentials' > I0713 18:43:56.636272 28243 slave.cpp:395] Using default 'basic' HTTP > authenticator > I0713 18:43:56.636615 28243 resources.cpp:572] Parsing resources as JSON > failed: disk(*):1024 > Trying semicolon-delimited string format instead > I0713 18:43:56.636878 28243 resources.cpp:572] Parsing resources as JSON > failed: disk(*):1024 > Trying semicolon-delimited string format instead > I0713 18:43:56.637318 28243 slave.cpp:594] Agent resources: disk(*):1024; > cpus(*):16; mem(*):47270; ports(*):[31000-32000] > I0713 18:43:56.637859 28243 slave.cpp:602] Agent attributes: [ ] > I0713 18:43:56.638073 28220 sched.cpp:226] Version: 1.1.0 > I0713 18:43:56.638074 28243 slave.cpp:607] Agent hostname: 500c3e866abe > I0713 18:43:56.640148 28253 sched.cpp:330] New master detected at > master@172.17.0.8:35418 > I0713 18:43:56.640650 28253 sched.cpp:396] Authenticating with master > master@172.17.0.8:35418 > I0713 18:43:56.640738 28253 sched.cpp:403] Using default CRAM-MD5 > authenticatee > I0713 18:43:56.640801 28239 state.cpp:57] Recovering state from > '/tmp/PersistentVolumeEndpointsTest_OfferCreateThenEndpointRemove_gqStXQ/meta' > I0713 18:43:56.640976 28243 authenticatee.cpp:121] Creating new client SASL > connection > I0713 18:43:56.641319 28253 status_update_manager.cpp:200] Recovering status > update manager > I0713 18:43:56.641477 28243 master.cpp:6006] Authenticating > scheduler-398078e0-6dae-4c02-8197-af69d9eb230a@172.17.0.8:35418 > I0713 18:43:56.641636 28239 authenticator.cpp:414] Starting authentication > session for crammd5_authenticatee(554)@172.17.0.8:35418 > I0713 18:43:56.641542 28240 containerizer.cpp:522] Recovering containerizer > I0713 18:43:56.642201 28239 authenticator.cpp:98] Creating new server SASL > connection > I0713 18:43:56.642602 28252 authenticatee.cpp:213] Received SASL > authentication mechanisms: CRAM-MD5 > I0713 18:43:56.642634 28252 authenticatee.cpp:239] Attempting to authenticate > with mechanism 'CRAM-MD5' > I0713 18:43:56.642714 28239 authenticator.cpp:204] Received SASL > authentication start > I0713 18:43:56.642792 28239 authenticator.cpp:326] Authentication requires > more steps > I0713 18:43:56.642882 28239 authenticatee.cpp:259] Received SASL > authentication step > I0713 18:43:56.642978 28239 authenticator.cpp:232] Received SASL > authentication step > I0713 18:43:56.643009 28239 auxprop.cpp:109] Request to lookup properties for > user: 'test-principal' realm: '500c3e866abe' server FQDN: '500c3e866abe' > SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false > SASL_AUXPROP_AUTHZID: false > I0713 18:43:56.643026 28239 auxprop.cpp:181] Looking up auxiliary property > '*userPassword' > I0713 18:43:56.643064 28239 auxprop.cpp:181] Looking up auxiliary property > '*cmusaslsecretCRAM-MD5' > I0713 18:43:56.643091 28239 auxprop.cpp:109] Request to lookup properties for > user: 'test-principal' realm: '500c3e866abe' server FQDN: '500c3e866abe' > SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false > SASL_AUXPROP_AUTHZID: true > I0713 18:43:56.643107 28239 auxprop.cpp:131] Skipping auxiliary property > '*userPassword' since SASL_AUXPROP_AUTHZID == true > I0713 18:43:56.643117 28239 auxprop.cpp:131] Skipping auxiliary property > '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true > I0713 18:43:56.643136 28239 authenticator.cpp:318] Authentication success > I0713 18:43:56.643290 28239 authenticatee.cpp:299] Authentication success > I0713 18:43:56.643379 28239 master.cpp:6036] Successfully authenticated > principal 'test-principal' at > scheduler-398078e0-6dae-4c02-8197-af69d9eb230a@172.17.0.8:35418 > I0713 18:43:56.643501 28244 authenticator.cpp:432] Authentication session > cleanup for crammd5_authenticatee(554)@172.17.0.8:35418 > I0713 18:43:56.643987 28244 sched.cpp:502] Successfully authenticated with > master master@172.17.0.8:35418 > I0713 18:43:56.644011 28244 sched.cpp:820] Sending SUBSCRIBE call to > master@172.17.0.8:35418 > I0713 18:43:56.644103 28244 sched.cpp:853] Will retry registration in > 809.142674ms if necessary > I0713 18:43:56.644287 28244 master.cpp:2550] Received SUBSCRIBE call for > framework 'default' at > scheduler-398078e0-6dae-4c02-8197-af69d9eb230a@172.17.0.8:35418 > I0713 18:43:56.644346 28244 master.cpp:2012] Authorizing framework principal > 'test-principal' to receive offers for role 'role1' > I0713 18:43:56.644675 28249 provisioner.cpp:253] Provisioner recovery complete > I0713 18:43:56.645089 28245 master.cpp:2626] Subscribing framework default > with checkpointing disabled and capabilities [ ] > I0713 18:43:56.645783 28249 hierarchical.cpp:271] Added framework > 2258d072-b0c9-4c40-874c-6cf933ee345a-0000 > I0713 18:43:56.645916 28249 hierarchical.cpp:1537] No allocations performed > I0713 18:43:56.646000 28249 hierarchical.cpp:1632] No inverse offers to send > out! > I0713 18:43:56.646083 28248 sched.cpp:743] Framework registered with > 2258d072-b0c9-4c40-874c-6cf933ee345a-0000 > I0713 18:43:56.646116 28249 hierarchical.cpp:1172] Performed allocation for 0 > agents in 249850ns > I0713 18:43:56.646163 28248 sched.cpp:757] Scheduler::registered took 21831ns > I0713 18:43:56.646317 28246 slave.cpp:4856] Finished recovery > I0713 18:43:56.663516 28246 slave.cpp:5028] Querying resource estimator for > oversubscribable resources > I0713 18:43:56.664029 28254 status_update_manager.cpp:174] Pausing sending > status updates > I0713 18:43:56.664043 28246 slave.cpp:969] New master detected at > master@172.17.0.8:35418 > I0713 18:43:56.664567 28246 slave.cpp:1028] Authenticating with master > master@172.17.0.8:35418 > I0713 18:43:56.665148 28246 slave.cpp:1039] Using default CRAM-MD5 > authenticatee > I0713 18:43:56.665555 28246 slave.cpp:1001] Detecting new master > I0713 18:43:56.665590 28244 authenticatee.cpp:121] Creating new client SASL > connection > I0713 18:43:56.665889 28246 slave.cpp:5042] Received oversubscribable > resources from the resource estimator > I0713 18:43:56.666071 28253 master.cpp:6006] Authenticating > slave(251)@172.17.0.8:35418 > I0713 18:43:56.666316 28244 authenticator.cpp:414] Starting authentication > session for crammd5_authenticatee(555)@172.17.0.8:35418 > I0713 18:43:56.666803 28244 authenticator.cpp:98] Creating new server SASL > connection > I0713 18:43:56.667027 28244 authenticatee.cpp:213] Received SASL > authentication mechanisms: CRAM-MD5 > I0713 18:43:56.667058 28244 authenticatee.cpp:239] Attempting to authenticate > with mechanism 'CRAM-MD5' > I0713 18:43:56.667140 28244 authenticator.cpp:204] Received SASL > authentication start > I0713 18:43:56.667201 28244 authenticator.cpp:326] Authentication requires > more steps > I0713 18:43:56.667275 28244 authenticatee.cpp:259] Received SASL > authentication step > I0713 18:43:56.667359 28244 authenticator.cpp:232] Received SASL > authentication step > I0713 18:43:56.667390 28244 auxprop.cpp:109] Request to lookup properties for > user: 'test-principal' realm: '500c3e866abe' server FQDN: '500c3e866abe' > SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false > SASL_AUXPROP_AUTHZID: false > I0713 18:43:56.667404 28244 auxprop.cpp:181] Looking up auxiliary property > '*userPassword' > I0713 18:43:56.667484 28244 auxprop.cpp:181] Looking up auxiliary property > '*cmusaslsecretCRAM-MD5' > I0713 18:43:56.667508 28244 auxprop.cpp:109] Request to lookup properties for > user: 'test-principal' realm: '500c3e866abe' server FQDN: '500c3e866abe' > SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false > SASL_AUXPROP_AUTHZID: true > I0713 18:43:56.667521 28244 auxprop.cpp:131] Skipping auxiliary property > '*userPassword' since SASL_AUXPROP_AUTHZID == true > I0713 18:43:56.667531 28244 auxprop.cpp:131] Skipping auxiliary property > '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true > I0713 18:43:56.667551 28244 authenticator.cpp:318] Authentication success > I0713 18:43:56.667709 28244 authenticatee.cpp:299] Authentication success > I0713 18:43:56.667801 28244 master.cpp:6036] Successfully authenticated > principal 'test-principal' at slave(251)@172.17.0.8:35418 > I0713 18:43:56.667876 28244 authenticator.cpp:432] Authentication session > cleanup for crammd5_authenticatee(555)@172.17.0.8:35418 > I0713 18:43:56.668190 28244 slave.cpp:1123] Successfully authenticated with > master master@172.17.0.8:35418 > I0713 18:43:56.668352 28244 slave.cpp:1529] Will retry registration in > 5.90776ms if necessary > I0713 18:43:56.668772 28251 master.cpp:4676] Registering agent at > slave(251)@172.17.0.8:35418 (500c3e866abe) with id > 2258d072-b0c9-4c40-874c-6cf933ee345a-S0 > I0713 18:43:56.669507 28251 registrar.cpp:464] Applied 1 operations in > 96357ns; attempting to update the 'registry' > I0713 18:43:56.670956 28253 log.cpp:577] Attempting to append 337 bytes to > the log > I0713 18:43:56.671103 28251 coordinator.cpp:348] Coordinator attempting to > write APPEND action at position 3 > I0713 18:43:56.672089 28254 replica.cpp:537] Replica received write request > for position 3 from (9577)@172.17.0.8:35418 > I0713 18:43:56.674686 28250 hierarchical.cpp:1537] No allocations performed > I0713 18:43:56.675036 28250 hierarchical.cpp:1632] No inverse offers to send > out! > I0713 18:43:56.675192 28250 hierarchical.cpp:1172] Performed allocation for 0 > agents in 574337ns > I0713 18:43:56.674938 28242 slave.cpp:1529] Will retry registration in > 31.506393ms if necessary > I0713 18:43:56.675686 28250 master.cpp:4664] Ignoring register agent message > from slave(251)@172.17.0.8:35418 (500c3e866abe) as admission is already in > progress > I0713 18:43:56.689913 28254 leveldb.cpp:341] Persisting action (356 bytes) to > leveldb took 17.767596ms > I0713 18:43:56.690068 28254 replica.cpp:712] Persisted action at 3 > I0713 18:43:56.691304 28254 replica.cpp:691] Replica received learned notice > for position 3 from @0.0.0.0:0 > I0713 18:43:56.708274 28239 slave.cpp:1529] Will retry registration in > 39.295397ms if necessary > I0713 18:43:56.708631 28245 master.cpp:4664] Ignoring register agent message > from slave(251)@172.17.0.8:35418 (500c3e866abe) as admission is already in > progress > I0713 18:43:56.726626 28239 hierarchical.cpp:1537] No allocations performed > I0713 18:43:56.726686 28239 hierarchical.cpp:1632] No inverse offers to send > out! > I0713 18:43:56.726758 28239 hierarchical.cpp:1172] Performed allocation for 0 > agents in 207473ns > I0713 18:43:56.731587 28254 leveldb.cpp:341] Persisting action (358 bytes) to > leveldb took 40.15631ms > I0713 18:43:56.731680 28254 replica.cpp:712] Persisted action at 3 > I0713 18:43:56.731724 28254 replica.cpp:697] Replica learned APPEND action at > position 3 > I0713 18:43:56.733752 28254 registrar.cpp:509] Successfully updated the > 'registry' in 64.109056ms > I0713 18:43:56.734061 28240 log.cpp:596] Attempting to truncate the log to 3 > I0713 18:43:56.734262 28240 coordinator.cpp:348] Coordinator attempting to > write TRUNCATE action at position 4 > I0713 18:43:56.734968 28254 master.cpp:4745] Registered agent > 2258d072-b0c9-4c40-874c-6cf933ee345a-S0 at slave(251)@172.17.0.8:35418 > (500c3e866abe) with disk(*):1024; cpus(*):16; mem(*):47270; > ports(*):[31000-32000] > I0713 18:43:56.735697 28252 hierarchical.cpp:478] Added agent > 2258d072-b0c9-4c40-874c-6cf933ee345a-S0 (500c3e866abe) with disk(*):1024; > cpus(*):16; mem(*):47270; ports(*):[31000-32000] (allocated: ) > I0713 18:43:56.736590 28252 hierarchical.cpp:1632] No inverse offers to send > out! > I0713 18:43:56.736665 28252 hierarchical.cpp:1195] Performed allocation for > agent 2258d072-b0c9-4c40-874c-6cf933ee345a-S0 in 924479ns > I0713 18:43:56.736768 28252 slave.cpp:1169] Registered with master > master@172.17.0.8:35418; given agent ID > 2258d072-b0c9-4c40-874c-6cf933ee345a-S0 > I0713 18:43:56.736790 28252 fetcher.cpp:86] Clearing fetcher cache > I0713 18:43:56.737208 28252 slave.cpp:1192] Checkpointing SlaveInfo to > '/tmp/PersistentVolumeEndpointsTest_OfferCreateThenEndpointRemove_gqStXQ/meta/slaves/2258d072-b0c9-4c40-874c-6cf933ee345a-S0/slave.info' > I0713 18:43:56.737787 28249 master.cpp:5835] Sending 1 offers to framework > 2258d072-b0c9-4c40-874c-6cf933ee345a-0000 (default) at > scheduler-398078e0-6dae-4c02-8197-af69d9eb230a@172.17.0.8:35418 > I0713 18:43:56.737946 28249 status_update_manager.cpp:181] Resuming sending > status updates > I0713 18:43:56.738322 28249 sched.cpp:917] Scheduler::resourceOffers took > 112243ns > I0713 18:43:56.739811 28252 slave.cpp:1229] Forwarding total oversubscribed > resources > I0713 18:43:56.740494 28252 slave.cpp:3760] Received ping from > slave-observer(245)@172.17.0.8:35418 > I0713 18:43:56.740677 28252 master.cpp:5128] Received update of agent > 2258d072-b0c9-4c40-874c-6cf933ee345a-S0 at slave(251)@172.17.0.8:35418 > (500c3e866abe) with total oversubscribed resources > I0713 18:43:56.741009 28252 hierarchical.cpp:542] Agent > 2258d072-b0c9-4c40-874c-6cf933ee345a-S0 (500c3e866abe) updated with > oversubscribed resources (total: disk(*):1024; cpus(*):16; mem(*):47270; > ports(*):[31000-32000], allocated: disk(*):1024; cpus(*):16; mem(*):47270; > ports(*):[31000-32000]) > I0713 18:43:56.741189 28252 hierarchical.cpp:1537] No allocations performed > I0713 18:43:56.741228 28252 hierarchical.cpp:1632] No inverse offers to send > out! > I0713 18:43:56.741289 28252 hierarchical.cpp:1195] Performed allocation for > agent 2258d072-b0c9-4c40-874c-6cf933ee345a-S0 in 219942ns > I0713 18:43:56.742151 28240 replica.cpp:537] Replica received write request > for position 4 from (9578)@172.17.0.8:35418 > I0713 18:43:56.744851 28254 master.cpp:3468] Processing ACCEPT call for > offers: [ 2258d072-b0c9-4c40-874c-6cf933ee345a-O0 ] on agent > 2258d072-b0c9-4c40-874c-6cf933ee345a-S0 at slave(251)@172.17.0.8:35418 > (500c3e866abe) for framework 2258d072-b0c9-4c40-874c-6cf933ee345a-0000 > (default) at scheduler-398078e0-6dae-4c02-8197-af69d9eb230a@172.17.0.8:35418 > I0713 18:43:56.744977 28254 master.cpp:3144] Authorizing principal > 'test-principal' to reserve resources 'disk(role1, test-principal):512' > I0713 18:43:56.747194 28254 master.cpp:3695] Applying RESERVE operation for > resources disk(role1, test-principal):512 from framework > 2258d072-b0c9-4c40-874c-6cf933ee345a-0000 (default) at > scheduler-398078e0-6dae-4c02-8197-af69d9eb230a@172.17.0.8:35418 to agent > 2258d072-b0c9-4c40-874c-6cf933ee345a-S0 at slave(251)@172.17.0.8:35418 > (500c3e866abe) > I0713 18:43:56.747602 28254 master.cpp:7098] Sending checkpointed resources > disk(role1, test-principal):512 to agent > 2258d072-b0c9-4c40-874c-6cf933ee345a-S0 at slave(251)@172.17.0.8:35418 > (500c3e866abe) > I0713 18:43:56.748193 28245 slave.cpp:2600] Updated checkpointed resources > from to disk(role1, test-principal):512 > I0713 18:43:56.750397 28239 hierarchical.cpp:683] Updated allocation of > framework 2258d072-b0c9-4c40-874c-6cf933ee345a-0000 on agent > 2258d072-b0c9-4c40-874c-6cf933ee345a-S0 from disk(*):1024; cpus(*):16; > mem(*):47270; ports(*):[31000-32000] to disk(*):512; cpus(*):16; > mem(*):47270; ports(*):[31000-32000]; disk(role1, test-principal):512 > I0713 18:43:56.751693 28239 hierarchical.cpp:924] Recovered disk(*):512; > cpus(*):16; mem(*):47270; ports(*):[31000-32000]; disk(role1, > test-principal):512 (total: disk(*):512; cpus(*):16; mem(*):47270; > ports(*):[31000-32000]; disk(role1, test-principal):512, allocated: ) on > agent 2258d072-b0c9-4c40-874c-6cf933ee345a-S0 from framework > 2258d072-b0c9-4c40-874c-6cf933ee345a-0000 > I0713 18:43:56.777899 28243 hierarchical.cpp:1632] No inverse offers to send > out! > I0713 18:43:56.777992 28243 hierarchical.cpp:1172] Performed allocation for 1 > agents in 1.044349ms > I0713 18:43:56.778578 28243 master.cpp:5835] Sending 1 offers to framework > 2258d072-b0c9-4c40-874c-6cf933ee345a-0000 (default) at > scheduler-398078e0-6dae-4c02-8197-af69d9eb230a@172.17.0.8:35418 > I0713 18:43:56.779105 28243 sched.cpp:917] Scheduler::resourceOffers took > 122192ns > I0713 18:43:56.781126 28243 master.cpp:3468] Processing ACCEPT call for > offers: [ 2258d072-b0c9-4c40-874c-6cf933ee345a-O1 ] on agent > 2258d072-b0c9-4c40-874c-6cf933ee345a-S0 at slave(251)@172.17.0.8:35418 > (500c3e866abe) for framework 2258d072-b0c9-4c40-874c-6cf933ee345a-0000 > (default) at scheduler-398078e0-6dae-4c02-8197-af69d9eb230a@172.17.0.8:35418 > I0713 18:43:56.781239 28243 master.cpp:3253] Authorizing principal > 'test-principal' to create volumes > I0713 18:43:56.783177 28245 master.cpp:3801] Applying CREATE operation for > volumes disk(role1, test-principal)[id1:volume_path]:1 from framework > 2258d072-b0c9-4c40-874c-6cf933ee345a-0000 (default) at > scheduler-398078e0-6dae-4c02-8197-af69d9eb230a@172.17.0.8:35418 to agent > 2258d072-b0c9-4c40-874c-6cf933ee345a-S0 at slave(251)@172.17.0.8:35418 > (500c3e866abe) > I0713 18:43:56.783653 28245 master.cpp:7098] Sending checkpointed resources > disk(role1, test-principal):511; disk(role1, > test-principal)[id1:volume_path]:1 to agent > 2258d072-b0c9-4c40-874c-6cf933ee345a-S0 at slave(251)@172.17.0.8:35418 > (500c3e866abe) > I0713 18:43:56.785676 28249 hierarchical.cpp:683] Updated allocation of > framework 2258d072-b0c9-4c40-874c-6cf933ee345a-0000 on agent > 2258d072-b0c9-4c40-874c-6cf933ee345a-S0 from disk(role1, test-principal):512; > disk(*):512; cpus(*):16; mem(*):47270; ports(*):[31000-32000] to disk(role1, > test-principal):511; disk(*):512; cpus(*):16; mem(*):47270; > ports(*):[31000-32000]; disk(role1, test-principal)[id1:volume_path]:1 > I0713 18:43:56.786803 28249 hierarchical.cpp:924] Recovered disk(role1, > test-principal):511; disk(*):512; cpus(*):16; mem(*):47270; > ports(*):[31000-32000]; disk(role1, test-principal)[id1:volume_path]:1 > (total: disk(*):512; cpus(*):16; mem(*):47270; ports(*):[31000-32000]; > disk(role1, test-principal):511; disk(role1, > test-principal)[id1:volume_path]:1, allocated: ) on agent > 2258d072-b0c9-4c40-874c-6cf933ee345a-S0 from framework > 2258d072-b0c9-4c40-874c-6cf933ee345a-0000 > I0713 18:43:56.790855 28241 slave.cpp:2600] Updated checkpointed resources > from disk(role1, test-principal):512 to disk(role1, test-principal):511; > disk(role1, test-principal)[id1:volume_path]:1 > I0713 18:43:56.798692 28240 leveldb.cpp:341] Persisting action (16 bytes) to > leveldb took 56.485928ms > I0713 18:43:56.798765 28240 replica.cpp:712] Persisted action at 4 > I0713 18:43:56.800184 28240 replica.cpp:691] Replica received learned notice > for position 4 from @0.0.0.0:0 > I0713 18:43:56.830730 28245 hierarchical.cpp:1632] No inverse offers to send > out! > I0713 18:43:56.830854 28245 hierarchical.cpp:1172] Performed allocation for 1 > agents in 1.388256ms > I0713 18:43:56.831647 28245 master.cpp:5835] Sending 1 offers to framework > 2258d072-b0c9-4c40-874c-6cf933ee345a-0000 (default) at > scheduler-398078e0-6dae-4c02-8197-af69d9eb230a@172.17.0.8:35418 > I0713 18:43:56.832273 28245 sched.cpp:917] Scheduler::resourceOffers took > 146725ns > I0713 18:43:56.841397 28245 process.cpp:3322] Handling HTTP event for process > 'master' with path: '/master/destroy-volumes' > I0713 18:43:56.845082 28245 http.cpp:381] HTTP POST for > /master/destroy-volumes from 172.17.0.8:57083 > I0713 18:43:56.847327 28240 leveldb.cpp:341] Persisting action (18 bytes) to > leveldb took 47.043061ms > I0713 18:43:56.847586 28240 leveldb.cpp:399] Deleting ~2 keys from leveldb > took 196582ns > I0713 18:43:56.847704 28240 replica.cpp:712] Persisted action at 4 > I0713 18:43:56.847744 28240 replica.cpp:697] Replica learned TRUNCATE action > at position 4 > I0713 18:43:56.848569 28245 master.cpp:3304] Authorizing principal > 'test-principal' to destroy volumes 'disk(role1, > test-principal)[id1:volume_path]:1' > I0713 18:43:56.854449 28247 sched.cpp:943] Rescinded offer > 2258d072-b0c9-4c40-874c-6cf933ee345a-O2 > I0713 18:43:56.854564 28247 sched.cpp:954] Scheduler::offerRescinded took > 53044ns > I0713 18:43:56.855819 28240 hierarchical.cpp:924] Recovered disk(role1, > test-principal):511; disk(role1, test-principal)[id1:volume_path]:1; > disk(*):512; cpus(*):16; mem(*):47270; ports(*):[31000-32000] (total: > disk(*):512; cpus(*):16; mem(*):47270; ports(*):[31000-32000]; disk(role1, > test-principal):511; disk(role1, test-principal)[id1:volume_path]:1, > allocated: ) on agent 2258d072-b0c9-4c40-874c-6cf933ee345a-S0 from framework > 2258d072-b0c9-4c40-874c-6cf933ee345a-0000 > I0713 18:43:56.856168 28240 hierarchical.cpp:961] Framework > 2258d072-b0c9-4c40-874c-6cf933ee345a-0000 filtered agent > 2258d072-b0c9-4c40-874c-6cf933ee345a-S0 for 5secs > I0713 18:43:56.874948 28245 master.cpp:7098] Sending checkpointed resources > disk(role1, test-principal):512 to agent > 2258d072-b0c9-4c40-874c-6cf933ee345a-S0 at slave(251)@172.17.0.8:35418 > (500c3e866abe) > I0713 18:43:56.883574 28244 hierarchical.cpp:1632] No inverse offers to send > out! > I0713 18:43:56.883744 28244 hierarchical.cpp:1172] Performed allocation for 1 > agents in 2.274914ms > I0713 18:43:56.887001 28242 slave.cpp:2575] Deleting persistent volume 'id1' > at > '/tmp/PersistentVolumeEndpointsTest_OfferCreateThenEndpointRemove_gqStXQ/volumes/roles/role1/id1' > I0713 18:43:56.887470 28242 slave.cpp:2600] Updated checkpointed resources > from disk(role1, test-principal):511; disk(role1, > test-principal)[id1:volume_path]:1 to disk(role1, test-principal):512 > I0713 18:43:56.888828 28245 master.cpp:5835] Sending 1 offers to framework > 2258d072-b0c9-4c40-874c-6cf933ee345a-0000 (default) at > scheduler-398078e0-6dae-4c02-8197-af69d9eb230a@172.17.0.8:35418 > I0713 18:43:56.889853 28239 sched.cpp:917] Scheduler::resourceOffers took > 99847ns > I0713 18:43:56.890542 28245 process.cpp:3322] Handling HTTP event for process > 'master' with path: '/master/unreserve' > I0713 18:43:56.894271 28245 http.cpp:381] HTTP POST for /master/unreserve > from 172.17.0.8:57084 > I0713 18:43:56.896220 28245 master.cpp:3201] Authorizing principal > 'test-principal' to unreserve resources 'disk(role1, test-principal):512' > I0713 18:43:56.902156 28242 hierarchical.cpp:924] Recovered disk(role1, > test-principal):512; disk(*):512; cpus(*):16; mem(*):47270; > ports(*):[31000-32000] (total: disk(*):512; cpus(*):16; mem(*):47270; > ports(*):[31000-32000]; disk(role1, test-principal):512, allocated: ) on > agent 2258d072-b0c9-4c40-874c-6cf933ee345a-S0 from framework > 2258d072-b0c9-4c40-874c-6cf933ee345a-0000 > I0713 18:43:56.902437 28242 hierarchical.cpp:961] Framework > 2258d072-b0c9-4c40-874c-6cf933ee345a-0000 filtered agent > 2258d072-b0c9-4c40-874c-6cf933ee345a-S0 for 5secs > I0713 18:43:56.904063 28239 sched.cpp:943] Rescinded offer > 2258d072-b0c9-4c40-874c-6cf933ee345a-O3 > ../../src/tests/persistent_volume_endpoints_tests.cpp:1626: Failure > Mock function called more times than expected - returning directly. > Function call: offerRescinded(0x7fff630731f0, @0x2b5db8010660 > 2258d072-b0c9-4c40-874c-6cf933ee345a-O3) > Expected: to be called once > Actual: called twice - over-saturated and active > I0713 18:43:56.905498 28239 sched.cpp:954] Scheduler::offerRescinded took > 1.139804ms > I0713 18:43:56.919566 28245 master.cpp:7098] Sending checkpointed resources > to agent 2258d072-b0c9-4c40-874c-6cf933ee345a-S0 at > slave(251)@172.17.0.8:35418 (500c3e866abe) > I0713 18:43:56.920109 28243 slave.cpp:2600] Updated checkpointed resources > from disk(role1, test-principal):512 to > I0713 18:43:56.920534 28220 sched.cpp:1987] Asked to stop the driver > I0713 18:43:56.920699 28251 sched.cpp:1187] Stopping framework > '2258d072-b0c9-4c40-874c-6cf933ee345a-0000' > I0713 18:43:56.920999 28250 master.cpp:6410] Processing TEARDOWN call for > framework 2258d072-b0c9-4c40-874c-6cf933ee345a-0000 (default) at > scheduler-398078e0-6dae-4c02-8197-af69d9eb230a@172.17.0.8:35418 > I0713 18:43:56.921031 28250 master.cpp:6422] Removing framework > 2258d072-b0c9-4c40-874c-6cf933ee345a-0000 (default) at > scheduler-398078e0-6dae-4c02-8197-af69d9eb230a@172.17.0.8:35418 > I0713 18:43:56.921340 28251 hierarchical.cpp:382] Deactivated framework > 2258d072-b0c9-4c40-874c-6cf933ee345a-0000 > I0713 18:43:56.921362 28242 slave.cpp:2292] Asked to shut down framework > 2258d072-b0c9-4c40-874c-6cf933ee345a-0000 by master@172.17.0.8:35418 > W0713 18:43:56.921393 28242 slave.cpp:2307] Cannot shut down unknown > framework 2258d072-b0c9-4c40-874c-6cf933ee345a-0000 > I0713 18:43:56.921924 28250 hierarchical.cpp:333] Removed framework > 2258d072-b0c9-4c40-874c-6cf933ee345a-0000 > I0713 18:43:56.926107 28220 slave.cpp:841] Agent terminating > I0713 18:43:56.926527 28247 master.cpp:1371] Agent > 2258d072-b0c9-4c40-874c-6cf933ee345a-S0 at slave(251)@172.17.0.8:35418 > (500c3e866abe) disconnected > I0713 18:43:56.926555 28247 master.cpp:2910] Disconnecting agent > 2258d072-b0c9-4c40-874c-6cf933ee345a-S0 at slave(251)@172.17.0.8:35418 > (500c3e866abe) > I0713 18:43:56.926601 28247 master.cpp:2929] Deactivating agent > 2258d072-b0c9-4c40-874c-6cf933ee345a-S0 at slave(251)@172.17.0.8:35418 > (500c3e866abe) > I0713 18:43:56.927222 28247 hierarchical.cpp:571] Agent > 2258d072-b0c9-4c40-874c-6cf933ee345a-S0 deactivated > I0713 18:43:56.938642 28242 hierarchical.cpp:1537] No allocations performed > I0713 18:43:56.939143 28242 hierarchical.cpp:1172] Performed allocation for 1 > agents in 591579ns > I0713 18:43:56.940840 28220 master.cpp:1218] Master terminating > I0713 18:43:56.941227 28249 hierarchical.cpp:510] Removed agent > 2258d072-b0c9-4c40-874c-6cf933ee345a-S0 > [ FAILED ] PersistentVolumeEndpointsTest.OfferCreateThenEndpointRemove (994 > ms) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)