[ https://issues.apache.org/jira/browse/MESOS-9260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chun-Hung Hsiao reassigned MESOS-9260: -------------------------------------- Assignee: Chun-Hung Hsiao > StorageLocalResourceProviderTest.ReconcileUnacknowledgedTerminalOperation is > flaky > ---------------------------------------------------------------------------------- > > Key: MESOS-9260 > URL: https://issues.apache.org/jira/browse/MESOS-9260 > Project: Mesos > Issue Type: Bug > Reporter: Benno Evers > Assignee: Chun-Hung Hsiao > Priority: Major > Labels: flaky-test > > Observed in an internal CI system: > {noformat} > ../../src/tests/storage_local_resource_provider_tests.cpp:4286 > Failed to wait 15secs for offers > {noformat} > Full log: > {noformat} > [ RUN ] > StorageLocalResourceProviderTest.ReconcileUnacknowledgedTerminalOperation > I0926 00:01:11.465590 11812 cluster.cpp:173] Creating default 'local' > authorizer > I0926 00:01:11.466861 1833 master.cpp:413] Master > f0a77094-641d-45e5-a890-46f2c2d8202c (ip-172-16-10-227.ec2.internal) started > on 172.16.10.227:43103 > I0926 00:01:11.466879 1833 master.cpp:416] Flags at startup: --acls="" > --agent_ping_timeout="15secs" --agent_reregister_timeout="10mins" > --allocation_interval="50ms" --allocator="hierarchical" > --authenticate_agents="true" --authenticate_frameworks="true" > --authenticate_http_frameworks="true" --authenticate_http_readonly="true" > --authenticate_http_readwrite="true" --authentication_v0_timeout="15secs" > --authenticators="crammd5" --authorizers="local" > --credentials="/tmp/r7LBlp/credentials" --filter_gpu_resources="true" > --framework_sorter="drf" --help="false" --hostname_lookup="true" > --http_authenticators="basic" --http_framework_authenticators="basic" > --initialize_driver_logging="true" --log_auto_initialize="true" > --logbufsecs="0" --logging_level="INFO" --max_agent_ping_timeouts="5" > --max_completed_frameworks="50" --max_completed_tasks_per_framework="1000" > --max_unreachable_tasks_per_framework="1000" --memory_profiling="false" > --min_allocatable_resources="cpus:0.01|mem:32" --port="5050" --quiet="false" > --recovery_agent_removal_limit="100%" --registry="in_memory" > --registry_fetch_timeout="1mins" --registry_gc_interval="15mins" > --registry_max_agent_age="2weeks" --registry_max_agent_count="102400" > --registry_store_timeout="100secs" --registry_strict="false" > --require_agent_domain="false" --role_sorter="drf" --root_submissions="true" > --version="false" --webui_dir="/usr/local/share/mesos/webui" > --work_dir="/tmp/r7LBlp/master" --zk_session_timeout="10secs" > I0926 00:01:11.467025 1833 master.cpp:465] Master only allowing > authenticated frameworks to register > I0926 00:01:11.467034 1833 master.cpp:471] Master only allowing > authenticated agents to register > I0926 00:01:11.467041 1833 master.cpp:477] Master only allowing > authenticated HTTP frameworks to register > I0926 00:01:11.467047 1833 credentials.hpp:37] Loading credentials for > authentication from '/tmp/r7LBlp/credentials' > I0926 00:01:11.467128 1833 master.cpp:521] Using default 'crammd5' > authenticator > I0926 00:01:11.467170 1833 http.cpp:1037] Creating default 'basic' HTTP > authenticator for realm 'mesos-master-readonly' > I0926 00:01:11.467207 1833 http.cpp:1037] Creating default 'basic' HTTP > authenticator for realm 'mesos-master-readwrite' > I0926 00:01:11.467229 1833 http.cpp:1037] Creating default 'basic' HTTP > authenticator for realm 'mesos-master-scheduler' > I0926 00:01:11.467248 1833 master.cpp:602] Authorization enabled > I0926 00:01:11.467972 1833 hierarchical.cpp:182] Initialized hierarchical > allocator process > I0926 00:01:11.468001 1833 whitelist_watcher.cpp:77] No whitelist given > I0926 00:01:11.468333 1833 master.cpp:2083] Elected as the leading master! > I0926 00:01:11.468343 1833 master.cpp:1638] Recovering from registrar > I0926 00:01:11.468374 1833 registrar.cpp:339] Recovering registrar > I0926 00:01:11.468508 1833 registrar.cpp:383] Successfully fetched the > registry (0B) in 0ns > I0926 00:01:11.468541 1833 registrar.cpp:487] Applied 1 operations in > 8882ns; attempting to update the registry > I0926 00:01:11.468659 1833 registrar.cpp:544] Successfully updated the > registry in 0ns > I0926 00:01:11.468688 1833 registrar.cpp:416] Successfully recovered > registrar > I0926 00:01:11.468765 1833 master.cpp:1752] Recovered 0 agents from the > registry (176B); allowing 10mins for agents to reregister > I0926 00:01:11.468796 1833 hierarchical.cpp:220] Skipping recovery of > hierarchical allocator: nothing to recover > W0926 00:01:11.470585 11812 process.cpp:2810] Attempted to spawn already > running process files@172.16.10.227:43103 > I0926 00:01:11.470960 11812 containerizer.cpp:305] Using isolation { > cgroups/cpu, filesystem/posix, environment_secret, cgroups/mem, network/cni } > I0926 00:01:11.472955 11812 linux_launcher.cpp:144] Using > /sys/fs/cgroup/freezer as the freezer hierarchy for the Linux launcher > I0926 00:01:11.473356 11812 provisioner.cpp:298] Using default backend > 'overlay' > I0926 00:01:11.478726 11812 cluster.cpp:485] Creating default 'local' > authorizer > I0926 00:01:11.479349 1840 slave.cpp:267] Mesos agent started on > (771)@172.16.10.227:43103 > I0926 00:01:11.479365 1840 slave.cpp:268] Flags at startup: --acls="" > --appc_simple_discovery_uri_prefix="http://" > --appc_store_dir="/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_06lIVO/store/appc" > --authenticate_http_executors="true" --authenticate_http_readonly="true" > --authenticate_http_readwrite="true" --authenticatee="crammd5" > --authentication_backoff_factor="1secs" --authentication_timeout_max="1mins" > --authentication_timeout_min="5secs" --authorizer="local" > --cgroups_cpu_enable_pids_and_tids_count="false" > --cgroups_destroy_timeout="1mins" --cgroups_enable_cfs="false" > --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" > --cgroups_root="mesos_test_a71fca40-7489-4600-b9b0-b37c10625cff" > --container_disk_watch_interval="15secs" --containerizers="mesos" > --credential="/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_06lIVO/credential" > --default_role="*" --disallow_sharing_agent_pid_namespace="false" > --disk_profile_adaptor="org_apache_mesos_UriDiskProfileAdaptor" > --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" > --docker_registry="https://registry-1.docker.io" --docker_remove_delay="6hrs" > --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" > --docker_store_dir="/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_06lIVO/store/docker" > --docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume" > --enforce_container_disk_quota="false" > --executor_registration_timeout="1mins" > --executor_reregistration_timeout="2secs" > --executor_shutdown_grace_period="5secs" > --fetcher_cache_dir="/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_06lIVO/fetch" > --fetcher_cache_size="2GB" --fetcher_stall_timeout="1mins" > --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" > --gc_non_executor_container_sandboxes="false" --help="false" > --hostname_lookup="true" --http_command_executor="false" > --http_credentials="/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_06lIVO/http_credentials" > --http_heartbeat_interval="30secs" --initialize_driver_logging="true" > --isolation="cgroups/cpu,cgroups/mem" > --jwt_secret_key="/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_06lIVO/jwt_secret_key" > --launcher="linux" > --launcher_dir="/home/ubuntu/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mesos-ec2-ubuntu-16.04/mesos/build/src" > --logbufsecs="0" --logging_level="INFO" > --max_completed_executors_per_framework="150" --memory_profiling="false" > --network_cni_metrics="true" --oversubscribed_resources_interval="15secs" > --perf_duration="10secs" --perf_interval="1mins" --port="5051" > --qos_correction_interval_min="0ns" --quiet="false" > --reconfiguration_policy="equal" --recover="reconnect" > --recovery_timeout="15mins" --registration_backoff_factor="10ms" > --resource_provider_config_dir="/tmp/r7LBlp/resource_provider_configs" > --resources="cpus:2;gpus:0;mem:1024;disk:1024;ports:[31000-32000]" > --revocable_cpu_low_priority="true" > --runtime_dir="/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_06lIVO" > --sandbox_directory="/mnt/mesos/sandbox" --strict="true" > --switch_user="true" --systemd_enable_support="true" > --systemd_runtime_directory="/run/systemd/system" --version="false" > --work_dir="/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_YdUE86" > --zk_session_timeout="10secs" > I0926 00:01:11.479526 1840 credentials.hpp:86] Loading credential for > authentication from > '/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_06lIVO/credential' > I0926 00:01:11.479574 1840 slave.cpp:300] Agent using credential for: > test-principal > I0926 00:01:11.479584 1840 credentials.hpp:37] Loading credentials for > authentication from > '/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_06lIVO/http_credentials' > I0926 00:01:11.479640 1840 http.cpp:1037] Creating default 'basic' HTTP > authenticator for realm 'mesos-agent-executor' > I0926 00:01:11.479673 1840 http.cpp:1058] Creating default 'jwt' HTTP > authenticator for realm 'mesos-agent-executor' > I0926 00:01:11.479717 1840 http.cpp:1037] Creating default 'basic' HTTP > authenticator for realm 'mesos-agent-readonly' > I0926 00:01:11.479737 1840 http.cpp:1058] Creating default 'jwt' HTTP > authenticator for realm 'mesos-agent-readonly' > I0926 00:01:11.479763 1840 http.cpp:1037] Creating default 'basic' HTTP > authenticator for realm 'mesos-agent-readwrite' > I0926 00:01:11.479780 1840 http.cpp:1058] Creating default 'jwt' HTTP > authenticator for realm 'mesos-agent-readwrite' > I0926 00:01:11.479838 1840 disk_profile_adaptor.cpp:84] Creating disk > profile adaptor module 'org_apache_mesos_UriDiskProfileAdaptor' > I0926 00:01:11.480406 1840 slave.cpp:615] Agent resources: > [{"name":"cpus","scalar":{"value":2.0},"type":"SCALAR"},{"name":"mem","scalar":{"value":1024.0},"type":"SCALAR"},{"name":"disk","scalar":{"value":1024.0},"type":"SCALAR"},{"name":"ports","ranges":{"range":[{"begin":31000,"end":32000}]},"type":"RANGES"}] > I0926 00:01:11.480468 1840 slave.cpp:623] Agent attributes: [ ] > I0926 00:01:11.480478 1840 slave.cpp:632] Agent hostname: > ip-172-16-10-227.ec2.internal > I0926 00:01:11.480512 1837 uri_disk_profile_adaptor.cpp:304] Updated disk > profile mapping to 1 active profiles > I0926 00:01:11.480752 1837 task_status_update_manager.cpp:181] Pausing > sending task status updates > I0926 00:01:11.480829 1837 state.cpp:66] Recovering state from > '/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_YdUE86/meta' > I0926 00:01:11.480952 1838 slave.cpp:6909] Finished recovering checkpointed > state from > '/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_YdUE86/meta', > beginning agent recovery > I0926 00:01:11.480989 1838 task_status_update_manager.cpp:207] Recovering > task status update manager > I0926 00:01:11.481066 1838 containerizer.cpp:727] Recovering Mesos containers > I0926 00:01:11.481117 1838 linux_launcher.cpp:286] Recovering Linux launcher > I0926 00:01:11.481254 1840 containerizer.cpp:1053] Recovering isolators > I0926 00:01:11.481770 1837 containerizer.cpp:1092] Recovering provisioner > I0926 00:01:11.481986 1835 provisioner.cpp:494] Provisioner recovery complete > I0926 00:01:11.482172 1833 composing.cpp:339] Finished recovering all > containerizers > I0926 00:01:11.482262 1834 slave.cpp:7138] Recovering executors > I0926 00:01:11.482285 1834 slave.cpp:7291] Finished recovery > I0926 00:01:11.482527 1836 task_status_update_manager.cpp:181] Pausing > sending task status updates > I0926 00:01:11.482537 1837 slave.cpp:1254] New master detected at > master@172.16.10.227:43103 > I0926 00:01:11.482563 1837 slave.cpp:1319] Detecting new master > I0926 00:01:11.482641 1833 slave.cpp:1346] Authenticating with master > master@172.16.10.227:43103 > I0926 00:01:11.482664 1833 slave.cpp:1355] Using default CRAM-MD5 > authenticatee > I0926 00:01:11.482738 1836 authenticatee.cpp:121] Creating new client SASL > connection > I0926 00:01:11.483110 1840 master.cpp:9653] Authenticating > slave(771)@172.16.10.227:43103 > I0926 00:01:11.483165 1835 authenticator.cpp:414] Starting authentication > session for crammd5-authenticatee(1491)@172.16.10.227:43103 > I0926 00:01:11.483230 1835 authenticator.cpp:98] Creating new server SASL > connection > I0926 00:01:11.483600 1835 authenticatee.cpp:213] Received SASL > authentication mechanisms: CRAM-MD5 > I0926 00:01:11.483618 1835 authenticatee.cpp:239] Attempting to authenticate > with mechanism 'CRAM-MD5' > I0926 00:01:11.483647 1835 authenticator.cpp:204] Received SASL > authentication start > I0926 00:01:11.483681 1835 authenticator.cpp:326] Authentication requires > more steps > I0926 00:01:11.483709 1835 authenticatee.cpp:259] Received SASL > authentication step > I0926 00:01:11.483744 1835 authenticator.cpp:232] Received SASL > authentication step > I0926 00:01:11.483757 1835 auxprop.cpp:109] Request to lookup properties for > user: 'test-principal' realm: 'ip-172-16-10-227.ec2.internal' server FQDN: > 'ip-172-16-10-227.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false > SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false > I0926 00:01:11.483767 1835 auxprop.cpp:181] Looking up auxiliary property > '*userPassword' > I0926 00:01:11.483778 1835 auxprop.cpp:181] Looking up auxiliary property > '*cmusaslsecretCRAM-MD5' > I0926 00:01:11.483786 1835 auxprop.cpp:109] Request to lookup properties for > user: 'test-principal' realm: 'ip-172-16-10-227.ec2.internal' server FQDN: > 'ip-172-16-10-227.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false > SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true > I0926 00:01:11.483793 1835 auxprop.cpp:131] Skipping auxiliary property > '*userPassword' since SASL_AUXPROP_AUTHZID == true > I0926 00:01:11.483798 1835 auxprop.cpp:131] Skipping auxiliary property > '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true > I0926 00:01:11.483811 1835 authenticator.cpp:318] Authentication success > I0926 00:01:11.483865 1840 authenticatee.cpp:299] Authentication success > I0926 00:01:11.483928 1835 master.cpp:9685] Successfully authenticated > principal 'test-principal' at slave(771)@172.16.10.227:43103 > I0926 00:01:11.483942 1839 authenticator.cpp:432] Authentication session > cleanup for crammd5-authenticatee(1491)@172.16.10.227:43103 > I0926 00:01:11.484007 1833 slave.cpp:1446] Successfully authenticated with > master master@172.16.10.227:43103 > I0926 00:01:11.484109 1833 slave.cpp:1877] Will retry registration in > 14.703994ms if necessary > I0926 00:01:11.484150 1835 master.cpp:6605] Received register agent message > from slave(771)@172.16.10.227:43103 (ip-172-16-10-227.ec2.internal) > I0926 00:01:11.484203 1835 master.cpp:3964] Authorizing agent providing > resources 'cpus:2; mem:1024; disk:1024; ports:[31000-32000]' with principal > 'test-principal' > I0926 00:01:11.484340 1840 master.cpp:6672] Authorized registration of agent > at slave(771)@172.16.10.227:43103 (ip-172-16-10-227.ec2.internal) > I0926 00:01:11.484375 1840 master.cpp:6787] Registering agent at > slave(771)@172.16.10.227:43103 (ip-172-16-10-227.ec2.internal) with id > f0a77094-641d-45e5-a890-46f2c2d8202c-S0 > I0926 00:01:11.484493 1840 registrar.cpp:487] Applied 1 operations in > 38466ns; attempting to update the registry > I0926 00:01:11.484637 1835 registrar.cpp:544] Successfully updated the > registry in 0ns > I0926 00:01:11.484691 1834 master.cpp:6835] Admitted agent > f0a77094-641d-45e5-a890-46f2c2d8202c-S0 at slave(771)@172.16.10.227:43103 > (ip-172-16-10-227.ec2.internal) > I0926 00:01:11.484822 1834 master.cpp:6880] Registered agent > f0a77094-641d-45e5-a890-46f2c2d8202c-S0 at slave(771)@172.16.10.227:43103 > (ip-172-16-10-227.ec2.internal) with cpus:2; mem:1024; disk:1024; > ports:[31000-32000] > I0926 00:01:11.484916 1834 hierarchical.cpp:601] Added agent > f0a77094-641d-45e5-a890-46f2c2d8202c-S0 (ip-172-16-10-227.ec2.internal) with > cpus:2; mem:1024; disk:1024; ports:[31000-32000] (allocated: {}) > I0926 00:01:11.484963 1834 hierarchical.cpp:1564] Performed allocation for 1 > agents in 9501ns > I0926 00:01:11.484994 1834 slave.cpp:1479] Registered with master > master@172.16.10.227:43103; given agent ID > f0a77094-641d-45e5-a890-46f2c2d8202c-S0 > I0926 00:01:11.485126 1834 slave.cpp:1499] Checkpointing SlaveInfo to > '/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_YdUE86/meta/slaves/f0a77094-641d-45e5-a890-46f2c2d8202c-S0/slave.info' > I0926 00:01:11.485360 1835 task_status_update_manager.cpp:188] Resuming > sending task status updates > I0926 00:01:11.485404 1834 slave.cpp:1548] Forwarding agent update > {"operations":{},"resource_version_uuid":{"value":"TUGtB/13S1q/oBij0hB5rA=="},"slave_id":{"value":"f0a77094-641d-45e5-a890-46f2c2d8202c-S0"},"update_oversubscribed_resources":false} > I0926 00:01:11.485775 1836 master.cpp:7939] Ignoring update on agent > f0a77094-641d-45e5-a890-46f2c2d8202c-S0 at slave(771)@172.16.10.227:43103 > (ip-172-16-10-227.ec2.internal) as it reports no changes > I0926 00:01:11.487349 1836 process.cpp:3569] Handling HTTP event for process > 'slave(771)' with path: '/slave(771)/api/v1' > I0926 00:01:11.487823 1840 http.cpp:1177] HTTP POST for /slave(771)/api/v1 > from 172.16.10.227:35910 > I0926 00:01:11.487898 1840 http.cpp:2074] Processing GET_CONTAINERS call > I0926 00:01:11.490869 1836 container_daemon.cpp:121] Launching container > 'org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE' > I0926 00:01:11.491497 1833 process.cpp:3569] Handling HTTP event for process > 'slave(771)' with path: '/slave(771)/api/v1' > I0926 00:01:11.491904 1833 http.cpp:1177] HTTP POST for /slave(771)/api/v1 > from 172.16.10.227:35912 > I0926 00:01:11.492103 1833 http.cpp:2440] Processing LAUNCH_CONTAINER call > for container > 'org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE' > I0926 00:01:11.492333 1833 http.cpp:2556] Creating sandbox > '/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_YdUE86/containers/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE' > I0926 00:01:11.493259 1836 containerizer.cpp:1280] Starting container > org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE > I0926 00:01:11.493693 1836 containerizer.cpp:1446] Checkpointed > ContainerConfig at > '/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_06lIVO/containers/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE/config' > I0926 00:01:11.493710 1836 containerizer.cpp:3118] Transitioning the state > of container > org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE > from PROVISIONING to PREPARING > I0926 00:01:11.493806 1834 cgroups.cpp:429] Creating cgroup at > '/sys/fs/cgroup/cpu,cpuacct/mesos_test_a71fca40-7489-4600-b9b0-b37c10625cff/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE' > for container > org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE > I0926 00:01:11.494220 1834 cgroups.cpp:429] Creating cgroup at > '/sys/fs/cgroup/memory/mesos_test_a71fca40-7489-4600-b9b0-b37c10625cff/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE' > for container > org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE > I0926 00:01:11.494746 1834 memory.cpp:478] Started listening for OOM events > for container > org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE > I0926 00:01:11.494796 1834 memory.cpp:590] Started listening on 'low' memory > pressure events for container > org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE > I0926 00:01:11.494828 1834 memory.cpp:590] Started listening on 'medium' > memory pressure events for container > org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE > I0926 00:01:11.494858 1834 memory.cpp:590] Started listening on 'critical' > memory pressure events for container > org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE > I0926 00:01:11.495095 1836 memory.cpp:198] Updated > 'memory.soft_limit_in_bytes' to 1GB for container > org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE > I0926 00:01:11.495132 1834 cpu.cpp:92] Updated 'cpu.shares' to 102 (cpus > 0.1) for container > org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE > I0926 00:01:11.495165 1836 memory.cpp:227] Updated 'memory.limit_in_bytes' > to 1GB for container > org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE > I0926 00:01:11.495867 1834 containerizer.cpp:1939] Launching > 'mesos-containerizer' with flags '--help="false" > --launch_info="{"command":{"arguments":["/home/ubuntu/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mesos-ec2-ubuntu-16.04/mesos/build/src/test-csi-plugin","--available_capacity=4GB","--volumes=","--work_dir=/tmp/r7LBlp/test_csi_plugin"],"shell":false,"value":"/home/ubuntu/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mesos-ec2-ubuntu-16.04/mesos/build/src/test-csi-plugin"},"environment":{"variables":[{"name":"MESOS_SANDBOX","type":"VALUE","value":"/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_YdUE86/containers/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE"},{"name":"CSI_ENDPOINT","type":"VALUE","value":"unix:///tmp/mesos-csi-wGzvgE/endpoint.sock"}]},"task_environment":{},"working_directory":"/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_YdUE86/containers/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE"}" > --pipe_read="75" --pipe_write="76" > --runtime_directory="/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_06lIVO/containers/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE" > --unshare_namespace_mnt="false"' > I0926 00:01:11.496028 1840 linux_launcher.cpp:492] Launching container > org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE > and cloning with namespaces > I0926 00:01:11.521980 1834 containerizer.cpp:3118] Transitioning the state > of container > org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE > from PREPARING to ISOLATING > I0926 00:01:11.523646 1839 containerizer.cpp:3118] Transitioning the state > of container > org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE > from ISOLATING to FETCHING > I0926 00:01:11.523715 1839 fetcher.cpp:369] Starting to fetch URIs for > container: > org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE, > directory: > /tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_YdUE86/containers/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE > I0926 00:01:11.523888 1839 containerizer.cpp:3118] Transitioning the state > of container > org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE > from FETCHING to RUNNING > I0926 00:01:11.525076 1835 container_daemon.cpp:140] Invoking post-start > hook for container > 'org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE' > I0926 00:01:11.525574 1835 hierarchical.cpp:1564] Performed allocation for 1 > agents in 18948ns > I0926 00:01:11.576836 1836 hierarchical.cpp:1564] Performed allocation for 1 > agents in 19631ns > I0926 00:01:11.627815 1835 hierarchical.cpp:1564] Performed allocation for 1 > agents in 21050ns > I0926 00:01:11.639616 6260 test_csi_plugin.cpp:302] ProbeRequest '{}' > I0926 00:01:11.640031 1839 container_daemon.cpp:171] Waiting for container > 'org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE' > I0926 00:01:11.640367 6260 test_csi_plugin.cpp:274] GetPluginInfoRequest '{}' > I0926 00:01:11.640795 1836 process.cpp:3569] Handling HTTP event for process > 'slave(771)' with path: '/slave(771)/api/v1' > I0926 00:01:11.640939 1836 provider.cpp:2178] Node plugin loaded: > {"name":"org.apache.mesos.csi.test","vendorVersion":"1.8.0"} > I0926 00:01:11.641376 1834 http.cpp:1177] HTTP POST for /slave(771)/api/v1 > from 172.16.10.227:35914 > I0926 00:01:11.641413 6261 test_csi_plugin.cpp:288] > GetPluginCapabilitiesRequest '{}' > I0926 00:01:11.641479 1834 http.cpp:2670] Processing WAIT_CONTAINER call for > container > 'org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE' > I0926 00:01:11.641993 6261 test_csi_plugin.cpp:274] GetPluginInfoRequest '{}' > I0926 00:01:11.642236 1839 provider.cpp:2219] Controller plugin loaded: > {"name":"org.apache.mesos.csi.test","vendorVersion":"1.8.0"} > I0926 00:01:11.642463 6261 test_csi_plugin.cpp:593] > ControllerGetCapabilitiesRequest '{}' > I0926 00:01:11.642946 6261 test_csi_plugin.cpp:870] > NodeGetCapabilitiesRequest '{}' > I0926 00:01:11.643378 6261 test_csi_plugin.cpp:857] NodeGetIdRequest '{}' > I0926 00:01:11.643676 1839 provider.cpp:661] Finished recovery for resource > provider with type 'org.apache.mesos.rp.local.storage' and name 'test > I0926 00:01:11.643731 1834 status_update_manager_process.hpp:379] Pausing > operation status update manager > I0926 00:01:11.643767 1839 http_connection.hpp:227] New endpoint detected at > http://172.16.10.227:43103/slave(771)/api/v1/resource_provider > I0926 00:01:11.644228 1838 http_connection.hpp:283] Connected with the > remote endpoint at > http://172.16.10.227:43103/slave(771)/api/v1/resource_provider > I0926 00:01:11.644340 1839 provider.cpp:524] Connected to resource provider > manager > I0926 00:01:11.644498 1835 http_connection.hpp:131] Sending 1 call to > http://172.16.10.227:43103/slave(771)/api/v1/resource_provider > I0926 00:01:11.644778 1837 process.cpp:3569] Handling HTTP event for process > 'slave(771)' with path: '/slave(771)/api/v1/resource_provider' > I0926 00:01:11.645036 1840 http.cpp:1177] HTTP POST for > /slave(771)/api/v1/resource_provider from 172.16.10.227:35918 > I0926 00:01:11.645167 1840 manager.cpp:736] Subscribing resource provider > {"default_reservations":[{"role":"storage","type":"DYNAMIC"}],"name":"test","storage":{"plugin":{"containers":[{"command":{"arguments":["/home/ubuntu/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mesos-ec2-ubuntu-16.04/mesos/build/src/test-csi-plugin","--available_capacity=4GB","--volumes=","--work_dir=/tmp/r7LBlp/test_csi_plugin"],"shell":false,"value":"/home/ubuntu/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mesos-ec2-ubuntu-16.04/mesos/build/src/test-csi-plugin"},"resources":[{"name":"cpus","scalar":{"value":0.1},"type":"SCALAR"},{"name":"mem","scalar":{"value":1024.0},"type":"SCALAR"}],"services":["CONTROLLER_SERVICE","NODE_SERVICE"]}],"name":"test_csi_plugin","type":"org.apache.mesos.csi.test"}},"type":"org.apache.mesos.rp.local.storage"} > I0926 00:01:11.647924 1840 slave.cpp:7643] Handling resource provider > message 'SUBSCRIBE: > {"default_reservations":[{"role":"storage","type":"DYNAMIC"}],"id":{"value":"520ef77a-5d14-4c9f-8077-0a6c4ae0b0b2"},"name":"test","storage":{"plugin":{"containers":[{"command":{"arguments":["/home/ubuntu/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mesos-ec2-ubuntu-16.04/mesos/build/src/test-csi-plugin","--available_capacity=4GB","--volumes=","--work_dir=/tmp/r7LBlp/test_csi_plugin"],"shell":false,"value":"/home/ubuntu/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mesos-ec2-ubuntu-16.04/mesos/build/src/test-csi-plugin"},"resources":[{"name":"cpus","scalar":{"value":0.1},"type":"SCALAR"},{"name":"mem","scalar":{"value":1024.0},"type":"SCALAR"}],"services":["CONTROLLER_SERVICE","NODE_SERVICE"]}],"name":"test_csi_plugin","type":"org.apache.mesos.csi.test"}},"type":"org.apache.mesos.rp.local.storage"}' > I0926 00:01:11.648219 1837 provider.cpp:546] Received SUBSCRIBED event > I0926 00:01:11.648238 1837 provider.cpp:1496] Subscribed with ID > 520ef77a-5d14-4c9f-8077-0a6c4ae0b0b2 > I0926 00:01:11.648468 1836 status_update_manager_process.hpp:314] Recovering > operation status update manager > I0926 00:01:11.648780 6261 test_csi_plugin.cpp:531] ListVolumesRequest '{}' > I0926 00:01:11.649070 1839 provider.cpp:3519] Sending UPDATE_STATE call with > resources '{}' and 0 operations to agent > f0a77094-641d-45e5-a890-46f2c2d8202c-S0 > I0926 00:01:11.649124 1839 provider.cpp:1110] Resource provider > 520ef77a-5d14-4c9f-8077-0a6c4ae0b0b2 is in READY state > I0926 00:01:11.649148 1834 http_connection.hpp:131] Sending 3 call to > http://172.16.10.227:43103/slave(771)/api/v1/resource_provider > I0926 00:01:11.649207 1839 status_update_manager_process.hpp:385] Resuming > operation status update manager > I0926 00:01:11.649250 1835 provider.cpp:1422] Updating profiles { test } for > resource provider 520ef77a-5d14-4c9f-8077-0a6c4ae0b0b2 > I0926 00:01:11.649550 1837 process.cpp:3569] Handling HTTP event for process > 'slave(771)' with path: '/slave(771)/api/v1/resource_provider' > I0926 00:01:11.649752 1837 http.cpp:1177] HTTP POST for > /slave(771)/api/v1/resource_provider from 172.16.10.227:35916 > I0926 00:01:11.649860 1837 manager.cpp:917] Received UPDATE_STATE call with > resources '[]' and 0 operations from resource provider > 520ef77a-5d14-4c9f-8077-0a6c4ae0b0b2 > I0926 00:01:11.649793 6261 test_csi_plugin.cpp:559] GetCapacityRequest > '{"volumeCapabilities":[{"mount":{},"accessMode":{"mode":"SINGLE_NODE_WRITER"}}]}' > I0926 00:01:11.649943 1837 slave.cpp:7643] Handling resource provider > message 'UPDATE_STATE: 520ef77a-5d14-4c9f-8077-0a6c4ae0b0b2 {}' > I0926 00:01:11.649966 1837 slave.cpp:7763] Forwarding new total resources > cpus:2; mem:1024; disk:1024; ports:[31000-32000] > I0926 00:01:11.650372 1837 provider.cpp:1297] Removing '{}' and adding > 'disk(reservations: [(DYNAMIC,storage)])[RAW(,test)]:4096' to the total > resources > I0926 00:01:11.650480 1833 hierarchical.cpp:740] Grew agent > f0a77094-641d-45e5-a890-46f2c2d8202c-S0 by {} (total), { } (used) > I0926 00:01:11.650538 1833 hierarchical.cpp:697] Agent > f0a77094-641d-45e5-a890-46f2c2d8202c-S0 (ip-172-16-10-227.ec2.internal) > updated with total resources cpus:2; mem:1024; disk:1024; ports:[31000-32000] > I0926 00:01:11.650691 1837 provider.cpp:3519] Sending UPDATE_STATE call with > resources 'disk(reservations: [(DYNAMIC,storage)])[RAW(,test)]:4096' and 0 > operations to agent f0a77094-641d-45e5-a890-46f2c2d8202c-S0 > I0926 00:01:11.650811 1833 http_connection.hpp:131] Sending 3 call to > http://172.16.10.227:43103/slave(771)/api/v1/resource_provider > I0926 00:01:11.651093 1840 process.cpp:3569] Handling HTTP event for process > 'slave(771)' with path: '/slave(771)/api/v1/resource_provider' > I0926 00:01:11.651756 11812 scheduler.cpp:189] Version: 1.8.0 > I0926 00:01:11.652252 1838 scheduler.cpp:355] Using default 'basic' HTTP > authenticatee > I0926 00:01:11.652364 1836 scheduler.cpp:538] New master detected at > master@172.16.10.227:43103 > I0926 00:01:11.652377 1836 scheduler.cpp:547] Waiting for 0ns before > initiating a re-(connection) attempt with the master > I0926 00:01:11.652827 1835 scheduler.cpp:429] Connected with the master at > http://172.16.10.227:43103/master/api/v1/scheduler > I0926 00:01:11.653015 1838 scheduler.cpp:248] Sending SUBSCRIBE call to > http://172.16.10.227:43103/master/api/v1/scheduler > I0926 00:01:11.653323 1833 process.cpp:3569] Handling HTTP event for process > 'master' with path: '/master/api/v1/scheduler' > I0926 00:01:11.653650 1840 http.cpp:1177] HTTP POST for > /master/api/v1/scheduler from 172.16.10.227:35922 > I0926 00:01:11.653709 1840 master.cpp:2502] Received subscription request > for HTTP framework 'default' > I0926 00:01:11.653733 1840 master.cpp:2155] Authorizing framework principal > 'test-principal' to receive offers for roles '{ storage }' > I0926 00:01:11.653841 1834 master.cpp:2637] Subscribing framework 'default' > with checkpointing disabled and capabilities [ MULTI_ROLE, > RESERVATION_REFINEMENT ] > I0926 00:01:11.654402 1834 master.cpp:9883] Adding framework > f0a77094-641d-45e5-a890-46f2c2d8202c-0000 (default) with roles { } suppressed > I0926 00:01:11.654527 1837 hierarchical.cpp:306] Added framework > f0a77094-641d-45e5-a890-46f2c2d8202c-0000 > I0926 00:01:11.654933 1836 scheduler.cpp:845] Enqueuing event SUBSCRIBED > received from http://172.16.10.227:43103/master/api/v1/scheduler > I0926 00:01:11.654948 1837 hierarchical.cpp:1564] Performed allocation for 1 > agents in 140494ns > I0926 00:01:11.655035 1835 master.cpp:9468] Sending offers [ > f0a77094-641d-45e5-a890-46f2c2d8202c-O0 ] to framework > f0a77094-641d-45e5-a890-46f2c2d8202c-0000 (default) > I0926 00:01:11.689805 1839 http.cpp:1177] HTTP POST for > /slave(771)/api/v1/resource_provider from 172.16.10.227:35916 > I0926 00:01:13.959466 1834 manager.cpp:917] Received UPDATE_STATE call with > resources > '[{"disk":{"source":{"profile":"test","type":"RAW"}},"name":"disk","provider_id":{"value":"520ef77a-5d14-4c9f-8077-0a6c4ae0b0b2"},"reservations":[{"role":"storage","type":"DYNAMIC"}],"scalar":{"value":4096.0},"type":"SCALAR"}]' > and 0 operations from resource provider 520ef77a-5d14-4c9f-8077-0a6c4ae0b0b2 > I0926 00:01:13.960250 1839 slave.cpp:7643] Handling resource provider > message 'UPDATE_STATE: 520ef77a-5d14-4c9f-8077-0a6c4ae0b0b2 > disk(reservations: [(DYNAMIC,storage)])[RAW(,test)]:4096' > I0926 00:01:13.960312 1839 slave.cpp:7763] Forwarding new total resources > cpus:2; mem:1024; disk:1024; ports:[31000-32000]; disk(reservations: > [(DYNAMIC,storage)])[RAW(,test)]:4096 > I0926 00:01:13.960785 1838 hierarchical.cpp:697] Agent > f0a77094-641d-45e5-a890-46f2c2d8202c-S0 (ip-172-16-10-227.ec2.internal) > updated with total resources cpus:2; mem:1024; disk:1024; > ports:[31000-32000]; disk(reservations: [(DYNAMIC,storage)])[RAW(,test)]:4096 > I0926 00:01:13.960876 1838 hierarchical.cpp:1564] Performed allocation for 1 > agents in 40437ns > I0926 00:01:13.965555 1836 scheduler.cpp:845] Enqueuing event HEARTBEAT > received from http://172.16.10.227:43103/master/api/v1/scheduler > I0926 00:01:13.966929 1836 scheduler.cpp:845] Enqueuing event OFFERS > received from http://172.16.10.227:43103/master/api/v1/scheduler > I0926 00:01:13.967149 1838 scheduler.cpp:248] Sending DECLINE call to > http://172.16.10.227:43103/master/api/v1/scheduler > I0926 00:01:13.967648 1835 process.cpp:3569] Handling HTTP event for process > 'master' with path: '/master/api/v1/scheduler' > I0926 00:01:13.967850 1837 hierarchical.cpp:1564] Performed allocation for 1 > agents in 33598ns > I0926 00:01:13.968111 1833 http.cpp:1177] HTTP POST for > /master/api/v1/scheduler from 172.16.10.227:35920 > I0926 00:01:13.968158 1833 master.cpp:5850] Processing DECLINE call for > offers: [ f0a77094-641d-45e5-a890-46f2c2d8202c-O0 ] for framework > f0a77094-641d-45e5-a890-46f2c2d8202c-0000 (default) > I0926 00:01:13.968245 1833 master.cpp:11462] Removing offer > f0a77094-641d-45e5-a890-46f2c2d8202c-O0 > I0926 00:01:13.968375 1833 hierarchical.cpp:1236] Recovered cpus(allocated: > storage):2; mem(allocated: storage):1024; disk(allocated: storage):1024; > ports(allocated: storage):[31000-32000] (total: cpus:2; mem:1024; disk:1024; > ports:[31000-32000]; disk(reservations: > [(DYNAMIC,storage)])[RAW(,test)]:4096, allocated: {}) on agent > f0a77094-641d-45e5-a890-46f2c2d8202c-S0 from framework > f0a77094-641d-45e5-a890-46f2c2d8202c-0000 > I0926 00:01:13.968408 1833 hierarchical.cpp:1282] Framework > f0a77094-641d-45e5-a890-46f2c2d8202c-0000 filtered agent > f0a77094-641d-45e5-a890-46f2c2d8202c-S0 for 5secs > ../../src/tests/storage_local_resource_provider_tests.cpp:4286: Failure > Failed to wait 15secs for offers > I0926 00:01:28.977944 1833 master.cpp:1366] Framework > f0a77094-641d-45e5-a890-46f2c2d8202c-0000 (default) disconnected > I0926 00:01:28.977985 1833 master.cpp:3230] Deactivating framework > f0a77094-641d-45e5-a890-46f2c2d8202c-0000 (default) > I0926 00:01:28.978009 1833 master.cpp:3207] Disconnecting framework > f0a77094-641d-45e5-a890-46f2c2d8202c-0000 (default) > I0926 00:01:28.978024 1833 master.cpp:1381] Giving framework > f0a77094-641d-45e5-a890-46f2c2d8202c-0000 (default) 0ns to failover > I0926 00:01:28.978085 1833 hierarchical.cpp:420] Deactivated framework > f0a77094-641d-45e5-a890-46f2c2d8202c-0000 > I0926 00:01:28.978204 1837 master.cpp:9261] Framework failover timeout, > removing framework f0a77094-641d-45e5-a890-46f2c2d8202c-0000 (default) > I0926 00:01:28.978226 1837 master.cpp:10197] Removing framework > f0a77094-641d-45e5-a890-46f2c2d8202c-0000 (default) > I0926 00:01:28.978302 1840 slave.cpp:3896] Asked to shut down framework > f0a77094-641d-45e5-a890-46f2c2d8202c-0000 by master@172.16.10.227:43103 > I0926 00:01:28.978327 1840 slave.cpp:3911] Cannot shut down unknown > framework f0a77094-641d-45e5-a890-46f2c2d8202c-0000 > I0926 00:01:28.978341 1837 hierarchical.cpp:359] Removed framework > f0a77094-641d-45e5-a890-46f2c2d8202c-0000 > ../../src/tests/storage_local_resource_provider_tests.cpp:4266: Failure > Actual function call count doesn't match EXPECT_CALL(*scheduler, offers(_, > v1::scheduler::OffersHaveAnyResource(isRaw)))... > Expected: to be called once > Actual: never called - unsatisfied and active > I0926 00:01:28.978569 11812 slave.cpp:909] Agent terminating > I0926 00:01:28.978735 11812 manager.cpp:161] Terminating resource provider > 520ef77a-5d14-4c9f-8077-0a6c4ae0b0b2 > E0926 00:01:28.979059 1836 http_connection.hpp:452] End-Of-File received > I0926 00:01:28.979130 1836 http_connection.hpp:217] Re-detecting endpoint > I0926 00:01:28.979238 1836 http_connection.hpp:227] New endpoint detected at > http://172.16.10.227:43103/slave(771)/api/v1/resource_provider > I0926 00:01:28.979274 1839 provider.cpp:536] Disconnected from resource > provider manager > I0926 00:01:28.979276 1836 http_connection.hpp:338] Ignoring disconnection > attempt from stale connection > I0926 00:01:28.979300 1836 http_connection.hpp:338] Ignoring disconnection > attempt from stale connection > I0926 00:01:28.979308 1839 status_update_manager_process.hpp:379] Pausing > operation status update manager > I0926 00:01:28.979518 1834 master.cpp:1251] Agent > f0a77094-641d-45e5-a890-46f2c2d8202c-S0 at slave(771)@172.16.10.227:43103 > (ip-172-16-10-227.ec2.internal) disconnected > I0926 00:01:28.979540 1834 master.cpp:3267] Disconnecting agent > f0a77094-641d-45e5-a890-46f2c2d8202c-S0 at slave(771)@172.16.10.227:43103 > (ip-172-16-10-227.ec2.internal) > I0926 00:01:28.979560 1834 master.cpp:3286] Deactivating agent > f0a77094-641d-45e5-a890-46f2c2d8202c-S0 at slave(771)@172.16.10.227:43103 > (ip-172-16-10-227.ec2.internal) > I0926 00:01:28.979599 1834 hierarchical.cpp:795] Agent > f0a77094-641d-45e5-a890-46f2c2d8202c-S0 deactivated > I0926 00:01:28.979914 1839 containerizer.cpp:2455] Destroying container > org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE > in RUNNING state > I0926 00:01:28.979934 1839 containerizer.cpp:3118] Transitioning the state > of container > org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE > from RUNNING to DESTROYING > I0926 00:01:28.980154 1839 linux_launcher.cpp:580] Asked to destroy > container > org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE > I0926 00:01:28.980207 1839 linux_launcher.cpp:622] Destroying cgroup > '/sys/fs/cgroup/freezer/mesos_test_a71fca40-7489-4600-b9b0-b37c10625cff/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE' > I0926 00:01:28.980270 1840 http_connection.hpp:283] Connected with the > remote endpoint at > http://172.16.10.227:43103/slave(771)/api/v1/resource_provider > I0926 00:01:28.980406 1840 hierarchical.cpp:1564] Performed allocation for 1 > agents in 26143ns > I0926 00:01:28.980520 1836 provider.cpp:524] Connected to resource provider > manager > I0926 00:01:28.980541 1838 cgroups.cpp:2838] Freezing cgroup > /sys/fs/cgroup/freezer/mesos_test_a71fca40-7489-4600-b9b0-b37c10625cff/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE > I0926 00:01:28.980742 1835 http_connection.hpp:131] Sending 1 call to > http://172.16.10.227:43103/slave(771)/api/v1/resource_provider > I0926 00:01:28.980916 1838 cgroups.cpp:1229] Successfully froze cgroup > /sys/fs/cgroup/freezer/mesos_test_a71fca40-7489-4600-b9b0-b37c10625cff/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE > after 349184ns > I0926 00:01:28.981250 1837 cgroups.cpp:2856] Thawing cgroup > /sys/fs/cgroup/freezer/mesos_test_a71fca40-7489-4600-b9b0-b37c10625cff/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE > I0926 00:01:28.981411 1837 cgroups.cpp:1258] Successfully thawed cgroup > /sys/fs/cgroup/freezer/mesos_test_a71fca40-7489-4600-b9b0-b37c10625cff/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE > after 124928ns > I0926 00:01:28.981528 1841 process.cpp:2735] Returning '404 Not Found' for > '/slave(771)/api/v1/resource_provider' > E0926 00:01:28.981875 1840 provider.cpp:1067] Failed to subscribe resource > provider with type 'org.apache.mesos.rp.local.storage' and name 'test': > Received '404 Not Found' () > I0926 00:01:29.031200 1835 hierarchical.cpp:1564] Performed allocation for 1 > agents in 26888ns > I0926 00:01:29.081390 1836 hierarchical.cpp:1564] Performed allocation for 1 > agents in 21841ns > I0926 00:01:29.081869 1836 containerizer.cpp:2957] Container > org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE > has exited > I0926 00:01:29.083161 1839 provisioner.cpp:597] Ignoring destroy request for > unknown container > org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE > I0926 00:01:29.083875 1834 container_daemon.cpp:189] Invoking post-stop hook > for container > 'org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE' > I0926 00:01:29.084064 1834 container_daemon.cpp:121] Launching container > 'org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE' > I0926 00:01:29.086195 1841 process.cpp:2735] Returning '404 Not Found' for > '/slave(771)/api/v1' > E0926 00:01:29.086671 1833 container_daemon.cpp:150] Failed to launch > container > 'org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE': > Failed to launch container > 'org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE': > Unexpected response '404 Not Found' () > I0926 00:01:29.099272 11812 master.cpp:1093] Master terminating > I0926 00:01:29.099445 1836 hierarchical.cpp:637] Removed agent > f0a77094-641d-45e5-a890-46f2c2d8202c-S0 > I0926 00:01:29.113350 1837 cgroups.cpp:2838] Freezing cgroup > /sys/fs/cgroup/freezer/mesos_test_a71fca40-7489-4600-b9b0-b37c10625cff > I0926 00:01:29.113492 1837 cgroups.cpp:1229] Successfully froze cgroup > /sys/fs/cgroup/freezer/mesos_test_a71fca40-7489-4600-b9b0-b37c10625cff after > 99072ns > I0926 00:01:29.113646 1837 cgroups.cpp:2856] Thawing cgroup > /sys/fs/cgroup/freezer/mesos_test_a71fca40-7489-4600-b9b0-b37c10625cff > I0926 00:01:29.113750 1837 cgroups.cpp:1258] Successfully thawed cgroup > /sys/fs/cgroup/freezer/mesos_test_a71fca40-7489-4600-b9b0-b37c10625cff after > 72960ns > [ FAILED ] > StorageLocalResourceProviderTest.ReconcileUnacknowledgedTerminalOperation > (17703 ms) > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)