Chun-Hung Hsiao created MESOS-9765:
--------------------------------------
Summary: Test `ROOT_CreateDestroyPersistentMountVolumeWithReboot`
is flaky.
Key: MESOS-9765
URL: https://issues.apache.org/jira/browse/MESOS-9765
Project: Mesos
Issue Type: Bug
Components: test
Reporter: Chun-Hung Hsiao
Assignee: Chun-Hung Hsiao
Observed a failure on test
{{CSIVersion/StorageLocalResourceProviderTest.ROOT_CreateDestroyPersistentMountVolumeWithReboot/v0}}
with the following error:
{noformat}
unknown file: Failure
Unexpected mock function call - returning directly.
Function call: statusUpdate(0x7ffea80beb70, @0x7f6027ff6b90 136-byte object
<E0-A2 02-36 60-7F 00-00 00-00 00-00 00-00 00-00 3E-EC 00-00 00-00 00-00 B8-0C
B0-33 60-7F 00-00 40-7C 0A-04 60-7F 00-00 30-0A 0C-04 60-7F 00-00 F0-7D 07-04
60-7F 00-00 20-88 0B-04 60-7F 00-00 ... 00-00 00-00 00-00 00-00 00-00 00-00
00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 A0-28 0C-04 60-7F
00-00 E7-52 88-A9 71-4C D7-41 00-00 00-00 01-00 00-00 08-00 00-00 03-00 00-00>)
Google Mock tried the following 6 expectations, but none matched:
/home/centos/workspace/mesos/Mesos_CI-build/FLAG/CMake/label/mesos-ec2-centos-7/mesos/src/tests/storage_local_resource_provider_tests.cpp:2552:
tried expectation #0: EXPECT_CALL(sched, statusUpdate(&driver,
TaskStatusStateEq(TASK_STARTING)))...
Expected arg #1: task status state eq TASK_STARTING
Actual: 136-byte object <E0-A2 02-36 60-7F 00-00 00-00 00-00 00-00
00-00 3E-EC 00-00 00-00 00-00 B8-0C B0-33 60-7F 00-00 40-7C 0A-04 60-7F 00-00
30-0A 0C-04 60-7F 00-00 F0-7D 07-04 60-7F 00-00 20-88 0B-04 60-7F 00-00 ...
00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00
00-00 00-00 00-00 A0-28 0C-04 60-7F 00-00 E7-52 88-A9 71-4C D7-41 00-00 00-00
01-00 00-00 08-00 00-00 03-00 00-00>
Expected: to be called once
Actual: called once - saturated and active
/home/centos/workspace/mesos/Mesos_CI-build/FLAG/CMake/label/mesos-ec2-centos-7/mesos/src/tests/storage_local_resource_provider_tests.cpp:2553:
tried expectation #1: EXPECT_CALL(sched, statusUpdate(&driver,
TaskStatusStateEq(TASK_RUNNING)))...
Expected arg #1: task status state eq TASK_RUNNING
Actual: 136-byte object <E0-A2 02-36 60-7F 00-00 00-00 00-00 00-00
00-00 3E-EC 00-00 00-00 00-00 B8-0C B0-33 60-7F 00-00 40-7C 0A-04 60-7F 00-00
30-0A 0C-04 60-7F 00-00 F0-7D 07-04 60-7F 00-00 20-88 0B-04 60-7F 00-00 ...
00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00
00-00 00-00 00-00 A0-28 0C-04 60-7F 00-00 E7-52 88-A9 71-4C D7-41 00-00 00-00
01-00 00-00 08-00 00-00 03-00 00-00>
Expected: to be called once
Actual: called once - saturated and active
/home/centos/workspace/mesos/Mesos_CI-build/FLAG/CMake/label/mesos-ec2-centos-7/mesos/src/tests/storage_local_resource_provider_tests.cpp:2554:
tried expectation #2: EXPECT_CALL(sched, statusUpdate(&driver,
TaskStatusStateEq(TASK_FINISHED)))...
Expected arg #1: task status state eq TASK_FINISHED
Actual: 136-byte object <E0-A2 02-36 60-7F 00-00 00-00 00-00 00-00
00-00 3E-EC 00-00 00-00 00-00 B8-0C B0-33 60-7F 00-00 40-7C 0A-04 60-7F 00-00
30-0A 0C-04 60-7F 00-00 F0-7D 07-04 60-7F 00-00 20-88 0B-04 60-7F 00-00 ...
00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00
00-00 00-00 00-00 A0-28 0C-04 60-7F 00-00 E7-52 88-A9 71-4C D7-41 00-00 00-00
01-00 00-00 08-00 00-00 03-00 00-00>
Expected: to be called once
Actual: called once - saturated and active
/home/centos/workspace/mesos/Mesos_CI-build/FLAG/CMake/label/mesos-ec2-centos-7/mesos/src/tests/storage_local_resource_provider_tests.cpp:2630:
tried expectation #3: EXPECT_CALL(sched, statusUpdate(&driver,
TaskStatusStateEq(TASK_STARTING)))...
Expected arg #1: task status state eq TASK_STARTING
Actual: 136-byte object <E0-A2 02-36 60-7F 00-00 00-00 00-00 00-00
00-00 3E-EC 00-00 00-00 00-00 B8-0C B0-33 60-7F 00-00 40-7C 0A-04 60-7F 00-00
30-0A 0C-04 60-7F 00-00 F0-7D 07-04 60-7F 00-00 20-88 0B-04 60-7F 00-00 ...
00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00
00-00 00-00 00-00 A0-28 0C-04 60-7F 00-00 E7-52 88-A9 71-4C D7-41 00-00 00-00
01-00 00-00 08-00 00-00 03-00 00-00>
Expected: to be called once
Actual: called once - saturated and active
/home/centos/workspace/mesos/Mesos_CI-build/FLAG/CMake/label/mesos-ec2-centos-7/mesos/src/tests/storage_local_resource_provider_tests.cpp:2631:
tried expectation #4: EXPECT_CALL(sched, statusUpdate(&driver,
TaskStatusStateEq(TASK_RUNNING)))...
Expected arg #1: task status state eq TASK_RUNNING
Actual: 136-byte object <E0-A2 02-36 60-7F 00-00 00-00 00-00 00-00
00-00 3E-EC 00-00 00-00 00-00 B8-0C B0-33 60-7F 00-00 40-7C 0A-04 60-7F 00-00
30-0A 0C-04 60-7F 00-00 F0-7D 07-04 60-7F 00-00 20-88 0B-04 60-7F 00-00 ...
00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00
00-00 00-00 00-00 A0-28 0C-04 60-7F 00-00 E7-52 88-A9 71-4C D7-41 00-00 00-00
01-00 00-00 08-00 00-00 03-00 00-00>
Expected: to be called once
Actual: never called - unsatisfied and active
/home/centos/workspace/mesos/Mesos_CI-build/FLAG/CMake/label/mesos-ec2-centos-7/mesos/src/tests/storage_local_resource_provider_tests.cpp:2632:
tried expectation #5: EXPECT_CALL(sched, statusUpdate(&driver,
TaskStatusStateEq(TASK_FINISHED)))...
Expected arg #1: task status state eq TASK_FINISHED
Actual: 136-byte object <E0-A2 02-36 60-7F 00-00 00-00 00-00 00-00
00-00 3E-EC 00-00 00-00 00-00 B8-0C B0-33 60-7F 00-00 40-7C 0A-04 60-7F 00-00
30-0A 0C-04 60-7F 00-00 F0-7D 07-04 60-7F 00-00 20-88 0B-04 60-7F 00-00 ...
00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00
00-00 00-00 00-00 A0-28 0C-04 60-7F 00-00 E7-52 88-A9 71-4C D7-41 00-00 00-00
01-00 00-00 08-00 00-00 03-00 00-00>
Expected: to be called once
Actual: never called - unsatisfied and active
{noformat}
The problem is that the task was OOM-killed:
{noformat}
I0503 01:04:24.410918 24105 memory.cpp:515] OOM detected for container
df47d711-3ee8-4430-b90d-b29bcb90d407
I0503 01:04:24.411118 24105 memory.cpp:555] Memory limit exceeded: Requested:
32MB Maximum Used: 32MB
MEMORY STATISTICS:
cache 471040
rss 33083392
rss_huge 14680064
mapped_file 0
swap 0
pgpgin 4662
pgpgout 47
pgfault 77439
pgmajfault 0
inactive_anon 471040
active_anon 33083392
inactive_file 0
active_file 0
unevictable 0
hierarchical_memory_limit 33554432
hierarchical_memsw_limit 9223372036854771712
total_cache 471040
total_rss 33083392
total_rss_huge 14680064
total_mapped_file 0
total_swap 0
total_pgpgin 4662
total_pgpgout 47
total_pgfault 77439
total_pgmajfault 0
total_inactive_anon 471040
total_active_anon 33083392
total_inactive_file 0
total_active_file 0
total_unevictable 0
I0503 01:04:24.412195 24105 containerizer.cpp:3147] Container
df47d711-3ee8-4430-b90d-b29bcb90d407 has reached its limit for resource
[{"name":"mem","scalar":{"value":32.0},"type":"SCALAR"}] and will be terminated
I0503 01:04:24.412314 24105 containerizer.cpp:2586] Destroying container
df47d711-3ee8-4430-b90d-b29bcb90d407 in RUNNING state
{noformat}
This might happen to other SLRP root tests as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
