Hi guys, I've been able to repro this more easily with some the libevent based clock I've been working on. I have a fix for this deadlock, just need to sync up with BenH to walk through it. We should have this fixed soon.
@Vinod thanks for the extra information, that's exactly where I was able to repro :-D Joris On Thu, Nov 20, 2014 at 7:14 PM, Jie Yu <[email protected]> wrote: > I've seen a similar issue on my dev box. > > > On Thursday, November 20, 2014, Vinod Kone <[email protected]> wrote: > >> This particular test got blocked at the very beginning of the test. All it >> did was start master, start slave, pause clock and wait for http request >> to >> /metrics/snapshot to return. >> >> That last part does AWAIT_READY() with default duration of 10s. AFAICT, >> that code (process::internal::await()) should return within 10s even if >> the >> clock is paused. So it is strange that it didn't. >> >> My suspicion with recent libprocess changes is because 1) i haven't seen >> this issue before and 2) seems to be related to the guts of >> libprocess/clock/timers that were recently touched. Of course, it could be >> totally unrelated to libprocess changes too. >> >> On Thu, Nov 20, 2014 at 5:34 PM, Benjamin Hindman <[email protected]> >> wrote: >> >> > What led you to that? >> > >> > On Thu, Nov 20, 2014 at 4:19 PM, Vinod Kone <[email protected]> >> wrote: >> > >> >> +benh, joris >> >> >> >> from the looks of it, i suspect the test is doing a busy loop in >> >> >> https://github.com/apache/mesos/blob/master/3rdparty/libprocess/include/process/gtest.hpp#L48 >> >> >> >> could this be related to recent libprocess changes? >> >> >> >> On Thu, Nov 20, 2014 at 4:07 PM, Vinod Kone <[email protected]> >> wrote: >> >> >> >>> too bad there is no gdb trace on this box to understand where this >> test >> >>> is stuck :( >> >>> >> >>> On Thu, Nov 20, 2014 at 3:55 PM, Apache Jenkins Server < >> >>> [email protected]> wrote: >> >>> >> >>>> See < >> >>>> >> https://builds.apache.org/job/Mesos-Trunk-Ubuntu-Build-Out-Of-Src-Disable-Java-Disable-Python-Disable-Webui/2581/changes >> >>>> > >> >>>> >> >>>> Changes: >> >>>> >> >>>> [niklas] Changed master label test to strictly enforce format. >> >>>> >> >>>> [niklas] Changed slave label test to strictly enforce format. >> >>>> >> >>>> [tnachen] Match future dispatch messages with type info. >> >>>> >> >>>> [tnachen] Fixed future dispatch in tests to match function type. >> >>>> >> >>>> ------------------------------------------ >> >>>> [...truncated 3727 lines...] >> >>>> make[3]: `no-executor-framework' is up to date. >> >>>> make[3]: `docker-no-executor-framework' is up to date. >> >>>> make[3]: `balloon-framework' is up to date. >> >>>> make[3]: `balloon-executor' is up to date. >> >>>> make[3]: `load-generator-framework' is up to date. >> >>>> make[3]: `setns-test-helper' is up to date. >> >>>> make[3]: `mesos-tests' is up to date. >> >>>> make[3]: Nothing to be done for >> >>>> `../../src/tests/balloon_framework_test.sh'. >> >>>> make[3]: Nothing to be done for >> >>>> `../../src/tests/low_level_scheduler_libprocess_test.sh'. >> >>>> make[3]: Nothing to be done for >> >>>> `../../src/tests/low_level_scheduler_pthread_test.sh'. >> >>>> make[3]: Nothing to be done for >> >>>> `../../src/tests/java_exception_test.sh'. >> >>>> make[3]: Nothing to be done for >> >>>> `../../src/tests/java_framework_test.sh'. >> >>>> make[3]: Nothing to be done for `../../src/tests/java_log_test.sh'. >> >>>> make[3]: Nothing to be done for >> >>>> `../../src/tests/no_executor_framework_test.sh'. >> >>>> make[3]: Nothing to be done for >> >>>> `../../src/tests/python_framework_test.sh'. >> >>>> make[3]: Nothing to be done for >> >>>> `../../src/tests/test_framework_test.sh'. >> >>>> make[3]: Leaving directory `< >> >>>> >> https://builds.apache.org/job/Mesos-Trunk-Ubuntu-Build-Out-Of-Src-Disable-Java-Disable-Python-Disable-Webui/ws/build/src >> >>>> '> >> >>>> make check-local >> >>>> make[3]: Entering directory `< >> >>>> >> https://builds.apache.org/job/Mesos-Trunk-Ubuntu-Build-Out-Of-Src-Disable-Java-Disable-Python-Disable-Webui/ws/build/src >> >>>> '> >> >>>> ./mesos-tests >> >>>> WARNING: Logging before InitGoogleLogging() is written to STDERR >> >>>> I1120 23:24:12.553694 30655 process.cpp:983] libprocess is >> initialized >> >>>> on 67.195.81.186:33912 for 16 cpus >> >>>> I1120 23:24:12.553962 30655 logging.cpp:177] Logging to STDERR >> >>>> Source directory: < >> >>>> >> https://builds.apache.org/job/Mesos-Trunk-Ubuntu-Build-Out-Of-Src-Disable-Java-Disable-Python-Disable-Webui/ws/ >> >>>> > >> >>>> Build directory: < >> >>>> >> https://builds.apache.org/job/Mesos-Trunk-Ubuntu-Build-Out-Of-Src-Disable-Java-Disable-Python-Disable-Webui/ws/build >> >>>> > >> >>>> ------------------------------------------------------------- >> >>>> We cannot run any cgroups tests that require mounting >> >>>> hierarchies because you have the following hierarchies mounted: >> >>>> /sys/fs/cgroup/blkio, /sys/fs/cgroup/cpu, /sys/fs/cgroup/cpuacct, >> >>>> /sys/fs/cgroup/cpuset, /sys/fs/cgroup/devices, >> /sys/fs/cgroup/freezer, >> >>>> /sys/fs/cgroup/hugetlb, /sys/fs/cgroup/memory, >> /sys/fs/cgroup/perf_event, >> >>>> /sys/fs/cgroup/systemd >> >>>> We'll disable the CgroupsNoHierarchyTest test fixture for now. >> >>>> ------------------------------------------------------------- >> >>>> Note: Google Test filter = >> >>>> >> *-DockerContainerizerTest.ROOT_DOCKER_Launch_Executor:DockerContainerizerTest.ROOT_DOCKER_Launch_Executor_Bridged:DockerContainerizerTest.ROOT_DOCKER_Launch:DockerContainerizerTest.ROOT_DOCKER_Kill:DockerContainerizerTest.ROOT_DOCKER_Usage:DockerContainerizerTest.ROOT_DOCKER_Update:DockerContainerizerTest.DISABLED_ROOT_DOCKER_Recover:DockerContainerizerTest.ROOT_DOCKER_Logs:DockerContainerizerTest.ROOT_DOCKER_Default_CMD:DockerContainerizerTest.ROOT_DOCKER_Default_CMD_Override:DockerContainerizerTest.ROOT_DOCKER_Default_CMD_Args:DockerContainerizerTest.ROOT_DOCKER_SlaveRecoveryTaskContainer:DockerContainerizerTest.DISABLED_ROOT_DOCKER_SlaveRecoveryExecutorContainer:DockerContainerizerTest.ROOT_DOCKER_PortMapping:DockerContainerizerTest.ROOT_DOCKER_LaunchSandboxWithColon:DockerContainerizerTest.ROOT_DOCKER_DestroyWhileFetching:DockerContainerizerTest.ROOT_DOCKER_DestroyWhilePulling:DockerTest.ROOT_DOCKER_interface:DockerTest.ROOT_DOCKER_CheckCommandWithShell:DockerTest.ROOT_DOCKER_CheckPortResource:DockerTest.ROOT_DOCKER_CancelPull:CpuIsolatorTest/1.UserCpuUsage:CpuIsolatorTest/1.SystemCpuUsage:LimitedCpuIsolatorTest.ROOT_CGROUPS_Cfs:LimitedCpuIsolatorTest.ROOT_CGROUPS_Cfs_Big_Quota:MemIsolatorTest/0.MemUsage:MemIsolatorTest/1.MemUsage:MemIsolatorTest/2.MemUsage:PerfEventIsolatorTest.ROOT_CGROUPS_Sample:SharedFilesystemIsolatorTest.ROOT_RelativeVolume:SharedFilesystemIsolatorTest.ROOT_AbsoluteVolume:NamespacesPidIsolatorTest.ROOT_PidNamespace:UserCgroupIsolatorTest/0.ROOT_CGROUPS_UserCgroup:UserCgroupIsolatorTest/1.ROOT_CGROUPS_UserCgroup:UserCgroupIsolatorTest/2.ROOT_CGROUPS_UserCgroup:MesosContainerizerSlaveRecoveryTest.CGROUPS_ROOT_PerfRollForward:MesosContainerizerSlaveRecoveryTest.CGROUPS_ROOT_PidNamespaceForward:MesosContainerizerSlaveRecoveryTest.CGROUPS_ROOT_PidNamespaceBackward:SlaveTest.ROOT_RunTaskWithCommandInfoWithoutUser:SlaveTest.DISABLED_ROOT_RunTaskWithCommandInfoWithUser:ContainerizerTest.ROOT_CGROUPS_BalloonFramework:CgroupsAnyHierarchyTest.ROOT_CGROUPS_Enabled:CgroupsAnyHierarchyTest.ROOT_CGROUPS_Subsystems:CgroupsAnyHierarchyTest.ROOT_CGROUPS_Mounted:CgroupsAnyHierarchyTest.ROOT_CGROUPS_Get:CgroupsAnyHierarchyTest.ROOT_CGROUPS_NestedCgroups:CgroupsAnyHierarchyTest.ROOT_CGROUPS_Tasks:CgroupsAnyHierarchyTest.ROOT_CGROUPS_Read:CgroupsAnyHierarchyTest.ROOT_CGROUPS_Write:CgroupsAnyHierarchyTest.ROOT_CGROUPS_Cfs_Big_Quota:CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_Busy:CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_SubsystemsHierarchy:CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_FindCgroupSubsystems:CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_MountedSubsystems:CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_CreateRemove:CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_Listen:CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_FreezeNonFreezer:CgroupsNoHierarchyTest.ROOT_CGROUPS_NOHIERARCHY_MountUnmountHierarchy:CgroupsAnyHierarchyWithCpuAcctMemoryTest.ROOT_CGROUPS_Stat:CgroupsAnyHierarchyWithFreezerTest.ROOT_CGROUPS_Freeze:CgroupsAnyHierarchyWithFreezerTest.ROOT_CGROUPS_Kill:CgroupsAnyHierarchyWithFreezerTest.ROOT_CGROUPS_Destroy:CgroupsAnyHierarchyWithFreezerTest.ROOT_CGROUPS_AssignThreads:CgroupsAnyHierarchyWithFreezerTest.ROOT_CGROUPS_DestroyStoppedProcess:CgroupsAnyHierarchyWithFreezerTest.ROOT_CGROUPS_DestroyTracedProcess:CgroupsAnyHierarchyWithPerfEventTest.ROOT_CGROUPS_Perf:NsTest.ROOT_setns:NsTest.ROOT_setnsMultipleThreads:NsTest.ROOT_getns:NsTest.ROOT_destroy:PerfTest.ROOT_Events:PerfTest.ROOT_SampleInit:SlaveCount/Registrar_BENCHMARK_Test.performance/0:SlaveCount/Registrar_BENCHMARK_Test.performance/1:SlaveCount/Registrar_BENCHMARK_Test.performance/2:SlaveCount/Registrar_BENCHMARK_Test.performance/3 >> >>>> Note: Randomizing tests' orders with a seed of 47110 . >> >>>> [==========] Running 437 tests from 68 test cases. >> >>>> [----------] Global test environment set-up. >> >>>> [----------] 4 tests from CRAMMD5Authentication/2, where TypeParam = >> >>>> >> mesos::internal::cram_md5::Authentication<mesos::internal::cram_md5::CRAMMD5Authenticatee, >> >>>> mesos::internal::tests::Module<mesos::internal::Authenticator, >> >>>> (mesos::internal::tests::ModuleID)3> > >> >>>> [ RUN ] CRAMMD5Authentication/2.failed1 >> >>>> Using temporary directory >> '/tmp/CRAMMD5Authentication_2_failed1_V0pTyx' >> >>>> I1120 23:24:12.685442 30676 authenticatee.hpp:114] Initializing >> client >> >>>> SASL >> >>>> I1120 23:24:12.686884 30676 authenticatee.hpp:138] Creating new >> client >> >>>> SASL connection >> >>>> I1120 23:24:12.687741 30678 authenticator.hpp:108] Initializing >> server >> >>>> SASL >> >>>> I1120 23:24:12.687944 30678 auxprop.cpp:63] Initialized in-memory >> >>>> auxiliary property plugin >> >>>> I1120 23:24:12.687966 30678 authenticator.hpp:170] Creating new >> server >> >>>> SASL connection >> >>>> I1120 23:24:12.688140 30678 authenticatee.hpp:229] Received SASL >> >>>> authentication mechanisms: CRAM-MD5 >> >>>> I1120 23:24:12.688180 30678 authenticatee.hpp:255] Attempting to >> >>>> authenticate with mechanism 'CRAM-MD5' >> >>>> I1120 23:24:12.688294 30678 authenticator.hpp:276] Received SASL >> >>>> authentication start >> >>>> I1120 23:24:12.688361 30678 authenticator.hpp:398] Authentication >> >>>> requires more steps >> >>>> I1120 23:24:12.688472 30673 authenticatee.hpp:275] Received SASL >> >>>> authentication step >> >>>> I1120 23:24:12.688617 30673 authenticator.hpp:304] Received SASL >> >>>> authentication step >> >>>> I1120 23:24:12.688649 30673 auxprop.cpp:99] Request to lookup >> >>>> properties for user: 'benh' realm: 'penates.apache.org' server >> FQDN: ' >> >>>> penates.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false >> >>>> SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false >> >>>> I1120 23:24:12.688663 30673 auxprop.cpp:171] Looking up auxiliary >> >>>> property '*userPassword' >> >>>> I1120 23:24:12.688814 30673 auxprop.cpp:171] Looking up auxiliary >> >>>> property '*cmusaslsecretCRAM-MD5' >> >>>> I1120 23:24:12.688846 30673 auxprop.cpp:99] Request to lookup >> >>>> properties for user: 'benh' realm: 'penates.apache.org' server >> FQDN: ' >> >>>> penates.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false >> >>>> SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true >> >>>> I1120 23:24:12.688858 30673 auxprop.cpp:121] Skipping auxiliary >> >>>> property '*userPassword' since SASL_AUXPROP_AUTHZID == true >> >>>> I1120 23:24:12.688863 30673 auxprop.cpp:121] Skipping auxiliary >> >>>> property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true >> >>>> W1120 23:24:12.688887 30673 authenticator.hpp:404] Authentication >> >>>> failure: authentication failure >> >>>> [ OK ] CRAMMD5Authentication/2.failed1 (11 ms) >> >>>> [ RUN ] CRAMMD5Authentication/2.AuthenticatorDestructionRace >> >>>> Using temporary directory >> >>>> '/tmp/CRAMMD5Authentication_2_AuthenticatorDestructionRace_CRclP9' >> >>>> I1120 23:24:12.690254 30684 authenticatee.hpp:138] Creating new >> client >> >>>> SASL connection >> >>>> I1120 23:24:12.690830 30676 authenticator.hpp:170] Creating new >> server >> >>>> SASL connection >> >>>> I1120 23:24:12.690989 30682 authenticatee.hpp:229] Received SASL >> >>>> authentication mechanisms: CRAM-MD5 >> >>>> I1120 23:24:12.691015 30682 authenticatee.hpp:255] Attempting to >> >>>> authenticate with mechanism 'CRAM-MD5' >> >>>> I1120 23:24:12.691097 30671 authenticator.hpp:276] Received SASL >> >>>> authentication start >> >>>> I1120 23:24:12.691135 30671 authenticator.hpp:398] Authentication >> >>>> requires more steps >> >>>> [ OK ] CRAMMD5Authentication/2.AuthenticatorDestructionRace (2 >> ms) >> >>>> [ RUN ] CRAMMD5Authentication/2.success >> >>>> Using temporary directory >> '/tmp/CRAMMD5Authentication_2_success_JZF85L' >> >>>> I1120 23:24:12.692466 30682 authenticatee.hpp:138] Creating new >> client >> >>>> SASL connection >> >>>> I1120 23:24:12.692939 30676 authenticator.hpp:170] Creating new >> server >> >>>> SASL connection >> >>>> I1120 23:24:12.693069 30684 authenticatee.hpp:229] Received SASL >> >>>> authentication mechanisms: CRAM-MD5 >> >>>> I1120 23:24:12.693094 30684 authenticatee.hpp:255] Attempting to >> >>>> authenticate with mechanism 'CRAM-MD5' >> >>>> I1120 23:24:12.693171 30682 authenticator.hpp:276] Received SASL >> >>>> authentication start >> >>>> I1120 23:24:12.693207 30682 authenticator.hpp:398] Authentication >> >>>> requires more steps >> >>>> I1120 23:24:12.693274 30684 authenticatee.hpp:275] Received SASL >> >>>> authentication step >> >>>> I1120 23:24:12.693366 30682 authenticator.hpp:304] Received SASL >> >>>> authentication step >> >>>> I1120 23:24:12.693394 30682 auxprop.cpp:99] Request to lookup >> >>>> properties for user: 'benh' realm: 'penates.apache.org' server >> FQDN: ' >> >>>> penates.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false >> >>>> SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false >> >>>> I1120 23:24:12.693408 30682 auxprop.cpp:171] Looking up auxiliary >> >>>> property '*userPassword' >> >>>> I1120 23:24:12.693431 30682 auxprop.cpp:171] Looking up auxiliary >> >>>> property '*cmusaslsecretCRAM-MD5' >> >>>> I1120 23:24:12.693449 30682 auxprop.cpp:99] Request to lookup >> >>>> properties for user: 'benh' realm: 'penates.apache.org' server >> FQDN: ' >> >>>> penates.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false >> >>>> SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true >> >>>> I1120 23:24:12.693456 30682 auxprop.cpp:121] Skipping auxiliary >> >>>> property '*userPassword' since SASL_AUXPROP_AUTHZID == true >> >>>> I1120 23:24:12.693461 30682 auxprop.cpp:121] Skipping auxiliary >> >>>> property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true >> >>>> I1120 23:24:12.693476 30682 authenticator.hpp:390] Authentication >> >>>> success >> >>>> I1120 23:24:12.693544 30684 authenticatee.hpp:315] Authentication >> >>>> success >> >>>> [ OK ] CRAMMD5Authentication/2.success (3 ms) >> >>>> [ RUN ] CRAMMD5Authentication/2.failed2 >> >>>> Using temporary directory >> '/tmp/CRAMMD5Authentication_2_failed2_yugjno' >> >>>> I1120 23:24:12.694799 30677 authenticatee.hpp:138] Creating new >> client >> >>>> SASL connection >> >>>> I1120 23:24:12.695314 30676 authenticator.hpp:170] Creating new >> server >> >>>> SASL connection >> >>>> I1120 23:24:12.695435 30674 authenticatee.hpp:229] Received SASL >> >>>> authentication mechanisms: CRAM-MD5 >> >>>> I1120 23:24:12.695457 30674 authenticatee.hpp:255] Attempting to >> >>>> authenticate with mechanism 'CRAM-MD5' >> >>>> I1120 23:24:12.695544 30670 authenticator.hpp:276] Received SASL >> >>>> authentication start >> >>>> I1120 23:24:12.695582 30670 authenticator.hpp:398] Authentication >> >>>> requires more steps >> >>>> I1120 23:24:12.695657 30674 authenticatee.hpp:275] Received SASL >> >>>> authentication step >> >>>> I1120 23:24:12.695745 30674 authenticator.hpp:304] Received SASL >> >>>> authentication step >> >>>> I1120 23:24:12.695770 30674 auxprop.cpp:99] Request to lookup >> >>>> properties for user: 'benh' realm: 'penates.apache.org' server >> FQDN: ' >> >>>> penates.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false >> >>>> SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false >> >>>> I1120 23:24:12.695778 30674 auxprop.cpp:171] Looking up auxiliary >> >>>> property '*userPassword' >> >>>> I1120 23:24:12.695788 30674 auxprop.cpp:171] Looking up auxiliary >> >>>> property '*cmusaslsecretCRAM-MD5' >> >>>> I1120 23:24:12.695797 30674 auxprop.cpp:99] Request to lookup >> >>>> properties for user: 'benh' realm: 'penates.apache.org' server >> FQDN: ' >> >>>> penates.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false >> >>>> SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true >> >>>> I1120 23:24:12.695803 30674 auxprop.cpp:121] Skipping auxiliary >> >>>> property '*userPassword' since SASL_AUXPROP_AUTHZID == true >> >>>> I1120 23:24:12.695809 30674 auxprop.cpp:121] Skipping auxiliary >> >>>> property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true >> >>>> W1120 23:24:12.695818 30674 authenticator.hpp:404] Authentication >> >>>> failure: user not found >> >>>> [ OK ] CRAMMD5Authentication/2.failed2 (2 ms) >> >>>> [----------] 4 tests from CRAMMD5Authentication/2 (18 ms total) >> >>>> >> >>>> [----------] 6 tests from RateLimitingTest >> >>>> [ RUN ] RateLimitingTest.NoRateLimiting >> >>>> Using temporary directory >> '/tmp/RateLimitingTest_NoRateLimiting_XteRE0' >> >>>> I1120 23:24:12.701375 30655 leveldb.cpp:176] Opened db in 3.4075ms >> >>>> I1120 23:24:12.703060 30655 leveldb.cpp:183] Compacted db in >> 1.64536ms >> >>>> I1120 23:24:12.703178 30655 leveldb.cpp:198] Created db iterator in >> >>>> 78140ns >> >>>> I1120 23:24:12.703205 30655 leveldb.cpp:204] Seeked to beginning of >> db >> >>>> in 3649ns >> >>>> I1120 23:24:12.703217 30655 leveldb.cpp:273] Iterated through 0 keys >> in >> >>>> the db in 400ns >> >>>> I1120 23:24:12.703368 30655 replica.cpp:741] Replica recovered with >> log >> >>>> positions 0 -> 0 with 1 holes and 0 unlearned >> >>>> I1120 23:24:12.705715 30678 recover.cpp:437] Starting replica >> recovery >> >>>> I1120 23:24:12.706840 30678 recover.cpp:463] Replica is in EMPTY >> status >> >>>> I1120 23:24:12.711232 30681 replica.cpp:638] Replica in EMPTY status >> >>>> received a broadcasted recover request >> >>>> I1120 23:24:12.712901 30683 recover.cpp:188] Received a recover >> >>>> response from a replica in EMPTY status >> >>>> I1120 23:24:12.713747 30673 recover.cpp:554] Updating replica status >> to >> >>>> STARTING >> >>>> I1120 23:24:12.714715 30683 leveldb.cpp:306] Persisting metadata (8 >> >>>> bytes) to leveldb took 708621ns >> >>>> I1120 23:24:12.714754 30683 replica.cpp:320] Persisted replica status >> >>>> to STARTING >> >>>> I1120 23:24:12.715100 30684 recover.cpp:463] Replica is in STARTING >> >>>> status >> >>>> I1120 23:24:12.716419 30679 replica.cpp:638] Replica in STARTING >> status >> >>>> received a broadcasted recover request >> >>>> I1120 23:24:12.716961 30676 recover.cpp:188] Received a recover >> >>>> response from a replica in STARTING status >> >>>> I1120 23:24:12.717490 30677 recover.cpp:554] Updating replica status >> to >> >>>> VOTING >> >>>> I1120 23:24:12.718242 30680 leveldb.cpp:306] Persisting metadata (8 >> >>>> bytes) to leveldb took 484920ns >> >>>> I1120 23:24:12.718276 30680 replica.cpp:320] Persisted replica status >> >>>> to VOTING >> >>>> I1120 23:24:12.718385 30672 recover.cpp:568] Successfully joined the >> >>>> Paxos group >> >>>> I1120 23:24:12.718698 30672 recover.cpp:452] Recover process >> terminated >> >>>> I1120 23:24:12.723183 30670 master.cpp:318] Master >> >>>> 20141120-232412-3125920579-33912-30655 (penates.apache.org) started >> on >> >>>> 67.195.81.186:33912 >> >>>> I1120 23:24:12.723278 30670 master.cpp:364] Master only allowing >> >>>> authenticated frameworks to register >> >>>> I1120 23:24:12.723353 30670 master.cpp:369] Master only allowing >> >>>> authenticated slaves to register >> >>>> I1120 23:24:12.723420 30670 credentials.hpp:36] Loading credentials >> for >> >>>> authentication from >> >>>> '/tmp/RateLimitingTest_NoRateLimiting_XteRE0/credentials' >> >>>> I1120 23:24:12.723983 30670 master.cpp:413] Authorization enabled >> >>>> I1120 23:24:12.724187 30670 master.cpp:460] Framework rate limiting >> >>>> enabled >> >>>> I1120 23:24:12.724884 30678 master.cpp:126] No whitelist given. >> >>>> Advertising offers for all slaves >> >>>> I1120 23:24:12.725131 30684 hierarchical_allocator_process.hpp:299] >> >>>> Initializing hierarchical allocator process with master : >> >>>> [email protected]:33912 >> >>>> I1120 23:24:12.728044 30676 master.cpp:1263] The newly elected leader >> >>>> is [email protected]:33912 with id >> >>>> 20141120-232412-3125920579-33912-30655 >> >>>> I1120 23:24:12.728078 30676 master.cpp:1276] Elected as the leading >> >>>> master! >> >>>> I1120 23:24:12.728116 30676 master.cpp:1094] Recovering from >> registrar >> >>>> I1120 23:24:12.728454 30679 registrar.cpp:313] Recovering registrar >> >>>> I1120 23:24:12.730170 30677 log.cpp:656] Attempting to start the >> writer >> >>>> I1120 23:24:12.733129 30676 replica.cpp:474] Replica received >> implicit >> >>>> promise request with proposal 1 >> >>>> I1120 23:24:12.733763 30676 leveldb.cpp:306] Persisting metadata (8 >> >>>> bytes) to leveldb took 598467ns >> >>>> I1120 23:24:12.733795 30676 replica.cpp:342] Persisted promised to 1 >> >>>> I1120 23:24:12.735090 30680 coordinator.cpp:230] Coordinator >> attemping >> >>>> to fill missing position >> >>>> I1120 23:24:12.737241 30677 replica.cpp:375] Replica received >> explicit >> >>>> promise request for position 0 with proposal 2 >> >>>> I1120 23:24:12.737690 30677 leveldb.cpp:343] Persisting action (8 >> >>>> bytes) to leveldb took 402621ns >> >>>> I1120 23:24:12.737721 30677 replica.cpp:676] Persisted action at 0 >> >>>> I1120 23:24:12.739233 30681 replica.cpp:508] Replica received write >> >>>> request for position 0 >> >>>> I1120 23:24:12.739351 30681 leveldb.cpp:438] Reading position from >> >>>> leveldb took 79235ns >> >>>> I1120 23:24:12.739878 30681 leveldb.cpp:343] Persisting action (14 >> >>>> bytes) to leveldb took 448572ns >> >>>> I1120 23:24:12.739907 30681 replica.cpp:676] Persisted action at 0 >> >>>> I1120 23:24:12.740696 30679 replica.cpp:655] Replica received learned >> >>>> notice for position 0 >> >>>> I1120 23:24:12.741052 30679 leveldb.cpp:343] Persisting action (16 >> >>>> bytes) to leveldb took 322694ns >> >>>> I1120 23:24:12.741083 30679 replica.cpp:676] Persisted action at 0 >> >>>> I1120 23:24:12.741117 30679 replica.cpp:661] Replica learned NOP >> action >> >>>> at position 0 >> >>>> I1120 23:24:12.741811 30672 log.cpp:672] Writer started with ending >> >>>> position 0 >> >>>> I1120 23:24:12.744951 30670 leveldb.cpp:438] Reading position from >> >>>> leveldb took 30629ns >> >>>> I1120 23:24:12.748874 30678 registrar.cpp:346] Successfully fetched >> the >> >>>> registry (0B) in 20.35584ms >> >>>> I1120 23:24:12.749171 30678 registrar.cpp:445] Applied 1 operations >> in >> >>>> 87917ns; attempting to update the 'registry' >> >>>> I1120 23:24:12.752719 30672 log.cpp:680] Attempting to append 140 >> bytes >> >>>> to the log >> >>>> I1120 23:24:12.752936 30671 coordinator.cpp:340] Coordinator >> attempting >> >>>> to write APPEND action at position 1 >> >>>> I1120 23:24:12.753829 30671 replica.cpp:508] Replica received write >> >>>> request for position 1 >> >>>> I1120 23:24:12.754359 30671 leveldb.cpp:343] Persisting action (159 >> >>>> bytes) to leveldb took 494376ns >> >>>> I1120 23:24:12.754387 30671 replica.cpp:676] Persisted action at 1 >> >>>> I1120 23:24:12.755213 30677 replica.cpp:655] Replica received learned >> >>>> notice for position 1 >> >>>> I1120 23:24:12.755621 30677 leveldb.cpp:343] Persisting action (161 >> >>>> bytes) to leveldb took 375152ns >> >>>> I1120 23:24:12.755648 30677 replica.cpp:676] Persisted action at 1 >> >>>> I1120 23:24:12.755669 30677 replica.cpp:661] Replica learned APPEND >> >>>> action at position 1 >> >>>> I1120 23:24:12.757231 30673 registrar.cpp:490] Successfully updated >> the >> >>>> 'registry' in 7.922944ms >> >>>> I1120 23:24:12.757463 30673 registrar.cpp:376] Successfully recovered >> >>>> registrar >> >>>> I1120 23:24:12.757665 30682 log.cpp:699] Attempting to truncate the >> log >> >>>> to 1 >> >>>> I1120 23:24:12.757859 30679 coordinator.cpp:340] Coordinator >> attempting >> >>>> to write TRUNCATE action at position 2 >> >>>> I1120 23:24:12.757899 30681 master.cpp:1121] Recovered 0 slaves from >> >>>> the Registry (102B) ; allowing 10mins for slaves to re-register >> >>>> I1120 23:24:12.758951 30680 replica.cpp:508] Replica received write >> >>>> request for position 2 >> >>>> I1120 23:24:12.759395 30680 leveldb.cpp:343] Persisting action (16 >> >>>> bytes) to leveldb took 408946ns >> >>>> I1120 23:24:12.759429 30680 replica.cpp:676] Persisted action at 2 >> >>>> I1120 23:24:12.760109 30675 replica.cpp:655] Replica received learned >> >>>> notice for position 2 >> >>>> I1120 23:24:12.760496 30675 leveldb.cpp:343] Persisting action (18 >> >>>> bytes) to leveldb took 356913ns >> >>>> I1120 23:24:12.760584 30675 leveldb.cpp:401] Deleting ~1 keys from >> >>>> leveldb took 45969ns >> >>>> I1120 23:24:12.760608 30675 replica.cpp:676] Persisted action at 2 >> >>>> I1120 23:24:12.760633 30675 replica.cpp:661] Replica learned TRUNCATE >> >>>> action at position 2 >> >>>> I1120 23:24:12.779798 30680 process.cpp:2881] Handling HTTP event for >> >>>> process 'metrics' with path: '/metrics/snapshot' >> >>>> Thu Nov 20 23:54:06 UTC 2014: process still running after 1800 >> seconds >> >>>> ========== >> >>>> >> >>>> Attaching gdb to 30437 bash ../support/timed_tests.sh GLOG_v=1 >> >>>> MESOS_VERBOSE=1 make check GTEST_SHUFFLE=1 GTEST_OUTPUT=xml: 1800 >> >>>> >> >>>> ========== >> >>>> ../support/timed_tests.sh: line 72: gdb: command not found >> >>>> ========== >> >>>> >> >>>> Attaching gdb to 30440 bash ../support/timed_tests.sh GLOG_v=1 >> >>>> MESOS_VERBOSE=1 make check GTEST_SHUFFLE=1 GTEST_OUTPUT=xml: 1800 >> >>>> >> >>>> ========== >> >>>> ../support/timed_tests.sh: line 72: gdb: command not found >> >>>> ========== >> >>>> >> >>>> Attaching gdb to 30442 make check GTEST_SHUFFLE=1 GTEST_OUTPUT=xml: >> >>>> >> >>>> ========== >> >>>> ../support/timed_tests.sh: line 72: gdb: command not found >> >>>> ========== >> >>>> >> >>>> Attaching gdb to 30446 /bin/bash -c fail=; \ if (target_option=k; >> case >> >>>> ${target_option-} in ?) ;; *) echo "am__make_running_with_option: >> internal >> >>>> error: invalid" "target option '${target_option-}' specified" >&2; >> exit 1;; >> >>>> esac; has_opt=no; sane_makeflags=$MAKEFLAGS; if test -n ' Makefile' >> && test >> >>>> -n '0'; then sane_makeflags=$MFLAGS; else case $MAKEFLAGS in *\\[\ >> \.]*) >> >>>> bs=\\; sane_makeflags=`printf '%s\n' "$MAKEFLAGS" | sed "s/$bs$bs[$bs >> >>>> $bs.]*//g"`;; esac; fi; skip_next=no; strip_trailopt () { flg=`printf >> >>>> '%s\n' "$flg" | sed "s/$1.*$//"`; }; for flg in $sane_makeflags; do >> test >> >>>> $skip_next = yes && { skip_next=no; continue; }; case $flg in >> *=*|--*) >> >>>> continue;; -*I) strip_trailopt 'I'; skip_next=yes;; -*I?*) >> strip_trailopt >> >>>> 'I';; -*O) strip_trailopt 'O'; skip_next=yes;; -*O?*) strip_trailopt >> 'O';; >> >>>> -*l) strip_trailopt 'l'; skip_next=yes;; -*l?*) strip_trailopt 'l';; >> >>>> -[dEDm]) skip_next=yes;; -[JT]) skip_next=yes;; esac; case $flg in >> >>>> *$target_option*) has_opt=yes; break;; esac; done; test $has_opt = >> yes); >> >>>> then \ failcom='fail=yes'; \ else \ failcom='exit 1'; \ fi; \ >> >>>> dot_seen=no; \ target=`echo check-recursive | sed s/-recursive//`; \ >> case >> >>>> "check-recursive" in \ distclean-* | maintainer-clean-*) list='. >> 3rdparty >> >>>> src ec2' ;; \ *) list='. 3rdparty src ec2' ;; \ esac; \ for subdir >> in >> >>>> $list; do \ echo "Making $target in $subdir"; \ if test >> "$subdir" = >> >>>> "."; then \ dot_seen=yes; \ local_target="$target-am"; \ >> else \ >> >>>> local_target="$target"; \ fi; \ (CDPATH="${ZSH_VERSION+.}:" >> && cd >> >>>> $subdir && make $local_target) \ || eval $failcom; \ done; \ if >> test >> >>>> "$dot_seen" = "no"; then \ make "$target-am" || exit 1; \ fi; >> test -z >> >>>> "$fail" >> >>>> >> >>>> ========== >> >>>> ../support/timed_tests.sh: line 72: gdb: command not found >> >>>> ========== >> >>>> >> >>>> Attaching gdb to 30642 /bin/bash -c fail=; \ if (target_option=k; >> case >> >>>> ${target_option-} in ?) ;; *) echo "am__make_running_with_option: >> internal >> >>>> error: invalid" "target option '${target_option-}' specified" >&2; >> exit 1;; >> >>>> esac; has_opt=no; sane_makeflags=$MAKEFLAGS; if test -n ' Makefile' >> && test >> >>>> -n '0'; then sane_makeflags=$MFLAGS; else case $MAKEFLAGS in *\\[\ >> \.]*) >> >>>> bs=\\; sane_makeflags=`printf '%s\n' "$MAKEFLAGS" | sed "s/$bs$bs[$bs >> >>>> $bs.]*//g"`;; esac; fi; skip_next=no; strip_trailopt () { flg=`printf >> >>>> '%s\n' "$flg" | sed "s/$1.*$//"`; }; for flg in $sane_makeflags; do >> test >> >>>> $skip_next = yes && { skip_next=no; continue; }; case $flg in >> *=*|--*) >> >>>> continue;; -*I) strip_trailopt 'I'; skip_next=yes;; -*I?*) >> strip_trailopt >> >>>> 'I';; -*O) strip_trailopt 'O'; skip_next=yes;; -*O?*) strip_trailopt >> 'O';; >> >>>> -*l) strip_trailopt 'l'; skip_next=yes;; -*l?*) strip_trailopt 'l';; >> >>>> -[dEDm]) skip_next=yes;; -[JT]) skip_next=yes;; esac; case $flg in >> >>>> *$target_option*) has_opt=yes; break;; esac; done; test $has_opt = >> yes); >> >>>> then \ failcom='fail=yes'; \ else \ failcom='exit 1'; \ fi; \ >> >>>> dot_seen=no; \ target=`echo check-recursive | sed s/-recursive//`; \ >> case >> >>>> "check-recursive" in \ distclean-* | maintainer-clean-*) list='. >> 3rdparty >> >>>> src ec2' ;; \ *) list='. 3rdparty src ec2' ;; \ esac; \ for subdir >> in >> >>>> $list; do \ echo "Making $target in $subdir"; \ if test >> "$subdir" = >> >>>> "."; then \ dot_seen=yes; \ local_target="$target-am"; \ >> else \ >> >>>> local_target="$target"; \ fi; \ (CDPATH="${ZSH_VERSION+.}:" >> && cd >> >>>> $subdir && make $local_target) \ || eval $failcom; \ done; \ if >> test >> >>>> "$dot_seen" = "no"; then \ make "$target-am" || exit 1; \ fi; >> test -z >> >>>> "$fail" >> >>>> >> >>>> ========== >> >>>> ../support/timed_tests.sh: line 72: gdb: command not found >> >>>> ========== >> >>>> >> >>>> Attaching gdb to 30643 make check >> >>>> >> >>>> ========== >> >>>> ../support/timed_tests.sh: line 72: gdb: command not found >> >>>> ========== >> >>>> >> >>>> Attaching gdb to 30645 make check-am >> >>>> >> >>>> ========== >> >>>> ../support/timed_tests.sh: line 72: gdb: command not found >> >>>> ========== >> >>>> >> >>>> Attaching gdb to 30652 make check-local >> >>>> >> >>>> ========== >> >>>> ../support/timed_tests.sh: line 72: gdb: command not found >> >>>> ========== >> >>>> >> >>>> Attaching gdb to 30655 < >> >>>> >> https://builds.apache.org/job/Mesos-Trunk-Ubuntu-Build-Out-Of-Src-Disable-Java-Disable-Python-Disable-Webui/ws/build/src/.libs/lt-mesos-tests >> >>>> > >> >>>> >> >>>> ========== >> >>>> ../support/timed_tests.sh: line 72: gdb: command not found >> >>>> Test failed and killing the stuck test process >> >>>> W1120 23:24:12.779798 30669 logging.cpp:81] RAW: Received signal >> >>>> SIGTERM from process 30652 of user 2396; exiting >> >>>> W1120 23:24:12.779798 30655 logging.cpp:81] RAW: Received signal >> >>>> SIGTERM from process 20380 of user 2396; exiting >> >>>> make: *** [check-recursive] Terminated >> >>>> /tmp/hudson9138584248396286316.sh: line 16: 30437 Terminated >> >>>> ../support/timed_tests.sh "GLOG_v=1 MESOS_VERBOSE=1 make check >> >>>> GTEST_SHUFFLE=1 GTEST_OUTPUT=xml:" 1800 >> >>>> make[3]: *** [check-local] Terminated >> >>>> make[2]: *** [check-am] Terminated >> >>>> make[1]: *** [check] Terminated >> >>>> Build step 'Execute shell' marked build as failure >> >>>> Recording test results >> >>>> >> >>> >> >>> >> >> >> > >> >
