Here is the RB: https://reviews.apache.org/r/28367/
Joris On Fri, Nov 21, 2014 at 11:06 AM, Joris Van Remoortere <jo...@mesosphere.io> wrote: > Hi guys, > > I've been able to repro this more easily with some the libevent based > clock I've been working on. > I have a fix for this deadlock, just need to sync up with BenH to walk > through it. We should have this fixed soon. > > @Vinod thanks for the extra information, that's exactly where I was able > to repro :-D > > Joris > > On Thu, Nov 20, 2014 at 7:14 PM, Jie Yu <yujie....@gmail.com> wrote: > >> I've seen a similar issue on my dev box. >> >> >> On Thursday, November 20, 2014, Vinod Kone <vinodk...@gmail.com> wrote: >> >>> This particular test got blocked at the very beginning of the test. All >>> it >>> did was start master, start slave, pause clock and wait for http request >>> to >>> /metrics/snapshot to return. >>> >>> That last part does AWAIT_READY() with default duration of 10s. AFAICT, >>> that code (process::internal::await()) should return within 10s even if >>> the >>> clock is paused. So it is strange that it didn't. >>> >>> My suspicion with recent libprocess changes is because 1) i haven't seen >>> this issue before and 2) seems to be related to the guts of >>> libprocess/clock/timers that were recently touched. Of course, it could >>> be >>> totally unrelated to libprocess changes too. >>> >>> On Thu, Nov 20, 2014 at 5:34 PM, Benjamin Hindman <b...@mesosphere.io> >>> wrote: >>> >>> > What led you to that? >>> > >>> > On Thu, Nov 20, 2014 at 4:19 PM, Vinod Kone <vinodk...@gmail.com> >>> wrote: >>> > >>> >> +benh, joris >>> >> >>> >> from the looks of it, i suspect the test is doing a busy loop in >>> >> >>> https://github.com/apache/mesos/blob/master/3rdparty/libprocess/include/process/gtest.hpp#L48 >>> >> >>> >> could this be related to recent libprocess changes? >>> >> >>> >> On Thu, Nov 20, 2014 at 4:07 PM, Vinod Kone <vinodk...@gmail.com> >>> wrote: >>> >> >>> >>> too bad there is no gdb trace on this box to understand where this >>> test >>> >>> is stuck :( >>> >>> >>> >>> On Thu, Nov 20, 2014 at 3:55 PM, Apache Jenkins Server < >>> >>> jenk...@builds.apache.org> wrote: >>> >>> >>> >>>> See < >>> >>>> >>> https://builds.apache.org/job/Mesos-Trunk-Ubuntu-Build-Out-Of-Src-Disable-Java-Disable-Python-Disable-Webui/2581/changes >>> >>>> > >>> >>>> >>> >>>> Changes: >>> >>>> >>> >>>> [niklas] Changed master label test to strictly enforce format. >>> >>>> >>> >>>> [niklas] Changed slave label test to strictly enforce format. >>> >>>> >>> >>>> [tnachen] Match future dispatch messages with type info. >>> >>>> >>> >>>> [tnachen] Fixed future dispatch in tests to match function type. >>> >>>> >>> >>>> ------------------------------------------ >>> >>>> [...truncated 3727 lines...] >>> >>>> make[3]: `no-executor-framework' is up to date. >>> >>>> make[3]: `docker-no-executor-framework' is up to date. >>> >>>> make[3]: `balloon-framework' is up to date. >>> >>>> make[3]: `balloon-executor' is up to date. >>> >>>> make[3]: `load-generator-framework' is up to date. >>> >>>> make[3]: `setns-test-helper' is up to date. >>> >>>> make[3]: `mesos-tests' is up to date. >>> >>>> make[3]: Nothing to be done for >>> >>>> `../../src/tests/balloon_framework_test.sh'. >>> >>>> make[3]: Nothing to be done for >>> >>>> `../../src/tests/low_level_scheduler_libprocess_test.sh'. >>> >>>> make[3]: Nothing to be done for >>> >>>> `../../src/tests/low_level_scheduler_pthread_test.sh'. >>> >>>> make[3]: Nothing to be done for >>> >>>> `../../src/tests/java_exception_test.sh'. >>> >>>> make[3]: Nothing to be done for >>> >>>> `../../src/tests/java_framework_test.sh'. >>> >>>> make[3]: Nothing to be done for `../../src/tests/java_log_test.sh'. >>> >>>> make[3]: Nothing to be done for >>> >>>> `../../src/tests/no_executor_framework_test.sh'. >>> >>>> make[3]: Nothing to be done for >>> >>>> `../../src/tests/python_framework_test.sh'. >>> >>>> make[3]: Nothing to be done for >>> >>>> `../../src/tests/test_framework_test.sh'. >>> >>>> make[3]: Leaving directory `< >>> >>>> >>> https://builds.apache.org/job/Mesos-Trunk-Ubuntu-Build-Out-Of-Src-Disable-Java-Disable-Python-Disable-Webui/ws/build/src >>> >>>> '> >>> >>>> make check-local >>> >>>> make[3]: Entering directory `< >>> >>>> >>> https://builds.apache.org/job/Mesos-Trunk-Ubuntu-Build-Out-Of-Src-Disable-Java-Disable-Python-Disable-Webui/ws/build/src >>> >>>> '> >>> >>>> ./mesos-tests >>> >>>> WARNING: Logging before InitGoogleLogging() is written to STDERR >>> >>>> I1120 23:24:12.553694 30655 process.cpp:983] libprocess is >>> initialized >>> >>>> on 67.195.81.186:33912 for 16 cpus >>> >>>> I1120 23:24:12.553962 30655 logging.cpp:177] Logging to STDERR >>> >>>> Source directory: < >>> >>>> >>> https://builds.apache.org/job/Mesos-Trunk-Ubuntu-Build-Out-Of-Src-Disable-Java-Disable-Python-Disable-Webui/ws/ >>> >>>> > >>> >>>> Build directory: < >>> >>>> >>> https://builds.apache.org/job/Mesos-Trunk-Ubuntu-Build-Out-Of-Src-Disable-Java-Disable-Python-Disable-Webui/ws/build >>> >>>> > >>> >>>> ------------------------------------------------------------- >>> >>>> We cannot run any cgroups tests that require mounting >>> >>>> hierarchies because you have the following hierarchies mounted: >>> >>>> /sys/fs/cgroup/blkio, /sys/fs/cgroup/cpu, /sys/fs/cgroup/cpuacct, >>> >>>> /sys/fs/cgroup/cpuset, /sys/fs/cgroup/devices, >>> /sys/fs/cgroup/freezer, >>> >>>> /sys/fs/cgroup/hugetlb, /sys/fs/cgroup/memory, >>> /sys/fs/cgroup/perf_event, >>> >>>> /sys/fs/cgroup/systemd >>> >>>> We'll disable the CgroupsNoHierarchyTest test fixture for now. >>> >>>> ------------------------------------------------------------- >>> >>>> Note: Google Test filter = >>> >>>> >>> *-DockerContainerizerTest.ROOT_DOCKER_Launch_Executor:DockerContainerizerTest.ROOT_DOCKER_Launch_Executor_Bridged:DockerContainerizerTest.ROOT_DOCKER_Launch:DockerContainerizerTest.ROOT_DOCKER_Kill:DockerContainerizerTest.ROOT_DOCKER_Usage:DockerContainerizerTest.ROOT_DOCKER_Update:DockerContainerizerTest.DISABLED_ROOT_DOCKER_Recover:DockerContainerizerTest.ROOT_DOCKER_Logs:DockerContainerizerTest.ROOT_DOCKER_Default_CMD:DockerContainerizerTest.ROOT_DOCKER_Default_CMD_Override:DockerContainerizerTest.ROOT_DOCKER_Default_CMD_Args:DockerContainerizerTest.ROOT_DOCKER_SlaveRecoveryTaskContainer:DockerContainerizerTest.DISABLED_ROOT_DOCKER_SlaveRecoveryExecutorContainer:DockerContainerizerTest.ROOT_DOCKER_PortMapping:DockerContainerizerTest.ROOT_DOCKER_LaunchSandboxWithColon:DockerContainerizerTest.ROOT_DOCKER_DestroyWhileFetching:DockerContainerizerTest.ROOT_DOCKER_DestroyWhilePulling:DockerTest.ROOT_DOCKER_interface:DockerTest.ROOT_DOCKER_CheckCommandWithShell:DockerTest.ROOT_DOCKER_CheckPortResource:DockerTest.ROOT_DOCKER_CancelPull:CpuIsolatorTest/1.UserCpuUsage:CpuIsolatorTest/1.SystemCpuUsage:LimitedCpuIsolatorTest.ROOT_CGROUPS_Cfs:LimitedCpuIsolatorTest.ROOT_CGROUPS_Cfs_Big_Quota:MemIsolatorTest/0.MemUsage:MemIsolatorTest/1.MemUsage:MemIsolatorTest/2.MemUsage:PerfEventIsolatorTest.ROOT_CGROUPS_Sample:SharedFilesystemIsolatorTest.ROOT_RelativeVolume:SharedFilesystemIsolatorTest.ROOT_AbsoluteVolume:NamespacesPidIsolatorTest.ROOT_PidNamespace:UserCgroupIsolatorTest/0.ROOT_CGROUPS_UserCgroup:UserCgroupIsolatorTest/1.ROOT_CGROUPS_UserCgroup:UserCgroupIsolatorTest/2.ROOT_CGROUPS_UserCgroup:MesosContainerizerSlaveRecoveryTest.CGROUPS_ROOT_PerfRollForward:MesosContainerizerSlaveRecoveryTest.CGROUPS_ROOT_PidNamespaceForward:MesosContainerizerSlaveRecoveryTest.CGROUPS_ROOT_PidNamespaceBackward:SlaveTest.ROOT_RunTaskWithCommandInfoWithoutUser:SlaveTest.DISABLED_ROOT_RunTaskWithCommandInfoWithUser:ContainerizerTest.ROOT_CGROUPS_BalloonFramework:CgroupsAnyHierarchyTest.ROOT_CGROUPS_Enabled:CgroupsAnyHierarchyTest.ROOT_CGROUPS_Subsystems:CgroupsAnyHierarchyTest.ROOT_CGROUPS_Mounted:CgroupsAnyHierarchyTest.ROOT_CGROUPS_Get:CgroupsAnyHierarchyTest.ROOT_CGROUPS_NestedCgroups:CgroupsAnyHierarchyTest.ROOT_CGROUPS_Tasks:CgroupsAnyHierarchyTest.ROOT_CGROUPS_Read:CgroupsAnyHierarchyTest.ROOT_CGROUPS_Write:CgroupsAnyHierarchyTest.ROOT_CGROUPS_Cfs_Big_Quota:CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_Busy:CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_SubsystemsHierarchy:CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_FindCgroupSubsystems:CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_MountedSubsystems:CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_CreateRemove:CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_Listen:CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_FreezeNonFreezer:CgroupsNoHierarchyTest.ROOT_CGROUPS_NOHIERARCHY_MountUnmountHierarchy:CgroupsAnyHierarchyWithCpuAcctMemoryTest.ROOT_CGROUPS_Stat:CgroupsAnyHierarchyWithFreezerTest.ROOT_CGROUPS_Freeze:CgroupsAnyHierarchyWithFreezerTest.ROOT_CGROUPS_Kill:CgroupsAnyHierarchyWithFreezerTest.ROOT_CGROUPS_Destroy:CgroupsAnyHierarchyWithFreezerTest.ROOT_CGROUPS_AssignThreads:CgroupsAnyHierarchyWithFreezerTest.ROOT_CGROUPS_DestroyStoppedProcess:CgroupsAnyHierarchyWithFreezerTest.ROOT_CGROUPS_DestroyTracedProcess:CgroupsAnyHierarchyWithPerfEventTest.ROOT_CGROUPS_Perf:NsTest.ROOT_setns:NsTest.ROOT_setnsMultipleThreads:NsTest.ROOT_getns:NsTest.ROOT_destroy:PerfTest.ROOT_Events:PerfTest.ROOT_SampleInit:SlaveCount/Registrar_BENCHMARK_Test.performance/0:SlaveCount/Registrar_BENCHMARK_Test.performance/1:SlaveCount/Registrar_BENCHMARK_Test.performance/2:SlaveCount/Registrar_BENCHMARK_Test.performance/3 >>> >>>> Note: Randomizing tests' orders with a seed of 47110 . >>> >>>> [==========] Running 437 tests from 68 test cases. >>> >>>> [----------] Global test environment set-up. >>> >>>> [----------] 4 tests from CRAMMD5Authentication/2, where TypeParam = >>> >>>> >>> mesos::internal::cram_md5::Authentication<mesos::internal::cram_md5::CRAMMD5Authenticatee, >>> >>>> mesos::internal::tests::Module<mesos::internal::Authenticator, >>> >>>> (mesos::internal::tests::ModuleID)3> > >>> >>>> [ RUN ] CRAMMD5Authentication/2.failed1 >>> >>>> Using temporary directory >>> '/tmp/CRAMMD5Authentication_2_failed1_V0pTyx' >>> >>>> I1120 23:24:12.685442 30676 authenticatee.hpp:114] Initializing >>> client >>> >>>> SASL >>> >>>> I1120 23:24:12.686884 30676 authenticatee.hpp:138] Creating new >>> client >>> >>>> SASL connection >>> >>>> I1120 23:24:12.687741 30678 authenticator.hpp:108] Initializing >>> server >>> >>>> SASL >>> >>>> I1120 23:24:12.687944 30678 auxprop.cpp:63] Initialized in-memory >>> >>>> auxiliary property plugin >>> >>>> I1120 23:24:12.687966 30678 authenticator.hpp:170] Creating new >>> server >>> >>>> SASL connection >>> >>>> I1120 23:24:12.688140 30678 authenticatee.hpp:229] Received SASL >>> >>>> authentication mechanisms: CRAM-MD5 >>> >>>> I1120 23:24:12.688180 30678 authenticatee.hpp:255] Attempting to >>> >>>> authenticate with mechanism 'CRAM-MD5' >>> >>>> I1120 23:24:12.688294 30678 authenticator.hpp:276] Received SASL >>> >>>> authentication start >>> >>>> I1120 23:24:12.688361 30678 authenticator.hpp:398] Authentication >>> >>>> requires more steps >>> >>>> I1120 23:24:12.688472 30673 authenticatee.hpp:275] Received SASL >>> >>>> authentication step >>> >>>> I1120 23:24:12.688617 30673 authenticator.hpp:304] Received SASL >>> >>>> authentication step >>> >>>> I1120 23:24:12.688649 30673 auxprop.cpp:99] Request to lookup >>> >>>> properties for user: 'benh' realm: 'penates.apache.org' server >>> FQDN: ' >>> >>>> penates.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false >>> >>>> SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false >>> >>>> I1120 23:24:12.688663 30673 auxprop.cpp:171] Looking up auxiliary >>> >>>> property '*userPassword' >>> >>>> I1120 23:24:12.688814 30673 auxprop.cpp:171] Looking up auxiliary >>> >>>> property '*cmusaslsecretCRAM-MD5' >>> >>>> I1120 23:24:12.688846 30673 auxprop.cpp:99] Request to lookup >>> >>>> properties for user: 'benh' realm: 'penates.apache.org' server >>> FQDN: ' >>> >>>> penates.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false >>> >>>> SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true >>> >>>> I1120 23:24:12.688858 30673 auxprop.cpp:121] Skipping auxiliary >>> >>>> property '*userPassword' since SASL_AUXPROP_AUTHZID == true >>> >>>> I1120 23:24:12.688863 30673 auxprop.cpp:121] Skipping auxiliary >>> >>>> property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true >>> >>>> W1120 23:24:12.688887 30673 authenticator.hpp:404] Authentication >>> >>>> failure: authentication failure >>> >>>> [ OK ] CRAMMD5Authentication/2.failed1 (11 ms) >>> >>>> [ RUN ] CRAMMD5Authentication/2.AuthenticatorDestructionRace >>> >>>> Using temporary directory >>> >>>> '/tmp/CRAMMD5Authentication_2_AuthenticatorDestructionRace_CRclP9' >>> >>>> I1120 23:24:12.690254 30684 authenticatee.hpp:138] Creating new >>> client >>> >>>> SASL connection >>> >>>> I1120 23:24:12.690830 30676 authenticator.hpp:170] Creating new >>> server >>> >>>> SASL connection >>> >>>> I1120 23:24:12.690989 30682 authenticatee.hpp:229] Received SASL >>> >>>> authentication mechanisms: CRAM-MD5 >>> >>>> I1120 23:24:12.691015 30682 authenticatee.hpp:255] Attempting to >>> >>>> authenticate with mechanism 'CRAM-MD5' >>> >>>> I1120 23:24:12.691097 30671 authenticator.hpp:276] Received SASL >>> >>>> authentication start >>> >>>> I1120 23:24:12.691135 30671 authenticator.hpp:398] Authentication >>> >>>> requires more steps >>> >>>> [ OK ] CRAMMD5Authentication/2.AuthenticatorDestructionRace >>> (2 ms) >>> >>>> [ RUN ] CRAMMD5Authentication/2.success >>> >>>> Using temporary directory >>> '/tmp/CRAMMD5Authentication_2_success_JZF85L' >>> >>>> I1120 23:24:12.692466 30682 authenticatee.hpp:138] Creating new >>> client >>> >>>> SASL connection >>> >>>> I1120 23:24:12.692939 30676 authenticator.hpp:170] Creating new >>> server >>> >>>> SASL connection >>> >>>> I1120 23:24:12.693069 30684 authenticatee.hpp:229] Received SASL >>> >>>> authentication mechanisms: CRAM-MD5 >>> >>>> I1120 23:24:12.693094 30684 authenticatee.hpp:255] Attempting to >>> >>>> authenticate with mechanism 'CRAM-MD5' >>> >>>> I1120 23:24:12.693171 30682 authenticator.hpp:276] Received SASL >>> >>>> authentication start >>> >>>> I1120 23:24:12.693207 30682 authenticator.hpp:398] Authentication >>> >>>> requires more steps >>> >>>> I1120 23:24:12.693274 30684 authenticatee.hpp:275] Received SASL >>> >>>> authentication step >>> >>>> I1120 23:24:12.693366 30682 authenticator.hpp:304] Received SASL >>> >>>> authentication step >>> >>>> I1120 23:24:12.693394 30682 auxprop.cpp:99] Request to lookup >>> >>>> properties for user: 'benh' realm: 'penates.apache.org' server >>> FQDN: ' >>> >>>> penates.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false >>> >>>> SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false >>> >>>> I1120 23:24:12.693408 30682 auxprop.cpp:171] Looking up auxiliary >>> >>>> property '*userPassword' >>> >>>> I1120 23:24:12.693431 30682 auxprop.cpp:171] Looking up auxiliary >>> >>>> property '*cmusaslsecretCRAM-MD5' >>> >>>> I1120 23:24:12.693449 30682 auxprop.cpp:99] Request to lookup >>> >>>> properties for user: 'benh' realm: 'penates.apache.org' server >>> FQDN: ' >>> >>>> penates.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false >>> >>>> SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true >>> >>>> I1120 23:24:12.693456 30682 auxprop.cpp:121] Skipping auxiliary >>> >>>> property '*userPassword' since SASL_AUXPROP_AUTHZID == true >>> >>>> I1120 23:24:12.693461 30682 auxprop.cpp:121] Skipping auxiliary >>> >>>> property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true >>> >>>> I1120 23:24:12.693476 30682 authenticator.hpp:390] Authentication >>> >>>> success >>> >>>> I1120 23:24:12.693544 30684 authenticatee.hpp:315] Authentication >>> >>>> success >>> >>>> [ OK ] CRAMMD5Authentication/2.success (3 ms) >>> >>>> [ RUN ] CRAMMD5Authentication/2.failed2 >>> >>>> Using temporary directory >>> '/tmp/CRAMMD5Authentication_2_failed2_yugjno' >>> >>>> I1120 23:24:12.694799 30677 authenticatee.hpp:138] Creating new >>> client >>> >>>> SASL connection >>> >>>> I1120 23:24:12.695314 30676 authenticator.hpp:170] Creating new >>> server >>> >>>> SASL connection >>> >>>> I1120 23:24:12.695435 30674 authenticatee.hpp:229] Received SASL >>> >>>> authentication mechanisms: CRAM-MD5 >>> >>>> I1120 23:24:12.695457 30674 authenticatee.hpp:255] Attempting to >>> >>>> authenticate with mechanism 'CRAM-MD5' >>> >>>> I1120 23:24:12.695544 30670 authenticator.hpp:276] Received SASL >>> >>>> authentication start >>> >>>> I1120 23:24:12.695582 30670 authenticator.hpp:398] Authentication >>> >>>> requires more steps >>> >>>> I1120 23:24:12.695657 30674 authenticatee.hpp:275] Received SASL >>> >>>> authentication step >>> >>>> I1120 23:24:12.695745 30674 authenticator.hpp:304] Received SASL >>> >>>> authentication step >>> >>>> I1120 23:24:12.695770 30674 auxprop.cpp:99] Request to lookup >>> >>>> properties for user: 'benh' realm: 'penates.apache.org' server >>> FQDN: ' >>> >>>> penates.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false >>> >>>> SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false >>> >>>> I1120 23:24:12.695778 30674 auxprop.cpp:171] Looking up auxiliary >>> >>>> property '*userPassword' >>> >>>> I1120 23:24:12.695788 30674 auxprop.cpp:171] Looking up auxiliary >>> >>>> property '*cmusaslsecretCRAM-MD5' >>> >>>> I1120 23:24:12.695797 30674 auxprop.cpp:99] Request to lookup >>> >>>> properties for user: 'benh' realm: 'penates.apache.org' server >>> FQDN: ' >>> >>>> penates.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false >>> >>>> SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true >>> >>>> I1120 23:24:12.695803 30674 auxprop.cpp:121] Skipping auxiliary >>> >>>> property '*userPassword' since SASL_AUXPROP_AUTHZID == true >>> >>>> I1120 23:24:12.695809 30674 auxprop.cpp:121] Skipping auxiliary >>> >>>> property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true >>> >>>> W1120 23:24:12.695818 30674 authenticator.hpp:404] Authentication >>> >>>> failure: user not found >>> >>>> [ OK ] CRAMMD5Authentication/2.failed2 (2 ms) >>> >>>> [----------] 4 tests from CRAMMD5Authentication/2 (18 ms total) >>> >>>> >>> >>>> [----------] 6 tests from RateLimitingTest >>> >>>> [ RUN ] RateLimitingTest.NoRateLimiting >>> >>>> Using temporary directory >>> '/tmp/RateLimitingTest_NoRateLimiting_XteRE0' >>> >>>> I1120 23:24:12.701375 30655 leveldb.cpp:176] Opened db in 3.4075ms >>> >>>> I1120 23:24:12.703060 30655 leveldb.cpp:183] Compacted db in >>> 1.64536ms >>> >>>> I1120 23:24:12.703178 30655 leveldb.cpp:198] Created db iterator in >>> >>>> 78140ns >>> >>>> I1120 23:24:12.703205 30655 leveldb.cpp:204] Seeked to beginning of >>> db >>> >>>> in 3649ns >>> >>>> I1120 23:24:12.703217 30655 leveldb.cpp:273] Iterated through 0 >>> keys in >>> >>>> the db in 400ns >>> >>>> I1120 23:24:12.703368 30655 replica.cpp:741] Replica recovered with >>> log >>> >>>> positions 0 -> 0 with 1 holes and 0 unlearned >>> >>>> I1120 23:24:12.705715 30678 recover.cpp:437] Starting replica >>> recovery >>> >>>> I1120 23:24:12.706840 30678 recover.cpp:463] Replica is in EMPTY >>> status >>> >>>> I1120 23:24:12.711232 30681 replica.cpp:638] Replica in EMPTY status >>> >>>> received a broadcasted recover request >>> >>>> I1120 23:24:12.712901 30683 recover.cpp:188] Received a recover >>> >>>> response from a replica in EMPTY status >>> >>>> I1120 23:24:12.713747 30673 recover.cpp:554] Updating replica >>> status to >>> >>>> STARTING >>> >>>> I1120 23:24:12.714715 30683 leveldb.cpp:306] Persisting metadata (8 >>> >>>> bytes) to leveldb took 708621ns >>> >>>> I1120 23:24:12.714754 30683 replica.cpp:320] Persisted replica >>> status >>> >>>> to STARTING >>> >>>> I1120 23:24:12.715100 30684 recover.cpp:463] Replica is in STARTING >>> >>>> status >>> >>>> I1120 23:24:12.716419 30679 replica.cpp:638] Replica in STARTING >>> status >>> >>>> received a broadcasted recover request >>> >>>> I1120 23:24:12.716961 30676 recover.cpp:188] Received a recover >>> >>>> response from a replica in STARTING status >>> >>>> I1120 23:24:12.717490 30677 recover.cpp:554] Updating replica >>> status to >>> >>>> VOTING >>> >>>> I1120 23:24:12.718242 30680 leveldb.cpp:306] Persisting metadata (8 >>> >>>> bytes) to leveldb took 484920ns >>> >>>> I1120 23:24:12.718276 30680 replica.cpp:320] Persisted replica >>> status >>> >>>> to VOTING >>> >>>> I1120 23:24:12.718385 30672 recover.cpp:568] Successfully joined the >>> >>>> Paxos group >>> >>>> I1120 23:24:12.718698 30672 recover.cpp:452] Recover process >>> terminated >>> >>>> I1120 23:24:12.723183 30670 master.cpp:318] Master >>> >>>> 20141120-232412-3125920579-33912-30655 (penates.apache.org) >>> started on >>> >>>> 67.195.81.186:33912 >>> >>>> I1120 23:24:12.723278 30670 master.cpp:364] Master only allowing >>> >>>> authenticated frameworks to register >>> >>>> I1120 23:24:12.723353 30670 master.cpp:369] Master only allowing >>> >>>> authenticated slaves to register >>> >>>> I1120 23:24:12.723420 30670 credentials.hpp:36] Loading credentials >>> for >>> >>>> authentication from >>> >>>> '/tmp/RateLimitingTest_NoRateLimiting_XteRE0/credentials' >>> >>>> I1120 23:24:12.723983 30670 master.cpp:413] Authorization enabled >>> >>>> I1120 23:24:12.724187 30670 master.cpp:460] Framework rate limiting >>> >>>> enabled >>> >>>> I1120 23:24:12.724884 30678 master.cpp:126] No whitelist given. >>> >>>> Advertising offers for all slaves >>> >>>> I1120 23:24:12.725131 30684 hierarchical_allocator_process.hpp:299] >>> >>>> Initializing hierarchical allocator process with master : >>> >>>> master@67.195.81.186:33912 >>> >>>> I1120 23:24:12.728044 30676 master.cpp:1263] The newly elected >>> leader >>> >>>> is master@67.195.81.186:33912 with id >>> >>>> 20141120-232412-3125920579-33912-30655 >>> >>>> I1120 23:24:12.728078 30676 master.cpp:1276] Elected as the leading >>> >>>> master! >>> >>>> I1120 23:24:12.728116 30676 master.cpp:1094] Recovering from >>> registrar >>> >>>> I1120 23:24:12.728454 30679 registrar.cpp:313] Recovering registrar >>> >>>> I1120 23:24:12.730170 30677 log.cpp:656] Attempting to start the >>> writer >>> >>>> I1120 23:24:12.733129 30676 replica.cpp:474] Replica received >>> implicit >>> >>>> promise request with proposal 1 >>> >>>> I1120 23:24:12.733763 30676 leveldb.cpp:306] Persisting metadata (8 >>> >>>> bytes) to leveldb took 598467ns >>> >>>> I1120 23:24:12.733795 30676 replica.cpp:342] Persisted promised to 1 >>> >>>> I1120 23:24:12.735090 30680 coordinator.cpp:230] Coordinator >>> attemping >>> >>>> to fill missing position >>> >>>> I1120 23:24:12.737241 30677 replica.cpp:375] Replica received >>> explicit >>> >>>> promise request for position 0 with proposal 2 >>> >>>> I1120 23:24:12.737690 30677 leveldb.cpp:343] Persisting action (8 >>> >>>> bytes) to leveldb took 402621ns >>> >>>> I1120 23:24:12.737721 30677 replica.cpp:676] Persisted action at 0 >>> >>>> I1120 23:24:12.739233 30681 replica.cpp:508] Replica received write >>> >>>> request for position 0 >>> >>>> I1120 23:24:12.739351 30681 leveldb.cpp:438] Reading position from >>> >>>> leveldb took 79235ns >>> >>>> I1120 23:24:12.739878 30681 leveldb.cpp:343] Persisting action (14 >>> >>>> bytes) to leveldb took 448572ns >>> >>>> I1120 23:24:12.739907 30681 replica.cpp:676] Persisted action at 0 >>> >>>> I1120 23:24:12.740696 30679 replica.cpp:655] Replica received >>> learned >>> >>>> notice for position 0 >>> >>>> I1120 23:24:12.741052 30679 leveldb.cpp:343] Persisting action (16 >>> >>>> bytes) to leveldb took 322694ns >>> >>>> I1120 23:24:12.741083 30679 replica.cpp:676] Persisted action at 0 >>> >>>> I1120 23:24:12.741117 30679 replica.cpp:661] Replica learned NOP >>> action >>> >>>> at position 0 >>> >>>> I1120 23:24:12.741811 30672 log.cpp:672] Writer started with ending >>> >>>> position 0 >>> >>>> I1120 23:24:12.744951 30670 leveldb.cpp:438] Reading position from >>> >>>> leveldb took 30629ns >>> >>>> I1120 23:24:12.748874 30678 registrar.cpp:346] Successfully fetched >>> the >>> >>>> registry (0B) in 20.35584ms >>> >>>> I1120 23:24:12.749171 30678 registrar.cpp:445] Applied 1 operations >>> in >>> >>>> 87917ns; attempting to update the 'registry' >>> >>>> I1120 23:24:12.752719 30672 log.cpp:680] Attempting to append 140 >>> bytes >>> >>>> to the log >>> >>>> I1120 23:24:12.752936 30671 coordinator.cpp:340] Coordinator >>> attempting >>> >>>> to write APPEND action at position 1 >>> >>>> I1120 23:24:12.753829 30671 replica.cpp:508] Replica received write >>> >>>> request for position 1 >>> >>>> I1120 23:24:12.754359 30671 leveldb.cpp:343] Persisting action (159 >>> >>>> bytes) to leveldb took 494376ns >>> >>>> I1120 23:24:12.754387 30671 replica.cpp:676] Persisted action at 1 >>> >>>> I1120 23:24:12.755213 30677 replica.cpp:655] Replica received >>> learned >>> >>>> notice for position 1 >>> >>>> I1120 23:24:12.755621 30677 leveldb.cpp:343] Persisting action (161 >>> >>>> bytes) to leveldb took 375152ns >>> >>>> I1120 23:24:12.755648 30677 replica.cpp:676] Persisted action at 1 >>> >>>> I1120 23:24:12.755669 30677 replica.cpp:661] Replica learned APPEND >>> >>>> action at position 1 >>> >>>> I1120 23:24:12.757231 30673 registrar.cpp:490] Successfully updated >>> the >>> >>>> 'registry' in 7.922944ms >>> >>>> I1120 23:24:12.757463 30673 registrar.cpp:376] Successfully >>> recovered >>> >>>> registrar >>> >>>> I1120 23:24:12.757665 30682 log.cpp:699] Attempting to truncate the >>> log >>> >>>> to 1 >>> >>>> I1120 23:24:12.757859 30679 coordinator.cpp:340] Coordinator >>> attempting >>> >>>> to write TRUNCATE action at position 2 >>> >>>> I1120 23:24:12.757899 30681 master.cpp:1121] Recovered 0 slaves from >>> >>>> the Registry (102B) ; allowing 10mins for slaves to re-register >>> >>>> I1120 23:24:12.758951 30680 replica.cpp:508] Replica received write >>> >>>> request for position 2 >>> >>>> I1120 23:24:12.759395 30680 leveldb.cpp:343] Persisting action (16 >>> >>>> bytes) to leveldb took 408946ns >>> >>>> I1120 23:24:12.759429 30680 replica.cpp:676] Persisted action at 2 >>> >>>> I1120 23:24:12.760109 30675 replica.cpp:655] Replica received >>> learned >>> >>>> notice for position 2 >>> >>>> I1120 23:24:12.760496 30675 leveldb.cpp:343] Persisting action (18 >>> >>>> bytes) to leveldb took 356913ns >>> >>>> I1120 23:24:12.760584 30675 leveldb.cpp:401] Deleting ~1 keys from >>> >>>> leveldb took 45969ns >>> >>>> I1120 23:24:12.760608 30675 replica.cpp:676] Persisted action at 2 >>> >>>> I1120 23:24:12.760633 30675 replica.cpp:661] Replica learned >>> TRUNCATE >>> >>>> action at position 2 >>> >>>> I1120 23:24:12.779798 30680 process.cpp:2881] Handling HTTP event >>> for >>> >>>> process 'metrics' with path: '/metrics/snapshot' >>> >>>> Thu Nov 20 23:54:06 UTC 2014: process still running after 1800 >>> seconds >>> >>>> ========== >>> >>>> >>> >>>> Attaching gdb to 30437 bash ../support/timed_tests.sh GLOG_v=1 >>> >>>> MESOS_VERBOSE=1 make check GTEST_SHUFFLE=1 GTEST_OUTPUT=xml: 1800 >>> >>>> >>> >>>> ========== >>> >>>> ../support/timed_tests.sh: line 72: gdb: command not found >>> >>>> ========== >>> >>>> >>> >>>> Attaching gdb to 30440 bash ../support/timed_tests.sh GLOG_v=1 >>> >>>> MESOS_VERBOSE=1 make check GTEST_SHUFFLE=1 GTEST_OUTPUT=xml: 1800 >>> >>>> >>> >>>> ========== >>> >>>> ../support/timed_tests.sh: line 72: gdb: command not found >>> >>>> ========== >>> >>>> >>> >>>> Attaching gdb to 30442 make check GTEST_SHUFFLE=1 GTEST_OUTPUT=xml: >>> >>>> >>> >>>> ========== >>> >>>> ../support/timed_tests.sh: line 72: gdb: command not found >>> >>>> ========== >>> >>>> >>> >>>> Attaching gdb to 30446 /bin/bash -c fail=; \ if (target_option=k; >>> case >>> >>>> ${target_option-} in ?) ;; *) echo "am__make_running_with_option: >>> internal >>> >>>> error: invalid" "target option '${target_option-}' specified" >&2; >>> exit 1;; >>> >>>> esac; has_opt=no; sane_makeflags=$MAKEFLAGS; if test -n ' Makefile' >>> && test >>> >>>> -n '0'; then sane_makeflags=$MFLAGS; else case $MAKEFLAGS in *\\[\ >>> \.]*) >>> >>>> bs=\\; sane_makeflags=`printf '%s\n' "$MAKEFLAGS" | sed >>> "s/$bs$bs[$bs >>> >>>> $bs.]*//g"`;; esac; fi; skip_next=no; strip_trailopt () { >>> flg=`printf >>> >>>> '%s\n' "$flg" | sed "s/$1.*$//"`; }; for flg in $sane_makeflags; do >>> test >>> >>>> $skip_next = yes && { skip_next=no; continue; }; case $flg in >>> *=*|--*) >>> >>>> continue;; -*I) strip_trailopt 'I'; skip_next=yes;; -*I?*) >>> strip_trailopt >>> >>>> 'I';; -*O) strip_trailopt 'O'; skip_next=yes;; -*O?*) >>> strip_trailopt 'O';; >>> >>>> -*l) strip_trailopt 'l'; skip_next=yes;; -*l?*) strip_trailopt 'l';; >>> >>>> -[dEDm]) skip_next=yes;; -[JT]) skip_next=yes;; esac; case $flg in >>> >>>> *$target_option*) has_opt=yes; break;; esac; done; test $has_opt = >>> yes); >>> >>>> then \ failcom='fail=yes'; \ else \ failcom='exit 1'; \ fi; \ >>> >>>> dot_seen=no; \ target=`echo check-recursive | sed s/-recursive//`; >>> \ case >>> >>>> "check-recursive" in \ distclean-* | maintainer-clean-*) list='. >>> 3rdparty >>> >>>> src ec2' ;; \ *) list='. 3rdparty src ec2' ;; \ esac; \ for >>> subdir in >>> >>>> $list; do \ echo "Making $target in $subdir"; \ if test >>> "$subdir" = >>> >>>> "."; then \ dot_seen=yes; \ local_target="$target-am"; \ >>> else \ >>> >>>> local_target="$target"; \ fi; \ (CDPATH="${ZSH_VERSION+.}:" >>> && cd >>> >>>> $subdir && make $local_target) \ || eval $failcom; \ done; \ if >>> test >>> >>>> "$dot_seen" = "no"; then \ make "$target-am" || exit 1; \ fi; >>> test -z >>> >>>> "$fail" >>> >>>> >>> >>>> ========== >>> >>>> ../support/timed_tests.sh: line 72: gdb: command not found >>> >>>> ========== >>> >>>> >>> >>>> Attaching gdb to 30642 /bin/bash -c fail=; \ if (target_option=k; >>> case >>> >>>> ${target_option-} in ?) ;; *) echo "am__make_running_with_option: >>> internal >>> >>>> error: invalid" "target option '${target_option-}' specified" >&2; >>> exit 1;; >>> >>>> esac; has_opt=no; sane_makeflags=$MAKEFLAGS; if test -n ' Makefile' >>> && test >>> >>>> -n '0'; then sane_makeflags=$MFLAGS; else case $MAKEFLAGS in *\\[\ >>> \.]*) >>> >>>> bs=\\; sane_makeflags=`printf '%s\n' "$MAKEFLAGS" | sed >>> "s/$bs$bs[$bs >>> >>>> $bs.]*//g"`;; esac; fi; skip_next=no; strip_trailopt () { >>> flg=`printf >>> >>>> '%s\n' "$flg" | sed "s/$1.*$//"`; }; for flg in $sane_makeflags; do >>> test >>> >>>> $skip_next = yes && { skip_next=no; continue; }; case $flg in >>> *=*|--*) >>> >>>> continue;; -*I) strip_trailopt 'I'; skip_next=yes;; -*I?*) >>> strip_trailopt >>> >>>> 'I';; -*O) strip_trailopt 'O'; skip_next=yes;; -*O?*) >>> strip_trailopt 'O';; >>> >>>> -*l) strip_trailopt 'l'; skip_next=yes;; -*l?*) strip_trailopt 'l';; >>> >>>> -[dEDm]) skip_next=yes;; -[JT]) skip_next=yes;; esac; case $flg in >>> >>>> *$target_option*) has_opt=yes; break;; esac; done; test $has_opt = >>> yes); >>> >>>> then \ failcom='fail=yes'; \ else \ failcom='exit 1'; \ fi; \ >>> >>>> dot_seen=no; \ target=`echo check-recursive | sed s/-recursive//`; >>> \ case >>> >>>> "check-recursive" in \ distclean-* | maintainer-clean-*) list='. >>> 3rdparty >>> >>>> src ec2' ;; \ *) list='. 3rdparty src ec2' ;; \ esac; \ for >>> subdir in >>> >>>> $list; do \ echo "Making $target in $subdir"; \ if test >>> "$subdir" = >>> >>>> "."; then \ dot_seen=yes; \ local_target="$target-am"; \ >>> else \ >>> >>>> local_target="$target"; \ fi; \ (CDPATH="${ZSH_VERSION+.}:" >>> && cd >>> >>>> $subdir && make $local_target) \ || eval $failcom; \ done; \ if >>> test >>> >>>> "$dot_seen" = "no"; then \ make "$target-am" || exit 1; \ fi; >>> test -z >>> >>>> "$fail" >>> >>>> >>> >>>> ========== >>> >>>> ../support/timed_tests.sh: line 72: gdb: command not found >>> >>>> ========== >>> >>>> >>> >>>> Attaching gdb to 30643 make check >>> >>>> >>> >>>> ========== >>> >>>> ../support/timed_tests.sh: line 72: gdb: command not found >>> >>>> ========== >>> >>>> >>> >>>> Attaching gdb to 30645 make check-am >>> >>>> >>> >>>> ========== >>> >>>> ../support/timed_tests.sh: line 72: gdb: command not found >>> >>>> ========== >>> >>>> >>> >>>> Attaching gdb to 30652 make check-local >>> >>>> >>> >>>> ========== >>> >>>> ../support/timed_tests.sh: line 72: gdb: command not found >>> >>>> ========== >>> >>>> >>> >>>> Attaching gdb to 30655 < >>> >>>> >>> https://builds.apache.org/job/Mesos-Trunk-Ubuntu-Build-Out-Of-Src-Disable-Java-Disable-Python-Disable-Webui/ws/build/src/.libs/lt-mesos-tests >>> >>>> > >>> >>>> >>> >>>> ========== >>> >>>> ../support/timed_tests.sh: line 72: gdb: command not found >>> >>>> Test failed and killing the stuck test process >>> >>>> W1120 23:24:12.779798 30669 logging.cpp:81] RAW: Received signal >>> >>>> SIGTERM from process 30652 of user 2396; exiting >>> >>>> W1120 23:24:12.779798 30655 logging.cpp:81] RAW: Received signal >>> >>>> SIGTERM from process 20380 of user 2396; exiting >>> >>>> make: *** [check-recursive] Terminated >>> >>>> /tmp/hudson9138584248396286316.sh: line 16: 30437 Terminated >>> >>>> ../support/timed_tests.sh "GLOG_v=1 MESOS_VERBOSE=1 make check >>> >>>> GTEST_SHUFFLE=1 GTEST_OUTPUT=xml:" 1800 >>> >>>> make[3]: *** [check-local] Terminated >>> >>>> make[2]: *** [check-am] Terminated >>> >>>> make[1]: *** [check] Terminated >>> >>>> Build step 'Execute shell' marked build as failure >>> >>>> Recording test results >>> >>>> >>> >>> >>> >>> >>> >> >>> > >>> >> >
