Thanks for filing E. Seems like its only failed once recently in this build:
https://builds.apache.org/job/HBase-1.2/jdk=latest1.8,label=Hadoop/lastCompletedBuild/testReport/org.apache.hadoop.hbase.regionserver/TestSplitWalDataLoss/test/history/ I've seen it fail in a few places too... Yeah, lets fix the flakey. St.Ack On Thu, Dec 10, 2015 at 2:44 PM, Elliott Clark <ecl...@apache.org> wrote: > I'm seeing this on non-apache build machines too. Filed HBASE-14962 > > On Wed, Dec 9, 2015 at 9:57 AM, Stack <st...@duboce.net> wrote: > > > This test usually passes. This is first fail in about 20 runs: > > > > > > > https://builds.apache.org/job/HBase-1.2/jdk=latest1.8,label=Hadoop/lastCompletedBuild/testReport/org.apache.hadoop.hbase.regionserver/TestSplitWalDataLoss/test/history/ > > > > St.Ack > > > > On Wed, Dec 9, 2015 at 9:48 AM, Apache Jenkins Server < > > jenk...@builds.apache.org> wrote: > > > > > See < > > > > > > https://builds.apache.org/job/HBase-1.2/jdk=latest1.8,label=Hadoop/432/changes > > > > > > > > > > Changes: > > > > > > [tedyu] HBASE-14954 IllegalArgumentException was thrown when doing > online > > > > > > ------------------------------------------ > > > [...truncated 45664 lines...] > > > at > > > > > > org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:190) > > > at > > > org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1430) > > > at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1021) > > > at > > > > > > org.apache.hadoop.hbase.regionserver.TestSplitWalDataLoss.test(TestSplitWalDataLoss.java:121) > > > > > > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 30.491 > > sec > > > - in org.apache.hadoop.hbase.regionserver.TestClusterId > > > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option > > > MaxPermSize=256m; support was removed in 8.0 > > > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option > > > MaxPermSize=256m; support was removed in 8.0 > > > Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 44.565 > > sec > > > - in org.apache.hadoop.hbase.regionserver.TestMajorCompaction > > > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option > > > MaxPermSize=256m; support was removed in 8.0 > > > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option > > > MaxPermSize=256m; support was removed in 8.0 > > > Running org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed > > > Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 55.637 > > sec > > > <<< FAILURE! - in > > org.apache.hadoop.hbase.regionserver.TestSplitWalDataLoss > > > test(org.apache.hadoop.hbase.regionserver.TestSplitWalDataLoss) Time > > > elapsed: 39.723 sec <<< ERROR! > > > org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: > > > Failed 1 action: TestSplitWalDataLoss:dataloss: 1 time, > > > at > > > > > > org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:228) > > > at > > > > > > org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1800(AsyncProcess.java:208) > > > at > > > > > > org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1712) > > > at > > > > > > org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:240) > > > at > > > > > > org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:190) > > > at > > > org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1430) > > > at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1021) > > > at > > > > > > org.apache.hadoop.hbase.regionserver.TestSplitWalDataLoss.test(TestSplitWalDataLoss.java:121) > > > > > > Running org.apache.hadoop.hbase.regionserver.TestCompactionState > > > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 34.549 > > sec > > > - in org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed > > > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option > > > MaxPermSize=256m; support was removed in 8.0 > > > Running org.apache.hadoop.hbase.regionserver.TestRegionReplicas > > > Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 94.584 > > sec > > > <<< FAILURE! - in > > org.apache.hadoop.hbase.regionserver.TestSplitWalDataLoss > > > test(org.apache.hadoop.hbase.regionserver.TestSplitWalDataLoss) Time > > > elapsed: 38.183 sec <<< ERROR! > > > org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: > > > Failed 1 action: TestSplitWalDataLoss:dataloss: 1 time, > > > at > > > > > > org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:228) > > > at > > > > > > org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1800(AsyncProcess.java:208) > > > at > > > > > > org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1712) > > > at > > > > > > org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:240) > > > at > > > > > > org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:190) > > > at > > > org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1430) > > > at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1021) > > > at > > > > > > org.apache.hadoop.hbase.regionserver.TestSplitWalDataLoss.test(TestSplitWalDataLoss.java:121) > > > > > > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option > > > MaxPermSize=256m; support was removed in 8.0 > > > Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 290.901 > > > sec - in org.apache.hadoop.hbase.regionserver.wal.TestLogRolling > > > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option > > > MaxPermSize=256m; support was removed in 8.0 > > > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option > > > MaxPermSize=256m; support was removed in 8.0 > > > Running org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter > > > Running org.apache.hadoop.hbase.regionserver.TestEncryptionKeyRotation > > > Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 50.722 > > sec > > > - in org.apache.hadoop.hbase.regionserver.TestCompactionState > > > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option > > > MaxPermSize=256m; support was removed in 8.0 > > > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option > > > MaxPermSize=256m; support was removed in 8.0 > > > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option > > > MaxPermSize=256m; support was removed in 8.0 > > > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option > > > MaxPermSize=256m; support was removed in 8.0 > > > Running > > org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler > > > Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.512 > sec > > > - in org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler > > > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option > > > MaxPermSize=256m; support was removed in 8.0 > > > Running > > org.apache.hadoop.hbase.regionserver.handler.TestCloseRegionHandler > > > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.099 > sec > > > - in > org.apache.hadoop.hbase.regionserver.handler.TestCloseRegionHandler > > > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option > > > MaxPermSize=256m; support was removed in 8.0 > > > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.298 > > sec > > > - in org.apache.hadoop.hbase.regionserver.TestEncryptionKeyRotation > > > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option > > > MaxPermSize=256m; support was removed in 8.0 > > > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option > > > MaxPermSize=256m; support was removed in 8.0 > > > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option > > > MaxPermSize=256m; support was removed in 8.0 > > > Running org.apache.hadoop.hbase.regionserver.TestFlushRegionEntry > > > Running org.apache.hadoop.hbase.regionserver.TestRegionServerMetrics > > > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.689 > sec > > > - in org.apache.hadoop.hbase.regionserver.TestFlushRegionEntry > > > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option > > > MaxPermSize=256m; support was removed in 8.0 > > > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option > > > MaxPermSize=256m; support was removed in 8.0 > > > Running org.apache.hadoop.hbase.TestLocalHBaseCluster > > > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.883 > > sec > > > - in org.apache.hadoop.hbase.TestLocalHBaseCluster > > > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option > > > MaxPermSize=256m; support was removed in 8.0 > > > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option > > > MaxPermSize=256m; support was removed in 8.0 > > > Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 170.868 > > > sec - in org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover > > > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option > > > MaxPermSize=256m; support was removed in 8.0 > > > Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 36.257 > > > sec - in org.apache.hadoop.hbase.regionserver.TestRegionServerMetrics > > > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option > > > MaxPermSize=256m; support was removed in 8.0 > > > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option > > > MaxPermSize=256m; support was removed in 8.0 > > > Running org.apache.hadoop.hbase.migration.TestNamespaceUpgrade > > > Running org.apache.hadoop.hbase.migration.TestUpgradeTo96 > > > Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.279 > sec > > > - in org.apache.hadoop.hbase.migration.TestUpgradeTo96 > > > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option > > > MaxPermSize=256m; support was removed in 8.0 > > > Running org.apache.hadoop.hbase.quotas.TestQuotaThrottle > > > Running org.apache.hadoop.hbase.quotas.TestQuotaAdmin > > > Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.186 > > sec > > > - in org.apache.hadoop.hbase.migration.TestNamespaceUpgrade > > > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option > > > MaxPermSize=256m; support was removed in 8.0 > > > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.726 > sec > > > - in org.apache.hadoop.hbase.quotas.TestQuotaAdmin > > > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option > > > MaxPermSize=256m; support was removed in 8.0 > > > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option > > > MaxPermSize=256m; support was removed in 8.0 > > > Running org.apache.hadoop.hbase.quotas.TestQuotaTableUtil > > > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.857 > sec > > > - in org.apache.hadoop.hbase.quotas.TestQuotaTableUtil > > > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option > > > MaxPermSize=256m; support was removed in 8.0 > > > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option > > > MaxPermSize=256m; support was removed in 8.0 > > > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option > > > MaxPermSize=256m; support was removed in 8.0 > > > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option > > > MaxPermSize=256m; support was removed in 8.0 > > > Running org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide > > > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.864 > sec > > > - in org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide > > > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option > > > MaxPermSize=256m; support was removed in 8.0 > > > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option > > > MaxPermSize=256m; support was removed in 8.0 > > > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option > > > MaxPermSize=256m; support was removed in 8.0 > > > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option > > > MaxPermSize=256m; support was removed in 8.0 > > > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option > > > MaxPermSize=256m; support was removed in 8.0 > > > Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.992 > > > sec - in org.apache.hadoop.hbase.quotas.TestQuotaThrottle > > > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option > > > MaxPermSize=256m; support was removed in 8.0 > > > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option > > > MaxPermSize=256m; support was removed in 8.0 > > > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option > > > MaxPermSize=256m; support was removed in 8.0 > > > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option > > > MaxPermSize=256m; support was removed in 8.0 > > > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option > > > MaxPermSize=256m; support was removed in 8.0 > > > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option > > > MaxPermSize=256m; support was removed in 8.0 > > > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option > > > MaxPermSize=256m; support was removed in 8.0 > > > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option > > > MaxPermSize=256m; support was removed in 8.0 > > > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option > > > MaxPermSize=256m; support was removed in 8.0 > > > Running org.apache.hadoop.hbase.constraint.TestConstraint > > > Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.374 > > sec > > > - in org.apache.hadoop.hbase.constraint.TestConstraint > > > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 164.494 > > > sec - in org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter > > > Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 175.005 > > > sec - in org.apache.hadoop.hbase.regionserver.TestRegionReplicas > > > > > > Results : > > > > > > Tests in error: > > > > > > > > > org.apache.hadoop.hbase.regionserver.TestSplitWalDataLoss.test(org.apache.hadoop.hbase.regionserver.TestSplitWalDataLoss) > > > Run 1: TestSplitWalDataLoss.test:121 » RetriesExhaustedWithDetails > > > Failed 1 action: T... > > > Run 2: TestSplitWalDataLoss.test:121 » RetriesExhaustedWithDetails > > > Failed 1 action: T... > > > Run 3: TestSplitWalDataLoss.test:121 » RetriesExhaustedWithDetails > > > Failed 1 action: T... > > > > > > Flaked tests: > > > > > > > > > org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager.testFailoverDeadServerCversionChange(org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager) > > > Run 1: > > > TestReplicationSourceManager.testFailoverDeadServerCversionChange:413 > > > expected:<14> but was:<15> > > > Run 2: PASS > > > > > > > > > Tests run: 2351, Failures: 0, Errors: 1, Skipped: 40, Flakes: 1 > > > > > > [INFO] > > > > ------------------------------------------------------------------------ > > > [INFO] Reactor Summary: > > > [INFO] > > > [INFO] Apache HBase ...................................... SUCCESS > > > [1:37.423s] > > > [INFO] Apache HBase - Checkstyle ......................... SUCCESS > > [8.625s] > > > [INFO] Apache HBase - Resource Bundle .................... SUCCESS > > [0.155s] > > > [INFO] Apache HBase - Annotations ........................ SUCCESS > > [1.033s] > > > [INFO] Apache HBase - Protocol ........................... SUCCESS > > > [18.835s] > > > [INFO] Apache HBase - Common ............................. SUCCESS > > > [3:14.914s] > > > [INFO] Apache HBase - Procedure .......................... SUCCESS > > > [3:21.571s] > > > [INFO] Apache HBase - Client ............................. SUCCESS > > > [1:25.066s] > > > [INFO] Apache HBase - Hadoop Compatibility ............... SUCCESS > > [8.593s] > > > [INFO] Apache HBase - Hadoop Two Compatibility ........... SUCCESS > > [8.395s] > > > [INFO] Apache HBase - Prefix Tree ........................ SUCCESS > > [7.729s] > > > [INFO] Apache HBase - Server ............................. FAILURE > > > [1:37:20.920s] > > > [INFO] Apache HBase - Testing Util ....................... SKIPPED > > > [INFO] Apache HBase - Thrift ............................. SKIPPED > > > [INFO] Apache HBase - Rest ............................... SKIPPED > > > [INFO] Apache HBase - Shell .............................. SKIPPED > > > [INFO] Apache HBase - Integration Tests .................. SKIPPED > > > [INFO] Apache HBase - Examples ........................... SKIPPED > > > [INFO] Apache HBase - External Block Cache ............... SKIPPED > > > [INFO] Apache HBase - Assembly ........................... SKIPPED > > > [INFO] Apache HBase - Shaded ............................. SKIPPED > > > [INFO] Apache HBase - Shaded - Client .................... SKIPPED > > > [INFO] Apache HBase - Shaded - Server .................... SKIPPED > > > [INFO] > > > > ------------------------------------------------------------------------ > > > [INFO] BUILD FAILURE > > > [INFO] > > > > ------------------------------------------------------------------------ > > > [INFO] Total time: 1:48:21.317s > > > [INFO] Finished at: Wed Dec 09 17:45:49 UTC 2015 > > > [INFO] Final Memory: 94M/719M > > > [INFO] > > > > ------------------------------------------------------------------------ > > > [ERROR] Failed to execute goal > > > org.apache.maven.plugins:maven-surefire-plugin:2.18.1:test > > > (secondPartTestsExecution) on project hbase-server: There are test > > failures. > > > [ERROR] > > > [ERROR] Please refer to < > > > > > > https://builds.apache.org/job/HBase-1.2/jdk=latest1.8,label=Hadoop/ws/hbase-server/target/surefire-reports > > > > > > for the individual test results. > > > [ERROR] -> [Help 1] > > > [ERROR] > > > [ERROR] To see the full stack trace of the errors, re-run Maven with > the > > > -e switch. > > > [ERROR] Re-run Maven using the -X switch to enable full debug logging. > > > [ERROR] > > > [ERROR] For more information about the errors and possible solutions, > > > please read the following articles: > > > [ERROR] [Help 1] > > > http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException > > > [ERROR] > > > [ERROR] After correcting the problems, you can resume the build with > the > > > command > > > [ERROR] mvn <goals> -rf :hbase-server > > > Build step 'Invoke top-level Maven targets' marked build as failure > > > Performing Post build task... > > > Match found for :.* : True > > > Logical operation result is TRUE > > > Running script : # Post-build task script. TODO: Check this in and > have > > > all builds reference check-in. > > > pwd && ls > > > # NOTE!!!! The below code has been copied and pasted from > > > ./dev-tools/run-test.sh > > > # Do not change here without syncing there and vice-versa. > > > ZOMBIE_TESTS_COUNT=`jps -v | grep surefirebooter | grep -e > '-Dhbase.test' > > > | wc -l` > > > if [[ $ZOMBIE_TESTS_COUNT != 0 ]] ; then > > > echo "Suspicious java process found - waiting 30s to see if there are > > > just slow to stop" > > > sleep 30 > > > ZOMBIE_TESTS_COUNT=`jps -v | grep surefirebooter | grep -e > > '-Dhbase.test' > > > | wc -l` > > > if [[ $ZOMBIE_TESTS_COUNT != 0 ]] ; then > > > echo " {color:red}There appear to be $ZOMBIE_TESTS_COUNT zombie > > > tests{color}, they should have been killed by surefire but survived" > > > jps -v | grep surefirebooter | grep -e '-Dhbase.test' > > > jps -v | grep surefirebooter | grep -e '-Dhbase.test' | cut -d ' ' > -f > > 1 > > > | xargs -n 1 jstack > > > # Exit with error > > > exit 1 > > > else > > > echo "We're ok: there is no zombie test, but some tests took some > time > > > to stop" > > > fi > > > else > > > echo "We're ok: there is no zombie test" > > > fi > > > [Hadoop] $ /bin/bash -xe /tmp/hudson6500769177688484037.sh > > > + pwd > > > < > https://builds.apache.org/job/HBase-1.2/jdk=latest1.8,label=Hadoop/ws/> > > > + ls > > > bin > > > CHANGES.txt > > > conf > > > dev-support > > > hbase-annotations > > > hbase-assembly > > > hbase-checkstyle > > > hbase-client > > > hbase-common > > > hbase-examples > > > hbase-external-blockcache > > > hbase-hadoop2-compat > > > hbase-hadoop-compat > > > hbase-it > > > hbase-native-client > > > hbase-prefix-tree > > > hbase-procedure > > > hbase-protocol > > > hbase-resource-bundle > > > hbase-rest > > > hbase-server > > > hbase-shaded > > > hbase-shell > > > hbase-spark > > > hbase-testing-util > > > hbase-thrift > > > LICENSE.txt > > > NOTICE.txt > > > pom.xml > > > README.txt > > > src > > > target > > > ++ jps -v > > > ++ grep surefirebooter > > > ++ wc -l > > > ++ grep -e -Dhbase.test > > > + ZOMBIE_TESTS_COUNT=0 > > > + [[ 0 != 0 ]] > > > + echo 'We'\''re ok: there is no zombie test' > > > We're ok: there is no zombie test > > > POST BUILD TASK : SUCCESS > > > END OF POST BUILD TASK : 0 > > > Archiving artifacts > > > Recording test results > > > > > >
