See <https://builds.apache.org/job/HBase-TRUNK/2646/changes>
Changes: [mbautin] [jira] [HBASE-4218] HFile data block encoding framework and delta encoding implementation (Jacek Midgal, Mikhail Bautin) Summary: Adding a framework that allows to "encode" keys in an HFile data block. We support two modes of encoding: (1) both on disk and in cache, and (2) in cache only. This is distinct from compression that is already being done in HBase, e.g. GZ or LZO. When data block encoding is enabled, we store blocks in cache in an uncompressed but encoded form. This allows to fit more blocks in cache and reduce the number of disk reads. The most common example of data block encoding is delta encoding, where we take advantage of the fact that HFile keys are sorted and share a lot of common prefixes, and only store the delta between each pair of consecutive keys. Initial encoding algorithms implemented are DIFF, FAST_DIFF, and PREFIX. This is based on the delta encoding patch developed by Jacek Midgal during his 2011 summer internship at Facebook. The original patch is available here: https://reviews.apache.org/r/2308/diff/. Test Plan: Unit tests. Distributed load test on a five-node cluster. Reviewers: JIRA, tedyu, stack, nspiegelberg, Kannan Reviewed By: Kannan CC: tedyu, todd, mbautin, stack, Kannan, mcorgan, gqchen Differential Revision: https://reviews.facebook.net/D447 [mbautin] adding mbautin as a developer to pom.xml [stack] HBASE-5278 HBase shell script refers to removed 'migrate' functionality [mbautin] HBASE-5230 : ensure that compactions do not cache-on-write data blocks [stack] More home page edits and ref to favicon.ico [tedyu] HBASE-4720 revert until agreement is reached on solution [tedyu] HBASE-4720 Implement atomic update operations (checkAndPut, checkAndDelete) for REST client/server (Mubarak) ------------------------------------------ [...truncated 2019 lines...] Running org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.668 sec Running org.apache.hadoop.hbase.coprocessor.TestMasterObserver Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.392 sec Running org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint Tests run: 3, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 27.936 sec Running org.apache.hadoop.hbase.coprocessor.TestWALObserver Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.277 sec Running org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.318 sec Running org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit Tests run: 30, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 246.195 sec Running org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.257 sec Running org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.165 sec Running org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 33.742 sec Running org.apache.hadoop.hbase.TestGlobalMemStoreSize Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.921 sec Running org.apache.hadoop.hbase.mapred.TestTableInputFormat Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.587 sec Running org.apache.hadoop.hbase.mapred.TestTableMapReduce Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 63.514 sec Running org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 86.201 sec Running org.apache.hadoop.hbase.mapreduce.TestTableMapReduce Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 93.368 sec Running org.apache.hadoop.hbase.regionserver.wal.TestLogRolling Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 435.433 sec Running org.apache.hadoop.hbase.mapreduce.TestImportTsv Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 87.353 sec Running org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 40.061 sec Running org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 32.202 sec Running org.apache.hadoop.hbase.mapreduce.TestImportExport Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 48.922 sec Running org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 218.005 sec Running org.apache.hadoop.hbase.thrift2.TestThriftHBaseServiceHandler Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.672 sec Running org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.165 sec Running org.apache.hadoop.hbase.catalog.TestMetaReaderEditor Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.007 sec Running org.apache.hadoop.hbase.catalog.TestCatalogTracker Tests run: 9, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 6.439 sec Running org.apache.hadoop.hbase.TestAcidGuarantees Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 101.633 sec Running org.apache.hadoop.hbase.client.TestTimestampsFilter Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.706 sec Running org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 81.041 sec Running org.apache.hadoop.hbase.client.TestShell Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 88.472 sec Running org.apache.hadoop.hbase.client.TestFromClientSide Tests run: 52, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 163.299 sec Running org.apache.hadoop.hbase.client.TestHCM Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.578 sec Running org.apache.hadoop.hbase.client.TestInstantSchemaChange Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 201.186 sec Running org.apache.hadoop.hbase.client.TestInstantSchemaChangeSplit Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 156.33 sec Running org.apache.hadoop.hbase.client.TestMetaScanner Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.784 sec Running org.apache.hadoop.hbase.client.TestHTableUtil Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.596 sec Running org.apache.hadoop.hbase.client.TestScannerTimeout Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 47.053 sec Running org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD Running org.apache.hadoop.hbase.client.TestHTablePool$TestHTableReusablePool Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.535 sec Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.174 sec Running org.apache.hadoop.hbase.client.TestHTablePool$TestHTableThreadLocalPool Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.695 sec Running org.apache.hadoop.hbase.client.replication.TestReplicationAdmin Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.933 sec Running org.apache.hadoop.hbase.client.TestInstantSchemaChangeFailover Tests run: 4, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 121.589 sec Running org.apache.hadoop.hbase.client.TestMultipleTimestamps Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.667 sec Running org.apache.hadoop.hbase.metrics.TestMetricsMBeanBase Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.115 sec Running org.apache.hadoop.hbase.client.TestMultiParallel Running org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 39.096 sec Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 566.895 sec Running org.apache.hadoop.hbase.constraint.TestConstraint Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.769 sec Running org.apache.hadoop.hbase.rest.TestMultiRowResource Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.793 sec Running org.apache.hadoop.hbase.rest.TestGzipFilter Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.075 sec Running org.apache.hadoop.hbase.rest.TestRowResource Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.295 sec Running org.apache.hadoop.hbase.rest.TestScannersWithFilters Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.169 sec Running org.apache.hadoop.hbase.rest.client.TestRemoteAdmin Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.734 sec Running org.apache.hadoop.hbase.rest.TestVersionResource Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.098 sec Running org.apache.hadoop.hbase.rest.client.TestRemoteTable Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.388 sec Running org.apache.hadoop.hbase.rest.TestStatusResource Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.322 sec Running org.apache.hadoop.hbase.rest.TestScannerResource Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.97 sec Running org.apache.hadoop.hbase.rest.TestTableResource Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.382 sec Running org.apache.hadoop.hbase.rest.TestSchemaResource Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.53 sec Running org.apache.hadoop.hbase.thrift.TestThriftServer Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.698 sec Running org.apache.hadoop.hbase.io.encoding.TestLoadAndSwitchEncodeOnDisk Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.91 sec Running org.apache.hadoop.hbase.client.TestAdmin Tests run: 35, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 185.375 sec Running org.apache.hadoop.hbase.thrift.TestThriftServerCmdLine Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 94.552 sec Running org.apache.hadoop.hbase.io.hfile.TestHFileSeek Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.419 sec Running org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.518 sec Running org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.684 sec Running org.apache.hadoop.hbase.io.hfile.TestHFileBlockIndex Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.021 sec Running org.apache.hadoop.hbase.io.hfile.TestLruBlockCache Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.779 sec Running org.apache.hadoop.hbase.io.hfile.TestHFilePerformance Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.329 sec Running org.apache.hadoop.hbase.io.hfile.TestFixedFileTrailer Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.621 sec Running org.apache.hadoop.hbase.io.encoding.TestUpgradeFromHFileV1ToEncoding Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 87.232 sec Running org.apache.hadoop.hbase.io.encoding.TestDataBlockEncoders Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 146.21 sec Running org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite Tests run: 36, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 43.67 sec Running org.apache.hadoop.hbase.avro.TestAvroServer Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.534 sec Running org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.086 sec # # There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (malloc) failed to allocate 32756 bytes for ChunkPool::allocate # An error report file with more information is saved as: # <https://builds.apache.org/job/HBase-TRUNK/ws/trunk/hs_err_pid25397.log> Aborted org.apache.maven.surefire.booter.SurefireBooterForkException: Error occurred in starting fork, check output in log at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:238) at org.apache.maven.plugin.surefire.booterclient.ForkStarter.access$000(ForkStarter.java:64) at org.apache.maven.plugin.surefire.booterclient.ForkStarter$ParallelFork.call(ForkStarter.java:303) at org.apache.maven.plugin.surefire.booterclient.ForkStarter$ParallelFork.call(ForkStarter.java:285) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Running org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.341 sec Running org.apache.hadoop.hbase.replication.TestReplicationSource Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.495 sec Running org.apache.hadoop.hbase.replication.TestReplicationPeer Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.708 sec Running org.apache.hadoop.hbase.replication.TestMultiSlaveReplication Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 40.155 sec Running org.apache.hadoop.hbase.ipc.TestDelayedRpc Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.867 sec Running org.apache.hadoop.hbase.io.encoding.TestChangingEncoding Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 254.266 sec No results for java.util.concurrent.FutureTask@d089a9 Running org.apache.hadoop.hbase.replication.TestMasterReplication Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 60.219 sec Running org.apache.hadoop.hbase.TestZooKeeper Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 45.583 sec <<< FAILURE! Running org.apache.hadoop.hbase.TestHBaseTestingUtility Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 59.134 sec Running org.apache.hadoop.hbase.replication.TestReplication Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 114.776 sec Results : Failed tests: testClientSessionExpired(org.apache.hadoop.hbase.TestZooKeeper) Tests run: 861, Failures: 1, Errors: 0, Skipped: 10 [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 35:17.730s [INFO] Finished at: Thu Jan 26 03:56:35 UTC 2012 [INFO] Final Memory: 47M/535M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12-TRUNK-HBASE-2:test (secondPartTestsExecution) on project hbase: There are test failures. [ERROR] [ERROR] Please refer to <https://builds.apache.org/job/HBase-TRUNK/ws/trunk/target/surefire-reports> for the individual test results. [ERROR] -> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12-TRUNK-HBASE-2:test (secondPartTestsExecution) on project hbase: There are test failures. Please refer to <https://builds.apache.org/job/HBase-TRUNK/ws/trunk/target/surefire-reports> for the individual test results. at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:213) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) at org.apache.maven.lifecycle.internal.MojoExecutor.executeForkedExecutions(MojoExecutor.java:365) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:199) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59) at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:319) at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156) at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537) at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196) at org.apache.maven.cli.MavenCli.main(MavenCli.java:141) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290) at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409) at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352) Caused by: org.apache.maven.plugin.MojoFailureException: There are test failures. Please refer to <https://builds.apache.org/job/HBase-TRUNK/ws/trunk/target/surefire-reports> for the individual test results. at org.apache.maven.plugin.surefire.SurefireHelper.reportExecution(SurefireHelper.java:87) at org.apache.maven.plugin.surefire.SurefirePlugin.writeSummary(SurefirePlugin.java:651) at org.apache.maven.plugin.surefire.SurefirePlugin.handleSummary(SurefirePlugin.java:625) at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:137) at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:98) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209) ... 23 more [ERROR] [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException Build step 'Invoke top-level Maven targets' marked build as failure Archiving artifacts Updating HBASE-4218 Updating HBASE-5278 Updating HBASE-5230 Updating HBASE-4720 Recording test results
