See <https://builds.apache.org/job/HBase-TRUNK-security/132/changes>
Changes: [larsh] HBASE-5541 Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks [larsh] HBASE-5526 ^Cnfigurable file and directory based umask (Jesse Yates) [mbautin] [jira] [HBASE-5074] Support checksums in HBase block cache Author: Dhruba Summary: HFile is enhanced to store a checksum for each block. HDFS checksum verification is avoided while reading data into the block cache. On a checksum verification failure, we retry the file system read request with hdfs checksums switched on (thanks Todd). I have a benchmark that shows that it reduces iops on the disk by about 40%. In this experiment, the entire memory on the regionserver is allocated to the regionserver's jvm and the OS buffer cache size is negligible. I also measured negligible (<5%) additional cpu usage while using hbase-level checksums. The salient points of this patch: 1. Each hfile's trailer used to have a 4 byte version number. I enhanced this so that these 4 bytes can be interpreted as a (major version number, minor version). Pre-existing hfiles have a minor version of 0. The new hfile format has a minor version of 1 (thanks Mikhail). The hfile major version remains unchanged at 2. The reason I did not introduce a new major version number is because the code changes needed to store/read checksums do not differ much from existing V2 writers/readers. 2. Introduced a HFileSystem object which is a encapsulates the FileSystem objects needed to access data from hfiles and hlogs. HDFS FileSystem objects already had the ability to switch off checksum verifications for reads. 3. The majority of the code changes are located in hbase.io.hfie package. The retry of a read on an initial checksum failure occurs inside the hbase.io.hfile package itself. The code changes to hbase.regionserver package are minor. 4. The format of a hfileblock is the header followed by the data followed by the checksum(s). Each 16 K (configurable) size of data has a 4 byte checksum. The hfileblock header has two additional fields: a 4 byte value to store the bytesPerChecksum and a 4 byte value to store the size of the user data (excluding the checksum data). This is well explained in the associated javadocs. 5. I added a test to test backward compatibility. I will be writing more unit tests that triggers checksum verification failures aggressively. I have left a few redundant log messages in the code (just for easier debugging) and will remove them in later stage of this patch. I will also be adding metrics on number of checksum verification failures/success in a later version of this diff. 6. By default, hbase-level checksums are switched on and hdfs level checksums are switched off for hfile-reads. No changes to Hlog code path here. Test Plan: The default setting is to switch on hbase checksums for hfile-reads, thus all existing tests actually validate the new code pieces. I will be writing more unit tests for triggering checksum verification failures. Reviewers: mbautin Reviewed By: mbautin CC: JIRA, tedyu, mbautin, dhruba, todd, stack Differential Revision: https://reviews.facebook.net/D1521 [larsh] HBASE-5529 MR test failures becuase MALLOC_ARENA_MAX is not set (Gregory Chanan) [tedyu] HBASE-5515 Add a processRow API that supports atomic multiple reads and writes on a row (Scott Chen) [tedyu] HBASE-5514 Compile against hadoop 0.24-SNAPSHOT (Mingjie) ------------------------------------------ [...truncated 2058 lines...] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.262 sec Running org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 497.418 sec Running org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit Tests run: 30, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 247.803 sec Running org.apache.hadoop.hbase.regionserver.TestRegionServerMetrics Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 32.798 sec Running org.apache.hadoop.hbase.regionserver.TestAtomicOperation Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.12 sec Running org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.268 sec Running org.apache.hadoop.hbase.regionserver.TestParallelPut Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.903 sec Running org.apache.hadoop.hbase.regionserver.TestRpcMetrics Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.686 sec Running org.apache.hadoop.hbase.regionserver.TestSeekOptimizations Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.152 sec Running org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.079 sec Running org.apache.hadoop.hbase.regionserver.handler.TestCloseRegionHandler Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.305 sec Running org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.073 sec Running org.apache.hadoop.hbase.regionserver.TestBlocksRead Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.709 sec Running org.apache.hadoop.hbase.regionserver.TestStoreScanner Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.089 sec Running org.apache.hadoop.hbase.regionserver.TestSplitLogWorker Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.122 sec Running org.apache.hadoop.hbase.regionserver.metrics.TestSchemaMetrics Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.845 sec Running org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.895 sec Running org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 40.793 sec Running org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 46.155 sec Running org.apache.hadoop.hbase.regionserver.TestMasterAddressManager Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.565 sec Running org.apache.hadoop.hbase.regionserver.TestHRegion Tests run: 61, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.618 sec Running org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.63 sec Running org.apache.hadoop.hbase.security.token.TestTokenAuthentication Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.786 sec Running org.apache.hadoop.hbase.security.token.TestZKSecretWatcher Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.388 sec Running org.apache.hadoop.hbase.security.access.TestAccessControlFilter Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.772 sec Running org.apache.hadoop.hbase.security.access.TestZKPermissionsWatcher Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.198 sec Running org.apache.hadoop.hbase.security.access.TestTablePermissions Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.062 sec Running org.apache.hadoop.hbase.security.access.TestAccessController Tests run: 21, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.649 sec Running org.apache.hadoop.hbase.ipc.TestDelayedRpc Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.192 sec Running org.apache.hadoop.hbase.TestGlobalMemStoreSize Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.408 sec Running org.apache.hadoop.hbase.TestFullLogReconstruction Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 34.268 sec Running org.apache.hadoop.hbase.constraint.TestConstraint Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.404 sec Running org.apache.hadoop.hbase.rest.TestSchemaResource Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.259 sec Running org.apache.hadoop.hbase.rest.client.TestRemoteTable Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.96 sec Running org.apache.hadoop.hbase.rest.client.TestRemoteAdmin Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.061 sec Running org.apache.hadoop.hbase.rest.TestMultiRowResource Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.934 sec Running org.apache.hadoop.hbase.rest.TestTableResource Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.459 sec Running org.apache.hadoop.hbase.rest.TestGzipFilter Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.926 sec Running org.apache.hadoop.hbase.rest.TestRowResource Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.024 sec Running org.apache.hadoop.hbase.rest.TestVersionResource Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.828 sec Running org.apache.hadoop.hbase.rest.TestScannerResource Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.652 sec Running org.apache.hadoop.hbase.rest.TestStatusResource Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.947 sec Running org.apache.hadoop.hbase.metrics.TestMetricsMBeanBase Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.093 sec Running org.apache.hadoop.hbase.io.hfile.TestHFileBlockIndex Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.317 sec Running org.apache.hadoop.hbase.rest.TestScannersWithFilters Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.928 sec Running org.apache.hadoop.hbase.io.hfile.TestLruBlockCache Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.687 sec Running org.apache.hadoop.hbase.io.hfile.TestHFileSeek Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.651 sec Running org.apache.hadoop.hbase.io.hfile.TestFixedFileTrailer Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.538 sec Running org.apache.hadoop.hbase.io.hfile.TestHFilePerformance Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.354 sec Running org.apache.hadoop.hbase.io.hfile.TestHFileBlockCompatibility Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.038 sec Running org.apache.hadoop.hbase.io.hfile.TestForceCacheImportantBlocks Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.576 sec Running org.apache.hadoop.hbase.io.hfile.TestChecksum Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.745 sec Running org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite Tests run: 36, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.486 sec Running org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.067 sec Running org.apache.hadoop.hbase.regionserver.wal.TestLogRolling Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 398.912 sec Running org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.882 sec Running org.apache.hadoop.hbase.io.hfile.TestHFileBlock Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 45.389 sec Running org.apache.hadoop.hbase.io.encoding.TestUpgradeFromHFileV1ToEncoding Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 81.351 sec Running org.apache.hadoop.hbase.io.encoding.TestLoadAndSwitchEncodeOnDisk Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.046 sec Running org.apache.hadoop.hbase.io.encoding.TestDataBlockEncoders Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 98.451 sec Running org.apache.hadoop.hbase.TestRegionRebalancing Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.102 sec Running org.apache.hadoop.hbase.mapred.TestTableInputFormat Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.587 sec Running org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.589 sec Running org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.93 sec Running org.apache.hadoop.hbase.replication.TestReplicationSource Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.229 sec Running org.apache.hadoop.hbase.mapred.TestTableMapReduce Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 62.291 sec Running org.apache.hadoop.hbase.replication.TestReplicationPeer Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.579 sec Running org.apache.hadoop.hbase.replication.TestMultiSlaveReplication Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 38.196 sec Running org.apache.hadoop.hbase.io.encoding.TestChangingEncoding Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 232.253 sec Running org.apache.hadoop.hbase.replication.TestMasterReplication Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 63.692 sec Running org.apache.hadoop.hbase.util.TestFSTableDescriptors Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.889 sec Running org.apache.hadoop.hbase.replication.TestReplication Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 139.965 sec <<< FAILURE! Running org.apache.hadoop.hbase.util.TestIdLock Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.294 sec Running org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.473 sec Running org.apache.hadoop.hbase.util.TestMergeTable Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.561 sec Running org.apache.hadoop.hbase.util.TestMiniClusterLoadSequential Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 87.833 sec Running org.apache.hadoop.hbase.util.TestHBaseFsck Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 38.101 sec Running org.apache.hadoop.hbase.util.TestMiniClusterLoadParallel Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 91.046 sec Running org.apache.hadoop.hbase.util.TestPoolMap$TestRoundRobinPoolType Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec Running org.apache.hadoop.hbase.util.TestPoolMap$TestThreadLocalPoolType Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 sec Running org.apache.hadoop.hbase.util.TestPoolMap$TestReusablePoolType Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.024 sec Running org.apache.hadoop.hbase.util.TestFSUtils Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.373 sec Running org.apache.hadoop.hbase.util.TestRegionSplitter Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.759 sec Running org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildBase Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.35 sec Running org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildOverlap Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.584 sec Running org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildHole Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.687 sec Running org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.161 sec Running org.apache.hadoop.hbase.util.TestMiniClusterLoadEncoded Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 66.404 sec Running org.apache.hadoop.hbase.TestInfoServers Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.273 sec Running org.apache.hadoop.hbase.util.TestMergeTool Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 332.903 sec Results : Failed tests: queueFailover(org.apache.hadoop.hbase.replication.TestReplication): Waited too much time for queueFailover replication Tests run: 930, Failures: 1, Errors: 0, Skipped: 10 [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 43:37.794s [INFO] Finished at: Fri Mar 09 05:48:26 UTC 2012 [INFO] Final Memory: 46M/773M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12-TRUNK-HBASE-2:test (secondPartTestsExecution) on project hbase: There are test failures. [ERROR] [ERROR] Please refer to /x1/jenkins/jenkins-slave/workspace/HBase-TRUNK-security/trunk/target/surefire-reports for the individual test results. [ERROR] -> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12-TRUNK-HBASE-2:test (secondPartTestsExecution) on project hbase: There are test failures. Please refer to /x1/jenkins/jenkins-slave/workspace/HBase-TRUNK-security/trunk/target/surefire-reports for the individual test results. at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:213) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) at org.apache.maven.lifecycle.internal.MojoExecutor.executeForkedExecutions(MojoExecutor.java:365) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:199) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59) at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:319) at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156) at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537) at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196) at org.apache.maven.cli.MavenCli.main(MavenCli.java:141) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290) at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409) at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352) Caused by: org.apache.maven.plugin.MojoFailureException: There are test failures. Please refer to /x1/jenkins/jenkins-slave/workspace/HBase-TRUNK-security/trunk/target/surefire-reports for the individual test results. at org.apache.maven.plugin.surefire.SurefireHelper.reportExecution(SurefireHelper.java:87) at org.apache.maven.plugin.surefire.SurefirePlugin.writeSummary(SurefirePlugin.java:651) at org.apache.maven.plugin.surefire.SurefirePlugin.handleSummary(SurefirePlugin.java:625) at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:137) at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:98) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209) ... 23 more [ERROR] [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException Build step 'Invoke top-level Maven targets' marked build as failure Archiving artifacts Updating HBASE-5515 Updating HBASE-5074 Updating HBASE-5541 Updating HBASE-5526 Updating HBASE-5514 Updating HBASE-5529 Recording test results
