[ https://issues.apache.org/jira/browse/HBASE-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HBASE-4908: ------------------------------- Attachment: D549.5.patch mbautin updated the revision "[jira] [HBASE-4908] HBase cluster test tool (port from 0.89-fb)". Reviewers: stack, Karthik, Kannan, nspiegelberg, JIRA This passes unit tests (I will confirm this again -- please do not commit yet), and is successfully running a load test in a 5-node cluster. The reader verifies keys sequentially in the current version, and if it catches up with the reader it sleeps until more keys are written. This is different from the original version of HBaseTest in 0.89-fb (renamed LoadTest in this version), and I think I will modify the reader to read some previous keys that are known to have been written when the "current key" of the reader catches up with the last key written by the writer. I also made the number of read errors to tolerate configurable (the --max_read_errors option). I think it will take me a couple more iterations on this diff before it can be committed. Sorry for a lot of updates and emails. REVISION DETAIL https://reviews.facebook.net/D549 AFFECTED FILES pom.xml src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java src/test/java/org/apache/hadoop/hbase/EmptyWatcher.java src/main/java/org/apache/hadoop/hbase/HConstants.java src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java src/main/java/org/apache/hadoop/hbase/util/Bytes.java src/main/java/org/apache/hadoop/hbase/util/HMerge.java src/main/java/org/apache/hadoop/hbase/util/Keying.java src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java src/main/java/org/apache/hadoop/hbase/zookeeper/ZKConfig.java src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java src/test/java/org/apache/hadoop/hbase/util/IntegrationTestTool.java src/test/java/org/apache/hadoop/hbase/util/LoadTest.java src/test/java/org/apache/hadoop/hbase/util/LoadTestKVGenerator.java src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReader.java src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java src/test/java/org/apache/hadoop/hbase/util/TestBytes.java src/test/java/org/apache/hadoop/hbase/util/TestLoadTestKVGenerator.java src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadParallel.java src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadSequential.java > HBase cluster test tool (port from 0.89-fb) > ------------------------------------------- > > Key: HBASE-4908 > URL: https://issues.apache.org/jira/browse/HBASE-4908 > Project: HBase > Issue Type: Sub-task > Reporter: Mikhail Bautin > Assignee: Mikhail Bautin > Attachments: D549.1.patch, D549.2.patch, D549.3.patch, D549.4.patch, > D549.5.patch > > > Porting one of our HBase cluster test tools (a single-process multi-threaded > load generator and verifier) from 0.89-fb to trunk. > I cleaned up the code a bit compared to what's in 0.89-fb, and discovered > that it has some features that I have not tried yet (some kind of a kill > test, and some way to run HBase as multiple processes on one machine). > The main utility of this piece of code for us has been the HBaseClusterTest > command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a > load test in our five-node dev cluster testing, e.g.: > hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn > load_test -read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression > GZIP > I will be using this code to load-test the delta encoding patch and making > fixes, but I am submitting the patch for early feedback. I will probably try > out its other functionality and comment on how it works. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira