I reran the tests just for read using --presplit 6 and enought lines to create 7 regions. Ran more than 12h for each release. 0.94.16 => 367 random get / second. 0.94.17 => 372 random get / second.
I initially had more than 400 gets/seconds for 0.94.16. But I'm not getting that anymore. So it's all +1 for me. JM 2014-02-23 13:54 GMT-05:00 Jean-Marc Spaggiari <[email protected]>: > Hey, I'm not getting where the 15% slowdown is coming too neither. I will > re-run the tests. I run the randomread on a 2M lines table written with > RandomWrite. It creates 3 region table so I did not used presplit. I run > the balancer after that. Since they are 3 nodes, whereever the regions are > assigned, HDFS blocks are local. So I'm not running major compaction. But I > can if required. I did not run it with MR. Only with 1 single client. I > will have the MR in the scope for the next release. > > For MAX_FILESIZE, I will first retry on my previous cluster with a > previous version and will com back if I'm able to get a difference in the > behaviors. But at the end, what I see in 0.94.17 is still good, and even if > different that what I think I saw before, it's better. More what I will > have expected. > > I'm rerunning the RandomRead with presplit and twice the rows. Will take > about 24h to get the results... > > > 2014-02-23 12:39 GMT-05:00 Ted Yu <[email protected]>: > > To make the read comparison fair, I think '--presplit=' should be used. >> >> Cheers >> >> >> On Sun, Feb 23, 2014 at 11:15 AM, lars hofhansl <[email protected]> wrote: >> >> > Thanks for the thorough testing. >> > >> > >> > Can you elaborate "page is split according to the MAX_FILESIZE parameter >> > until all regions are according to it"? >> > As for the 15% slowdown in the random read test... I have looked through >> > the issues and did not see anything warranting that, it's also nothing I >> > had observed in my testing. >> > >> > Did you run this with M/R? HBASE-10501 will reduce the default number of >> > regions per RegionServer (would be as high as 19 per table before even >> with >> > little data), so maybe you're seeing a bit less parallelism (for small >> data >> > sets). >> > >> > >> > -- Lars >> > >> > >> > >> > ________________________________ >> > From: Jean-Marc Spaggiari <[email protected]> >> > To: dev <[email protected]> >> > Sent: Sunday, February 23, 2014 5:47 AM >> > Subject: Re: [VOTE] The 2nd hbase 0.94.17 release candidate is available >> > for download >> > >> > >> > Haha ;) Yes. UI. Under the page details in the WebUI, In previous >> release >> > (maybe not 0.94.16 because I did not test it) when clicking split in the >> > WebUI, table was split in half. Now, page is split according to the >> > MAX_FILESIZE >> > parameter until all regions are according to it. >> > >> > I'm not 100% sure about it so might be wrong. But I'm not sure I have >> seen >> > that before. >> > >> > >> > >> > 2014-02-23 8:32 GMT-05:00 Ted Yu <[email protected]>: >> > >> > > bq. Click split under UA >> > > >> > > I guess you meant 'under UI' >> > > >> > > Cheers >> > > >> > > On Feb 23, 2014, at 4:47 AM, Jean-Marc Spaggiari < >> > [email protected]> >> > > wrote: >> > > >> > > > Tested, tl.tr: +1 ;) >> > > > >> > > > All tests are done with Oracle JDK 1.7.0_45 + Hadoop 1.2.1 (Except >> for >> > > the >> > > > 0.96.1 test where Hadoop 2.2.0 is used) >> > > > >> > > > Checked the jars, the signature, the CHANGES.TXT file, the >> > documentation >> > > => >> > > > Passed. >> > > > Checked logs, UI, HBCK, test suite => Passed >> > > > Run all Performances tests => Passed. 0.94.17 performances just a >> bit >> > > under >> > > > 0.94.16. RandomReadTest 15% slower with 0.94.17. Will retest it. >> Will >> > > post >> > > > all the numbers in detail in my blog later today. >> > > > >> > > > Only concern is again the IntegrationTestLoadAndVerify result where >> I >> > > don't >> > > > have the exptected result, but it's the same thing with 0.94.16 and >> > > 0.96.1 >> > > > too. With difference Hadoop version. >> > > > >> > > > >> > > > 0.94.16 + Hadoop 1.2.1: >> > > > 14/02/21 21:22:19 INFO mapred.JobClient: >> > > > org.apache.hadoop.hbase.test.IntegrationTestLoadAndVerify$Counters >> > > > 14/02/21 21:22:19 INFO mapred.JobClient: ROWS_WRITTEN=0 >> > > > 14/02/21 21:22:19 INFO mapred.JobClient: >> REFERENCES_CHECKED=9855584 >> > > > >> > > > 0.94.17 + Hadoop 1.2.1: >> > > > 14/02/21 21:31:28 INFO mapred.JobClient: >> > > > org.apache.hadoop.hbase.test.IntegrationTestLoadAndVerify$Counters >> > > > 14/02/21 21:31:28 INFO mapred.JobClient: ROWS_WRITTEN=0 >> > > > 14/02/21 21:31:28 INFO mapred.JobClient: >> REFERENCES_CHECKED=9855909 >> > > > >> > > > 0.96.1 + Hadoop 2.2.0: >> > > > org.apache.hadoop.hbase.test.IntegrationTestLoadAndVerify$Counters >> > > > REFERENCES_CHECKED=9855588 >> > > > >> > > > Create table, put, scan, compact => Passed. >> > > > Offline merged all regions into a single one, hbck, split => Passed. >> > > > Rolling restart => Failed once because of jar push issue (my bad), >> > > retried >> > > > => Passed. >> > > > Ran all tests (LoadTestTool, IntegrationTestLoadAndVerify, >> > > > HLogPerformanceEvaluation, IntegrationTestBigLinkedList) on a 4 >> nodes >> > > > cluster => Passed. >> > > > When we click split in the UA, seems to trigger major compaction, >> which >> > > was >> > > > not the case before (Not sure in which version this has been >> > introduced) >> > > > >> > > > Summary: >> > > > - IntgrationTestLoadAndVerify results still "strange"; >> > > > - Split behaviour in the UA seem the have changed. >> > > > - +1 >> > > > >> > > > JM >> > > > PS: Go Canada Go! >> > > > >> > > > 2014-02-21 10:01 GMT-05:00 Ted Yu <[email protected]>: >> > > > >> > > >> +1 based on the following: >> > > >> >> > > >> 1. Ran test suite using JDK 1.7 - pass >> > > >> 2. compiled Phoenix with locally built 0.94.17 - pass >> > > >> 3. checked the documentation >> > > >> >> > > >> Cheers >> > > >> >> > > >> >> > > >> On Thu, Feb 20, 2014 at 11:18 PM, lars hofhansl <[email protected]> >> > > wrote: >> > > >> >> > > >>> +1 >> > > >>> >> > > >>> - checked documentation and tarball >> > > >>> >> > > >>> - Ran in local and distributes mode >> > > >>> - checked the UI pages >> > > >>> >> > > >>> - loaded 100m KeyValues >> > > >>> - tested scan performance >> > > >>> >> > > >>> - converted between FAST_DIFF and NONE encoding >> > > >>> - tested flushes and compactions >> > > >>> >> > > >>> >> > > >>> -- Lars >> > > >>> >> > > >>> >> > > >>> ________________________________ >> > > >>> From: lars hofhansl <[email protected]> >> > > >>> To: hbase-dev <[email protected]> >> > > >>> Sent: Tuesday, February 18, 2014 8:31 PM >> > > >>> Subject: [VOTE] The 2nd hbase 0.94.17 release candidate is >> available >> > > for >> > > >>> download >> > > >>> >> > > >>> >> > > >>> The 2nd 0.94.17 RC is available for download at >> > > >>> http://people.apache.org/~larsh/hbase-0.94.17-rc1/ >> > > >>> Signed with my code signing key: C7CFE328 >> > > >>> >> > > >>> HBase 0.94.17 is a bug fix release with 35 fixes: >> > > >>> [HBASE-7963] - HBase VerifyReplication not working when >> security >> > > >>> enabled >> > > >>> [HBASE-10249] - TestReplicationSyncUpTool fails because >> failover >> > > >> takes >> > > >>> too long >> > > >>> [HBASE-10274] - MiniZookeeperCluster should close ZKDatabase >> when >> > > >>> shutdown ZooKeeperServers >> > > >>> [HBASE-10319] - HLog should roll periodically to allow DN >> > > >> decommission >> > > >>> to eventually complete. >> > > >>> [HBASE-10320] - Avoid ArrayList.iterator() >> ExplicitColumnTracker >> > > >>> [HBASE-10335] - AuthFailedException in zookeeper may block >> > > >> replication >> > > >>> forever >> > > >>> [HBASE-10340] - [BACKPORT] HBASE-9892 Add info port to >> ServerName >> > to >> > > >>> support multi instances in a node >> > > >>> [HBASE-10363] - [0.94] TestInputSampler and >> TestInputSamplerTool >> > > fail >> > > >>> under hadoop 2.0/23 profiles. >> > > >>> [HBASE-10371] - Compaction creates empty hfile, then selects >> this >> > > >> file >> > > >>> for compaction and creates empty hfile and over again >> > > >>> [HBASE-10383] - Secure Bulk Load for 'completebulkload' fails >> for >> > > >>> version 0.94.15 >> > > >>> [HBASE-10400] - [hbck] Continue if region dir missing on region >> > > merge >> > > >>> attempt >> > > >>> [HBASE-10401] - [hbck] perform overlap group merges in parallel >> > > >>> [HBASE-10448] - ZKUtil create and watch methods don't set >> watch in >> > > >>> some cases >> > > >>> [HBASE-10470] - Import generates huge log file while importing >> > large >> > > >>> amounts of data >> > > >>> [HBASE-10481] - API Compatibility JDiff script does not >> properly >> > > >>> handle arguments in reverse order >> > > >>> [HBASE-10482] - ReplicationSyncUp doesn't clean up its ZK, >> needed >> > > for >> > > >>> tests >> > > >>> [HBASE-10485] - PrefixFilter#filterKeyValue() should perform >> > > >> filtering >> > > >>> on row key >> > > >>> [HBASE-10489] - TestImportExport fails in 0.94 with Hadoop2 >> > > >>> [HBASE-10493] - InclusiveStopFilter#filterKeyValue() should >> > perform >> > > >>> filtering on row key >> > > >>> [HBASE-10501] - Improve >> IncreasingToUpperBoundRegionSplitPolicy to >> > > >>> avoid too many regions >> > > >>> [HBASE-10505] - Import.filterKv does not call >> Filter.filterRowKey >> > > >>> [HBASE-10506] - Fail-fast if client connection is lost before >> the >> > > >> real >> > > >>> call be executed in RPC layer >> > > >>> [HBASE-10508] - Backport HBASE-10365 'HBaseFsck should clean up >> > > >>> connection properly when repair is completed' to 0.94 and 0.96 >> > > >>> [HBASE-10539] - HRegion.addAndGetGlobalMemstoreSize returns >> > previous >> > > >>> size >> > > >>> [HBASE-10545] - RS Hangs waiting on region to close on >> shutdown; >> > has >> > > >>> to timeout before can go down >> > > >>> [HBASE-10546] - Two scanner objects are open for each hbase map >> > task >> > > >>> but only one scanner object is closed >> > > >>> [HBASE-10551] - Change local mode back to one RS thread by >> default >> > > >>> [HBASE-10552] - >> > > >> HFilePerformanceEvaluation.GaussianRandomReadBenchmark >> > > >>> fails sometimes. >> > > >>> [HBASE-10555] - Backport HBASE-8519 to 0.94, Backup master will >> > > never >> > > >>> come up if primary master dies during initialization >> > > >>> [HBASE-10562] - Fix TestMultiTableInputFormat for Hadoop 2 in >> 0.94 >> > > >>> [HBASE-10212] - New rpc metric: number of active handler >> > > >>> [HBASE-10423] - Report back the message of split or rollback >> > failure >> > > >>> to the master >> > > >>> [HBASE-10457] - Print corrupted file information in >> SnapshotInfo >> > > tool >> > > >>> without -file option >> > > >>> [HBASE-10473] - Add utility for adorning http Context >> > > >>> [HBASE-10480] - TestLogRollPeriod#testWithEdits may fail due to >> > > >>> insufficient waiting >> > > >>> >> > > >>> Notable in this release is HBASE-10501, which changes how regions >> are >> > > >>> distributed for an initially empty table. >> > > >>> >> > > >>> The list of changes is also available here: >> > > >> >> > > >> > >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753&version=12325845 >> > > >>> >> > > >>> Here're the jenkins runs for this RC: >> > > >>> https://builds.apache.org/job/HBase-0.94.17/15/ and >> > > >>> https://builds.apache.org/job/HBase-0.94.17-security/5/ >> > > >>> >> > > >>> Please try out the RC, check out the doc, take it for a spin, etc, >> > and >> > > >>> vote +1/-1 by EOD February 24th on whether we should release this >> as >> > > >>> 0.94.17. >> > > >>> >> > > >>> As before, I realize that release testing is like pulling teeth, >> but >> > > >>> please give this is quick run, so we can get these bug fixes out >> > there. >> > > >>> >> > > >>> Thanks. >> > > >>> >> > > >>> -- Lars >> > > >> >> > > >> > >> > >
