I tried using reseek() as suggested, along with my patch from HBASE-8306 (30% selection rate, random distribution and FAST_DIFF encoding on both column families). I got uneven results:
2013-04-09 16:59:01,324 INFO [main] regionserver.TestJoinedScanners(167): Slow scanner finished in 7.529083 seconds, got 1546 rows 2013-04-09 16:59:06,760 INFO [main] regionserver.TestJoinedScanners(167): Joined scanner finished in 5.43579 seconds, got 1546 rows ... 2013-04-09 16:59:12,711 INFO [main] regionserver.TestJoinedScanners(167): Slow scanner finished in 5.95016 seconds, got 1546 rows 2013-04-09 16:59:20,240 INFO [main] regionserver.TestJoinedScanners(167): Joined scanner finished in 7.529044 seconds, got 1546 rows FYI On Tue, Apr 9, 2013 at 4:47 PM, lars hofhansl <la...@apache.org> wrote: > We did some tests here. > I ran this through the profiler against a local RegionServer and found the > part that causes the slowdown is a seek called here: > boolean mayHaveData = > (nextJoinedKv != null && > nextJoinedKv.matchingRow(currentRow, offset, length)) > || > (this.joinedHeap.seek(KeyValue.createFirstOnRow(currentRow, offset, length)) > && joinedHeap.peek() != null > && joinedHeap.peek().matchingRow(currentRow, offset, > length)); > > Looking at the code, this is needed because the joinedHeap can fall > behind, and hence we have to catch it up. > The key observation, though, is that the joined heap can only ever be > behind, and hence we do not need a seek, but only a reseek. > > Deploying a RegionServer with the seek replaced with reseek we see an > improvement in *all* cases. > > I'll file a jira with a fix later. > > -- Lars > > > > ________________________________ > From: James Taylor <jtay...@salesforce.com> > To: user@hbase.apache.org > Sent: Monday, April 8, 2013 6:53 PM > Subject: Re: Essential column family performance > > Good idea, Sergey. We'll rerun with larger non essential column family > values and see if there's a crossover point. One other difference for us > is that we're using FAST_DIFF encoding. We'll try with no encoding too. > Our table has 20 million rows across four regions servers. > > Regarding the parallelization we do, we run multiple scans in parallel > instead of one single scan over the table. We use the region boundaries > of the table to divide up the work evenly, adding a start/stop key for > each scan that corresponds to the region boundaries. Our client then > does a final merge/aggregation step (i.e. adding up the count it gets > back from the scan for each region). > > On 04/08/2013 01:34 PM, Sergey Shelukhin wrote: > > IntegrationTestLazyCfLoading uses randomly distributed keys with the > > following condition for filtering: > > 1 == (Long.parseLong(Bytes.toString(rowKey, 0, 4), 16) & 1); where rowKey > > is hex string of MD5 key. > > Then, there are 2 "lazy" CFs, each of which has a value of 4-64k. > > This test also showed significant improvement IIRC, so random > distribution > > and high %%ge of values selected should not be a problem as such. > > > > My hunch would be that the additional cost of seeks/merging the results > > from two CFs outweights the benefit of lazy loading on such small values > > for the "lazy" CF with lots of data selected. This feature definitely > makes > > no sense if you are selecting all values, because then extra work is > being > > done for no benefit (everything is read anyway). > > So the use cases would be larger "lazy" CFs or/and low percentage of > values > > selected. > > > > Can you try to increase the 2nd CF values' size and rerun the test? > > > > > > On Mon, Apr 8, 2013 at 10:38 AM, James Taylor <jtay...@salesforce.com > >wrote: > > > >> In the TestJoinedScanners.java, is the 40% randomly distributed or > >> sequential? > >> > >> In our test, the % is randomly distributed. Also, our custom filter does > >> the same thing that SingleColumnValueFilter does. On the client-side, > we'd > >> execute the query in parallel, through multiple scans along the region > >> boundaries. Would that have a negative impact on performance for this > >> "essential column family" feature? > >> > >> Thanks, > >> > >> James > >> > >> > >> On 04/08/2013 10:10 AM, Anoop John wrote: > >> > >>> Agree here. The effectiveness depends on what % of data satisfies the > >>> condition, how it is distributed across HFile blocks. We will get > >>> performance gain when the we will be able to skip some HFile blocks > (from > >>> non essential CFs). Can test with different HFile block size (lower > >>> value)? > >>> > >>> -Anoop- > >>> > >>> > >>> On Mon, Apr 8, 2013 at 8:19 PM, Ted Yu <yuzhih...@gmail.com> wrote: > >>> > >>> I made the following change in TestJoinedScanners.java: > >>>> - int flag_percent = 1; > >>>> + int flag_percent = 40; > >>>> > >>>> The test took longer but still favors joined scanner. > >>>> I got some new results: > >>>> > >>>> 2013-04-08 07:46:06,959 INFO [main] regionserver.** > >>>> TestJoinedScanners(157): > >>>> Slow scanner finished in 7.424388 seconds, got 2050 rows > >>>> ... > >>>> 2013-04-08 07:46:12,010 INFO [main] regionserver.** > >>>> TestJoinedScanners(157): > >>>> Joined scanner finished in 5.05063 seconds, got 2050 rows > >>>> > >>>> 2013-04-08 07:46:18,358 INFO [main] regionserver.** > >>>> TestJoinedScanners(157): > >>>> Slow scanner finished in 6.348517 seconds, got 2050 rows > >>>> ... > >>>> 2013-04-08 07:46:22,946 INFO [main] regionserver.** > >>>> TestJoinedScanners(157): > >>>> Joined scanner finished in 4.587545 seconds, got 2050 rows > >>>> > >>>> Looks like effectiveness of joined scanner is affected by > distribution of > >>>> data. > >>>> > >>>> Cheers > >>>> > >>>> On Sun, Apr 7, 2013 at 8:52 PM, lars hofhansl <la...@apache.org> > wrote: > >>>> > >>>> Looking at the joined scanner test code, it sets it up such that 1% > of > >>>> the > >>>> > >>>>> rows match, which would somewhat be in line with James' results. > >>>>> > >>>>> In my own testing a while ago I found a 100% improvement with 0% > match. > >>>>> > >>>>> > >>>>> -- Lars > >>>>> > >>>>> > >>>>> > >>>>> ______________________________**__ > >>>>> From: Ted Yu <yuzhih...@gmail.com> > >>>>> To: user@hbase.apache.org > >>>>> Sent: Sunday, April 7, 2013 4:13 PM > >>>>> Subject: Re: Essential column family performance > >>>>> > >>>>> I have attached 5416-TestJoinedScanners-0.94.**txt to HBASE-5416 for > >>>>> your > >>>>> reference. > >>>>> > >>>>> On my MacBook, I got the following results from the test: > >>>>> > >>>>> 2013-04-07 16:08:17,474 INFO [main] > >>>>> > >>>> regionserver.**TestJoinedScanners(157): > >>>> > >>>>> Slow scanner finished in 7.973822 seconds, got 100 rows > >>>>> ... > >>>>> 2013-04-07 16:08:17,946 INFO [main] > >>>>> > >>>> regionserver.**TestJoinedScanners(157): > >>>> > >>>>> Joined scanner finished in 0.47235 seconds, got 100 rows > >>>>> > >>>>> Cheers > >>>>> > >>>>> On Sun, Apr 7, 2013 at 4:03 PM, Ted Yu <yuzhih...@gmail.com> wrote: > >>>>> > >>>>> Looking at > >>>>>> https://issues.apache.org/**jira/secure/attachment/** > >>>> 12564340/5416-0.94-v3.txt< > https://issues.apache.org/jira/secure/attachment/12564340/5416-0.94-v3.txt > > > >>>> , > >>>> > >>>>> I found that it didn't contain TestJoinedScanners which shows > >>>>> > >>>>>> difference in scanner performance: > >>>>>> > >>>>>> LOG.info((slow ? "Slow" : "Joined") + " scanner finished in " + > >>>>>> Double.toString(timeSec) > >>>>>> > >>>>>> + " seconds, got " + Long.toString(rows_count/2) + " rows"); > >>>>>> > >>>>>> The test uses SingleColumnValueFilter: > >>>>>> > >>>>>> SingleColumnValueFilter filter = new SingleColumnValueFilter( > >>>>>> > >>>>>> cf_essential, col_name, CompareFilter.CompareOp.EQUAL, > >>>>>> > >>>>> flag_yes); > >>>>> It is possible that the custom filter you were using would exhibit > >>>>>> different access pattern compared to SingleColumnValueFilter. e.g. > does > >>>>>> your filter utilize hint ? > >>>>>> It would be easier for me and other people to reproduce the issue > you > >>>>>> experienced if you put your scenario in some test similar to > >>>>>> TestJoinedScanners. > >>>>>> > >>>>>> Will take a closer look at the code Monday. > >>>>>> > >>>>>> Cheers > >>>>>> > >>>>>> On Sun, Apr 7, 2013 at 11:37 AM, James Taylor < > jtay...@salesforce.com > >>>>>> wrote: > >>>>>> > >>>>>> Yes, on 0.94.6. We have our own custom filter derived from > FilterBase, > >>>>>> so > >>>>>> filterIfMissing isn't the issue - the results of the scan are > correct. > >>>>>>> I can see that if the essential column family has more data > compared > >>>>>>> > >>>>>> to > >>>>> the non essential column family that the results would eventually > even > >>>>>> out. > >>>>>> I was hoping to always be able to enable the essential column family > >>>>>>> feature. Is there an inherent reason why performance would degrade > >>>>>>> > >>>>>> like > >>>>> this? Does it boil down to a single sequential scan versus many > seeks? > >>>>>>> Thanks, > >>>>>>> > >>>>>>> James > >>>>>>> > >>>>>>> > >>>>>>> On 04/07/2013 07:44 AM, Ted Yu wrote: > >>>>>>> > >>>>>>> James: > >>>>>>>> Your test was based on 0.94.6.1, right ? > >>>>>>>> > >>>>>>>> What Filter were you using ? > >>>>>>>> > >>>>>>>> If you used SingleColumnValueFilter, have you seen my comment > here ? > >>>>>>>> https://issues.apache.org/****jira/browse/HBASE-5416?**< > https://issues.apache.org/**jira/browse/HBASE-5416?**> > >>>>>>>> focusedCommentId=13541229&****page=com.atlassian.jira.** > >>>>>>>> plugin.system.issuetabpanels:****comment-tabpanel#comment-**** > >>>>>>>> 13541229< > >>>>>>>> > >>>>>>> https://issues.apache.org/**jira/browse/HBASE-5416?** > >>>> focusedCommentId=13541229&**page=com.atlassian.jira.** > >>>> plugin.system.issuetabpanels:**comment-tabpanel#comment-**13541229< > https://issues.apache.org/jira/browse/HBASE-5416?focusedCommentId=13541229&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13541229 > > > >>>> > >>>>> BTW the use case Max Lapan tried to address has non essential > column > >>>>>>>> family > >>>>>>>> carrying considerably more data compared to essential column > family. > >>>>>>>> > >>>>>>>> Cheers > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> On Sat, Apr 6, 2013 at 11:05 PM, James Taylor < > >>>>>>>> > >>>>>>> jtay...@salesforce.com > >>>>> wrote: > >>>>>>>> Hello, > >>>>>>>> > >>>>>>>>> We're doing some performance testing of the essential column > family > >>>>>>>>> feature, and we're seeing some performance degradation when > >>>>>>>>> > >>>>>>>> comparing > >>>>> with > >>>>>>>>> and without the feature enabled: > >>>>>>>>> > >>>>>>>>> Performance of scan relative > >>>>>>>>> % of rows selected to not enabling the feature > >>>>>>>>> --------------------- ------------------------------******-- > >>>>>>>>> > >>>>>>>>> 100% 1.0x > >>>>>>>>> 80% 2.0x > >>>>>>>>> 60% 2.3x > >>>>>>>>> 40% 2.2x > >>>>>>>>> 20% 1.5x > >>>>>>>>> 10% 1.0x > >>>>>>>>> 5% 0.67x > >>>>>>>>> 0% 0.30% > >>>>>>>>> > >>>>>>>>> In our scenario, we have two column families. The key value from > the > >>>>>>>>> essential column family is used in the filter, while the key > value > >>>>>>>>> > >>>>>>>> from > >>>>>> the > >>>>>>>>> other, non essential column family is returned by the scan. Each > row > >>>>>>>>> contains values for both key values, with the values being > >>>>>>>>> > >>>>>>>> relatively > >>>>> narrow (less than 50 bytes). In this scenario, the only time we're > >>>>>>>>> seeing a > >>>>>>>>> performance gain is when less than 10% of the rows are selected. > >>>>>>>>> > >>>>>>>>> Is this a reasonable test? Has anyone else measured this? > >>>>>>>>> > >>>>>>>>> Thanks, > >>>>>>>>> > >>>>>>>>> James > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> >
