Re: Custom Filter and SEEK_NEXT_USING_HINT issue
Anoop, Ramkrishna Thank you for explanation! I've got it. On Mon, Jan 21, 2013 at 12:59 PM, Anoop Sam John wrote: > > I suppose if scanning process has started at once on > all regions, then I would find in log files at least one value per region, > but I have found one value per region only for those regions, that resides > before the particular one. > > @Eugeny - FuzzyFilter like any other filter works at the server side. The > scanning from client side will be like sequential starting from the 1st > region (Region with empty startkey or the corresponding region which > contains the startkey whatever you mentioned in your scan). From client, > request will go to RS for scanning a region. Once that region is over the > next region will be contacted for scan(from client) and so on. There is no > parallel scanning of multiple regions from client side. [This is when > using a HTable scan APIs] > > When MR used for scanning, we will be doing parallel scans from all the > regions. Here will be having mappers per region. But the normal scan from > client side will be sequential on the regions not parallel. > > -Anoop- > > From: Eugeny Morozov [emoro...@griddynamics.com] > Sent: Monday, January 21, 2013 1:46 PM > To: user@hbase.apache.org > Cc: Alex Baranau > Subject: Re: Custom Filter and SEEK_NEXT_USING_HINT issue > > Finally, the mystery has been solved. > > Small remark before I explain everything. > > The situation with only region is absolutely the same: > Fzzy: 1Q7iQ9JA > Next fzzy: F7dtxwqVQ_Pw <-- the value I'm trying to find. > Fzzy: F7dt8QWPSIDw > Somehow FuzzyRowFilter has just omit my value here. > > > So, the explanation. > In javadoc for FuzzyRowFilter question mark is used as substitution for > unknown value. Of course it's possible to use anything including zero > instead of question mark. > For quite some time we used literals to encode our keys. Literals like > you've seen already: 1Q7iQ9JA or F7dt8QWPSIDw. But that's Base64 form > of just 8 bytes, which requires 1.5 times more space. So we've decided to > store raw version - just byte[8]. But unfortunately the symbol '?' is > exactly in the middle of the byte (according to ascii table > http://www.asciitable.com/), which means with FuzzyRowFilter we skip half > of values in some cases. In the same time question mark is exactly before > any letter that could be used in key. > > Despite the fact we have integration tests - that's just a coincidence we > haven't such an example in there. > > So, as an advice - always use zero instead of question mark for > FuzzyRowFilter. > > Thank's to everyone! > > P.S. But the question with region scanning order is still here. I do not > understand why with FuzzyFilter it goes from one region to another until it > stops at the value. I suppose if scanning process has started at once on > all regions, then I would find in log files at least one value per region, > but I have found one value per region only for those regions, that resides > before the particular one. > > > On Mon, Jan 21, 2013 at 4:22 AM, Michael Segel >wrote: > > > If its the same class and its not a patch, then the first class loaded > > wins. > > > > So if you have a Class Foo and HBase has a Class Foo, your code will > never > > see the light of day. > > > > Perhaps I'm stating the obvious but its something to think about when > > working w Hadoop. > > > > On Jan 19, 2013, at 3:36 AM, Eugeny Morozov > > wrote: > > > > > Ted, > > > > > > that is correct. > > > HBase 0.92.x and we use part of the patch 6509. > > > > > > I use the filter as a custom filter, it lives in separate jar file and > > goes > > > to HBase's classpath. I did not patch HBase. > > > Moreover I do not use protobuf's descriptions that comes with the > filter > > in > > > patch. Only two classes I have - FuzzyRowFilter itself and its test > > class. > > > > > > And it works perfectly on small dataset like 100 rows (1 region). But > > when > > > my dataset is more than 10mln (260 regions), it somehow loosing rows. > I'm > > > not sure, but it seems to me it is not fault of the filter. > > > > > > > > > On Sat, Jan 19, 2013 at 3:56 AM, Ted Yu wrote: > > > > > >> To my knowledge CDH-4.1.2 is based on HBase 0.92.x > > >> > > >> Looks like you were using patch from HBASE-6509 which was integrated > to > > >> trunk only. > >
Re: Custom Filter and SEEK_NEXT_USING_HINT issue
Finally, the mystery has been solved. Small remark before I explain everything. The situation with only region is absolutely the same: Fzzy: 1Q7iQ9JA Next fzzy: F7dtxwqVQ_Pw <-- the value I'm trying to find. Fzzy: F7dt8QWPSIDw Somehow FuzzyRowFilter has just omit my value here. So, the explanation. In javadoc for FuzzyRowFilter question mark is used as substitution for unknown value. Of course it's possible to use anything including zero instead of question mark. For quite some time we used literals to encode our keys. Literals like you've seen already: 1Q7iQ9JA or F7dt8QWPSIDw. But that's Base64 form of just 8 bytes, which requires 1.5 times more space. So we've decided to store raw version - just byte[8]. But unfortunately the symbol '?' is exactly in the middle of the byte (according to ascii table http://www.asciitable.com/), which means with FuzzyRowFilter we skip half of values in some cases. In the same time question mark is exactly before any letter that could be used in key. Despite the fact we have integration tests - that's just a coincidence we haven't such an example in there. So, as an advice - always use zero instead of question mark for FuzzyRowFilter. Thank's to everyone! P.S. But the question with region scanning order is still here. I do not understand why with FuzzyFilter it goes from one region to another until it stops at the value. I suppose if scanning process has started at once on all regions, then I would find in log files at least one value per region, but I have found one value per region only for those regions, that resides before the particular one. On Mon, Jan 21, 2013 at 4:22 AM, Michael Segel wrote: > If its the same class and its not a patch, then the first class loaded > wins. > > So if you have a Class Foo and HBase has a Class Foo, your code will never > see the light of day. > > Perhaps I'm stating the obvious but its something to think about when > working w Hadoop. > > On Jan 19, 2013, at 3:36 AM, Eugeny Morozov > wrote: > > > Ted, > > > > that is correct. > > HBase 0.92.x and we use part of the patch 6509. > > > > I use the filter as a custom filter, it lives in separate jar file and > goes > > to HBase's classpath. I did not patch HBase. > > Moreover I do not use protobuf's descriptions that comes with the filter > in > > patch. Only two classes I have - FuzzyRowFilter itself and its test > class. > > > > And it works perfectly on small dataset like 100 rows (1 region). But > when > > my dataset is more than 10mln (260 regions), it somehow loosing rows. I'm > > not sure, but it seems to me it is not fault of the filter. > > > > > > On Sat, Jan 19, 2013 at 3:56 AM, Ted Yu wrote: > > > >> To my knowledge CDH-4.1.2 is based on HBase 0.92.x > >> > >> Looks like you were using patch from HBASE-6509 which was integrated to > >> trunk only. > >> Please confirm. > >> > >> Copying Alex who wrote the patch. > >> > >> Cheers > >> > >> On Fri, Jan 18, 2013 at 3:28 PM, Eugeny Morozov > >> wrote: > >> > >>> Hi, folks! > >>> > >>> HBase, Hadoop, etc version is CDH-4.1.2 > >>> > >>> I'm using custom FuzzyRowFilter, which I get from > >>> > >>> > >> > http://blog.sematext.com/2012/08/09/consider-using-fuzzyrowfilter-when-in-need-for-secondary-indexes-in-hbase/and > >>> suddenly after quite a time we found that it starts loosing data. > >>> > >>> Basically the idea of FuzzyRowFilter is that it tries to find key that > >> has > >>> been provided and if there is no such a key - but more exists in table > - > >> it > >>> returns SEEK_NEXT_USING_HINT. And in getNextKeyHint(...) it builds > >> required > >>> key. As I understand, HBase in this key will fast-forward to required > >> key - > >>> it must be similar or same as to get Scan with setStartRow. > >>> > >>> I'm trying to find key F7dt8QWPSIDw, it is definitely in HBase - I'm > able > >>> to get it using Scan.setStartRow. > >>> For FuzzyFilter I'm using empty Scan - I didn't specify start row, stop > >> row > >>> or anything related. > >>> That's what happening: > >>> > >>> Fzzy: 1Q7iQ9JA > >>> Next fzzy: F7dtxwqVQ_Pw > >>> Fzzy: AQAAnA96rxTg > >>> Next fzzy: F7dtxwqVQ_Pw > >>> Fzzy: AgAADQWPSIDw > >>> Next fzzy: F7dtxwqVQ_Pw > >>> Fzzy:
Re: Custom Filter and SEEK_NEXT_USING_HINT issue
Ted, thanks for the question. There are results of investigation. It seems I am mistaken. I thought that scanners are assigned to each regions to scan (and do that in parallel) and that means each scanner should start from the beginning of its region and then fall down to the required record. But currently we have 256 splits in the table by the first byte of values: start - end NA - \x01 \x01 - \x02 \x02 - \x03 ... \xFE - \xFF \xFF - NA And it turns out that the values I've seen are the values from different regions, except two last values - they both reside in just one region: 1Q7iQ9JA : [0 <-- that's the value's first byte (meaning particular region here) AQAAnA96rxTg : [1 AgAADQWPSIDw : [2 ... EwAAEwqVQrTw : [19 FAAACQqVQrTw : [20 FQAAIAqVQrTw : [21 FgAAeAWPSIDw : [22 FwAAAw33Zb9Q : [23 F7dt8QWPSIDw : [23 1. I still don't get, why it skips required value. 2. The only explanation to have such an output I've found is that scanning is searching regions one by one until it found the value. Should it be so? Shouldn't it start from the beginning (if there is no setStartRow) (and in parallel for all regions at once) and in second step (after filter's getHint method) know exactly where to go? On Sat, Jan 19, 2013 at 5:16 PM, Ted wrote: > In your original email you said the first key looked like start key of a > region, can you verify that ? > > Thanks > > On Jan 19, 2013, at 1:36 AM, Eugeny Morozov > wrote: > > > Ted, > > > > that is correct. > > HBase 0.92.x and we use part of the patch 6509. > > > > I use the filter as a custom filter, it lives in separate jar file and > goes > > to HBase's classpath. I did not patch HBase. > > Moreover I do not use protobuf's descriptions that comes with the filter > in > > patch. Only two classes I have - FuzzyRowFilter itself and its test > class. > > > > And it works perfectly on small dataset like 100 rows (1 region). But > when > > my dataset is more than 10mln (260 regions), it somehow loosing rows. I'm > > not sure, but it seems to me it is not fault of the filter. > > > > > > On Sat, Jan 19, 2013 at 3:56 AM, Ted Yu wrote: > > > >> To my knowledge CDH-4.1.2 is based on HBase 0.92.x > >> > >> Looks like you were using patch from HBASE-6509 which was integrated to > >> trunk only. > >> Please confirm. > >> > >> Copying Alex who wrote the patch. > >> > >> Cheers > >> > >> On Fri, Jan 18, 2013 at 3:28 PM, Eugeny Morozov > >> wrote: > >> > >>> Hi, folks! > >>> > >>> HBase, Hadoop, etc version is CDH-4.1.2 > >>> > >>> I'm using custom FuzzyRowFilter, which I get from > >> > http://blog.sematext.com/2012/08/09/consider-using-fuzzyrowfilter-when-in-need-for-secondary-indexes-in-hbase/and > >>> suddenly after quite a time we found that it starts loosing data. > >>> > >>> Basically the idea of FuzzyRowFilter is that it tries to find key that > >> has > >>> been provided and if there is no such a key - but more exists in table > - > >> it > >>> returns SEEK_NEXT_USING_HINT. And in getNextKeyHint(...) it builds > >> required > >>> key. As I understand, HBase in this key will fast-forward to required > >> key - > >>> it must be similar or same as to get Scan with setStartRow. > >>> > >>> I'm trying to find key F7dt8QWPSIDw, it is definitely in HBase - I'm > able > >>> to get it using Scan.setStartRow. > >>> For FuzzyFilter I'm using empty Scan - I didn't specify start row, stop > >> row > >>> or anything related. > >>> That's what happening: > >>> > >>> Fzzy: 1Q7iQ9JA > >>> Next fzzy: F7dtxwqVQ_Pw > >>> Fzzy: AQAAnA96rxTg > >>> Next fzzy: F7dtxwqVQ_Pw > >>> Fzzy: AgAADQWPSIDw > >>> Next fzzy: F7dtxwqVQ_Pw > >>> Fzzy: AwAA-Q33Zb9Q > >>> Next fzzy: F7dtxwqVQ_Pw > >>> Fzzy: BAAAOg8oyu7A > >>> Next fzzy: F7dtxwqVQ_Pw > >>> Fzzy: BQAA9gqVQrTw > >>> Next fzzy: F7dtxwqVQ_Pw > >>> Fzzy: BgABZQ7iQ9JA > >>> Next fzzy: F7dtxwqVQ_Pw > >>> Fzzy: BwAAbgrpAojg > >>> Next fzzy: F7dtxwqVQ_Pw > >>> Fzzy: CAAAUQWPSIDw > >>> Next fzzy: F7dtxwqVQ_Pw > >>> Fzzy: CQABVgqVQrTw > >>> Next fzzy: F7dtxwqVQ_Pw > >>> Fzzy: CgAAOQ7iQ9JA > >>> Next fzzy: F7dtxwqVQ_Pw > >>> Fzzy: CwAALwqVQrTw > &
Re: Custom Filter and SEEK_NEXT_USING_HINT issue
Ted, that is correct. HBase 0.92.x and we use part of the patch 6509. I use the filter as a custom filter, it lives in separate jar file and goes to HBase's classpath. I did not patch HBase. Moreover I do not use protobuf's descriptions that comes with the filter in patch. Only two classes I have - FuzzyRowFilter itself and its test class. And it works perfectly on small dataset like 100 rows (1 region). But when my dataset is more than 10mln (260 regions), it somehow loosing rows. I'm not sure, but it seems to me it is not fault of the filter. On Sat, Jan 19, 2013 at 3:56 AM, Ted Yu wrote: > To my knowledge CDH-4.1.2 is based on HBase 0.92.x > > Looks like you were using patch from HBASE-6509 which was integrated to > trunk only. > Please confirm. > > Copying Alex who wrote the patch. > > Cheers > > On Fri, Jan 18, 2013 at 3:28 PM, Eugeny Morozov > wrote: > > > Hi, folks! > > > > HBase, Hadoop, etc version is CDH-4.1.2 > > > > I'm using custom FuzzyRowFilter, which I get from > > > > > http://blog.sematext.com/2012/08/09/consider-using-fuzzyrowfilter-when-in-need-for-secondary-indexes-in-hbase/and > > suddenly after quite a time we found that it starts loosing data. > > > > Basically the idea of FuzzyRowFilter is that it tries to find key that > has > > been provided and if there is no such a key - but more exists in table - > it > > returns SEEK_NEXT_USING_HINT. And in getNextKeyHint(...) it builds > required > > key. As I understand, HBase in this key will fast-forward to required > key - > > it must be similar or same as to get Scan with setStartRow. > > > > I'm trying to find key F7dt8QWPSIDw, it is definitely in HBase - I'm able > > to get it using Scan.setStartRow. > > For FuzzyFilter I'm using empty Scan - I didn't specify start row, stop > row > > or anything related. > > That's what happening: > > > > Fzzy: 1Q7iQ9JA > > Next fzzy: F7dtxwqVQ_Pw > > Fzzy: AQAAnA96rxTg > > Next fzzy: F7dtxwqVQ_Pw > > Fzzy: AgAADQWPSIDw > > Next fzzy: F7dtxwqVQ_Pw > > Fzzy: AwAA-Q33Zb9Q > > Next fzzy: F7dtxwqVQ_Pw > > Fzzy: BAAAOg8oyu7A > > Next fzzy: F7dtxwqVQ_Pw > > Fzzy: BQAA9gqVQrTw > > Next fzzy: F7dtxwqVQ_Pw > > Fzzy: BgABZQ7iQ9JA > > Next fzzy: F7dtxwqVQ_Pw > > Fzzy: BwAAbgrpAojg > > Next fzzy: F7dtxwqVQ_Pw > > Fzzy: CAAAUQWPSIDw > > Next fzzy: F7dtxwqVQ_Pw > > Fzzy: CQABVgqVQrTw > > Next fzzy: F7dtxwqVQ_Pw > > Fzzy: CgAAOQ7iQ9JA > > Next fzzy: F7dtxwqVQ_Pw > > Fzzy: CwAALwqVQrTw > > Next fzzy: F7dtxwqVQ_Pw > > Fzzy: DAAAMwWPSIDw > > Next fzzy: F7dtxwqVQ_Pw > > Fzzy: DQAADgjqzsIQ > > Next fzzy: F7dtxwqVQ_Pw > > Fzzy: DgAAOgCcWv9g > > Next fzzy: F7dtxwqVQ_Pw > > Fzzy: DwAAKg7iQ9JA > > Next fzzy: F7dtxwqVQ_Pw > > Fzzy: EAAAugqVQrTw > > Next fzzy: F7dtxwqVQ_Pw > > Fzzy: EQAAJAqVQrTw > > Next fzzy: F7dtxwqVQ_Pw > > Fzzy: EgAABgIOMBgg > > Next fzzy: F7dtxwqVQ_Pw > > Fzzy: EwAAEwqVQrTw > > Next fzzy: F7dtxwqVQ_Pw > > Fzzy: FAAACQqVQrTw > > Next fzzy: F7dtxwqVQ_Pw > > Fzzy: FQAAIAqVQrTw > > Next fzzy: F7dtxwqVQ_Pw > > Fzzy: FgAAeAWPSIDw > > Next fzzy: F7dtxwqVQ_Pw > > Fzzy: FwAAAw33Zb9Q > > Next fzzy: F7dtxwqVQ_Pw > > Fzzy: F7dt8QWPSIDw > > > > It's obvious that my FuzzyRowFilter knows what to search and every time > it > > repeats its question. > > The very first key - I suppose is just the first key of a region where my > > key is located. > > The very last key - is the key that is already bigger than what I'm > trying > > to find - that's the reason why FuzzyFilter stopped there. > > > > Do you know any issue with SEEK_NEXT_USING_HINT? I've searched, but > > unsuccessfully. > > Do you have any idea how to explain these many trials? > > > > Thanks in advance. > > -- > > Evgeny Morozov > > Developer Grid Dynamics > > Skype: morozov.evgeny > > www.griddynamics.com > > emoro...@griddynamics.com > > > -- Evgeny Morozov Developer Grid Dynamics Skype: morozov.evgeny www.griddynamics.com emoro...@griddynamics.com
Custom Filter and SEEK_NEXT_USING_HINT issue
Hi, folks! HBase, Hadoop, etc version is CDH-4.1.2 I'm using custom FuzzyRowFilter, which I get from http://blog.sematext.com/2012/08/09/consider-using-fuzzyrowfilter-when-in-need-for-secondary-indexes-in-hbase/and suddenly after quite a time we found that it starts loosing data. Basically the idea of FuzzyRowFilter is that it tries to find key that has been provided and if there is no such a key - but more exists in table - it returns SEEK_NEXT_USING_HINT. And in getNextKeyHint(...) it builds required key. As I understand, HBase in this key will fast-forward to required key - it must be similar or same as to get Scan with setStartRow. I'm trying to find key F7dt8QWPSIDw, it is definitely in HBase - I'm able to get it using Scan.setStartRow. For FuzzyFilter I'm using empty Scan - I didn't specify start row, stop row or anything related. That's what happening: Fzzy: 1Q7iQ9JA Next fzzy: F7dtxwqVQ_Pw Fzzy: AQAAnA96rxTg Next fzzy: F7dtxwqVQ_Pw Fzzy: AgAADQWPSIDw Next fzzy: F7dtxwqVQ_Pw Fzzy: AwAA-Q33Zb9Q Next fzzy: F7dtxwqVQ_Pw Fzzy: BAAAOg8oyu7A Next fzzy: F7dtxwqVQ_Pw Fzzy: BQAA9gqVQrTw Next fzzy: F7dtxwqVQ_Pw Fzzy: BgABZQ7iQ9JA Next fzzy: F7dtxwqVQ_Pw Fzzy: BwAAbgrpAojg Next fzzy: F7dtxwqVQ_Pw Fzzy: CAAAUQWPSIDw Next fzzy: F7dtxwqVQ_Pw Fzzy: CQABVgqVQrTw Next fzzy: F7dtxwqVQ_Pw Fzzy: CgAAOQ7iQ9JA Next fzzy: F7dtxwqVQ_Pw Fzzy: CwAALwqVQrTw Next fzzy: F7dtxwqVQ_Pw Fzzy: DAAAMwWPSIDw Next fzzy: F7dtxwqVQ_Pw Fzzy: DQAADgjqzsIQ Next fzzy: F7dtxwqVQ_Pw Fzzy: DgAAOgCcWv9g Next fzzy: F7dtxwqVQ_Pw Fzzy: DwAAKg7iQ9JA Next fzzy: F7dtxwqVQ_Pw Fzzy: EAAAugqVQrTw Next fzzy: F7dtxwqVQ_Pw Fzzy: EQAAJAqVQrTw Next fzzy: F7dtxwqVQ_Pw Fzzy: EgAABgIOMBgg Next fzzy: F7dtxwqVQ_Pw Fzzy: EwAAEwqVQrTw Next fzzy: F7dtxwqVQ_Pw Fzzy: FAAACQqVQrTw Next fzzy: F7dtxwqVQ_Pw Fzzy: FQAAIAqVQrTw Next fzzy: F7dtxwqVQ_Pw Fzzy: FgAAeAWPSIDw Next fzzy: F7dtxwqVQ_Pw Fzzy: FwAAAw33Zb9Q Next fzzy: F7dtxwqVQ_Pw Fzzy: F7dt8QWPSIDw It's obvious that my FuzzyRowFilter knows what to search and every time it repeats its question. The very first key - I suppose is just the first key of a region where my key is located. The very last key - is the key that is already bigger than what I'm trying to find - that's the reason why FuzzyFilter stopped there. Do you know any issue with SEEK_NEXT_USING_HINT? I've searched, but unsuccessfully. Do you have any idea how to explain these many trials? Thanks in advance. -- Evgeny Morozov Developer Grid Dynamics Skype: morozov.evgeny www.griddynamics.com emoro...@griddynamics.com
Re: Many scanner opening
Lars, We tried, but I didn't know there is such a contention issue. We have two different column families. First one contains data, that are partially used as a filter. And actual data lives in second column family. So, outer scanner (the first one) goes through the table and filter out keys that contain required data. Then, these keys are moved to the inner (second) scanner. BTW, second scanner utilizes FuzzyRowFilter: http://blog.sematext.com/2012/08/09/consider-using-fuzzyrowfilter-when-in-need-for-secondary-indexes-in-hbase/ We have pretty small cluster - only 18 mappers, but looks like it's enough to get contention =) On Thu, Dec 20, 2012 at 10:51 PM, lars hofhansl wrote: > Cool. > > You probably made it less likely that your scanners will scan the same > HFile in parallel. > > -- Lars > > > > ________ > From: Eugeny Morozov > To: user@hbase.apache.org; lars hofhansl > Sent: Thursday, December 20, 2012 2:32 AM > Subject: Re: Many scanner opening > > Lars, > > Cool stuff! Thanks a lot! I'm not sure I can apply the patch, cause we're > using CDH-4.1.1, but increasing size of internal scanner does the trick - > decreased number of scanners. > At least temporarily it's good enough. > > Thanks! > > On Wed, Dec 19, 2012 at 6:23 AM, lars hofhansl > wrote: > > > You might have run into HBASE-7336. > > (Not available in any official release, yet) > > > > If you're using 0.94 (and probably 0.92) you can just apply this patch > > (it's save and simple). > > > > > > > > > > From: Eugeny Morozov > > To: user@hbase.apache.org > > Sent: Tuesday, December 18, 2012 12:01 AM > > Subject: Many scanner opening > > > > Hello! > > > > We faced an issue recently that the more map tasks are completed, the > > longer it takes to complete one more map task. > > > > In our architecture we have two scanners to read the table. The first > one, > > which is called 'outer' scanner is reading table and filter some rowkeys. > > These rowkeys are used as a filter for second scanner - 'internal'. Thus > we > > constantly open 'internal' scanner with different filters. > > > > As an additional symptoms we see that our cluster practically does > nothing > > - there is no CPU loading, no disk loading, no network, etc. Most of the > > time it means we are waiting on some locks, but I'm not sure. > > > > I would appreciate any ideas or suggestions to understand the case. > > Thank you in advance. > > -- > > Evgeny Morozov > > Developer Grid Dynamics > > Skype: morozov.evgeny > > www.griddynamics.com > > emoro...@griddynamics.com > > > > > > -- > Evgeny Morozov > Developer Grid Dynamics > Skype: morozov.evgeny > www.griddynamics.com > emoro...@griddynamics.com > -- Evgeny Morozov Developer Grid Dynamics Skype: morozov.evgeny www.griddynamics.com emoro...@griddynamics.com
Re: Many scanner opening
Lars, Cool stuff! Thanks a lot! I'm not sure I can apply the patch, cause we're using CDH-4.1.1, but increasing size of internal scanner does the trick - decreased number of scanners. At least temporarily it's good enough. Thanks! On Wed, Dec 19, 2012 at 6:23 AM, lars hofhansl wrote: > You might have run into HBASE-7336. > (Not available in any official release, yet) > > If you're using 0.94 (and probably 0.92) you can just apply this patch > (it's save and simple). > > > > > From: Eugeny Morozov > To: user@hbase.apache.org > Sent: Tuesday, December 18, 2012 12:01 AM > Subject: Many scanner opening > > Hello! > > We faced an issue recently that the more map tasks are completed, the > longer it takes to complete one more map task. > > In our architecture we have two scanners to read the table. The first one, > which is called 'outer' scanner is reading table and filter some rowkeys. > These rowkeys are used as a filter for second scanner - 'internal'. Thus we > constantly open 'internal' scanner with different filters. > > As an additional symptoms we see that our cluster practically does nothing > - there is no CPU loading, no disk loading, no network, etc. Most of the > time it means we are waiting on some locks, but I'm not sure. > > I would appreciate any ideas or suggestions to understand the case. > Thank you in advance. > -- > Evgeny Morozov > Developer Grid Dynamics > Skype: morozov.evgeny > www.griddynamics.com > emoro...@griddynamics.com > -- Evgeny Morozov Developer Grid Dynamics Skype: morozov.evgeny www.griddynamics.com emoro...@griddynamics.com
Many scanner opening
Hello! We faced an issue recently that the more map tasks are completed, the longer it takes to complete one more map task. In our architecture we have two scanners to read the table. The first one, which is called 'outer' scanner is reading table and filter some rowkeys. These rowkeys are used as a filter for second scanner - 'internal'. Thus we constantly open 'internal' scanner with different filters. As an additional symptoms we see that our cluster practically does nothing - there is no CPU loading, no disk loading, no network, etc. Most of the time it means we are waiting on some locks, but I'm not sure. I would appreciate any ideas or suggestions to understand the case. Thank you in advance. -- Evgeny Morozov Developer Grid Dynamics Skype: morozov.evgeny www.griddynamics.com emoro...@griddynamics.com
Re: Debugging Coprocessor code in Eclipse
Anil, you could've also get some benefit from using HBaseTestingUtility. It is able to run HBase cluster in standalone mode all-in-one JVM. Of course it requires to have some code to create tables, assign coprocessor to table and populate it with data. And then run client code against it. All of that could be done using Unit tests framework, which will give you ability to test the whole server. It also possible that HBase will be crashed by itself due to timeouts, but having this kind of preparation would allow you to restart it in a matter of seconds to continue debuggind. Hope this helps. On Tue, Oct 16, 2012 at 8:46 PM, anil gupta wrote: > Hi Ram, > > Thanks for your reply. I'll be trying your suggestions soon with the my > local standalone installation of HBase and update this thread. > > Thanks, > Anil Gupta > > On Mon, Oct 15, 2012 at 12:03 AM, Ramkrishna.S.Vasudevan < > ramkrishna.vasude...@huawei.com> wrote: > > > Hi Anil > > > > We also do a lot of stuff with coprocessors MasterObservers, > > RegionObservers > > and WALObservers. > > Just start your master and RS in debug mode and connect remotely from > > eclipse. This should be fine. Whenever the code goes to the > > RegionObserver > > or any observers automatically you will be able to debug. Only thing is > > ensure that the JAR that is running remotely and your current source code > > are in sync. Otherwise you may not go thro the exact lines. > > > > You can add debug mode to your JAVA_OPTS like this > > exec "$JAVA" $JAVA_HEAP_MAX $HBASE_OPTS > > -Xrunjdwp:transport=dt_socket,address=4530,server=y,suspend=n -classpath > > "$CLASSPATH" $CLASS "$@ > > > > Hope this helps. > > > > Regards > > Ram > > > > > > > -Original Message- > > > From: anil gupta [mailto:anilgupt...@gmail.com] > > > Sent: Monday, October 15, 2012 5:29 AM > > > To: user@hbase.apache.org > > > Subject: Debugging Coprocessor code in Eclipse > > > > > > Hi All, > > > > > > I am using HBase0.92.1 and currently working on HBase RegionObserver > > > and > > > some other coprocessor stuff. For testing this stuff, i have to deploy > > > my > > > jar on the cluster and put in a lot of logging to understand the > > > problems, > > > if any. > > > Is there a way to debug my region observer within eclipse. Please share > > > your experiences/suggestions on debugging co-processors. > > > > > > I am able to debug filters and other client side code from eclipse by > > > remotely connecting to cluster. But, i am unable to find a way to debug > > > the > > > code which runs within RegionServers. > > > > > > Here is my random thought: > > > I think i would be able to debug the co-processor code if i run the > > > Stand-alone HBase instance from a eclipse java project which contains > > > the > > > java source code of HBase0.92.1 and my Region Observer. > > > > > > If the above seems like a feasible idea, then please let me know how to > > > run > > > the HBase Standalone instance from a eclipse java project. Run > > > Configurations?Jars? > > > > > > -- > > > Thanks & Regards, > > > Anil Gupta > > > > > > > -- > Thanks & Regards, > Anil Gupta > -- Evgeny Morozov Developer Grid Dynamics Skype: morozov.evgeny www.griddynamics.com emoro...@griddynamics.com
Re: Does TotalOrderPartitioner refresh its partitions selection tree
Chris, In this case nothing scared actually happens. * If partitions are the same, then HBase simply copies all your HFiles during bulkloading procedure. * If partitions are changed, then it still copies them, but in addition, some of these files (according to number of split regions) would be also split and the whole procedure would be slightly longer. To understand the idea, imagine you have only one HFile, but some regions in HBase. On Mon, Oct 8, 2012 at 9:33 PM, chris Behin wrote: > Hil all, > > I am reading TotalOrderPartitioner code to understand its structure, > > As I understood ( correct me if I am wrong ) TotalOrderPartitioner.setConf > method reads splitPoints from partFile via readPartitions method > ( TotalOrderPartitioner.java line 82 ) > and creates a buildTrie or BinarySearchNode according to this partition > information ( TotalOrderPartitioner.java line 97 and 108 ) > > I wonder what happens if regions are changed during the execution of > mapreduce job. Is it a problem? > Because TotalOrderPartitioner does not refresh its partitioner during its > execution ( as I understand ). > > You may say that this is an exception case and the possibility is low, > I just ask for understanding the mechanism.. > > best regards... > -- Evgeny Morozov Developer Grid Dynamics Skype: morozov.evgeny www.griddynamics.com emoro...@griddynamics.com
Re: Questions on Table design for time series data
I'd suggest to think about manual major compactions and splits. Using manual compactions and bulkload allows to split HFiles manually. Like if you would like to read last 3 months more often that all others data, then you could have three HFiles for each month and one HFile for whole other stuff. Using scan.setTimestamps would allow to filter out only those three HFiles, thus scan would be faster. Moreover if you have TTL about one month there is no need to run it everyday (as in auto mode). Especially, when using bulkloads you basically control the size of output HFiles by having input of particular size. Say, you give input for last two weeks and have one HFile per regions for last two weeks. Using this new feature known as Coprocessor, you could hook up to the compactSelection process and alter the compaction chosing HFiles you would like to process. That allow to combine particular HFiles. All of that allow to run major compaction just once-twice in month - major compaction over huge amount of data is a heavy operation - the rare, the better. Though without monitoring and measurement it looks like early optimization. On Wed, Oct 3, 2012 at 10:59 PM, Jacques wrote: > We're all volunteers here so we don't always have the time to fully > understand and plan others' schemas. > > In general your questions seemed to be worried about a lot of things that > may or may not matter depending on the specifics of your implementation. > Without knowing those specifics it is hard to be super definitive. You > seem to be very worried about the cost of compactions and retention. Is > that because you're having issues now? > > Short answers: > > q1: Unless you have a good reason for splitting up into two tables, I'd > keep as one. Pros: Easier to understand/better matches intellectual > understanding/allows checkAndPuts across both families/data is colocated > (server, not disk) on retrieval if you want to work with both groups > simultaneously using get, MR, etc. Con: There will be some extra > merge/flush activity if the two columns grow at substantially different > rates. > > q2: 365*1000 regions is problematic (if that is what you're suggesting). > Even with HFilev2 and partially loaded multi-level indexes, there is still > quite a bit of overhead per region. I pointed you at the Jive thing in > part since hashing that value as a bucket seems a lot more reasonable. > Additional Random idea: if you know retention policy on insert and your > data is immutable post insertion, consider shifting the insert timestamp > and maintain a single ttl. Would require more client side code but would > allow configurable ttls while utilizing existing HBase infrastructure. > > q3: Sounds like you're prematurely optimizing here. Maybe others would > disagree. I'd use ttl until you find that isn't performant enough. The > tension between flexibility and speed is clear here. I'd say you either > need to pick specific ttls and optimize for that scenario via region > pruning (e.g. separate tables for each ttl type) or you need to use a more > general approach that leverages the per value ttl and compaction > methodology. There is enough operational work managing an HBase/HDFS > cluster without having to worry about specialized region management. > > Jacques > > On Wed, Oct 3, 2012 at 11:31 AM, Karthikeyan Muthukumarasamy < > mkarthik.h...@gmail.com> wrote: > > > Hi Jacques, > > Thanks for the response! > > Yes, I have seen the video before. It suggets usage of TTL based > retention > > implementation. In their usecase, Jive has a fixed retention say 3 months > > and so they can pre-create regions for so many buckets, their bucket id > is > > DAY_OF_YEAR%retention_in_days. But, in our usecase, the retention period > is > > configurable, so pre-creationg regions based on retention will not work. > > Thats why we went for MMDD based buckets which is immune to retention > > period changes. > > Now that you know that Ive gone through that video from Jive, I would > > request you to re-read my specific questions and share your suggestions. > > Thanks & Regards > > MK > > > > > > > > On Wed, Oct 3, 2012 at 11:51 PM, Jacques wrote: > > > > > I would suggest you watch this video: > > > > > > > > > http://www.cloudera.com/resource/video-hbasecon-2012-real-performance-gains-with-real-time-data/ > > > > > > The jive guys solved a lot of the problems you're talking about and > > discuss > > > it in that case study. > > > > > > > > > > > > On Wed, Oct 3, 2012 at 6:27 AM, Karthikeyan Muthukumarasamy < > > > mkarthik.h...@gmail.com> wrote: > > > > > > > Hi, > > > > Our usecase is as follows: > > > > We have time series data continuously flowing into the system and has > > to > > > be > > > > stored in HBase. > > > > Subscriber Mobile Number (a.k.a MSISDN) is the primary identifier > based > > > on > > > > which data is stored and later retrieved. > > > > There are two sets of parameters that get stored in every record in > > > HBa
Re: Bulk Loads and Updates
Hi! Sure, you do, but don't forget to sort all KV pairs before put them into context. Or else you'd get some "unsorted" expection. If you have them completely the same and you need to reduce number of same lines you could use Combiner, but their behavior is not deterministic, so basically there is no guarantee that it'll be run and how many times. On Thu, Oct 4, 2012 at 12:22 AM, gordoslocos wrote: > Thank you Paul. > > I was just thinking that I could use add a reducer to the step that > prepares the data to build custom logic around having multiple entries > which produce the same rowkey. What do u think? > > Sent from my iPhone > > On 03/10/2012, at 17:12, Paul Mackles wrote: > > > Keys in hbase are a combination of rowkey/column/timestamp. > > > > Two records with the same rowkey but different column will result in two > > different cells with the same rowkey which is probably what you expect. > > > > For two records with the same rowkey and same column, the timestamp will > > normally differentiate them but in the case of a bulk load, the timestamp > > could be the same so it may actually be a tie and both will be stored. > > There are no updates in bulk loads. > > > > All 20 versions will get loaded but the 10 oldest will be deleted during > > the next major compaction. > > > > I would definitely recommend setting up small scale tests for all of the > > above scenarios to confirm. > > > > On 10/3/12 3:35 PM, "Juan P." wrote: > > > >> Hi guys, > >> I've been reading up on bulk load using MapReduce jobs and I wanted to > >> validate something. > >> > >> If I the input I wanted to load into HBase produced the same key for > >> several lines. How will HBase handle that? > >> > >> I understand the MapReduce job will create StoreFiles which the region > >> servers just pick up and make available to the users. But is there a > >> validation to treat the first as insert and the rest as updates? > >> > >> What about the limit on the number of versions of a key HBase can have? > If > >> I want to have 10 versions, but the bulk load has 20 values for the same > >> key, will it only keep the last 10? > >> > >> Thanks, > >> Juan > > > -- Evgeny Morozov Developer Grid Dynamics Skype: morozov.evgeny www.griddynamics.com emoro...@griddynamics.com
Re: Distribution of regions to servers
Dan, see inlined. On Thu, Sep 27, 2012 at 5:30 AM, Dan Han wrote: > Hi, Eugeny , > >Thanks for your response. I answered your questions inline in Blue. > And I'd like to give an example to describe my problem. > > Let's think about two data schemas for the same dataset. > The two data schemas have different composite row keys. Just the first idea. If you have different schemas, then it would be much simpler to have two different tables with these schemas. Because in this case HBase itself automatically distribute each of the tables' regions evenly across the cluster. You could actually use the same coprocessor for both of the tables. In case you're using two different column families, you could specify different BLOCKSIZE (default value is '65536''). You could set this option different in 10 times for CFs (as the difference in between your schemas). I believe this would decrease number of readings for larger data chunks. In general it is actually not good to have two (or more) really different in size column families, because they have compaction and flushing based on region, which means that if HBase start compacting small column family it will do the same for big one. http://hbase.apache.org/book.html#number.of.cfs BTW, I don't think that coprocessors are good choice to have data mining. The reason is that it is kind of dangerous. Since coprocessor are server side creatures - they live in Region Server - they simply could get the whole system down. Expensive analysis creates heap and CPU pressure, which in turn lead to GC pauses and even more CPU pressure. Consider to use PIG and HBaseStorage to load data from HBase. But there is > a same part in both schemas, which represents a sequence ID. > In 1st schema, one row contains 1KB information; > while in 2nd schema, one row contains 10KB information. > So the number of rows in one region in 1st schema is more than > that in 2nd schema, right? If the queried data is based on the sequence ID, > as one region in 1st schema is responsible for more number of rows than > that in 2nd schema, > there would be more computation and long execution time for the > corresponding coprocessor. > So in this case, if the regions are not distributed well, > some region servers will suffer in excess workload. > That is why I want to do some management of regions to get better load > balance based on large queries. > > Hope it makes sense to you. > > Best Wishes > Dan Han > > > On Wed, Sep 26, 2012 at 3:19 PM, Eugeny Morozov > wrote: > > > Dan, > > > > I have additional questions. > > What is the access pattern of your queries? I mean that f.e. > PrefixFilters > > have to be applied for all KeyValue pairs in HFiles, which could be slow. > > Or f.e. scanner setCaching option is able to decrease number of network > > hops to get data from RegionServer. > > > > I set the range of the rows and the related columns to narrow down the > scan scope, > and I used PrefixFilter/ColumnFilter/BinaryFilter to get the rows. > I set a little cache (5KB), but I kept it the same for all evaluated > data schema. > Because I mainly focus on evaluate the performance of queries under the > different data schemas. > > > > Additionally, coprocessors are able to use InternalScanner instead of > > ResultScanner, which is also could help greatly. > > > > yes, I used InternalScanner. > > > > > Also, the more dimension you specify, the more precise your query is, the > > less data is about to be processed - family, columns, timeranges, etc. > > > > > > On Wed, Sep 26, 2012 at 7:39 PM, Dan Han wrote: > > > > > Thanks for your swift response, Ramkrishna and Anoop. And I will > > > explicate what we are doing now below. > > > > > >We are trying to explore a systematic way to design the appropriate > > data > > > schema for various applications in HBase. So we first designed several > > data > > > schemas for each dataset and evaluate them with the same queries. The > > > queries are designed based on the requirements, such as selecting the > > data > > > with a matching expression, finding the difference between two > > > snapshots. The queries were processed with user-level Coprocessor. > > > > > >In our experiments, we found that under some data schemas, the > queries > > > cannot get any results because of the connection timeout and RS crash > > > sometimes. We observed that in this case, the queried data were > centered > > in > > > a few regions locating in a few region servers. We think the failure > > might > > &
Re: Distribution of regions to servers
Dan, I have additional questions. What is the access pattern of your queries? I mean that f.e. PrefixFilters have to be applied for all KeyValue pairs in HFiles, which could be slow. Or f.e. scanner setCaching option is able to decrease number of network hops to get data from RegionServer. Additionally, coprocessors are able to use InternalScanner instead of ResultScanner, which is also could help greatly. Also, the more dimension you specify, the more precise your query is, the less data is about to be processed - family, columns, timeranges, etc. On Wed, Sep 26, 2012 at 7:39 PM, Dan Han wrote: > Thanks for your swift response, Ramkrishna and Anoop. And I will > explicate what we are doing now below. > >We are trying to explore a systematic way to design the appropriate data > schema for various applications in HBase. So we first designed several data > schemas for each dataset and evaluate them with the same queries. The > queries are designed based on the requirements, such as selecting the data > with a matching expression, finding the difference between two > snapshots. The queries were processed with user-level Coprocessor. > >In our experiments, we found that under some data schemas, the queries > cannot get any results because of the connection timeout and RS crash > sometimes. We observed that in this case, the queried data were centered in > a few regions locating in a few region servers. We think the failure might > be caused by the excess workload in these few region servers and the > inappropriate load balance. To our best knowledge, this case can be avoided > and improved by the well-distributed regions across the region servers. > > Therefore, we have been thinking to add a monitoring and management > component between the client and server, which can schedule the > queries/jobs from client side and distribute the regions dynamically > according to the current workload of each region server, the incoming > queries and data locality. > > Does it make sense? Just my two cents. Any comments? > > Best Wishes > Dan Han > > On Tue, Sep 25, 2012 at 10:44 PM, Anoop Sam John > wrote: > > > Hi > > Can u share more details pls? What work you are doing within the CPs > > > > -Anoop- > > > > From: Dan Han [dannahan2...@gmail.com] > > Sent: Wednesday, September 26, 2012 5:55 AM > > To: user@hbase.apache.org > > Subject: Distribution of regions to servers > > > > Hi all, > > > >I am doing some experiments on HBase with Coprocessor. I found that > the > > performance > > of Coprocessor is impacted much by the distribution of the regions. I am > > kind of interested in > > going deep into this problem and see if I can do something. > > > > I only searched out the discussion in the following link. > > > > > http://search-hadoop.com/m/Vjhgj1lqw7Y1/hbase+distribution+region&subj=distribution+of+regions+to+servers > > > > I am wondering if there is any further discussion or any on-going work? > Can > > someone point it to me if there is? > > Thanks in advance. > > > > Best Wishes > > Dan Han > > > -- Evgeny Morozov Developer Grid Dynamics Skype: morozov.evgeny www.griddynamics.com emoro...@griddynamics.com
Re: Simple way to unit test HBase Map reduce jobs?
Hi, Elazar, I've found that MRUnit is pretty convenient way to test MR jobs. On the other hand there is HBaseTestingUtility, which is helpful to run miniCluster. Hope this helps. On Mon, Sep 24, 2012 at 8:43 PM, Elazar Leibovich wrote: > Is there a way similar to miniserver to test map reduce jobs on something > similar to miniCluster? > -- Evgeny Morozov Developer Grid Dynamics Skype: morozov.evgeny www.griddynamics.com emoro...@griddynamics.com
Re: scan.setTimeRange performance
Hi, Jean-Daniel, thanks for the reply.
I've found the reason. And it's quite simple to understand. Don't know why
I've missed it.
The reason for slow processing was the fact that specified time range was
too thin.
So, firstly Region Server filter out HFiles, which it will scan.
Then, it reads them (or just one HFile as in my case) and trying to find
first 10 to 50 values that fall into given time range. If time range is
thin, then Region Server must read the HFile almost completely. At the
contrast, whey there is no time range, then it just return first 10 to 50
values.
That's the difference.
That's actually the reason I'm sure that time range is working correctly =)
On Mon, Sep 24, 2012 at 11:15 PM, Jean-Daniel Cryans wrote:
> Hi Eugeny,
>
> The mailing list stripped your attachement (as it often does) so you
> might want to put it on a public web server.
>
> I don't have much to contribute except than to point to a recent
> conversation that you can find here:
> http://comments.gmane.org/gmane.comp.java.hadoop.hbase.user/28722
>
> Hope this helps,
>
> J-D
>
> On Fri, Sep 21, 2012 at 5:03 AM, Eugeny Morozov
> wrote:
> > Hello!
> >
> > It is known and I saw it in the code that time range set by
> > scan.setTimeRange is used to filter out HFiles for further scan.
> > Which means that speed of following scanner.next must be almost zero in
> case
> > if I set time range far away in future. I am sure that I do not have
> HFiles
> > that fall into the set time range period.
> >
> > But - and here is the question - surprisingly scanning with set time
> range
> > is far longer than without it.
> >
> > My results are following:
> > Use range [false]. Time spent (avg): [0]
> > Use range [true]. Time spent (avg): [525]
> >
> > There are KeyValues listed, when time range is not used.
> >
> > The code is following:
> > public static void run(boolean useRange, HTable table) throws
> Exception
> > {
> > Scan scan = new Scan().addFamily( family ).setCaching( -1
> > ).setCacheBlocks( false );
> > scan.setStartRow( random start row );
> > if (useRange) scan.setTimeRange(1348114401600L, 1348114401700L);
> >
> > ResultScanner scanner = table.getScanner(scan);
> > for(int i = 0 ; i < N; i++) { // There were bunch of measures,
> where
> > N was from 10 to 50
> > long time = System.currentTimeMillis();
> > result = scanner.next();
> > sum += (System.currentTimeMillis() - time) / N;
> > }
> > }
> >
> > Of course such a measurements are include all sort of noise like network
> > overhead, etc, but I'm using virtual machine on my own box, and at the
> time
> > I do measurement there is no other activity neither on my own box or this
> > virtual machine, so such a noise is minimum.
> >
> > Also I've used YourKit to measure tracing and sampling of running
> > HRegionServer, but didn't found anything suspicious. Though I didn't
> look at
> > heap and GC perf. Tracing is in attach.
> >
> > So, the question is why is it so slow when time range is set and so fast
> > without it?
> > --
> > Evgeny Morozov
> > Developer Grid Dynamics
> > Skype: morozov.evgeny
> > www.griddynamics.com
> > emoro...@griddynamics.com
>
--
Evgeny Morozov
Developer Grid Dynamics
Skype: morozov.evgeny
www.griddynamics.com
emoro...@griddynamics.com
scan.setTimeRange performance
Hello!
It is known and I saw it in the code that time range set by
scan.setTimeRange is used to filter out HFiles for further scan.
Which means that speed of following scanner.next must be almost zero in
case if I set time range far away in future. I am sure that I do not have
HFiles that fall into the set time range period.
But - and here is the question - surprisingly scanning with set time range
is far longer than without it.
My results are following:
Use range [false]. Time spent (avg): [0]
Use range [true]. Time spent (avg): [525]
There are KeyValues listed, when time range is not used.
The code is following:
public static void run(boolean useRange, HTable table) throws Exception
{
Scan scan = new Scan().addFamily( family ).setCaching( -1
).setCacheBlocks( false );
scan.setStartRow( random start row );
if (useRange) scan.setTimeRange(1348114401600L, 1348114401700L);
ResultScanner scanner = table.getScanner(scan);
for(int i = 0 ; i < N; i++) { // There were bunch of measures,
where N was from 10 to 50
long time = System.currentTimeMillis();
result = scanner.next();
sum += (System.currentTimeMillis() - time) / N;
}
}
Of course such a measurements are include all sort of noise like network
overhead, etc, but I'm using virtual machine on my own box, and at the time
I do measurement there is no other activity neither on my own box or this
virtual machine, so such a noise is minimum.
Also I've used YourKit to measure tracing and sampling of running
HRegionServer, but didn't found anything suspicious. Though I didn't look
at heap and GC perf. Tracing is in attach.
So, the question is why is it so slow when time range is set and so fast
without it?
--
Evgeny Morozov
Developer Grid Dynamics
Skype: morozov.evgeny
www.griddynamics.com
emoro...@griddynamics.com
