[jira] [Commented] (HBASE-15381) Implement a distributed MOB compaction by procedure
[ https://issues.apache.org/jira/browse/HBASE-15381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357845#comment-16357845 ] Jingcheng Du commented on HBASE-15381: -- Please go ahead [~huaxiang], thanks a lot. > Implement a distributed MOB compaction by procedure > --- > > Key: HBASE-15381 > URL: https://issues.apache.org/jira/browse/HBASE-15381 > Project: HBase > Issue Type: Improvement > Components: mob >Reporter: Jingcheng Du >Assignee: Jingcheng Du >Priority: Major > Attachments: HBASE-15381-v2.patch, HBASE-15381-v3.patch, > HBASE-15381-v4.patch, HBASE-15381-v5.patch, HBASE-15381-v6.patch, > HBASE-15381.patch, mob distributed compaction design-v2.pdf, mob distributed > compaction design.pdf > > > In MOB, there is a periodical compaction which runs in HMaster (It can be > disabled by configuration), some small mob files are merged into bigger ones. > Now the compaction only runs in HMaster which is not efficient and might > impact the running of HMaster. In this JIRA, a distributed MOB compaction is > introduced, it is triggered by HMaster, but all the compaction jobs are > distributed to HRegionServers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19750) print pretty rowkey in getExceptionMessageAdditionalDetail
[
https://issues.apache.org/jira/browse/HBASE-19750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16338948#comment-16338948
]
Jingcheng Du commented on HBASE-19750:
--
Thanks [~suxingfate] for the patch. +1 on the 005 patch.
Hi [~carp84], what's your idea on the latest patch? Thanks.
> print pretty rowkey in getExceptionMessageAdditionalDetail
> --
>
> Key: HBASE-19750
> URL: https://issues.apache.org/jira/browse/HBASE-19750
> Project: HBase
> Issue Type: Improvement
> Components: Client
>Reporter: Wang, Xinglong
>Assignee: Wang, Xinglong
>Priority: Minor
> Attachments: HBASE-19750.001.patch, HBASE-19750.002.patch,
> HBASE-19750.003.patch, HBASE-19750.004.patch, HBASE-19750.005.patch
>
>
> Sometimes the rowkey is binary format and is not able to print out human
> readable string. In this case, the exception will still try to call the
> toString() method and result in something like '�\(�\'.
> It will be very inefficient to trouble shooting the issue when we get such
> kind of exception. We can't identify the problematic row key based on the
> printout.
> The idea here is that print out the rowkey use Bytes.toStringBinary() in
> additional with Bytes.toString(row).
> If the row is serialized from human readable string, then Bytes.toString(row)
> makes more sense. When it's from human unreadable string, then
> Bytes.toStringBinary(row) will help.
> The output of Bytes.toStringBinary(row) anyway can be applied to hbase shell
> to do the scan so that we can easily identify the corresponding row.
> {code:java}
> 2017-12-16 07:25:41,304 INFO [main]
> org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl: recovered from
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
> attempts=36, exceptions:
> Sat Dec 16 07:25:41 GMT-07:00 2017, null, java.net.SocketTimeoutException:
> callTimeout=25, callDuration=250473: row '�\(�\' on table 'mytable' at
> region=mytable,\xDF\x5C(\xF5\xC2\x8F\x5C\x1B,1412216342143.5d74ce411eecd40001d9bf6e62f0b607.,
> hostname=mycluster.internal.xx.com,60020,1503881012672, seqNum=6265890293
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:271)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:203)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
> at
> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
> at
> org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:403)
> at
> org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:364)
> at
> org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.nextKeyValue(TableRecordReaderImpl.java:205)
> at
> org.apache.hadoop.hbase.mapreduce.TableRecordReader.nextKeyValue(TableRecordReader.java:147)
> at
> org.apache.hadoop.hbase.mapreduce.TableInputFormatBase$1.nextKeyValue(TableInputFormatBase.java:216)
> at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:556)
> at
> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> {code}
> Current code:
> RegionServerCallable.java
> {code:java}
> public String getExceptionMessageAdditionalDetail() {
> return "row '" + Bytes.toString(row) + "' on table '" + tableName + "' at
> " + location;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-18693) adding an option to restore_snapshot to move mob files from archive dir to working dir
[ https://issues.apache.org/jira/browse/HBASE-18693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312387#comment-16312387 ] Jingcheng Du commented on HBASE-18693: -- Thanks [~huaxiang]! I am +1 to V3. > adding an option to restore_snapshot to move mob files from archive dir to > working dir > -- > > Key: HBASE-18693 > URL: https://issues.apache.org/jira/browse/HBASE-18693 > Project: HBase > Issue Type: Improvement > Components: mob >Affects Versions: 2.0.0-alpha-2 >Reporter: huaxiang sun >Assignee: huaxiang sun > Attachments: HBASE-18693.master.001.patch, > HBASE-18693.master.002.patch, HBASE-18693.master.003.patch > > > Today, there is a single mob region where mob files for all user regions are > saved. There could be many files (one million) in a single mob directory. > When one mob table is restored or cloned from snapshot, links are created for > these mob files. This creates a scaling issue for mob compaction. In mob > compaction's select() logic, for each hFileLink, it needs to call NN's > getFileStatus() to get the size of the linked hfile. Assume that one such > call takes 20ms, 20ms * 100 = 6 hours. > To avoid this overhead, we want to add an option so that restore_snapshot can > move mob files from archive dir to working dir. clone_snapshot is more > complicated as it can clone a snapshot to a different table so moving that > can destroy the snapshot. No option will be added for clone_snapshot. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19650) ExpiredMobFileCleaner has wrong logic about TTL check
[ https://issues.apache.org/jira/browse/HBASE-19650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16307632#comment-16307632 ] Jingcheng Du commented on HBASE-19650: -- Hi [~javaman_chen], would you mind providing a new patch? And please don't change the old test case if it doesn't go to wrong. If necessary you can add a simple test that addresses your case. Thanks. > ExpiredMobFileCleaner has wrong logic about TTL check > - > > Key: HBASE-19650 > URL: https://issues.apache.org/jira/browse/HBASE-19650 > Project: HBase > Issue Type: Bug > Components: mob >Reporter: chenxu >Assignee: chenxu > Attachments: HBASE-19650-master-v1.patch > > > If today is 2017-12-28 00:00:01, and TTL is set to 86400, when > MobUtils.cleanExpiredMobFiles execute, expireDate will be 1514304000749, but > fileDate is 151430400. So the fileDate before the expireDate and mobfiles > generated in is 2017-12-27 will all are deleted. But in fact, we want to > delete files before 2017-12-27. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19650) ExpiredMobFileCleaner has wrong logic about TTL check
[
https://issues.apache.org/jira/browse/HBASE-19650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16305922#comment-16305922
]
Jingcheng Du commented on HBASE-19650:
--
Thanks [~javaman_chen].
I reviewed the code, it is because only second/minute/hour in Calendar are
reset to 0, but millisecond is not.
The way to fix this issue is to set the millisecond to 0 as well. Is it okay
for you?
{code}
+ calendar.set(Calendar.MILLISECOND, 0);
{code}
> ExpiredMobFileCleaner has wrong logic about TTL check
> -
>
> Key: HBASE-19650
> URL: https://issues.apache.org/jira/browse/HBASE-19650
> Project: HBase
> Issue Type: Bug
> Components: mob
>Reporter: chenxu
>Assignee: chenxu
> Attachments: HBASE-19650-master-v1.patch
>
>
> If today is 2017-12-28 00:00:01, and TTL is set to 86400, when
> MobUtils.cleanExpiredMobFiles execute, expireDate will be 1514304000749, but
> fileDate is 151430400. So the fileDate before the expireDate and mobfiles
> generated in is 2017-12-27 will all are deleted. But in fact, we want to
> delete files before 2017-12-27.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19650) ExpiredMobFileCleaner has wrong logic about TTL check
[ https://issues.apache.org/jira/browse/HBASE-19650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16305076#comment-16305076 ] Jingcheng Du commented on HBASE-19650: -- Thanks for the patch [~javaman_chen]. You encounter this issue when you run the mob file cleaner? Or do you have test to re-produce this case? As far as I know, it won't happen. The comparison is based on date, the hour/minute/second are not included. In your description, the file date and expired date are 2017-12-27, and the files in 2017-12-27 won't be archived. > ExpiredMobFileCleaner has wrong logic about TTL check > - > > Key: HBASE-19650 > URL: https://issues.apache.org/jira/browse/HBASE-19650 > Project: HBase > Issue Type: Bug > Components: mob >Reporter: chenxu >Assignee: chenxu > Attachments: HBASE-19650-master-v1.patch > > > If today is 2017-12-28 00:00:01, and TTL is set to 86400, when > MobUtils.cleanExpiredMobFiles execute, expireDate will be 1514304000749, but > fileDate is 151430400. So the fileDate before the expireDate and mobfiles > generated in is 2017-12-27 will all are deleted. But in fact, we want to > delete files before 2017-12-27. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19507) Get or Scan Mob by rowkey return error value when run compact_mob or major_compact_mob after change MOB_THRESHOLD bigger
[
https://issues.apache.org/jira/browse/HBASE-19507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16291999#comment-16291999
]
Jingcheng Du commented on HBASE-19507:
--
Thanks [~wangyuan].
MOB required HFileV3.
You might need to change hbase.mob.file.compaction.mergeable.threshold and
re-run the steps you mentioned, and change this threshold back after all data
are recovered.
Otherwise, you have to write some code to recover the data.
# Scan all the MOB files.
# Convert the read cells to a ref cell and put it back to HBase table.
> Get or Scan Mob by rowkey return error value when run compact_mob or
> major_compact_mob after change MOB_THRESHOLD bigger
>
>
> Key: HBASE-19507
> URL: https://issues.apache.org/jira/browse/HBASE-19507
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: WangYuan
>Assignee: huaxiang sun
>
> 1、
> create 'abc',{NAME => 'cf', MOB_THRESHOLD => '10', IS_MOB => 'true'}
> put 'abc','1','cf:a','1'
> put 'abc','2','cf:a','2'
> put 'abc','3','cf:a','3'
> put 'abc','4','cf:a','y1'
> put 'abc','5','cf:a','y2'
> put 'abc','6','cf:a','y3'
>
> hbase(main):011:0> scan 'abc'
> ROWCOLUMN+CELL
>
>
> 1 column=cf:a,
> timestamp=1513171753098, value=1
>
> 2 column=cf:a,
> timestamp=1513171753208, value=2
>
> 3 column=cf:a,
> timestamp=1513171753246, value=3
>
> 4 column=cf:a,
> timestamp=1513171753273, value=y1
>
> 5 column=cf:a,
> timestamp=1513171753301, value=y2
>
> 6 column=cf:a,
> timestamp=1513171754282, value=y3
>
> hbase(main):012:0> flush 'abc'
> hbase(main):012:0> major_compact 'abc'
> hbase(main):012:0> major_compact_mob 'abc'
> 2、
> [See Hfile]:
> hbase org.apache.hadoop.hbase.io.hfile.HFile -f
> /hbase/data/default/abc/a31b3146cba0d4569a7bf44e70e299c9/cf/22a432ba5c2c4802bedd947b99626f10
> -p
> K: 1/cf:a/1513172294864/Put/vlen=5/seqid=4 V: 1
> K: 2/cf:a/1513172294892/Put/vlen=5/seqid=5 V: 2
> K: 3/cf:a/1513172294914/Put/vlen=5/seqid=6 V: 3
> K: 4/cf:a/1513172294954/Put/vlen=76/seqid=7 V:
> \x00\x00\x00\x0Ed41d8cd98f00b204e9800998ecf8427e20171213ce022548c4c3498e864fda289b81e711
> T[0]: T[1]: abc
> K: 5/cf:a/1513172294982/Put/vlen=76/seqid=8 V:
> \x00\x00\x00\x0Ed41d8cd98f00b204e9800998ecf8427e20171213ce022548c4c3498e864fda289b81e711
> T[0]: T[1]: abc
> K: 6/cf:a/1513172296455/Put/vlen=76/seqid=9 V:
> \x00\x00\x00\x0Ed41d8cd98f00b204e9800998ecf8427e20171213ce022548c4c3498e864fda289b81e711
> T[0]: T[1]: abc
> Scanned kv count -> 6
> [See Mobfile]:
> hbase org.apache.hadoop.hbase.io.hfile.HFile -f
> /hbase/mobdir/data/default/abc/07aab825b62dd9111831839cc9039df9/cf/d41d8cd98f00b204e9800998ecf8427e20171213bd8cfaf146684d4096ebf7994f050e96
> -p
> K: 4/cf:a/1513172924196/Put/vlen=14/seqid=7 V: y1
> K: 5/cf:a/1513172924214/Put/vlen=14/seqid=8 V: y2
> K: 6/cf:a/1513172925768/Put/vlen=14/seqid=9 V: y3
> 3、
> alter 'abc',{NAME => 'cf', MOB_THRESHOLD => '10240' }
> put
>
[jira] [Resolved] (HBASE-19507) Get or Scan Mob by rowkey return error value when run compact_mob or major_compact_mob after change MOB_THRESHOLD bigger
[
https://issues.apache.org/jira/browse/HBASE-19507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingcheng Du resolved HBASE-19507.
--
Resolution: Not A Bug
> Get or Scan Mob by rowkey return error value when run compact_mob or
> major_compact_mob after change MOB_THRESHOLD bigger
>
>
> Key: HBASE-19507
> URL: https://issues.apache.org/jira/browse/HBASE-19507
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: WangYuan
>Assignee: huaxiang sun
>
> 1、
> create 'abc',{NAME => 'cf', MOB_THRESHOLD => '10', IS_MOB => 'true'}
> put 'abc','1','cf:a','1'
> put 'abc','2','cf:a','2'
> put 'abc','3','cf:a','3'
> put 'abc','4','cf:a','y1'
> put 'abc','5','cf:a','y2'
> put 'abc','6','cf:a','y3'
>
> hbase(main):011:0> scan 'abc'
> ROWCOLUMN+CELL
>
>
> 1 column=cf:a,
> timestamp=1513171753098, value=1
>
> 2 column=cf:a,
> timestamp=1513171753208, value=2
>
> 3 column=cf:a,
> timestamp=1513171753246, value=3
>
> 4 column=cf:a,
> timestamp=1513171753273, value=y1
>
> 5 column=cf:a,
> timestamp=1513171753301, value=y2
>
> 6 column=cf:a,
> timestamp=1513171754282, value=y3
>
> hbase(main):012:0> flush 'abc'
> hbase(main):012:0> major_compact 'abc'
> hbase(main):012:0> major_compact_mob 'abc'
> 2、
> [See Hfile]:
> hbase org.apache.hadoop.hbase.io.hfile.HFile -f
> /hbase/data/default/abc/a31b3146cba0d4569a7bf44e70e299c9/cf/22a432ba5c2c4802bedd947b99626f10
> -p
> K: 1/cf:a/1513172294864/Put/vlen=5/seqid=4 V: 1
> K: 2/cf:a/1513172294892/Put/vlen=5/seqid=5 V: 2
> K: 3/cf:a/1513172294914/Put/vlen=5/seqid=6 V: 3
> K: 4/cf:a/1513172294954/Put/vlen=76/seqid=7 V:
> \x00\x00\x00\x0Ed41d8cd98f00b204e9800998ecf8427e20171213ce022548c4c3498e864fda289b81e711
> T[0]: T[1]: abc
> K: 5/cf:a/1513172294982/Put/vlen=76/seqid=8 V:
> \x00\x00\x00\x0Ed41d8cd98f00b204e9800998ecf8427e20171213ce022548c4c3498e864fda289b81e711
> T[0]: T[1]: abc
> K: 6/cf:a/1513172296455/Put/vlen=76/seqid=9 V:
> \x00\x00\x00\x0Ed41d8cd98f00b204e9800998ecf8427e20171213ce022548c4c3498e864fda289b81e711
> T[0]: T[1]: abc
> Scanned kv count -> 6
> [See Mobfile]:
> hbase org.apache.hadoop.hbase.io.hfile.HFile -f
> /hbase/mobdir/data/default/abc/07aab825b62dd9111831839cc9039df9/cf/d41d8cd98f00b204e9800998ecf8427e20171213bd8cfaf146684d4096ebf7994f050e96
> -p
> K: 4/cf:a/1513172924196/Put/vlen=14/seqid=7 V: y1
> K: 5/cf:a/1513172924214/Put/vlen=14/seqid=8 V: y2
> K: 6/cf:a/1513172925768/Put/vlen=14/seqid=9 V: y3
> 3、
> alter 'abc',{NAME => 'cf', MOB_THRESHOLD => '10240' }
> put
>
[jira] [Commented] (HBASE-19507) Get or Scan Mob by rowkey return error value when run compact_mob or major_compact_mob after change MOB_THRESHOLD bigger
[
https://issues.apache.org/jira/browse/HBASE-19507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16291930#comment-16291930
]
Jingcheng Du commented on HBASE-19507:
--
Thanks a lot Huaxiang. You mean you cannot re-produce this issue in master
branch or CDH5.7.1? Thanks!
> Get or Scan Mob by rowkey return error value when run compact_mob or
> major_compact_mob after change MOB_THRESHOLD bigger
>
>
> Key: HBASE-19507
> URL: https://issues.apache.org/jira/browse/HBASE-19507
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: WangYuan
>Assignee: huaxiang sun
>
> 1、
> create 'abc',{NAME => 'cf', MOB_THRESHOLD => '10', IS_MOB => 'true'}
> put 'abc','1','cf:a','1'
> put 'abc','2','cf:a','2'
> put 'abc','3','cf:a','3'
> put 'abc','4','cf:a','y1'
> put 'abc','5','cf:a','y2'
> put 'abc','6','cf:a','y3'
>
> hbase(main):011:0> scan 'abc'
> ROWCOLUMN+CELL
>
>
> 1 column=cf:a,
> timestamp=1513171753098, value=1
>
> 2 column=cf:a,
> timestamp=1513171753208, value=2
>
> 3 column=cf:a,
> timestamp=1513171753246, value=3
>
> 4 column=cf:a,
> timestamp=1513171753273, value=y1
>
> 5 column=cf:a,
> timestamp=1513171753301, value=y2
>
> 6 column=cf:a,
> timestamp=1513171754282, value=y3
>
> hbase(main):012:0> flush 'abc'
> hbase(main):012:0> major_compact 'abc'
> hbase(main):012:0> major_compact_mob 'abc'
> 2、
> [See Hfile]:
> hbase org.apache.hadoop.hbase.io.hfile.HFile -f
> /hbase/data/default/abc/a31b3146cba0d4569a7bf44e70e299c9/cf/22a432ba5c2c4802bedd947b99626f10
> -p
> K: 1/cf:a/1513172294864/Put/vlen=5/seqid=4 V: 1
> K: 2/cf:a/1513172294892/Put/vlen=5/seqid=5 V: 2
> K: 3/cf:a/1513172294914/Put/vlen=5/seqid=6 V: 3
> K: 4/cf:a/1513172294954/Put/vlen=76/seqid=7 V:
> \x00\x00\x00\x0Ed41d8cd98f00b204e9800998ecf8427e20171213ce022548c4c3498e864fda289b81e711
> T[0]: T[1]: abc
> K: 5/cf:a/1513172294982/Put/vlen=76/seqid=8 V:
> \x00\x00\x00\x0Ed41d8cd98f00b204e9800998ecf8427e20171213ce022548c4c3498e864fda289b81e711
> T[0]: T[1]: abc
> K: 6/cf:a/1513172296455/Put/vlen=76/seqid=9 V:
> \x00\x00\x00\x0Ed41d8cd98f00b204e9800998ecf8427e20171213ce022548c4c3498e864fda289b81e711
> T[0]: T[1]: abc
> Scanned kv count -> 6
> [See Mobfile]:
> hbase org.apache.hadoop.hbase.io.hfile.HFile -f
> /hbase/mobdir/data/default/abc/07aab825b62dd9111831839cc9039df9/cf/d41d8cd98f00b204e9800998ecf8427e20171213bd8cfaf146684d4096ebf7994f050e96
> -p
> K: 4/cf:a/1513172924196/Put/vlen=14/seqid=7 V: y1
> K: 5/cf:a/1513172924214/Put/vlen=14/seqid=8 V: y2
> K: 6/cf:a/1513172925768/Put/vlen=14/seqid=9 V: y3
> 3、
> alter 'abc',{NAME => 'cf', MOB_THRESHOLD => '10240' }
> put
>
[jira] [Comment Edited] (HBASE-19507) Get or Scan Mob by rowkey return error value when run compact_mob or major_compact_mob after change MOB_THRESHOLD bigger
[
https://issues.apache.org/jira/browse/HBASE-19507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16290387#comment-16290387
]
Jingcheng Du edited comment on HBASE-19507 at 12/14/17 6:14 AM:
CDH 5.7.1 was released since June 2016. And after that users reported data loss
in the JIRA https://issues.apache.org/jira/browse/HBASE-16578. I think 5.7.1
doesn't include this fix.
The data in MOB files should be correct in your production. If you can apply
this above mentioned JIRA, then apply it. And run mob major compaction again,
and I think it will fix this issue.
[~huaxiang], any ideas on this? Thanks.
was (Author: jingcheng.du):
CDH 5.7.1 was released since June 2016. And after that users reported data loss
in the JIRA https://issues.apache.org/jira/browse/HBASE-16578. I think 5.7.1
doesn't include this fix.
The data in MOB file should be correct in your production. If you can apply
this above mentioned JIRA, then apply it. And run mob major compaction again,
and I think it will fix this issue.
[~huaxiang], any ideas on this? Thanks.
> Get or Scan Mob by rowkey return error value when run compact_mob or
> major_compact_mob after change MOB_THRESHOLD bigger
>
>
> Key: HBASE-19507
> URL: https://issues.apache.org/jira/browse/HBASE-19507
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: WangYuan
>
> 1、
> create 'abc',{NAME => 'cf', MOB_THRESHOLD => '10', IS_MOB => 'true'}
> put 'abc','1','cf:a','1'
> put 'abc','2','cf:a','2'
> put 'abc','3','cf:a','3'
> put 'abc','4','cf:a','y1'
> put 'abc','5','cf:a','y2'
> put 'abc','6','cf:a','y3'
>
> hbase(main):011:0> scan 'abc'
> ROWCOLUMN+CELL
>
>
> 1 column=cf:a,
> timestamp=1513171753098, value=1
>
> 2 column=cf:a,
> timestamp=1513171753208, value=2
>
> 3 column=cf:a,
> timestamp=1513171753246, value=3
>
> 4 column=cf:a,
> timestamp=1513171753273, value=y1
>
> 5 column=cf:a,
> timestamp=1513171753301, value=y2
>
> 6 column=cf:a,
> timestamp=1513171754282, value=y3
>
> hbase(main):012:0> flush 'abc'
> hbase(main):012:0> major_compact 'abc'
> hbase(main):012:0> major_compact_mob 'abc'
> 2、
> [See Hfile]:
> hbase org.apache.hadoop.hbase.io.hfile.HFile -f
> /hbase/data/default/abc/a31b3146cba0d4569a7bf44e70e299c9/cf/22a432ba5c2c4802bedd947b99626f10
> -p
> K: 1/cf:a/1513172294864/Put/vlen=5/seqid=4 V: 1
> K: 2/cf:a/1513172294892/Put/vlen=5/seqid=5 V: 2
> K: 3/cf:a/1513172294914/Put/vlen=5/seqid=6 V: 3
> K: 4/cf:a/1513172294954/Put/vlen=76/seqid=7 V:
> \x00\x00\x00\x0Ed41d8cd98f00b204e9800998ecf8427e20171213ce022548c4c3498e864fda289b81e711
> T[0]: T[1]: abc
> K: 5/cf:a/1513172294982/Put/vlen=76/seqid=8 V:
> \x00\x00\x00\x0Ed41d8cd98f00b204e9800998ecf8427e20171213ce022548c4c3498e864fda289b81e711
> T[0]: T[1]: abc
> K: 6/cf:a/1513172296455/Put/vlen=76/seqid=9 V:
> \x00\x00\x00\x0Ed41d8cd98f00b204e9800998ecf8427e20171213ce022548c4c3498e864fda289b81e711
> T[0]: T[1]: abc
> Scanned kv count -> 6
> [See Mobfile]:
> hbase org.apache.hadoop.hbase.io.hfile.HFile -f
> /hbase/mobdir/data/default/abc/07aab825b62dd9111831839cc9039df9/cf/d41d8cd98f00b204e9800998ecf8427e20171213bd8cfaf146684d4096ebf7994f050e96
> -p
> K: 4/cf:a/1513172924196/Put/vlen=14/seqid=7 V: y1
> K: 5/cf:a/1513172924214/Put/vlen=14/seqid=8 V: y2
> K: 6/cf:a/1513172925768/Put/vlen=14/seqid=9 V: y3
> 3、
> alter 'abc',{NAME => 'cf', MOB_THRESHOLD => '10240' }
> put
>
[jira] [Commented] (HBASE-19507) Get or Scan Mob by rowkey return error value when run compact_mob or major_compact_mob after change MOB_THRESHOLD bigger
[
https://issues.apache.org/jira/browse/HBASE-19507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16290389#comment-16290389
]
Jingcheng Du commented on HBASE-19507:
--
[~wangyuan], please run your case in the master branch, and check if it can
pass in open source code. Thanks.
> Get or Scan Mob by rowkey return error value when run compact_mob or
> major_compact_mob after change MOB_THRESHOLD bigger
>
>
> Key: HBASE-19507
> URL: https://issues.apache.org/jira/browse/HBASE-19507
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: WangYuan
>
> 1、
> create 'abc',{NAME => 'cf', MOB_THRESHOLD => '10', IS_MOB => 'true'}
> put 'abc','1','cf:a','1'
> put 'abc','2','cf:a','2'
> put 'abc','3','cf:a','3'
> put 'abc','4','cf:a','y1'
> put 'abc','5','cf:a','y2'
> put 'abc','6','cf:a','y3'
>
> hbase(main):011:0> scan 'abc'
> ROWCOLUMN+CELL
>
>
> 1 column=cf:a,
> timestamp=1513171753098, value=1
>
> 2 column=cf:a,
> timestamp=1513171753208, value=2
>
> 3 column=cf:a,
> timestamp=1513171753246, value=3
>
> 4 column=cf:a,
> timestamp=1513171753273, value=y1
>
> 5 column=cf:a,
> timestamp=1513171753301, value=y2
>
> 6 column=cf:a,
> timestamp=1513171754282, value=y3
>
> hbase(main):012:0> flush 'abc'
> hbase(main):012:0> major_compact 'abc'
> hbase(main):012:0> major_compact_mob 'abc'
> 2、
> [See Hfile]:
> hbase org.apache.hadoop.hbase.io.hfile.HFile -f
> /hbase/data/default/abc/a31b3146cba0d4569a7bf44e70e299c9/cf/22a432ba5c2c4802bedd947b99626f10
> -p
> K: 1/cf:a/1513172294864/Put/vlen=5/seqid=4 V: 1
> K: 2/cf:a/1513172294892/Put/vlen=5/seqid=5 V: 2
> K: 3/cf:a/1513172294914/Put/vlen=5/seqid=6 V: 3
> K: 4/cf:a/1513172294954/Put/vlen=76/seqid=7 V:
> \x00\x00\x00\x0Ed41d8cd98f00b204e9800998ecf8427e20171213ce022548c4c3498e864fda289b81e711
> T[0]: T[1]: abc
> K: 5/cf:a/1513172294982/Put/vlen=76/seqid=8 V:
> \x00\x00\x00\x0Ed41d8cd98f00b204e9800998ecf8427e20171213ce022548c4c3498e864fda289b81e711
> T[0]: T[1]: abc
> K: 6/cf:a/1513172296455/Put/vlen=76/seqid=9 V:
> \x00\x00\x00\x0Ed41d8cd98f00b204e9800998ecf8427e20171213ce022548c4c3498e864fda289b81e711
> T[0]: T[1]: abc
> Scanned kv count -> 6
> [See Mobfile]:
> hbase org.apache.hadoop.hbase.io.hfile.HFile -f
> /hbase/mobdir/data/default/abc/07aab825b62dd9111831839cc9039df9/cf/d41d8cd98f00b204e9800998ecf8427e20171213bd8cfaf146684d4096ebf7994f050e96
> -p
> K: 4/cf:a/1513172924196/Put/vlen=14/seqid=7 V: y1
> K: 5/cf:a/1513172924214/Put/vlen=14/seqid=8 V: y2
> K: 6/cf:a/1513172925768/Put/vlen=14/seqid=9 V: y3
> 3、
> alter 'abc',{NAME => 'cf', MOB_THRESHOLD => '10240' }
> put
>
[jira] [Commented] (HBASE-19507) Get or Scan Mob by rowkey return error value when run compact_mob or major_compact_mob after change MOB_THRESHOLD bigger
[
https://issues.apache.org/jira/browse/HBASE-19507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16290387#comment-16290387
]
Jingcheng Du commented on HBASE-19507:
--
CDH 5.7.1 was released since June 2016. And after that users reported data loss
in the JIRA https://issues.apache.org/jira/browse/HBASE-16578. I think 5.7.1
doesn't include this fix.
The data in MOB file should be correct in your production. If you can apply
this above mentioned JIRA, then apply it. And run mob major compaction again,
and I think it will fix this issue.
[~huaxiang], any ideas on this? Thanks.
> Get or Scan Mob by rowkey return error value when run compact_mob or
> major_compact_mob after change MOB_THRESHOLD bigger
>
>
> Key: HBASE-19507
> URL: https://issues.apache.org/jira/browse/HBASE-19507
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: WangYuan
>
> 1、
> create 'abc',{NAME => 'cf', MOB_THRESHOLD => '10', IS_MOB => 'true'}
> put 'abc','1','cf:a','1'
> put 'abc','2','cf:a','2'
> put 'abc','3','cf:a','3'
> put 'abc','4','cf:a','y1'
> put 'abc','5','cf:a','y2'
> put 'abc','6','cf:a','y3'
>
> hbase(main):011:0> scan 'abc'
> ROWCOLUMN+CELL
>
>
> 1 column=cf:a,
> timestamp=1513171753098, value=1
>
> 2 column=cf:a,
> timestamp=1513171753208, value=2
>
> 3 column=cf:a,
> timestamp=1513171753246, value=3
>
> 4 column=cf:a,
> timestamp=1513171753273, value=y1
>
> 5 column=cf:a,
> timestamp=1513171753301, value=y2
>
> 6 column=cf:a,
> timestamp=1513171754282, value=y3
>
> hbase(main):012:0> flush 'abc'
> hbase(main):012:0> major_compact 'abc'
> hbase(main):012:0> major_compact_mob 'abc'
> 2、
> [See Hfile]:
> hbase org.apache.hadoop.hbase.io.hfile.HFile -f
> /hbase/data/default/abc/a31b3146cba0d4569a7bf44e70e299c9/cf/22a432ba5c2c4802bedd947b99626f10
> -p
> K: 1/cf:a/1513172294864/Put/vlen=5/seqid=4 V: 1
> K: 2/cf:a/1513172294892/Put/vlen=5/seqid=5 V: 2
> K: 3/cf:a/1513172294914/Put/vlen=5/seqid=6 V: 3
> K: 4/cf:a/1513172294954/Put/vlen=76/seqid=7 V:
> \x00\x00\x00\x0Ed41d8cd98f00b204e9800998ecf8427e20171213ce022548c4c3498e864fda289b81e711
> T[0]: T[1]: abc
> K: 5/cf:a/1513172294982/Put/vlen=76/seqid=8 V:
> \x00\x00\x00\x0Ed41d8cd98f00b204e9800998ecf8427e20171213ce022548c4c3498e864fda289b81e711
> T[0]: T[1]: abc
> K: 6/cf:a/1513172296455/Put/vlen=76/seqid=9 V:
> \x00\x00\x00\x0Ed41d8cd98f00b204e9800998ecf8427e20171213ce022548c4c3498e864fda289b81e711
> T[0]: T[1]: abc
> Scanned kv count -> 6
> [See Mobfile]:
> hbase org.apache.hadoop.hbase.io.hfile.HFile -f
> /hbase/mobdir/data/default/abc/07aab825b62dd9111831839cc9039df9/cf/d41d8cd98f00b204e9800998ecf8427e20171213bd8cfaf146684d4096ebf7994f050e96
> -p
> K: 4/cf:a/1513172924196/Put/vlen=14/seqid=7 V: y1
> K: 5/cf:a/1513172924214/Put/vlen=14/seqid=8 V: y2
> K: 6/cf:a/1513172925768/Put/vlen=14/seqid=9 V: y3
> 3、
> alter 'abc',{NAME => 'cf', MOB_THRESHOLD => '10240' }
> put
>
[jira] [Commented] (HBASE-19507) Get or Scan Mob by rowkey return error value when run compact_mob or major_compact_mob after change MOB_THRESHOLD bigger
[
https://issues.apache.org/jira/browse/HBASE-19507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16290273#comment-16290273
]
Jingcheng Du commented on HBASE-19507:
--
Thanks for the reporting.
What is the distribution of this HBase (CDH or HDP?) since we don't have MOB in
branch-1.x.
It looks like the cells that contain the ref links don't have the ref tags.
Would you mind running the tests in the HBase master branch? So that we can
make sure this is not an issue from backport. Thanks.
> Get or Scan Mob by rowkey return error value when run compact_mob or
> major_compact_mob after change MOB_THRESHOLD bigger
>
>
> Key: HBASE-19507
> URL: https://issues.apache.org/jira/browse/HBASE-19507
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: WangYuan
>
> 1、
> create 'abc',{NAME => 'cf', MOB_THRESHOLD => '10', IS_MOB => 'true'}
> put 'abc','1','cf:a','1'
> put 'abc','2','cf:a','2'
> put 'abc','3','cf:a','3'
> put 'abc','4','cf:a','y1'
> put 'abc','5','cf:a','y2'
> put 'abc','6','cf:a','y3'
>
> hbase(main):011:0> scan 'abc'
> ROWCOLUMN+CELL
>
>
> 1 column=cf:a,
> timestamp=1513171753098, value=1
>
> 2 column=cf:a,
> timestamp=1513171753208, value=2
>
> 3 column=cf:a,
> timestamp=1513171753246, value=3
>
> 4 column=cf:a,
> timestamp=1513171753273, value=y1
>
> 5 column=cf:a,
> timestamp=1513171753301, value=y2
>
> 6 column=cf:a,
> timestamp=1513171754282, value=y3
>
> hbase(main):012:0> flush 'abc'
> hbase(main):012:0> major_compact 'abc'
> hbase(main):012:0> major_compact_mob 'abc'
> 2、
> [See Hfile]:
> hbase org.apache.hadoop.hbase.io.hfile.HFile -f
> /hbase/data/default/abc/a31b3146cba0d4569a7bf44e70e299c9/cf/22a432ba5c2c4802bedd947b99626f10
> -p
> K: 1/cf:a/1513172294864/Put/vlen=5/seqid=4 V: 1
> K: 2/cf:a/1513172294892/Put/vlen=5/seqid=5 V: 2
> K: 3/cf:a/1513172294914/Put/vlen=5/seqid=6 V: 3
> K: 4/cf:a/1513172294954/Put/vlen=76/seqid=7 V:
> \x00\x00\x00\x0Ed41d8cd98f00b204e9800998ecf8427e20171213ce022548c4c3498e864fda289b81e711
> T[0]: T[1]: abc
> K: 5/cf:a/1513172294982/Put/vlen=76/seqid=8 V:
> \x00\x00\x00\x0Ed41d8cd98f00b204e9800998ecf8427e20171213ce022548c4c3498e864fda289b81e711
> T[0]: T[1]: abc
> K: 6/cf:a/1513172296455/Put/vlen=76/seqid=9 V:
> \x00\x00\x00\x0Ed41d8cd98f00b204e9800998ecf8427e20171213ce022548c4c3498e864fda289b81e711
> T[0]: T[1]: abc
> Scanned kv count -> 6
> [See Mobfile]:
> hbase org.apache.hadoop.hbase.io.hfile.HFile -f
> /hbase/mobdir/data/default/abc/07aab825b62dd9111831839cc9039df9/cf/d41d8cd98f00b204e9800998ecf8427e20171213bd8cfaf146684d4096ebf7994f050e96
> -p
> K: 4/cf:a/1513172924196/Put/vlen=14/seqid=7 V: y1
> K: 5/cf:a/1513172924214/Put/vlen=14/seqid=8 V: y2
> K: 6/cf:a/1513172925768/Put/vlen=14/seqid=9 V: y3
> 3、
> alter 'abc',{NAME => 'cf', MOB_THRESHOLD => '10240' }
> put
>
[jira] [Commented] (HBASE-18693) adding an option to restore_snapshot to move mob files from archive dir to working dir
[ https://issues.apache.org/jira/browse/HBASE-18693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16287449#comment-16287449 ] Jingcheng Du commented on HBASE-18693: -- Thanks [~huaxiang] for the patch! Have you already posted the v3 patch to RB? > adding an option to restore_snapshot to move mob files from archive dir to > working dir > -- > > Key: HBASE-18693 > URL: https://issues.apache.org/jira/browse/HBASE-18693 > Project: HBase > Issue Type: Improvement > Components: mob >Affects Versions: 2.0.0-alpha-2 >Reporter: huaxiang sun >Assignee: huaxiang sun > Attachments: HBASE-18693.master.001.patch, > HBASE-18693.master.002.patch, HBASE-18693.master.003.patch > > > Today, there is a single mob region where mob files for all user regions are > saved. There could be many files (one million) in a single mob directory. > When one mob table is restored or cloned from snapshot, links are created for > these mob files. This creates a scaling issue for mob compaction. In mob > compaction's select() logic, for each hFileLink, it needs to call NN's > getFileStatus() to get the size of the linked hfile. Assume that one such > call takes 20ms, 20ms * 100 = 6 hours. > To avoid this overhead, we want to add an option so that restore_snapshot can > move mob files from archive dir to working dir. clone_snapshot is more > complicated as it can clone a snapshot to a different table so moving that > can destroy the snapshot. No option will be added for clone_snapshot. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HBASE-18602) rsgroup cleanup unassign code
[
https://issues.apache.org/jira/browse/HBASE-18602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203409#comment-16203409
]
Jingcheng Du edited comment on HBASE-18602 at 10/13/17 11:33 AM:
-
bq. Applying your idea, the FavoredStochasticBalancer#testMisplacedRegions pass
now.
Yes [~chia7712], this one can pass if removing the unassign operations and add
hosting server for the misplaced RegionPlans.
I ran the other two, and they were timed out in my env, where they waited for
the RS stop in {{stopServersAndWaitUntilProcessed}} and timeout, and the
assignment is not started yet. Not sure why RS cannot be stopped.
was (Author: jingcheng.du):
bq. Applying your idea, the FavoredStochasticBalancer#testMisplacedRegions pass
now.
Yes [~chia7712], this one can pass.
I ran the other two, and they were timed out in my env, where they waited for
the RS stop in {{stopServersAndWaitUntilProcessed}} and timeout, and the
assignment is not started yet. Not sure why RS cannot be stopped.
> rsgroup cleanup unassign code
> -
>
> Key: HBASE-18602
> URL: https://issues.apache.org/jira/browse/HBASE-18602
> Project: HBase
> Issue Type: Improvement
> Components: rsgroup
>Reporter: Wang, Xinglong
>Assignee: Wang, Xinglong
>Priority: Minor
> Attachments: HBASE-18602-master-v1.patch,
> HBASE-18602-master-v2.patch, HBASE-18602-master-v3.patch
>
>
> While walking through rsgroup code, I found that variable misplacedRegions
> has never been added any element into. This makes the unassign region code is
> not functional. And according to my test, it is actually unnecessary to do
> that.
> RSGroupBasedLoadBalancer.java
> {code:java}
> private Map correctAssignments(
>Map existingAssignments)
> throws HBaseIOException{
> Map correctAssignments = new TreeMap<>();
> List misplacedRegions = new LinkedList<>();
> correctAssignments.put(LoadBalancer.BOGUS_SERVER_NAME, new
> LinkedList<>());
> for (Map.Entry assignments :
> existingAssignments.entrySet()){
> ServerName sName = assignments.getKey();
> correctAssignments.put(sName, new LinkedList<>());
> List regions = assignments.getValue();
> for (HRegionInfo region : regions) {
> RSGroupInfo info = null;
> try {
> info = rsGroupInfoManager.getRSGroup(
> rsGroupInfoManager.getRSGroupOfTable(region.getTable()));
> } catch (IOException exp) {
> LOG.debug("RSGroup information null for region of table " +
> region.getTable(),
> exp);
> }
> if ((info == null) || (!info.containsServer(sName.getAddress( {
> correctAssignments.get(LoadBalancer.BOGUS_SERVER_NAME).add(region);
> } else {
> correctAssignments.get(sName).add(region);
> }
> }
> }
> //TODO bulk unassign?
> //unassign misplaced regions, so that they are assigned to correct groups.
> for(HRegionInfo info: misplacedRegions) {
> try {
> this.masterServices.getAssignmentManager().unassign(info);
> } catch (IOException e) {
> throw new HBaseIOException(e);
> }
> }
> return correctAssignments;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18602) rsgroup cleanup unassign code
[
https://issues.apache.org/jira/browse/HBASE-18602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203409#comment-16203409
]
Jingcheng Du commented on HBASE-18602:
--
bq. Applying your idea, the FavoredStochasticBalancer#testMisplacedRegions pass
now.
Yes [~chia7712], this one can pass.
I ran the other two, and they were timed out in my env, where they waited for
the RS stop in {{stopServersAndWaitUntilProcessed}} and timeout, and the
assignment is not started yet. Not sure why RS cannot be stopped.
> rsgroup cleanup unassign code
> -
>
> Key: HBASE-18602
> URL: https://issues.apache.org/jira/browse/HBASE-18602
> Project: HBase
> Issue Type: Improvement
> Components: rsgroup
>Reporter: Wang, Xinglong
>Assignee: Wang, Xinglong
>Priority: Minor
> Attachments: HBASE-18602-master-v1.patch,
> HBASE-18602-master-v2.patch, HBASE-18602-master-v3.patch
>
>
> While walking through rsgroup code, I found that variable misplacedRegions
> has never been added any element into. This makes the unassign region code is
> not functional. And according to my test, it is actually unnecessary to do
> that.
> RSGroupBasedLoadBalancer.java
> {code:java}
> private Map correctAssignments(
>Map existingAssignments)
> throws HBaseIOException{
> Map correctAssignments = new TreeMap<>();
> List misplacedRegions = new LinkedList<>();
> correctAssignments.put(LoadBalancer.BOGUS_SERVER_NAME, new
> LinkedList<>());
> for (Map.Entry assignments :
> existingAssignments.entrySet()){
> ServerName sName = assignments.getKey();
> correctAssignments.put(sName, new LinkedList<>());
> List regions = assignments.getValue();
> for (HRegionInfo region : regions) {
> RSGroupInfo info = null;
> try {
> info = rsGroupInfoManager.getRSGroup(
> rsGroupInfoManager.getRSGroupOfTable(region.getTable()));
> } catch (IOException exp) {
> LOG.debug("RSGroup information null for region of table " +
> region.getTable(),
> exp);
> }
> if ((info == null) || (!info.containsServer(sName.getAddress( {
> correctAssignments.get(LoadBalancer.BOGUS_SERVER_NAME).add(region);
> } else {
> correctAssignments.get(sName).add(region);
> }
> }
> }
> //TODO bulk unassign?
> //unassign misplaced regions, so that they are assigned to correct groups.
> for(HRegionInfo info: misplacedRegions) {
> try {
> this.masterServices.getAssignmentManager().unassign(info);
> } catch (IOException e) {
> throw new HBaseIOException(e);
> }
> }
> return correctAssignments;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18602) rsgroup cleanup unassign code
[
https://issues.apache.org/jira/browse/HBASE-18602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203324#comment-16203324
]
Jingcheng Du commented on HBASE-18602:
--
Thanks [~chia7712]. Will take a look at it when having time.
Hi [~suxingfate], would you mind adding a unit test for misplaced region of
rsgroup? I doubt moving region will throw NPE when hosting server is null in
RegionPlan. Thanks.
> rsgroup cleanup unassign code
> -
>
> Key: HBASE-18602
> URL: https://issues.apache.org/jira/browse/HBASE-18602
> Project: HBase
> Issue Type: Improvement
> Components: rsgroup
>Reporter: Wang, Xinglong
>Assignee: Wang, Xinglong
>Priority: Minor
> Attachments: HBASE-18602-master-v1.patch,
> HBASE-18602-master-v2.patch, HBASE-18602-master-v3.patch
>
>
> While walking through rsgroup code, I found that variable misplacedRegions
> has never been added any element into. This makes the unassign region code is
> not functional. And according to my test, it is actually unnecessary to do
> that.
> RSGroupBasedLoadBalancer.java
> {code:java}
> private Map correctAssignments(
>Map existingAssignments)
> throws HBaseIOException{
> Map correctAssignments = new TreeMap<>();
> List misplacedRegions = new LinkedList<>();
> correctAssignments.put(LoadBalancer.BOGUS_SERVER_NAME, new
> LinkedList<>());
> for (Map.Entry assignments :
> existingAssignments.entrySet()){
> ServerName sName = assignments.getKey();
> correctAssignments.put(sName, new LinkedList<>());
> List regions = assignments.getValue();
> for (HRegionInfo region : regions) {
> RSGroupInfo info = null;
> try {
> info = rsGroupInfoManager.getRSGroup(
> rsGroupInfoManager.getRSGroupOfTable(region.getTable()));
> } catch (IOException exp) {
> LOG.debug("RSGroup information null for region of table " +
> region.getTable(),
> exp);
> }
> if ((info == null) || (!info.containsServer(sName.getAddress( {
> correctAssignments.get(LoadBalancer.BOGUS_SERVER_NAME).add(region);
> } else {
> correctAssignments.get(sName).add(region);
> }
> }
> }
> //TODO bulk unassign?
> //unassign misplaced regions, so that they are assigned to correct groups.
> for(HRegionInfo info: misplacedRegions) {
> try {
> this.masterServices.getAssignmentManager().unassign(info);
> } catch (IOException e) {
> throw new HBaseIOException(e);
> }
> }
> return correctAssignments;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18602) rsgroup cleanup unassign code
[
https://issues.apache.org/jira/browse/HBASE-18602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201403#comment-16201403
]
Jingcheng Du commented on HBASE-18602:
--
Hi [~suxingfate], the patch cannot be applied, you have to rebase the code and
update your patch. Thanks.
The regions with BOGUS_SERVER_NAME are regarded as misplaced regions, and these
regions have been handled and re-assigned/moved by the callers, for instance
HMaster. It is not necessary to un-assign the misplaced regions in
{{correctAssignments}} method, which means we can removed the unused
{{misplacedRegions}}.
In the same way, the unassign in FavoredStochasticBalancer#balanceCluster is
not necessary. And meanwhile in most of load balancer, we only decide how to
move the regions without doing the real move actions, whereas in
FavoredStochasticBalancer#balanceCluster we do the real move actions.
Your thoughts? [~chia7712]. Thanks.
> rsgroup cleanup unassign code
> -
>
> Key: HBASE-18602
> URL: https://issues.apache.org/jira/browse/HBASE-18602
> Project: HBase
> Issue Type: Improvement
> Components: rsgroup
>Reporter: Wang, Xinglong
>Priority: Minor
> Attachments: HBASE-18602-master-v1.patch
>
>
> While walking through rsgroup code, I found that variable misplacedRegions
> has never been added any element into. This makes the unassign region code is
> not functional. And according to my test, it is actually unnecessary to do
> that.
> RSGroupBasedLoadBalancer.java
> {code:java}
> private Map correctAssignments(
>Map existingAssignments)
> throws HBaseIOException{
> Map correctAssignments = new TreeMap<>();
> List misplacedRegions = new LinkedList<>();
> correctAssignments.put(LoadBalancer.BOGUS_SERVER_NAME, new
> LinkedList<>());
> for (Map.Entry assignments :
> existingAssignments.entrySet()){
> ServerName sName = assignments.getKey();
> correctAssignments.put(sName, new LinkedList<>());
> List regions = assignments.getValue();
> for (HRegionInfo region : regions) {
> RSGroupInfo info = null;
> try {
> info = rsGroupInfoManager.getRSGroup(
> rsGroupInfoManager.getRSGroupOfTable(region.getTable()));
> } catch (IOException exp) {
> LOG.debug("RSGroup information null for region of table " +
> region.getTable(),
> exp);
> }
> if ((info == null) || (!info.containsServer(sName.getAddress( {
> correctAssignments.get(LoadBalancer.BOGUS_SERVER_NAME).add(region);
> } else {
> correctAssignments.get(sName).add(region);
> }
> }
> }
> //TODO bulk unassign?
> //unassign misplaced regions, so that they are assigned to correct groups.
> for(HRegionInfo info: misplacedRegions) {
> try {
> this.masterServices.getAssignmentManager().unassign(info);
> } catch (IOException e) {
> throw new HBaseIOException(e);
> }
> }
> return correctAssignments;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18693) adding an option to restore_snapshot to move mob files from archive dir to working dir
[ https://issues.apache.org/jira/browse/HBASE-18693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198114#comment-16198114 ] Jingcheng Du commented on HBASE-18693: -- Thanks a lot [~huaxiang], I've updated the comments in RB. > adding an option to restore_snapshot to move mob files from archive dir to > working dir > -- > > Key: HBASE-18693 > URL: https://issues.apache.org/jira/browse/HBASE-18693 > Project: HBase > Issue Type: Improvement > Components: mob >Affects Versions: 2.0.0-alpha-2 >Reporter: huaxiang sun >Assignee: huaxiang sun > Attachments: HBASE-18693.master.001.patch > > > Today, there is a single mob region where mob files for all user regions are > saved. There could be many files (one million) in a single mob directory. > When one mob table is restored or cloned from snapshot, links are created for > these mob files. This creates a scaling issue for mob compaction. In mob > compaction's select() logic, for each hFileLink, it needs to call NN's > getFileStatus() to get the size of the linked hfile. Assume that one such > call takes 20ms, 20ms * 100 = 6 hours. > To avoid this overhead, we want to add an option so that restore_snapshot can > move mob files from archive dir to working dir. clone_snapshot is more > complicated as it can clone a snapshot to a different table so moving that > can destroy the snapshot. No option will be added for clone_snapshot. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18693) adding an option to restore_snapshot to move mob files from archive dir to working dir
[ https://issues.apache.org/jira/browse/HBASE-18693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198008#comment-16198008 ] Jingcheng Du commented on HBASE-18693: -- Sure, [~huaxiang]. I am looking at it. It may take a few days. Thanks. > adding an option to restore_snapshot to move mob files from archive dir to > working dir > -- > > Key: HBASE-18693 > URL: https://issues.apache.org/jira/browse/HBASE-18693 > Project: HBase > Issue Type: Improvement > Components: mob >Affects Versions: 2.0.0-alpha-2 >Reporter: huaxiang sun >Assignee: huaxiang sun > Attachments: HBASE-18693.master.001.patch > > > Today, there is a single mob region where mob files for all user regions are > saved. There could be many files (one million) in a single mob directory. > When one mob table is restored or cloned from snapshot, links are created for > these mob files. This creates a scaling issue for mob compaction. In mob > compaction's select() logic, for each hFileLink, it needs to call NN's > getFileStatus() to get the size of the linked hfile. Assume that one such > call takes 20ms, 20ms * 100 = 6 hours. > To avoid this overhead, we want to add an option so that restore_snapshot can > move mob files from archive dir to working dir. clone_snapshot is more > complicated as it can clone a snapshot to a different table so moving that > can destroy the snapshot. No option will be added for clone_snapshot. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18693) adding an option to restore_snapshot to move mob files from archive dir to working dir
[ https://issues.apache.org/jira/browse/HBASE-18693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16149191#comment-16149191 ] Jingcheng Du commented on HBASE-18693: -- bq. I got what was your concern. restore_snapshot always restores to the same table, that is why I add an option here. clone_snapshot is a different story, it can be cloned to different tables. If the option is added to clone_snapshot, it will corrupt the snapshot. You are right. I am +1 on this option. Thanks! > adding an option to restore_snapshot to move mob files from archive dir to > working dir > -- > > Key: HBASE-18693 > URL: https://issues.apache.org/jira/browse/HBASE-18693 > Project: HBase > Issue Type: Improvement > Components: mob >Affects Versions: 2.0.0-alpha-2 >Reporter: huaxiang sun >Assignee: huaxiang sun > > Today, there is a single mob region where mob files for all user regions are > saved. There could be many files (one million) in a single mob directory. > When one mob table is restored or cloned from snapshot, links are created for > these mob files. This creates a scaling issue for mob compaction. In mob > compaction's select() logic, for each hFileLink, it needs to call NN's > getFileStatus() to get the size of the linked hfile. Assume that one such > call takes 20ms, 20ms * 100 = 6 hours. > To avoid this overhead, we want to add an option so that restore_snapshot can > move mob files from archive dir to working dir. clone_snapshot is more > complicated as it can clone a snapshot to a different table so moving that > can destroy the snapshot. No option will be added for clone_snapshot. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18693) adding an option to restore_snapshot to move mob files from archive dir to working dir
[ https://issues.apache.org/jira/browse/HBASE-18693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147396#comment-16147396 ] Jingcheng Du commented on HBASE-18693: -- Thanks Huaxiang. bq. The snapshot itself is not destroyed after moving mob files from archive to working directory. I do not see an issue to restore a snapshot twice here. Can you share more details? Restoring a snapshot to the same table is okay. What if we try to restore the snapshot in another table? The same MOB file can be in different locations? No, right? bq. For one of our use cases, user exported a snapshot with millions of mob files and restored the table at a remote cluster. The select() took more than one day to complete before actual compaction happened. We did the hack to skip hfile links so compaction could happen within several minutes. Even compacting links in a longer interval, this is still a huge overhead. What do you think? You are right, this is a problem. How about select files with multiple threads, each thread handle part of the files selection? Thanks. > adding an option to restore_snapshot to move mob files from archive dir to > working dir > -- > > Key: HBASE-18693 > URL: https://issues.apache.org/jira/browse/HBASE-18693 > Project: HBase > Issue Type: Improvement > Components: mob >Affects Versions: 2.0.0-alpha-2 >Reporter: huaxiang sun >Assignee: huaxiang sun > > Today, there is a single mob region where mob files for all user regions are > saved. There could be many files (one million) in a single mob directory. > When one mob table is restored or cloned from snapshot, links are created for > these mob files. This creates a scaling issue for mob compaction. In mob > compaction's select() logic, for each hFileLink, it needs to call NN's > getFileStatus() to get the size of the linked hfile. Assume that one such > call takes 20ms, 20ms * 100 = 6 hours. > To avoid this overhead, we want to add an option so that restore_snapshot can > move mob files from archive dir to working dir. clone_snapshot is more > complicated as it can clone a snapshot to a different table so moving that > can destroy the snapshot. No option will be added for clone_snapshot. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Issue Comment Deleted] (HBASE-18693) adding an option to restore_snapshot to move mob files from archive dir to working dir
[ https://issues.apache.org/jira/browse/HBASE-18693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingcheng Du updated HBASE-18693: - Comment: was deleted (was: Thanks Huaxiang. HDFS move doesn't copy the data, right, it doesn't, it is supposed to be a rename operation. My concern is if we restore a snapshot twice which is possible, how to handle such operations? In HBase, we compact the hfile links in compaction, I think compacting hfile links in MOB compaction is reasonable too. Or we can skip the hfile links in most of MOB compaction, and compact the links in a longer interval (like a month)? I prefer the 1st option. What's your idea? Thanks. ) > adding an option to restore_snapshot to move mob files from archive dir to > working dir > -- > > Key: HBASE-18693 > URL: https://issues.apache.org/jira/browse/HBASE-18693 > Project: HBase > Issue Type: Improvement > Components: mob >Affects Versions: 2.0.0-alpha-2 >Reporter: huaxiang sun >Assignee: huaxiang sun > > Today, there is a single mob region where mob files for all user regions are > saved. There could be many files (one million) in a single mob directory. > When one mob table is restored or cloned from snapshot, links are created for > these mob files. This creates a scaling issue for mob compaction. In mob > compaction's select() logic, for each hFileLink, it needs to call NN's > getFileStatus() to get the size of the linked hfile. Assume that one such > call takes 20ms, 20ms * 100 = 6 hours. > To avoid this overhead, we want to add an option so that restore_snapshot can > move mob files from archive dir to working dir. clone_snapshot is more > complicated as it can clone a snapshot to a different table so moving that > can destroy the snapshot. No option will be added for clone_snapshot. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18693) adding an option to restore_snapshot to move mob files from archive dir to working dir
[ https://issues.apache.org/jira/browse/HBASE-18693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144864#comment-16144864 ] Jingcheng Du commented on HBASE-18693: -- Thanks Huaxiang. HDFS move doesn't copy the data, right, it doesn't, it is supposed to be a rename operation. My concern is if we restore a snapshot twice which is possible, how to handle such operations? In HBase, we compact the hfile links in compaction, I think compacting hfile links in MOB compaction is reasonable too. Or we can skip the hfile links in most of MOB compaction, and compact the links in a longer interval (like a month)? I prefer the 1st option. What's your idea? Thanks. > adding an option to restore_snapshot to move mob files from archive dir to > working dir > -- > > Key: HBASE-18693 > URL: https://issues.apache.org/jira/browse/HBASE-18693 > Project: HBase > Issue Type: Improvement > Components: mob >Affects Versions: 2.0.0-alpha-2 >Reporter: huaxiang sun >Assignee: huaxiang sun > > Today, there is a single mob region where mob files for all user regions are > saved. There could be many files (one million) in a single mob directory. > When one mob table is restored or cloned from snapshot, links are created for > these mob files. This creates a scaling issue for mob compaction. In mob > compaction's select() logic, for each hFileLink, it needs to call NN's > getFileStatus() to get the size of the linked hfile. Assume that one such > call takes 20ms, 20ms * 100 = 6 hours. > To avoid this overhead, we want to add an option so that restore_snapshot can > move mob files from archive dir to working dir. clone_snapshot is more > complicated as it can clone a snapshot to a different table so moving that > can destroy the snapshot. No option will be added for clone_snapshot. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18693) adding an option to restore_snapshot to move mob files from archive dir to working dir
[ https://issues.apache.org/jira/browse/HBASE-18693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144865#comment-16144865 ] Jingcheng Du commented on HBASE-18693: -- HDFS move doesn't copy the data, right, it doesn't, it is supposed to be a rename operation. My concern is if we restore a snapshot twice which is possible, how to handle such operations? In HBase, we compact the hfile links in compaction, I think compacting hfile links in MOB compaction is reasonable too. Or we can skip the hfile links in most of MOB compaction, and compact the links in a longer interval (like a month)? I prefer the 1st option. What's your idea? Thanks. > adding an option to restore_snapshot to move mob files from archive dir to > working dir > -- > > Key: HBASE-18693 > URL: https://issues.apache.org/jira/browse/HBASE-18693 > Project: HBase > Issue Type: Improvement > Components: mob >Affects Versions: 2.0.0-alpha-2 >Reporter: huaxiang sun >Assignee: huaxiang sun > > Today, there is a single mob region where mob files for all user regions are > saved. There could be many files (one million) in a single mob directory. > When one mob table is restored or cloned from snapshot, links are created for > these mob files. This creates a scaling issue for mob compaction. In mob > compaction's select() logic, for each hFileLink, it needs to call NN's > getFileStatus() to get the size of the linked hfile. Assume that one such > call takes 20ms, 20ms * 100 = 6 hours. > To avoid this overhead, we want to add an option so that restore_snapshot can > move mob files from archive dir to working dir. clone_snapshot is more > complicated as it can clone a snapshot to a different table so moving that > can destroy the snapshot. No option will be added for clone_snapshot. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HBASE-18693) adding an option to restore_snapshot to move mob files from archive dir to working dir
[ https://issues.apache.org/jira/browse/HBASE-18693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144792#comment-16144792 ] Jingcheng Du edited comment on HBASE-18693 at 8/29/17 6:02 AM: --- Thanks [~huangxiangang]. The restore operation should be a fast one, moving data in such an operation is not proper I think. Could we just skip the hfile link, or just compact them no matter what size it is in the compaction? How about this? was (Author: jingcheng.du): Thanks [~huangxiangang]. The restore operations should be a fast one, moving data in such an operation is not proper I think. Could we just skip the hfile link, or just compact them no matter what size it is in the compaction? How about this? > adding an option to restore_snapshot to move mob files from archive dir to > working dir > -- > > Key: HBASE-18693 > URL: https://issues.apache.org/jira/browse/HBASE-18693 > Project: HBase > Issue Type: Improvement > Components: mob >Affects Versions: 2.0.0-alpha-2 >Reporter: huaxiang sun >Assignee: huaxiang sun > > Today, there is a single mob region where mob files for all user regions are > saved. There could be many files (one million) in a single mob directory. > When one mob table is restored or cloned from snapshot, links are created for > these mob files. This creates a scaling issue for mob compaction. In mob > compaction's select() logic, for each hFileLink, it needs to call NN's > getFileStatus() to get the size of the linked hfile. Assume that one such > call takes 20ms, 20ms * 100 = 6 hours. > To avoid this overhead, we want to add an option so that restore_snapshot can > move mob files from archive dir to working dir. clone_snapshot is more > complicated as it can clone a snapshot to a different table so moving that > can destroy the snapshot. No option will be added for clone_snapshot. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18693) adding an option to restore_snapshot to move mob files from archive dir to working dir
[ https://issues.apache.org/jira/browse/HBASE-18693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144792#comment-16144792 ] Jingcheng Du commented on HBASE-18693: -- Thanks [~huangxiangang]. The restore operations should be a fast one, moving data in such an operation is not proper I think. Could we just skip the hfile link, or just compact them no matter what size it is in the compaction? How about this? > adding an option to restore_snapshot to move mob files from archive dir to > working dir > -- > > Key: HBASE-18693 > URL: https://issues.apache.org/jira/browse/HBASE-18693 > Project: HBase > Issue Type: Improvement > Components: mob >Affects Versions: 2.0.0-alpha-2 >Reporter: huaxiang sun >Assignee: huaxiang sun > > Today, there is a single mob region where mob files for all user regions are > saved. There could be many files (one million) in a single mob directory. > When one mob table is restored or cloned from snapshot, links are created for > these mob files. This creates a scaling issue for mob compaction. In mob > compaction's select() logic, for each hFileLink, it needs to call NN's > getFileStatus() to get the size of the linked hfile. Assume that one such > call takes 20ms, 20ms * 100 = 6 hours. > To avoid this overhead, we want to add an option so that restore_snapshot can > move mob files from archive dir to working dir. clone_snapshot is more > complicated as it can clone a snapshot to a different table so moving that > can destroy the snapshot. No option will be added for clone_snapshot. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18287) Remove log warning in PartitionedMobCompactor.java#getFileStatus
[ https://issues.apache.org/jira/browse/HBASE-18287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingcheng Du updated HBASE-18287: - Status: Open (was: Patch Available) > Remove log warning in PartitionedMobCompactor.java#getFileStatus > - > > Key: HBASE-18287 > URL: https://issues.apache.org/jira/browse/HBASE-18287 > Project: HBase > Issue Type: Bug > Components: mob >Affects Versions: 2.0.0-alpha-1 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Attachments: HBASE-18287.master.001.patch, > HBASE-18287.master.002.patch > > > It is confusing as the FileNotFound is expected here. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18287) Remove log warning in PartitionedMobCompactor.java#getFileStatus
[ https://issues.apache.org/jira/browse/HBASE-18287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137981#comment-16137981 ] Jingcheng Du commented on HBASE-18287: -- Resubmit the patch to trigger the test again. +1 on the patch V2. Thanks for the patch [~huaxiang]! > Remove log warning in PartitionedMobCompactor.java#getFileStatus > - > > Key: HBASE-18287 > URL: https://issues.apache.org/jira/browse/HBASE-18287 > Project: HBase > Issue Type: Bug > Components: mob >Affects Versions: 2.0.0-alpha-1 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Attachments: HBASE-18287.master.001.patch, > HBASE-18287.master.002.patch > > > It is confusing as the FileNotFound is expected here. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18287) Remove log warning in PartitionedMobCompactor.java#getFileStatus
[ https://issues.apache.org/jira/browse/HBASE-18287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingcheng Du updated HBASE-18287: - Status: Patch Available (was: Open) > Remove log warning in PartitionedMobCompactor.java#getFileStatus > - > > Key: HBASE-18287 > URL: https://issues.apache.org/jira/browse/HBASE-18287 > Project: HBase > Issue Type: Bug > Components: mob >Affects Versions: 2.0.0-alpha-1 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Attachments: HBASE-18287.master.001.patch, > HBASE-18287.master.002.patch > > > It is confusing as the FileNotFound is expected here. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18287) Remove log warning in PartitionedMobCompactor.java#getFileStatus
[ https://issues.apache.org/jira/browse/HBASE-18287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136356#comment-16136356 ] Jingcheng Du commented on HBASE-18287: -- Thanks [~huaxiang] for the patch. We should not remove the code of catching FileNotFoundException. We will do nothing when we catch the FNFE, but do not remove it. > Remove log warning in PartitionedMobCompactor.java#getFileStatus > - > > Key: HBASE-18287 > URL: https://issues.apache.org/jira/browse/HBASE-18287 > Project: HBase > Issue Type: Bug > Components: mob >Affects Versions: 2.0.0-alpha-1 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Attachments: HBASE-18287.master.001.patch > > > It is confusing as the FileNotFound is expected here. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HBASE-18381) HBase regionserver crashes when reading MOB file with column qualifier >64MB
[
https://issues.apache.org/jira/browse/HBASE-18381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089289#comment-16089289
]
Jingcheng Du edited comment on HBASE-18381 at 7/17/17 4:25 AM:
---
Thanks [~te...@apache.org], this is not related with MOB, this is because of
the default protobuf size limit of 64MB, non-MOB cells have the same limit too.
Your changes in HFile are not necessary in the patch, the test can pass with
only the changes in the configurations which means the test should pass if the
configuration is set properly in branch-2. And I don't think this would be an
issue in branch-2 and latter branches.
64MB is too large and we have filed a chapter to explain why we don't recommend
it in https://hbase.apache.org/book.html#faq. Besides, we have a test to
address the large cell size in TestMobStoreScanner#testGetMassive.
Hi [~huaxiang],
[~djelinski] ran into this issue in CDH release (on HBase 1.2.0-cdh5.10.0),
would you mind trying in your env? Thanks a lot.
was (Author: jingcheng.du):
Thanks [~te...@apache.org], this is not related with MOB, this is because of
the default protobuf size limit of 64MB, non-MOB cells have the same limit too.
Your changes in HFile are not necessary in the patch, the test can pass with
only the changes in the configurations which means the test should pass if the
configuration is set properly in branch-2. And I don't think this would be an
issue in branch-2 and latter branches.
64MB is too large and we have filed a chapter to explain why we don't recommend
it in https://hbase.apache.org/book.html#faq. Besides, we have a test to
address the large cell size in TestMobStoreScanner#testGetMassive.
Hi [~huaxiang], [~djelinski] ran into this issue in CDH release (on HBase
1.2.0-cdh5.10.0), would you mind trying in your env? Thanks a lot.
> HBase regionserver crashes when reading MOB file with column qualifier >64MB
>
>
> Key: HBASE-18381
> URL: https://issues.apache.org/jira/browse/HBASE-18381
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 2.0.0-alpha-1
> Environment: HBase 1.2.0-cdh5.10.0
>Reporter: Daniel Jelinski
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 2.0.0-beta-1
>
> Attachments: 18381.v1.txt, 18381.v2.txt
>
>
> After putting a cell with 64MB column qualifier to a MOB-enabled table,
> region server crashes when flushing data. Subsequent WAL replay attempts also
> result in region server crashes.
> Gist of code used to create the table:
> private String table = "poisonPill";
> private byte[] familyBytes = Bytes.toBytes("cf");
> private void createTable(Connection conn) throws IOException {
>Admin hbase_admin = conn.getAdmin();
>HTableDescriptor htable = new HTableDescriptor(TableName.valueOf(table));
>HColumnDescriptor hfamily = new HColumnDescriptor(familyBytes);
>hfamily.setMobEnabled(true);
>htable.setConfiguration("hfile.format.version","3");
>htable.addFamily(hfamily);
>hbase_admin.createTable(htable);
> }
> private void killTable(Connection conn) throws IOException {
>Table tbl = conn.getTable(TableName.valueOf(table));
>byte[] data = new byte[1<<26];
>byte[] smalldata = new byte[0];
>Put put = new Put(Bytes.toBytes("1"));
>put.addColumn(familyBytes, data, smalldata);
>tbl.put(put);
> }
> Region server logs (redacted):
> 2017-07-11 09:34:53,747 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Flushing 1/1 column families, memstore=64.00 MB; WAL is null, using passed
> sequenceid=7
> 2017-07-11 09:34:53,757 WARN org.apache.hadoop.hbase.io.hfile.HFileWriterV2:
> A minimum HFile version of 3 is required to support cell attributes/tags.
> Consider setting hfile.format.version accordingly.
> 2017-07-11 09:34:54,504 INFO
> org.apache.hadoop.hbase.mob.DefaultMobStoreFlusher: Flushed, sequenceid=7,
> memsize=67109096, hasBloomFilter=true, into tmp file
> hdfs://sandbox/hbase/data/default/poisonPill/f82e20f32302dfdd95c89ecc3be5a211/.tmp/7858d223eddd4199ad220fc77bb612eb
> 2017-07-11 09:34:54,694 ERROR org.apache.hadoop.hbase.regionserver.HStore:
> Failed to open store file :
> hdfs://sandbox/hbase/data/default/poisonPill/f82e20f32302dfdd95c89ecc3be5a211/.tmp/7858d223eddd4199ad220fc77bb612eb,
> keeping it in tmp location
> org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading HFile
> Trailer from file
> hdfs://sandbox/hbase/data/default/poisonPill/f82e20f32302dfdd95c89ecc3be5a211/.tmp/7858d223eddd4199ad220fc77bb612eb
> at
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:497)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:525)
> at
>
[jira] [Commented] (HBASE-18381) HBase regionserver crashes when reading MOB file with column qualifier >64MB
[
https://issues.apache.org/jira/browse/HBASE-18381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089289#comment-16089289
]
Jingcheng Du commented on HBASE-18381:
--
Thanks [~te...@apache.org], this is not related with MOB, this is because of
the default protobuf size limit of 64MB, non-MOB cells have the same limit too.
Your changes in HFile are not necessary in the patch, the test can pass with
only the changes in the configurations which means the test should pass if the
configuration is set properly in branch-2. And I don't think this would be an
issue in branch-2 and latter branches.
64MB is too large and we have filed a chapter to explain why we don't recommend
it in https://hbase.apache.org/book.html#faq. Besides, we have a test to
address the large cell size in TestMobStoreScanner#testGetMassive.
Hi [~huaxiang], [~djelinski] ran into this issue in CDH release (on HBase
1.2.0-cdh5.10.0), would you mind trying in your env? Thanks a lot.
> HBase regionserver crashes when reading MOB file with column qualifier >64MB
>
>
> Key: HBASE-18381
> URL: https://issues.apache.org/jira/browse/HBASE-18381
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 2.0.0-alpha-1
> Environment: HBase 1.2.0-cdh5.10.0
>Reporter: Daniel Jelinski
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 2.0.0-beta-1
>
> Attachments: 18381.v1.txt, 18381.v2.txt
>
>
> After putting a cell with 64MB column qualifier to a MOB-enabled table,
> region server crashes when flushing data. Subsequent WAL replay attempts also
> result in region server crashes.
> Gist of code used to create the table:
> private String table = "poisonPill";
> private byte[] familyBytes = Bytes.toBytes("cf");
> private void createTable(Connection conn) throws IOException {
>Admin hbase_admin = conn.getAdmin();
>HTableDescriptor htable = new HTableDescriptor(TableName.valueOf(table));
>HColumnDescriptor hfamily = new HColumnDescriptor(familyBytes);
>hfamily.setMobEnabled(true);
>htable.setConfiguration("hfile.format.version","3");
>htable.addFamily(hfamily);
>hbase_admin.createTable(htable);
> }
> private void killTable(Connection conn) throws IOException {
>Table tbl = conn.getTable(TableName.valueOf(table));
>byte[] data = new byte[1<<26];
>byte[] smalldata = new byte[0];
>Put put = new Put(Bytes.toBytes("1"));
>put.addColumn(familyBytes, data, smalldata);
>tbl.put(put);
> }
> Region server logs (redacted):
> 2017-07-11 09:34:53,747 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Flushing 1/1 column families, memstore=64.00 MB; WAL is null, using passed
> sequenceid=7
> 2017-07-11 09:34:53,757 WARN org.apache.hadoop.hbase.io.hfile.HFileWriterV2:
> A minimum HFile version of 3 is required to support cell attributes/tags.
> Consider setting hfile.format.version accordingly.
> 2017-07-11 09:34:54,504 INFO
> org.apache.hadoop.hbase.mob.DefaultMobStoreFlusher: Flushed, sequenceid=7,
> memsize=67109096, hasBloomFilter=true, into tmp file
> hdfs://sandbox/hbase/data/default/poisonPill/f82e20f32302dfdd95c89ecc3be5a211/.tmp/7858d223eddd4199ad220fc77bb612eb
> 2017-07-11 09:34:54,694 ERROR org.apache.hadoop.hbase.regionserver.HStore:
> Failed to open store file :
> hdfs://sandbox/hbase/data/default/poisonPill/f82e20f32302dfdd95c89ecc3be5a211/.tmp/7858d223eddd4199ad220fc77bb612eb,
> keeping it in tmp location
> org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading HFile
> Trailer from file
> hdfs://sandbox/hbase/data/default/poisonPill/f82e20f32302dfdd95c89ecc3be5a211/.tmp/7858d223eddd4199ad220fc77bb612eb
> at
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:497)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:525)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1105)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:265)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:404)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:509)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:499)
> at
> org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:675)
> at
> org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:667)
> at
> org.apache.hadoop.hbase.regionserver.HStore.validateStoreFile(HStore.java:1746)
> at
> org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:942)
> at
>
[jira] [Comment Edited] (HBASE-18379) SnapshotManager#checkSnapshotSupport() should better handle malfunctioning hdfs snapshot
[
https://issues.apache.org/jira/browse/HBASE-18379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16086906#comment-16086906
]
Jingcheng Du edited comment on HBASE-18379 at 7/14/17 6:24 AM:
---
I am okay with it. So which branch do we want to work against in this JIRA?
Just master branch?
was (Author: jingcheng.du):
I am okay with it. So which branch we want to against in this JIRA? Just master
branch?
> SnapshotManager#checkSnapshotSupport() should better handle malfunctioning
> hdfs snapshot
>
>
> Key: HBASE-18379
> URL: https://issues.apache.org/jira/browse/HBASE-18379
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>
> The following was observed by a customer which prevented master from coming
> up:
> {code}
> 2017-07-13 13:25:07,898 FATAL [xyz:16000.activeMasterManager] master.HMaster:
> Failed to become active master
> java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative
> path in absolute URI: Daily_Snapshot_Apps_2017-xx
> at org.apache.hadoop.fs.Path.initialize(Path.java:205)
> at org.apache.hadoop.fs.Path.(Path.java:171)
> at org.apache.hadoop.fs.Path.(Path.java:93)
> at
> org.apache.hadoop.hdfs.protocol.HdfsFileStatus.getFullPath(HdfsFileStatus.java:230)
> at
> org.apache.hadoop.hdfs.protocol.HdfsFileStatus.makeQualified(HdfsFileStatus.java:263)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:911)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:113)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:966)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:962)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:962)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1534)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1574)
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.getCompletedSnapshots(SnapshotManager.java:206)
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.checkSnapshotSupport(SnapshotManager.java:1011)
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.initialize(SnapshotManager.java:1070)
> at
> org.apache.hadoop.hbase.procedure.MasterProcedureManagerHost.initialize(MasterProcedureManagerHost.java:50)
> at
> org.apache.hadoop.hbase.master.HMaster.initializeZKBasedSystemTrackers(HMaster.java:667)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:732)
> at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:213)
> at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1863)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.net.URISyntaxException: Relative path in absolute URI:
> Daily_Snapshot_Apps_2017-xx
> at java.net.URI.checkPath(URI.java:1823)
> at java.net.URI.(URI.java:745)
> at org.apache.hadoop.fs.Path.initialize(Path.java:202)
> {code}
> Turns out the exception can be reproduced using hdfs command line accessing
> .snapshot directory.
> SnapshotManager#checkSnapshotSupport() should better handle malfunctioning
> hdfs snapshot so that master starts up.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18379) SnapshotManager#checkSnapshotSupport() should better handle malfunctioning hdfs snapshot
[
https://issues.apache.org/jira/browse/HBASE-18379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16086906#comment-16086906
]
Jingcheng Du commented on HBASE-18379:
--
I am okay with it. So which branch we want to against in this JIRA? Just master
branch?
> SnapshotManager#checkSnapshotSupport() should better handle malfunctioning
> hdfs snapshot
>
>
> Key: HBASE-18379
> URL: https://issues.apache.org/jira/browse/HBASE-18379
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>
> The following was observed by a customer which prevented master from coming
> up:
> {code}
> 2017-07-13 13:25:07,898 FATAL [xyz:16000.activeMasterManager] master.HMaster:
> Failed to become active master
> java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative
> path in absolute URI: Daily_Snapshot_Apps_2017-xx
> at org.apache.hadoop.fs.Path.initialize(Path.java:205)
> at org.apache.hadoop.fs.Path.(Path.java:171)
> at org.apache.hadoop.fs.Path.(Path.java:93)
> at
> org.apache.hadoop.hdfs.protocol.HdfsFileStatus.getFullPath(HdfsFileStatus.java:230)
> at
> org.apache.hadoop.hdfs.protocol.HdfsFileStatus.makeQualified(HdfsFileStatus.java:263)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:911)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:113)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:966)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:962)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:962)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1534)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1574)
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.getCompletedSnapshots(SnapshotManager.java:206)
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.checkSnapshotSupport(SnapshotManager.java:1011)
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.initialize(SnapshotManager.java:1070)
> at
> org.apache.hadoop.hbase.procedure.MasterProcedureManagerHost.initialize(MasterProcedureManagerHost.java:50)
> at
> org.apache.hadoop.hbase.master.HMaster.initializeZKBasedSystemTrackers(HMaster.java:667)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:732)
> at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:213)
> at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1863)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.net.URISyntaxException: Relative path in absolute URI:
> Daily_Snapshot_Apps_2017-xx
> at java.net.URI.checkPath(URI.java:1823)
> at java.net.URI.(URI.java:745)
> at org.apache.hadoop.fs.Path.initialize(Path.java:202)
> {code}
> Turns out the exception can be reproduced using hdfs command line accessing
> .snapshot directory.
> SnapshotManager#checkSnapshotSupport() should better handle malfunctioning
> hdfs snapshot so that master starts up.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18379) SnapshotManager#checkSnapshotSupport() should better handle malfunctioning hdfs snapshot
[
https://issues.apache.org/jira/browse/HBASE-18379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16086833#comment-16086833
]
Jingcheng Du commented on HBASE-18379:
--
It is the place. But the thrown exception is IllegalArgumentException which is
a RuntimeException.
So we need catch IOE and this RuntimeException at this time, and add another
RuntimeException when we find in the future? Or directly catch an Exception
here?
I am fine to catch exceptions here for checking the snapshots in the old
snapshot directory. But cannot confirm if the change is necessary for the path
issues.
Your ideas? Thanks.
> SnapshotManager#checkSnapshotSupport() should better handle malfunctioning
> hdfs snapshot
>
>
> Key: HBASE-18379
> URL: https://issues.apache.org/jira/browse/HBASE-18379
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>
> The following was observed by a customer which prevented master from coming
> up:
> {code}
> 2017-07-13 13:25:07,898 FATAL [xyz:16000.activeMasterManager] master.HMaster:
> Failed to become active master
> java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative
> path in absolute URI: Daily_Snapshot_Apps_2017-xx
> at org.apache.hadoop.fs.Path.initialize(Path.java:205)
> at org.apache.hadoop.fs.Path.(Path.java:171)
> at org.apache.hadoop.fs.Path.(Path.java:93)
> at
> org.apache.hadoop.hdfs.protocol.HdfsFileStatus.getFullPath(HdfsFileStatus.java:230)
> at
> org.apache.hadoop.hdfs.protocol.HdfsFileStatus.makeQualified(HdfsFileStatus.java:263)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:911)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:113)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:966)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:962)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:962)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1534)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1574)
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.getCompletedSnapshots(SnapshotManager.java:206)
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.checkSnapshotSupport(SnapshotManager.java:1011)
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.initialize(SnapshotManager.java:1070)
> at
> org.apache.hadoop.hbase.procedure.MasterProcedureManagerHost.initialize(MasterProcedureManagerHost.java:50)
> at
> org.apache.hadoop.hbase.master.HMaster.initializeZKBasedSystemTrackers(HMaster.java:667)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:732)
> at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:213)
> at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1863)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.net.URISyntaxException: Relative path in absolute URI:
> Daily_Snapshot_Apps_2017-xx
> at java.net.URI.checkPath(URI.java:1823)
> at java.net.URI.(URI.java:745)
> at org.apache.hadoop.fs.Path.initialize(Path.java:202)
> {code}
> Turns out the exception can be reproduced using hdfs command line accessing
> .snapshot directory.
> SnapshotManager#checkSnapshotSupport() should better handle malfunctioning
> hdfs snapshot so that master starts up.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (HBASE-18379) SnapshotManager#checkSnapshotSupport() should better handle malfunctioning hdfs snapshot
[
https://issues.apache.org/jira/browse/HBASE-18379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16086759#comment-16086759
]
Jingcheng Du edited comment on HBASE-18379 at 7/14/17 2:52 AM:
---
Thanks [~te...@apache.org].
I checked the code in URI, this exception happens only when it has a schema,
but the left part doesn't start with /.
According to the stack trace, the whole path is Daily_Snapshot_Apps_2017-xx, no
any schema (the part before ":"), this should pass the new Path step.
Do you have ideas why this happens? This just tries to create a path instance.
was (Author: jingcheng.du):
Thanks [~te...@apache.org].
I checked the code in URI, this exception happens only when it has a schema,
but the left part doesn't start with /.
According to the stack trace, the whole path is Daily_Snapshot_Apps_2017-xx, no
any schema (the part before ":"), this should can pass the new Path step.
Do you have ideas why this happens? This just tries to create a path instance.
> SnapshotManager#checkSnapshotSupport() should better handle malfunctioning
> hdfs snapshot
>
>
> Key: HBASE-18379
> URL: https://issues.apache.org/jira/browse/HBASE-18379
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>
> The following was observed by a customer which prevented master from coming
> up:
> {code}
> 2017-07-13 13:25:07,898 FATAL [xyz:16000.activeMasterManager] master.HMaster:
> Failed to become active master
> java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative
> path in absolute URI: Daily_Snapshot_Apps_2017-xx
> at org.apache.hadoop.fs.Path.initialize(Path.java:205)
> at org.apache.hadoop.fs.Path.(Path.java:171)
> at org.apache.hadoop.fs.Path.(Path.java:93)
> at
> org.apache.hadoop.hdfs.protocol.HdfsFileStatus.getFullPath(HdfsFileStatus.java:230)
> at
> org.apache.hadoop.hdfs.protocol.HdfsFileStatus.makeQualified(HdfsFileStatus.java:263)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:911)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:113)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:966)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:962)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:962)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1534)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1574)
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.getCompletedSnapshots(SnapshotManager.java:206)
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.checkSnapshotSupport(SnapshotManager.java:1011)
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.initialize(SnapshotManager.java:1070)
> at
> org.apache.hadoop.hbase.procedure.MasterProcedureManagerHost.initialize(MasterProcedureManagerHost.java:50)
> at
> org.apache.hadoop.hbase.master.HMaster.initializeZKBasedSystemTrackers(HMaster.java:667)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:732)
> at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:213)
> at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1863)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.net.URISyntaxException: Relative path in absolute URI:
> Daily_Snapshot_Apps_2017-xx
> at java.net.URI.checkPath(URI.java:1823)
> at java.net.URI.(URI.java:745)
> at org.apache.hadoop.fs.Path.initialize(Path.java:202)
> {code}
> Turns out the exception can be reproduced using hdfs command line accessing
> .snapshot directory.
> SnapshotManager#checkSnapshotSupport() should better handle malfunctioning
> hdfs snapshot so that master starts up.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18379) SnapshotManager#checkSnapshotSupport() should better handle malfunctioning hdfs snapshot
[
https://issues.apache.org/jira/browse/HBASE-18379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16086759#comment-16086759
]
Jingcheng Du commented on HBASE-18379:
--
Thanks [~te...@apache.org].
I checked the code in URI, this exception happens only when it has a schema,
but the left part doesn't start with /.
According to the stack trace, the whole path is Daily_Snapshot_Apps_2017-xx, no
any schema (the part before ":"), this should can pass the new Path step.
Do you have ideas why this happens? This just tries to create a path instance.
> SnapshotManager#checkSnapshotSupport() should better handle malfunctioning
> hdfs snapshot
>
>
> Key: HBASE-18379
> URL: https://issues.apache.org/jira/browse/HBASE-18379
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>
> The following was observed by a customer which prevented master from coming
> up:
> {code}
> 2017-07-13 13:25:07,898 FATAL [xyz:16000.activeMasterManager] master.HMaster:
> Failed to become active master
> java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative
> path in absolute URI: Daily_Snapshot_Apps_2017-xx
> at org.apache.hadoop.fs.Path.initialize(Path.java:205)
> at org.apache.hadoop.fs.Path.(Path.java:171)
> at org.apache.hadoop.fs.Path.(Path.java:93)
> at
> org.apache.hadoop.hdfs.protocol.HdfsFileStatus.getFullPath(HdfsFileStatus.java:230)
> at
> org.apache.hadoop.hdfs.protocol.HdfsFileStatus.makeQualified(HdfsFileStatus.java:263)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:911)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:113)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:966)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:962)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:962)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1534)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1574)
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.getCompletedSnapshots(SnapshotManager.java:206)
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.checkSnapshotSupport(SnapshotManager.java:1011)
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.initialize(SnapshotManager.java:1070)
> at
> org.apache.hadoop.hbase.procedure.MasterProcedureManagerHost.initialize(MasterProcedureManagerHost.java:50)
> at
> org.apache.hadoop.hbase.master.HMaster.initializeZKBasedSystemTrackers(HMaster.java:667)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:732)
> at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:213)
> at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1863)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.net.URISyntaxException: Relative path in absolute URI:
> Daily_Snapshot_Apps_2017-xx
> at java.net.URI.checkPath(URI.java:1823)
> at java.net.URI.(URI.java:745)
> at org.apache.hadoop.fs.Path.initialize(Path.java:202)
> {code}
> Turns out the exception can be reproduced using hdfs command line accessing
> .snapshot directory.
> SnapshotManager#checkSnapshotSupport() should better handle malfunctioning
> hdfs snapshot so that master starts up.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18312) Ineffective handling of FileNotFoundException in FileLink$FileLinkInputStream.tryOpen()
[
https://issues.apache.org/jira/browse/HBASE-18312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16075772#comment-16075772
]
Jingcheng Du commented on HBASE-18312:
--
+1.
> Ineffective handling of FileNotFoundException in
> FileLink$FileLinkInputStream.tryOpen()
> ---
>
> Key: HBASE-18312
> URL: https://issues.apache.org/jira/browse/HBASE-18312
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
> Attachments: 18312.v1.txt, 18312.v2.txt, 18312.v4.txt, 18312.v5.txt,
> 18312.v6.txt
>
>
> Found the following in region server log:
> {code}
> 2017-07-03 11:22:04,669 WARN
> [regionserver/a.b.c.d:16020-shortCompactions-1499094046361]
> retry.RetryInvocationHandler: Exception while invoking
> ClientNamenodeProtocolTranslatorPB.getBlockLocations over e.f.g.h:8020. Not
> retrying because try once and fail.
> org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File
> does not exist: /hbase/data/default/X/4d61af9d1cbcc5fe2a5cbddbfc92fe7e/
> K/47222a9cbd294f499f49de92ecf330ee
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61)
> ...
> at
> org.apache.hadoop.hbase.io.FileLink$FileLinkInputStream.tryOpen(FileLink.java:291)
> at
> org.apache.hadoop.hbase.io.FileLink$FileLinkInputStream.(FileLink.java:122)
> at
> org.apache.hadoop.hbase.io.FileLink$FileLinkInputStream.(FileLink.java:113)
> at org.apache.hadoop.hbase.io.FileLink.open(FileLink.java:404)
> at
> org.apache.hadoop.hbase.io.FSDataInputStreamWrapper.(FSDataInputStreamWrapper.java:98)
> at
> org.apache.hadoop.hbase.io.FSDataInputStreamWrapper.(FSDataInputStreamWrapper.java:83)
> {code}
> Here is related code:
> {code}
> private FSDataInputStream tryOpen() throws IOException {
> for (Path path: fileLink.getLocations()) {
> ...
> } catch (FileNotFoundException e) {
> // Try another file location
> }
> {code}
> The intention is to try possible locations for the linked file.
> However, RemoteException was the exception encountered. This makes the above
> catch clause ineffective.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18312) Ineffective handling of FileNotFoundException in FileLink$FileLinkInputStream.tryOpen()
[
https://issues.apache.org/jira/browse/HBASE-18312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16074557#comment-16074557
]
Jingcheng Du commented on HBASE-18312:
--
Revert the changes of the modifier of FileLinkInputStream from default back to
private?
> Ineffective handling of FileNotFoundException in
> FileLink$FileLinkInputStream.tryOpen()
> ---
>
> Key: HBASE-18312
> URL: https://issues.apache.org/jira/browse/HBASE-18312
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
> Attachments: 18312.v1.txt, 18312.v2.txt, 18312.v4.txt, 18312.v5.txt
>
>
> Found the following in region server log:
> {code}
> 2017-07-03 11:22:04,669 WARN
> [regionserver/a.b.c.d:16020-shortCompactions-1499094046361]
> retry.RetryInvocationHandler: Exception while invoking
> ClientNamenodeProtocolTranslatorPB.getBlockLocations over e.f.g.h:8020. Not
> retrying because try once and fail.
> org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File
> does not exist: /hbase/data/default/X/4d61af9d1cbcc5fe2a5cbddbfc92fe7e/
> K/47222a9cbd294f499f49de92ecf330ee
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61)
> ...
> at
> org.apache.hadoop.hbase.io.FileLink$FileLinkInputStream.tryOpen(FileLink.java:291)
> at
> org.apache.hadoop.hbase.io.FileLink$FileLinkInputStream.(FileLink.java:122)
> at
> org.apache.hadoop.hbase.io.FileLink$FileLinkInputStream.(FileLink.java:113)
> at org.apache.hadoop.hbase.io.FileLink.open(FileLink.java:404)
> at
> org.apache.hadoop.hbase.io.FSDataInputStreamWrapper.(FSDataInputStreamWrapper.java:98)
> at
> org.apache.hadoop.hbase.io.FSDataInputStreamWrapper.(FSDataInputStreamWrapper.java:83)
> {code}
> Here is related code:
> {code}
> private FSDataInputStream tryOpen() throws IOException {
> for (Path path: fileLink.getLocations()) {
> ...
> } catch (FileNotFoundException e) {
> // Try another file location
> }
> {code}
> The intention is to try possible locations for the linked file.
> However, RemoteException was the exception encountered. This makes the above
> catch clause ineffective.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (HBASE-18312) Ineffective handling of FileNotFoundException in FileLink$FileLinkInputStream.tryOpen()
[
https://issues.apache.org/jira/browse/HBASE-18312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16074557#comment-16074557
]
Jingcheng Du edited comment on HBASE-18312 at 7/5/17 10:39 AM:
---
bq. static class FileLinkInputStream extends InputStream
Revert the changes of the modifier of FileLinkInputStream from default back to
private?
was (Author: jingcheng.du):
Revert the changes of the modifier of FileLinkInputStream from default back to
private?
> Ineffective handling of FileNotFoundException in
> FileLink$FileLinkInputStream.tryOpen()
> ---
>
> Key: HBASE-18312
> URL: https://issues.apache.org/jira/browse/HBASE-18312
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
> Attachments: 18312.v1.txt, 18312.v2.txt, 18312.v4.txt, 18312.v5.txt
>
>
> Found the following in region server log:
> {code}
> 2017-07-03 11:22:04,669 WARN
> [regionserver/a.b.c.d:16020-shortCompactions-1499094046361]
> retry.RetryInvocationHandler: Exception while invoking
> ClientNamenodeProtocolTranslatorPB.getBlockLocations over e.f.g.h:8020. Not
> retrying because try once and fail.
> org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File
> does not exist: /hbase/data/default/X/4d61af9d1cbcc5fe2a5cbddbfc92fe7e/
> K/47222a9cbd294f499f49de92ecf330ee
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61)
> ...
> at
> org.apache.hadoop.hbase.io.FileLink$FileLinkInputStream.tryOpen(FileLink.java:291)
> at
> org.apache.hadoop.hbase.io.FileLink$FileLinkInputStream.(FileLink.java:122)
> at
> org.apache.hadoop.hbase.io.FileLink$FileLinkInputStream.(FileLink.java:113)
> at org.apache.hadoop.hbase.io.FileLink.open(FileLink.java:404)
> at
> org.apache.hadoop.hbase.io.FSDataInputStreamWrapper.(FSDataInputStreamWrapper.java:98)
> at
> org.apache.hadoop.hbase.io.FSDataInputStreamWrapper.(FSDataInputStreamWrapper.java:83)
> {code}
> Here is related code:
> {code}
> private FSDataInputStream tryOpen() throws IOException {
> for (Path path: fileLink.getLocations()) {
> ...
> } catch (FileNotFoundException e) {
> // Try another file location
> }
> {code}
> The intention is to try possible locations for the linked file.
> However, RemoteException was the exception encountered. This makes the above
> catch clause ineffective.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18312) Ineffective handling of FileNotFoundException in FileLink$FileLinkInputStream.tryOpen()
[
https://issues.apache.org/jira/browse/HBASE-18312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16074202#comment-16074202
]
Jingcheng Du commented on HBASE-18312:
--
Thanks [~te...@apache.org].
bq. FileLink.FileLinkInputStream stream = new FileLink.FileLinkInputStream(fs,
link, 1024);
Mind using FileLink.open() to instead of the above, and close the FileLink
after that? I am +1 after this. Thanks!
> Ineffective handling of FileNotFoundException in
> FileLink$FileLinkInputStream.tryOpen()
> ---
>
> Key: HBASE-18312
> URL: https://issues.apache.org/jira/browse/HBASE-18312
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
> Attachments: 18312.v1.txt, 18312.v2.txt, 18312.v4.txt
>
>
> Found the following in region server log:
> {code}
> 2017-07-03 11:22:04,669 WARN
> [regionserver/a.b.c.d:16020-shortCompactions-1499094046361]
> retry.RetryInvocationHandler: Exception while invoking
> ClientNamenodeProtocolTranslatorPB.getBlockLocations over e.f.g.h:8020. Not
> retrying because try once and fail.
> org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File
> does not exist: /hbase/data/default/X/4d61af9d1cbcc5fe2a5cbddbfc92fe7e/
> K/47222a9cbd294f499f49de92ecf330ee
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61)
> ...
> at
> org.apache.hadoop.hbase.io.FileLink$FileLinkInputStream.tryOpen(FileLink.java:291)
> at
> org.apache.hadoop.hbase.io.FileLink$FileLinkInputStream.(FileLink.java:122)
> at
> org.apache.hadoop.hbase.io.FileLink$FileLinkInputStream.(FileLink.java:113)
> at org.apache.hadoop.hbase.io.FileLink.open(FileLink.java:404)
> at
> org.apache.hadoop.hbase.io.FSDataInputStreamWrapper.(FSDataInputStreamWrapper.java:98)
> at
> org.apache.hadoop.hbase.io.FSDataInputStreamWrapper.(FSDataInputStreamWrapper.java:83)
> {code}
> Here is related code:
> {code}
> private FSDataInputStream tryOpen() throws IOException {
> for (Path path: fileLink.getLocations()) {
> ...
> } catch (FileNotFoundException e) {
> // Try another file location
> }
> {code}
> The intention is to try possible locations for the linked file.
> However, RemoteException was the exception encountered. This makes the above
> catch clause ineffective.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18312) Ineffective handling of FileNotFoundException in FileLink$FileLinkInputStream.tryOpen()
[
https://issues.apache.org/jira/browse/HBASE-18312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16073516#comment-16073516
]
Jingcheng Du commented on HBASE-18312:
--
Thanks Ted, mind providing an unit test for this?
> Ineffective handling of FileNotFoundException in
> FileLink$FileLinkInputStream.tryOpen()
> ---
>
> Key: HBASE-18312
> URL: https://issues.apache.org/jira/browse/HBASE-18312
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
> Attachments: 18312.v1.txt, 18312.v2.txt
>
>
> Found the following in region server log:
> {code}
> 2017-07-03 11:22:04,669 WARN
> [regionserver/a.b.c.d:16020-shortCompactions-1499094046361]
> retry.RetryInvocationHandler: Exception while invoking
> ClientNamenodeProtocolTranslatorPB.getBlockLocations over e.f.g.h:8020. Not
> retrying because try once and fail.
> org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File
> does not exist: /hbase/data/default/X/4d61af9d1cbcc5fe2a5cbddbfc92fe7e/
> K/47222a9cbd294f499f49de92ecf330ee
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61)
> ...
> at
> org.apache.hadoop.hbase.io.FileLink$FileLinkInputStream.tryOpen(FileLink.java:291)
> at
> org.apache.hadoop.hbase.io.FileLink$FileLinkInputStream.(FileLink.java:122)
> at
> org.apache.hadoop.hbase.io.FileLink$FileLinkInputStream.(FileLink.java:113)
> at org.apache.hadoop.hbase.io.FileLink.open(FileLink.java:404)
> at
> org.apache.hadoop.hbase.io.FSDataInputStreamWrapper.(FSDataInputStreamWrapper.java:98)
> at
> org.apache.hadoop.hbase.io.FSDataInputStreamWrapper.(FSDataInputStreamWrapper.java:83)
> {code}
> Here is related code:
> {code}
> private FSDataInputStream tryOpen() throws IOException {
> for (Path path: fileLink.getLocations()) {
> ...
> } catch (FileNotFoundException e) {
> // Try another file location
> }
> {code}
> The intention is to try possible locations for the linked file.
> However, RemoteException was the exception encountered. This makes the above
> catch clause ineffective.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18287) Remove log warning in PartitionedMobCompactor.java#getFileStatus
[ https://issues.apache.org/jira/browse/HBASE-18287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16067677#comment-16067677 ] Jingcheng Du commented on HBASE-18287: -- Sorry, I misunderstood the purpose. You are right, the warning message doesn't make sense in each look up, we can refactor the code in HFileLink way, we can remove the method PartitionedMobCompactor#getFileStatus and move the fs.getFileStatus to ParititionedMobCompactor#getLinkedFileStatus. But the difference is we don't throw the FNNFE at the end of the method, we just log warning when we cannot find the files in all possible locations. How about this? > Remove log warning in PartitionedMobCompactor.java#getFileStatus > - > > Key: HBASE-18287 > URL: https://issues.apache.org/jira/browse/HBASE-18287 > Project: HBase > Issue Type: Bug > Components: mob >Affects Versions: 2.0.0-alpha-1 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > > It is confusing as the FileNotFound is expected here. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18287) Remove log warning in PartitionedMobCompactor.java#getFileStatus
[ https://issues.apache.org/jira/browse/HBASE-18287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16067599#comment-16067599 ] Jingcheng Du commented on HBASE-18287: -- Thanks [~huaxiang]. Actually you can read the java doc of this API, it tells it throws two kinds of Exception, one is FNNFE, the other is IOE although only IOE is declared in the method. We have the similar usages in HFileLink. I think it is okay to keep the current code. > Remove log warning in PartitionedMobCompactor.java#getFileStatus > - > > Key: HBASE-18287 > URL: https://issues.apache.org/jira/browse/HBASE-18287 > Project: HBase > Issue Type: Bug > Components: mob >Affects Versions: 2.0.0-alpha-1 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > > It is confusing as the FileNotFound is expected here. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18049) It is not necessary to re-open the region when MOB files cannot be found
[
https://issues.apache.org/jira/browse/HBASE-18049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingcheng Du updated HBASE-18049:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
> It is not necessary to re-open the region when MOB files cannot be found
>
>
> Key: HBASE-18049
> URL: https://issues.apache.org/jira/browse/HBASE-18049
> Project: HBase
> Issue Type: Bug
> Components: Scanners
>Reporter: Jingcheng Du
>Assignee: Jingcheng Du
> Fix For: 2.0.0
>
> Attachments: HBASE-18049.patch, HBASE-18049-V2.patch,
> HBASE-18049-V3.patch
>
>
> In HBASE-17712, we try to re-open the region when store files cannot be
> found. This is useful for store files in a region, but is not necessary when
> the MOB files cannot be found, because the store files in a region only
> contain the references to the MOB files and a re-open of a region doesn't
> help the lost MOB files.
> In this JIRA, we will directly throw DNRIOE only when the MOB files are not
> found in {{MobStoreScanner}} and {{ReversedMobStoreScanner}}. Other logics
> keep the same.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-18049) It is not necessary to re-open the region when MOB files cannot be found
[
https://issues.apache.org/jira/browse/HBASE-18049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16015584#comment-16015584
]
Jingcheng Du commented on HBASE-18049:
--
Push to master branch. Thanks [~yuzhih...@gmail.com] and [~anoop.hbase] for the
review and comments.
> It is not necessary to re-open the region when MOB files cannot be found
>
>
> Key: HBASE-18049
> URL: https://issues.apache.org/jira/browse/HBASE-18049
> Project: HBase
> Issue Type: Bug
> Components: Scanners
>Reporter: Jingcheng Du
>Assignee: Jingcheng Du
> Fix For: 2.0.0
>
> Attachments: HBASE-18049.patch, HBASE-18049-V2.patch,
> HBASE-18049-V3.patch
>
>
> In HBASE-17712, we try to re-open the region when store files cannot be
> found. This is useful for store files in a region, but is not necessary when
> the MOB files cannot be found, because the store files in a region only
> contain the references to the MOB files and a re-open of a region doesn't
> help the lost MOB files.
> In this JIRA, we will directly throw DNRIOE only when the MOB files are not
> found in {{MobStoreScanner}} and {{ReversedMobStoreScanner}}. Other logics
> keep the same.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-18049) It is not necessary to re-open the region when MOB files cannot be found
[
https://issues.apache.org/jira/browse/HBASE-18049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingcheng Du updated HBASE-18049:
-
Attachment: HBASE-18049-V3.patch
Upload a new patch according to [~anoop.hbase]'s comments. Thanks.
Will commit this patch soon.
> It is not necessary to re-open the region when MOB files cannot be found
>
>
> Key: HBASE-18049
> URL: https://issues.apache.org/jira/browse/HBASE-18049
> Project: HBase
> Issue Type: Bug
> Components: Scanners
>Reporter: Jingcheng Du
>Assignee: Jingcheng Du
> Fix For: 2.0.0
>
> Attachments: HBASE-18049.patch, HBASE-18049-V2.patch,
> HBASE-18049-V3.patch
>
>
> In HBASE-17712, we try to re-open the region when store files cannot be
> found. This is useful for store files in a region, but is not necessary when
> the MOB files cannot be found, because the store files in a region only
> contain the references to the MOB files and a re-open of a region doesn't
> help the lost MOB files.
> In this JIRA, we will directly throw DNRIOE only when the MOB files are not
> found in {{MobStoreScanner}} and {{ReversedMobStoreScanner}}. Other logics
> keep the same.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-18049) It is not necessary to re-open the region when MOB files cannot be found
[
https://issues.apache.org/jira/browse/HBASE-18049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingcheng Du updated HBASE-18049:
-
Attachment: HBASE-18049-V2.patch
Refine the patch according to [~anoop.hbase]'s comments.
Hi [~anoop.hbase], the patch V2 is okay to commit? Thanks.
> It is not necessary to re-open the region when MOB files cannot be found
>
>
> Key: HBASE-18049
> URL: https://issues.apache.org/jira/browse/HBASE-18049
> Project: HBase
> Issue Type: Bug
> Components: Scanners
>Reporter: Jingcheng Du
>Assignee: Jingcheng Du
> Fix For: 2.0.0
>
> Attachments: HBASE-18049.patch, HBASE-18049-V2.patch
>
>
> In HBASE-17712, we try to re-open the region when store files cannot be
> found. This is useful for store files in a region, but is not necessary when
> the MOB files cannot be found, because the store files in a region only
> contain the references to the MOB files and a re-open of a region doesn't
> help the lost MOB files.
> In this JIRA, we will directly throw DNRIOE only when the MOB files are not
> found in {{MobStoreScanner}} and {{ReversedMobStoreScanner}}. Other logics
> keep the same.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-18049) It is not necessary to re-open the region when MOB files cannot be found
[
https://issues.apache.org/jira/browse/HBASE-18049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011767#comment-16011767
]
Jingcheng Du commented on HBASE-18049:
--
Thanks [~yuzhih...@gmail.com] and [~anoop.hbase] for the comments!
bq. So here in new change as well, we need getCause() check?
Right, we should add this.
bq. A solution to restore the state and data is to do a MOB major compaction
right?
Right, this is a simple solution which can probably fix the issue. But I think
users need to investigate the cases a little before running such commands.
Given such info in logs might mislead users to directly run the compaction? I
think we'd better not add this for now?
> It is not necessary to re-open the region when MOB files cannot be found
>
>
> Key: HBASE-18049
> URL: https://issues.apache.org/jira/browse/HBASE-18049
> Project: HBase
> Issue Type: Bug
> Components: Scanners
>Reporter: Jingcheng Du
>Assignee: Jingcheng Du
> Fix For: 2.0.0
>
> Attachments: HBASE-18049.patch
>
>
> In HBASE-17712, we try to re-open the region when store files cannot be
> found. This is useful for store files in a region, but is not necessary when
> the MOB files cannot be found, because the store files in a region only
> contain the references to the MOB files and a re-open of a region doesn't
> help the lost MOB files.
> In this JIRA, we will directly throw DNRIOE only when the MOB files are not
> found in {{MobStoreScanner}} and {{ReversedMobStoreScanner}}. Other logics
> keep the same.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-18049) It is not necessary to re-open the region when MOB files cannot be found
[
https://issues.apache.org/jira/browse/HBASE-18049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingcheng Du updated HBASE-18049:
-
Description:
In HBASE-17712, we try to re-open the region when store files cannot be found.
This is useful for store files in a region, but is not necessary when the MOB
files cannot be found, because the store files in a region only contain the
references to the MOB files and a re-open of a region doesn't help the lost MOB
files.
In this JIRA, we will directly throw DNRIOE only when the MOB files are not
found in {{MobStoreScanner}} and {{ReversedMobStoreScanner}}. Other logics keep
the same.
was:
In HBASE-17712, we try to re-open the region when store files cannot be found.
This is useful for store files in a region, but is not necessary when the MOB
files cannot be found because the store files in a region only contain the
references to the MOB files.
In this JIRA, we will directly throw DNRIOE only when the MOB files are not
found in {{MobStoreScanner}} and {{ReversedMobStoreScanner}}. Other logics keep
the same.
> It is not necessary to re-open the region when MOB files cannot be found
>
>
> Key: HBASE-18049
> URL: https://issues.apache.org/jira/browse/HBASE-18049
> Project: HBase
> Issue Type: Bug
> Components: Scanners
>Reporter: Jingcheng Du
>Assignee: Jingcheng Du
> Fix For: 2.0.0
>
> Attachments: HBASE-18049.patch
>
>
> In HBASE-17712, we try to re-open the region when store files cannot be
> found. This is useful for store files in a region, but is not necessary when
> the MOB files cannot be found, because the store files in a region only
> contain the references to the MOB files and a re-open of a region doesn't
> help the lost MOB files.
> In this JIRA, we will directly throw DNRIOE only when the MOB files are not
> found in {{MobStoreScanner}} and {{ReversedMobStoreScanner}}. Other logics
> keep the same.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-18049) It is not necessary to re-open the region when MOB files cannot be found
[
https://issues.apache.org/jira/browse/HBASE-18049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingcheng Du updated HBASE-18049:
-
Attachment: HBASE-18049.patch
Upload the first patch for review.
> It is not necessary to re-open the region when MOB files cannot be found
>
>
> Key: HBASE-18049
> URL: https://issues.apache.org/jira/browse/HBASE-18049
> Project: HBase
> Issue Type: Bug
> Components: Scanners
>Reporter: Jingcheng Du
>Assignee: Jingcheng Du
> Fix For: 2.0.0
>
> Attachments: HBASE-18049.patch
>
>
> In HBASE-17712, we try to re-open the region when store files cannot be
> found. This is useful for store files in a region, but is not necessary when
> the MOB files cannot be found because the store files in a region only
> contain the references to the MOB files.
> In this JIRA, we will directly throw DNRIOE only when the MOB files are not
> found in {{MobStoreScanner}} and {{ReversedMobStoreScanner}}. Other logics
> keep the same.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-18049) It is not necessary to re-open the region when MOB files cannot be found
[
https://issues.apache.org/jira/browse/HBASE-18049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingcheng Du updated HBASE-18049:
-
Status: Patch Available (was: Open)
> It is not necessary to re-open the region when MOB files cannot be found
>
>
> Key: HBASE-18049
> URL: https://issues.apache.org/jira/browse/HBASE-18049
> Project: HBase
> Issue Type: Bug
> Components: Scanners
>Reporter: Jingcheng Du
>Assignee: Jingcheng Du
> Fix For: 2.0.0
>
> Attachments: HBASE-18049.patch
>
>
> In HBASE-17712, we try to re-open the region when store files cannot be
> found. This is useful for store files in a region, but is not necessary when
> the MOB files cannot be found because the store files in a region only
> contain the references to the MOB files.
> In this JIRA, we will directly throw DNRIOE only when the MOB files are not
> found in {{MobStoreScanner}} and {{ReversedMobStoreScanner}}. Other logics
> keep the same.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (HBASE-18049) It is not necessary to re-open the region when MOB files cannot be found
Jingcheng Du created HBASE-18049:
Summary: It is not necessary to re-open the region when MOB files
cannot be found
Key: HBASE-18049
URL: https://issues.apache.org/jira/browse/HBASE-18049
Project: HBase
Issue Type: Bug
Components: Scanners
Reporter: Jingcheng Du
Assignee: Jingcheng Du
Fix For: 2.0.0
In HBASE-17712, we try to re-open the region when store files cannot be found.
This is useful for store files in a region, but is not necessary when the MOB
files cannot be found because the store files in a region only contain the
references to the MOB files.
In this JIRA, we will directly throw DNRIOE only when the MOB files are not
found in {{MobStoreScanner}} and {{ReversedMobStoreScanner}}. Other logics keep
the same.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HBASE-17690) Clean up MOB code
[ https://issues.apache.org/jira/browse/HBASE-17690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingcheng Du reassigned HBASE-17690: Assignee: Jingcheng Du (was: Xiang Li) > Clean up MOB code > - > > Key: HBASE-17690 > URL: https://issues.apache.org/jira/browse/HBASE-17690 > Project: HBase > Issue Type: Improvement > Components: mob >Reporter: Jingcheng Du >Assignee: Jingcheng Du > Fix For: 2.0.0 > > Attachments: HBASE-17690.patch, HBASE-17690-v1.patch > > > Clean up the code in MOB. > # Fix the incorrect description in comments. > # Fix the warning and remove redundant import in code. > # Remove the references to the deprecated code. > # Add throughput controller for DefaultMobStoreFlusher and > DefaultMobStoreCompactor. > The RB link is https://reviews.apache.org/r/57078/. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HBASE-17138) Backport read-path offheap (HBASE-11425) to branch-1
[
https://issues.apache.org/jira/browse/HBASE-17138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15994227#comment-15994227
]
Jingcheng Du edited comment on HBASE-17138 at 5/3/17 3:52 PM:
--
The changes in {{RegionScanner}} in master branch is neither binary compatible
nor source compatible for branch-1, and changes in {{RegionCoprocessorHost}}
might be only binary compatible. If users move to a new 1.x version which
includes the off-heap read path, they have to change the customized region
scanners and coprocessors.
Do we allow this kind of backward compatibility break in minor release?
[~busbey] [~anoop.hbase] [~Apache9]
According to the compatibility matrix in HBase
(https://hbase.apache.org/book.html#hbase.versioning), such changes are
Server-Side Limited API compatibility, but the changes in {{RegionScanner}}
break both binary and source compatibility.
was (Author: jingcheng.du):
The changes in {{RegionScanner}} in master branch is neither binary compatible
nor source compatible for branch-1, and changes in {{RegionCoprocessorHost}}
might be only binary compatible. If users move to a new 1.x version which
includes the off-heap read path, they have to change the customized region
scanners and coprocessors.
Do we allow this kind of backward compatibility break in minor release?
[~busbey] [~anoop.hbase] [~Apache9]
According to the compatibility matrix in HBase
(https://hbase.apache.org/book.html#hbase.versioning), such changes are
Server-Side Limited API compatibility, it is allowed in HBase minor releases.
> Backport read-path offheap (HBASE-11425) to branch-1
>
>
> Key: HBASE-17138
> URL: https://issues.apache.org/jira/browse/HBASE-17138
> Project: HBase
> Issue Type: Improvement
>Reporter: Yu Li
>Assignee: Yu Sun
> Attachments:
> 0001-fix-EHB-511-Resolve-client-compatibility-issue-introduced-by-offheap-change.patch,
> 0001-to-EHB-446-offheap-hfile-format-should-keep-compatible-v3.patch,
> 0001-to-EHB-456-Cell-should-be-compatible-with-branch-1.1.2.patch
>
>
> From the
> [thread|http://mail-archives.apache.org/mod_mbox/hbase-user/201611.mbox/%3CCAM7-19%2Bn7cEiY4H9iLQ3N9V0NXppOPduZwk-hhgNLEaJfiV3kA%40mail.gmail.com%3E]
> of sharing our experience and performance data of read-path offheap usage in
> Alibaba search, we could see people are positive to have HBASE-11425 in
> branch-1, so I'd like to create a JIRA and move the discussion and decision
> making here.
> Echoing some comments from the mail thread:
> Bryan:
> Is the backported patch available anywhere? If it ends up not getting
> officially backported to branch-1 due to 2.0 around the corner, some of us
> who build our own deploy may want to integrate into our builds
> Andrew:
> Yes, please, the patches will be useful to the community even if we decide
> not to backport into an official 1.x release.
> Enis:
> I don't see any reason why we cannot backport to branch-1.
> Ted:
> Opening a JIRA would be fine. This makes it easier for people to obtain the
> patch(es)
> Nick:
> From the DISCUSS thread re: EOL of 1.1, it seems we'll continue to
> support 1.x releases for some time... I would guess these will be
> maintained until 2.2 at least. Therefore, offheap patches that have seen
> production exposure seem like a reasonable candidate for backport, perhaps in
> a 1.4 or 1.5 release timeframe.
> Anoop:
> Because of some compatibility issues, we decide that this will be done in 2.0
> only.. Ya as Andy said, it would be great to share the 1.x backported
> patches.
> The following is all the jira ids we have back ported:
> HBASE-10930 Change Filters and GetClosestRowBeforeTracker to work with Cells
> (Ram)
> HBASE-13373 Squash HFileReaderV3 together with HFileReaderV2 and
> AbstractHFileReader; ditto for Scanners and BlockReader, etc.
> HBASE-13429 Remove deprecated seek/reseek methods from HFileScanner.
> HBASE-13450 - Purge RawBytescomparator from the writers and readers for
> HBASE-10800 (Ram)
> HBASE-13501 - Deprecate/Remove getComparator() in HRegionInfo.
> HBASE-12048 Remove deprecated APIs from Filter.
> HBASE-10800 - Use CellComparator instead of KVComparator (Ram)
> HBASE-13679 Change ColumnTracker and SQM to deal with Cell instead of byte[],
> int, int.
> HBASE-13642 Deprecate RegionObserver#postScannerFilterRow CP hook with
> byte[],int,int args in favor of taking Cell arg.
> HBASE-13641 Deperecate Filter#filterRowKey(byte[] buffer, int offset, int
> length) in favor of filterRowKey(Cell firstRowCell).
> HBASE-13827 Delayed scanner close in KeyValueHeap and StoreScanner.
> HBASE-13871 Change RegionScannerImpl to deal with Cell instead of byte[],
> int, int.
> HBASE-11911 Break up tests into more fine grained categories (Alex Newman)
> HBASE-12059 Create
[jira] [Comment Edited] (HBASE-17138) Backport read-path offheap (HBASE-11425) to branch-1
[
https://issues.apache.org/jira/browse/HBASE-17138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15994227#comment-15994227
]
Jingcheng Du edited comment on HBASE-17138 at 5/3/17 3:46 PM:
--
The changes in {{RegionScanner}} in master branch is neither binary compatible
nor source compatible for branch-1, and changes in {{RegionCoprocessorHost}}
might be only binary compatible. If users move to a new 1.x version which
includes the off-heap read path, they have to change the customized region
scanners and coprocessors.
Do we allow this kind of backward compatibility break in minor release?
[~busbey] [~anoop.hbase] [~Apache9]
According to the compatibility matrix in HBase
(https://hbase.apache.org/book.html#hbase.versioning), such changes are
Server-Side Limited API compatibility, it is allowed in HBase minor releases.
was (Author: jingcheng.du):
The changes in {{RegionScanner}} in master branch is only binary compatible
(not source compatible) for branch-1 and changes in {{RegionCoprocessorHost}}
might be only binary compatible too. If users move to a new 1.x version which
includes the off-heap read path, they have to change the customized region
scanners and coprocessors before recompiling.
Do we allow this kind of backward compatibility break in minor release?
[~busbey] [~anoop.hbase] [~Apache9]
According to the compatibility matrix in HBase
(https://hbase.apache.org/book.html#hbase.versioning), such changes are
Server-Side Limited API compatibility, it is allowed in HBase minor releases.
> Backport read-path offheap (HBASE-11425) to branch-1
>
>
> Key: HBASE-17138
> URL: https://issues.apache.org/jira/browse/HBASE-17138
> Project: HBase
> Issue Type: Improvement
>Reporter: Yu Li
>Assignee: Yu Sun
> Attachments:
> 0001-fix-EHB-511-Resolve-client-compatibility-issue-introduced-by-offheap-change.patch,
> 0001-to-EHB-446-offheap-hfile-format-should-keep-compatible-v3.patch,
> 0001-to-EHB-456-Cell-should-be-compatible-with-branch-1.1.2.patch
>
>
> From the
> [thread|http://mail-archives.apache.org/mod_mbox/hbase-user/201611.mbox/%3CCAM7-19%2Bn7cEiY4H9iLQ3N9V0NXppOPduZwk-hhgNLEaJfiV3kA%40mail.gmail.com%3E]
> of sharing our experience and performance data of read-path offheap usage in
> Alibaba search, we could see people are positive to have HBASE-11425 in
> branch-1, so I'd like to create a JIRA and move the discussion and decision
> making here.
> Echoing some comments from the mail thread:
> Bryan:
> Is the backported patch available anywhere? If it ends up not getting
> officially backported to branch-1 due to 2.0 around the corner, some of us
> who build our own deploy may want to integrate into our builds
> Andrew:
> Yes, please, the patches will be useful to the community even if we decide
> not to backport into an official 1.x release.
> Enis:
> I don't see any reason why we cannot backport to branch-1.
> Ted:
> Opening a JIRA would be fine. This makes it easier for people to obtain the
> patch(es)
> Nick:
> From the DISCUSS thread re: EOL of 1.1, it seems we'll continue to
> support 1.x releases for some time... I would guess these will be
> maintained until 2.2 at least. Therefore, offheap patches that have seen
> production exposure seem like a reasonable candidate for backport, perhaps in
> a 1.4 or 1.5 release timeframe.
> Anoop:
> Because of some compatibility issues, we decide that this will be done in 2.0
> only.. Ya as Andy said, it would be great to share the 1.x backported
> patches.
> The following is all the jira ids we have back ported:
> HBASE-10930 Change Filters and GetClosestRowBeforeTracker to work with Cells
> (Ram)
> HBASE-13373 Squash HFileReaderV3 together with HFileReaderV2 and
> AbstractHFileReader; ditto for Scanners and BlockReader, etc.
> HBASE-13429 Remove deprecated seek/reseek methods from HFileScanner.
> HBASE-13450 - Purge RawBytescomparator from the writers and readers for
> HBASE-10800 (Ram)
> HBASE-13501 - Deprecate/Remove getComparator() in HRegionInfo.
> HBASE-12048 Remove deprecated APIs from Filter.
> HBASE-10800 - Use CellComparator instead of KVComparator (Ram)
> HBASE-13679 Change ColumnTracker and SQM to deal with Cell instead of byte[],
> int, int.
> HBASE-13642 Deprecate RegionObserver#postScannerFilterRow CP hook with
> byte[],int,int args in favor of taking Cell arg.
> HBASE-13641 Deperecate Filter#filterRowKey(byte[] buffer, int offset, int
> length) in favor of filterRowKey(Cell firstRowCell).
> HBASE-13827 Delayed scanner close in KeyValueHeap and StoreScanner.
> HBASE-13871 Change RegionScannerImpl to deal with Cell instead of byte[],
> int, int.
> HBASE-11911 Break up tests into more fine grained categories (Alex Newman)
> HBASE-12059 Create hbase-annotations module
> HBASE-12106
[jira] [Comment Edited] (HBASE-17138) Backport read-path offheap (HBASE-11425) to branch-1
[
https://issues.apache.org/jira/browse/HBASE-17138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15994227#comment-15994227
]
Jingcheng Du edited comment on HBASE-17138 at 5/3/17 10:39 AM:
---
The changes in {{RegionScanner}} in master branch is only binary compatible
(not source compatible) for branch-1 and changes in {{RegionCoprocessorHost}}
might be only binary compatible too. If users move to a new 1.x version which
includes the off-heap read path, they have to change the customized region
scanners and coprocessors before recompiling.
Do we allow this kind of backward compatibility break in minor release?
[~busbey] [~anoop.hbase] [~Apache9]
According to the compatibility matrix in HBase
(https://hbase.apache.org/book.html#hbase.versioning), such changes are
Server-Side Limited API compatibility, it is allowed in HBase minor releases.
was (Author: jingcheng.du):
The changes in {{RegionScanner}} in master branch is only binary compatible
(not source compatible) for branch-1, and its annotations are
{{@InterfaceAudience.LimitedPrivate(HBaseInterfaceAudience.COPROC)}} and
{{@InterfaceStability.Evolving}}.
According to the compatibility matrix in HBase
(https://hbase.apache.org/book.html#hbase.versioning), does it mean we allow
the changes of binary compatibility in branch-1 minor releases? [~busbey]
[~anoop.hbase] [~Apache9]
> Backport read-path offheap (HBASE-11425) to branch-1
>
>
> Key: HBASE-17138
> URL: https://issues.apache.org/jira/browse/HBASE-17138
> Project: HBase
> Issue Type: Improvement
>Reporter: Yu Li
>Assignee: Yu Sun
> Attachments:
> 0001-fix-EHB-511-Resolve-client-compatibility-issue-introduced-by-offheap-change.patch,
> 0001-to-EHB-446-offheap-hfile-format-should-keep-compatible-v3.patch,
> 0001-to-EHB-456-Cell-should-be-compatible-with-branch-1.1.2.patch
>
>
> From the
> [thread|http://mail-archives.apache.org/mod_mbox/hbase-user/201611.mbox/%3CCAM7-19%2Bn7cEiY4H9iLQ3N9V0NXppOPduZwk-hhgNLEaJfiV3kA%40mail.gmail.com%3E]
> of sharing our experience and performance data of read-path offheap usage in
> Alibaba search, we could see people are positive to have HBASE-11425 in
> branch-1, so I'd like to create a JIRA and move the discussion and decision
> making here.
> Echoing some comments from the mail thread:
> Bryan:
> Is the backported patch available anywhere? If it ends up not getting
> officially backported to branch-1 due to 2.0 around the corner, some of us
> who build our own deploy may want to integrate into our builds
> Andrew:
> Yes, please, the patches will be useful to the community even if we decide
> not to backport into an official 1.x release.
> Enis:
> I don't see any reason why we cannot backport to branch-1.
> Ted:
> Opening a JIRA would be fine. This makes it easier for people to obtain the
> patch(es)
> Nick:
> From the DISCUSS thread re: EOL of 1.1, it seems we'll continue to
> support 1.x releases for some time... I would guess these will be
> maintained until 2.2 at least. Therefore, offheap patches that have seen
> production exposure seem like a reasonable candidate for backport, perhaps in
> a 1.4 or 1.5 release timeframe.
> Anoop:
> Because of some compatibility issues, we decide that this will be done in 2.0
> only.. Ya as Andy said, it would be great to share the 1.x backported
> patches.
> The following is all the jira ids we have back ported:
> HBASE-10930 Change Filters and GetClosestRowBeforeTracker to work with Cells
> (Ram)
> HBASE-13373 Squash HFileReaderV3 together with HFileReaderV2 and
> AbstractHFileReader; ditto for Scanners and BlockReader, etc.
> HBASE-13429 Remove deprecated seek/reseek methods from HFileScanner.
> HBASE-13450 - Purge RawBytescomparator from the writers and readers for
> HBASE-10800 (Ram)
> HBASE-13501 - Deprecate/Remove getComparator() in HRegionInfo.
> HBASE-12048 Remove deprecated APIs from Filter.
> HBASE-10800 - Use CellComparator instead of KVComparator (Ram)
> HBASE-13679 Change ColumnTracker and SQM to deal with Cell instead of byte[],
> int, int.
> HBASE-13642 Deprecate RegionObserver#postScannerFilterRow CP hook with
> byte[],int,int args in favor of taking Cell arg.
> HBASE-13641 Deperecate Filter#filterRowKey(byte[] buffer, int offset, int
> length) in favor of filterRowKey(Cell firstRowCell).
> HBASE-13827 Delayed scanner close in KeyValueHeap and StoreScanner.
> HBASE-13871 Change RegionScannerImpl to deal with Cell instead of byte[],
> int, int.
> HBASE-11911 Break up tests into more fine grained categories (Alex Newman)
> HBASE-12059 Create hbase-annotations module
> HBASE-12106 Move test annotations to test artifact (Enis Soztutar)
> HBASE-13916 Create MultiByteBuffer an aggregation of ByteBuffers.
> HBASE-15679 Assertion on wrong variable in
[jira] [Comment Edited] (HBASE-17138) Backport read-path offheap (HBASE-11425) to branch-1
[
https://issues.apache.org/jira/browse/HBASE-17138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15994227#comment-15994227
]
Jingcheng Du edited comment on HBASE-17138 at 5/3/17 10:10 AM:
---
The changes in {{RegionScanner}} in master branch is only binary compatible
(not source compatible) for branch-1, and its annotations are
{{@InterfaceAudience.LimitedPrivate(HBaseInterfaceAudience.COPROC)}} and
{{@InterfaceStability.Evolving}}.
According to the compatibility matrix in HBase
(https://hbase.apache.org/book.html#hbase.versioning), does it mean we allow
the changes of binary compatibility in branch-1 minor releases? [~busbey]
[~anoop.hbase] [~Apache9]
was (Author: jingcheng.du):
The changes in {{RegionScanner}} in master branch is only binary compatible for
branch-1, and its annotations are
{{@InterfaceAudience.LimitedPrivate(HBaseInterfaceAudience.COPROC)}} and
{{@InterfaceStability.Evolving}}.
According to the compatibility matrix in HBase
(https://hbase.apache.org/book.html#hbase.versioning), does it mean we allow
the changes of binary compatibility in branch-1 minor releases? [~busbey]
[~anoop.hbase] [~Apache9]
> Backport read-path offheap (HBASE-11425) to branch-1
>
>
> Key: HBASE-17138
> URL: https://issues.apache.org/jira/browse/HBASE-17138
> Project: HBase
> Issue Type: Improvement
>Reporter: Yu Li
>Assignee: Yu Sun
> Attachments:
> 0001-fix-EHB-511-Resolve-client-compatibility-issue-introduced-by-offheap-change.patch,
> 0001-to-EHB-446-offheap-hfile-format-should-keep-compatible-v3.patch,
> 0001-to-EHB-456-Cell-should-be-compatible-with-branch-1.1.2.patch
>
>
> From the
> [thread|http://mail-archives.apache.org/mod_mbox/hbase-user/201611.mbox/%3CCAM7-19%2Bn7cEiY4H9iLQ3N9V0NXppOPduZwk-hhgNLEaJfiV3kA%40mail.gmail.com%3E]
> of sharing our experience and performance data of read-path offheap usage in
> Alibaba search, we could see people are positive to have HBASE-11425 in
> branch-1, so I'd like to create a JIRA and move the discussion and decision
> making here.
> Echoing some comments from the mail thread:
> Bryan:
> Is the backported patch available anywhere? If it ends up not getting
> officially backported to branch-1 due to 2.0 around the corner, some of us
> who build our own deploy may want to integrate into our builds
> Andrew:
> Yes, please, the patches will be useful to the community even if we decide
> not to backport into an official 1.x release.
> Enis:
> I don't see any reason why we cannot backport to branch-1.
> Ted:
> Opening a JIRA would be fine. This makes it easier for people to obtain the
> patch(es)
> Nick:
> From the DISCUSS thread re: EOL of 1.1, it seems we'll continue to
> support 1.x releases for some time... I would guess these will be
> maintained until 2.2 at least. Therefore, offheap patches that have seen
> production exposure seem like a reasonable candidate for backport, perhaps in
> a 1.4 or 1.5 release timeframe.
> Anoop:
> Because of some compatibility issues, we decide that this will be done in 2.0
> only.. Ya as Andy said, it would be great to share the 1.x backported
> patches.
> The following is all the jira ids we have back ported:
> HBASE-10930 Change Filters and GetClosestRowBeforeTracker to work with Cells
> (Ram)
> HBASE-13373 Squash HFileReaderV3 together with HFileReaderV2 and
> AbstractHFileReader; ditto for Scanners and BlockReader, etc.
> HBASE-13429 Remove deprecated seek/reseek methods from HFileScanner.
> HBASE-13450 - Purge RawBytescomparator from the writers and readers for
> HBASE-10800 (Ram)
> HBASE-13501 - Deprecate/Remove getComparator() in HRegionInfo.
> HBASE-12048 Remove deprecated APIs from Filter.
> HBASE-10800 - Use CellComparator instead of KVComparator (Ram)
> HBASE-13679 Change ColumnTracker and SQM to deal with Cell instead of byte[],
> int, int.
> HBASE-13642 Deprecate RegionObserver#postScannerFilterRow CP hook with
> byte[],int,int args in favor of taking Cell arg.
> HBASE-13641 Deperecate Filter#filterRowKey(byte[] buffer, int offset, int
> length) in favor of filterRowKey(Cell firstRowCell).
> HBASE-13827 Delayed scanner close in KeyValueHeap and StoreScanner.
> HBASE-13871 Change RegionScannerImpl to deal with Cell instead of byte[],
> int, int.
> HBASE-11911 Break up tests into more fine grained categories (Alex Newman)
> HBASE-12059 Create hbase-annotations module
> HBASE-12106 Move test annotations to test artifact (Enis Soztutar)
> HBASE-13916 Create MultiByteBuffer an aggregation of ByteBuffers.
> HBASE-15679 Assertion on wrong variable in
> TestReplicationThrottler#testThrottling
> HBASE-13931 Move Unsafe based operations to UnsafeAccess.
> HBASE-12345 Unsafe based ByteBuffer Comparator.
> HBASE-13998 Remove CellComparator#compareRows(byte[] left,
[jira] [Comment Edited] (HBASE-17138) Backport read-path offheap (HBASE-11425) to branch-1
[
https://issues.apache.org/jira/browse/HBASE-17138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15994227#comment-15994227
]
Jingcheng Du edited comment on HBASE-17138 at 5/3/17 10:05 AM:
---
The changes in {{RegionScanner}} in master branch is only binary compatible for
branch-1, and its annotations are
{{@InterfaceAudience.LimitedPrivate(HBaseInterfaceAudience.COPROC)}} and
{{@InterfaceStability.Evolving}}.
According to the compatibility matrix in HBase
(https://hbase.apache.org/book.html#hbase.versioning), does it mean we allow
the changes of binary compatibility in branch-1 minor releases? [~busbey]
[~anoop.hbase] [~Apache9]
was (Author: jingcheng.du):
The changes in {{RegionScanner}} is only binary compatibility, and its
annotations are
{{@InterfaceAudience.LimitedPrivate(HBaseInterfaceAudience.COPROC)}} and
{{@InterfaceStability.Evolving}}.
According to the compatibility matrix in HBase
(https://hbase.apache.org/book.html#hbase.versioning), does it mean we allow
the changes of binary compatibility in branch-1 minor releases? [~busbey]
[~anoop.hbase] [~Apache9]
> Backport read-path offheap (HBASE-11425) to branch-1
>
>
> Key: HBASE-17138
> URL: https://issues.apache.org/jira/browse/HBASE-17138
> Project: HBase
> Issue Type: Improvement
>Reporter: Yu Li
>Assignee: Yu Sun
> Attachments:
> 0001-fix-EHB-511-Resolve-client-compatibility-issue-introduced-by-offheap-change.patch,
> 0001-to-EHB-446-offheap-hfile-format-should-keep-compatible-v3.patch,
> 0001-to-EHB-456-Cell-should-be-compatible-with-branch-1.1.2.patch
>
>
> From the
> [thread|http://mail-archives.apache.org/mod_mbox/hbase-user/201611.mbox/%3CCAM7-19%2Bn7cEiY4H9iLQ3N9V0NXppOPduZwk-hhgNLEaJfiV3kA%40mail.gmail.com%3E]
> of sharing our experience and performance data of read-path offheap usage in
> Alibaba search, we could see people are positive to have HBASE-11425 in
> branch-1, so I'd like to create a JIRA and move the discussion and decision
> making here.
> Echoing some comments from the mail thread:
> Bryan:
> Is the backported patch available anywhere? If it ends up not getting
> officially backported to branch-1 due to 2.0 around the corner, some of us
> who build our own deploy may want to integrate into our builds
> Andrew:
> Yes, please, the patches will be useful to the community even if we decide
> not to backport into an official 1.x release.
> Enis:
> I don't see any reason why we cannot backport to branch-1.
> Ted:
> Opening a JIRA would be fine. This makes it easier for people to obtain the
> patch(es)
> Nick:
> From the DISCUSS thread re: EOL of 1.1, it seems we'll continue to
> support 1.x releases for some time... I would guess these will be
> maintained until 2.2 at least. Therefore, offheap patches that have seen
> production exposure seem like a reasonable candidate for backport, perhaps in
> a 1.4 or 1.5 release timeframe.
> Anoop:
> Because of some compatibility issues, we decide that this will be done in 2.0
> only.. Ya as Andy said, it would be great to share the 1.x backported
> patches.
> The following is all the jira ids we have back ported:
> HBASE-10930 Change Filters and GetClosestRowBeforeTracker to work with Cells
> (Ram)
> HBASE-13373 Squash HFileReaderV3 together with HFileReaderV2 and
> AbstractHFileReader; ditto for Scanners and BlockReader, etc.
> HBASE-13429 Remove deprecated seek/reseek methods from HFileScanner.
> HBASE-13450 - Purge RawBytescomparator from the writers and readers for
> HBASE-10800 (Ram)
> HBASE-13501 - Deprecate/Remove getComparator() in HRegionInfo.
> HBASE-12048 Remove deprecated APIs from Filter.
> HBASE-10800 - Use CellComparator instead of KVComparator (Ram)
> HBASE-13679 Change ColumnTracker and SQM to deal with Cell instead of byte[],
> int, int.
> HBASE-13642 Deprecate RegionObserver#postScannerFilterRow CP hook with
> byte[],int,int args in favor of taking Cell arg.
> HBASE-13641 Deperecate Filter#filterRowKey(byte[] buffer, int offset, int
> length) in favor of filterRowKey(Cell firstRowCell).
> HBASE-13827 Delayed scanner close in KeyValueHeap and StoreScanner.
> HBASE-13871 Change RegionScannerImpl to deal with Cell instead of byte[],
> int, int.
> HBASE-11911 Break up tests into more fine grained categories (Alex Newman)
> HBASE-12059 Create hbase-annotations module
> HBASE-12106 Move test annotations to test artifact (Enis Soztutar)
> HBASE-13916 Create MultiByteBuffer an aggregation of ByteBuffers.
> HBASE-15679 Assertion on wrong variable in
> TestReplicationThrottler#testThrottling
> HBASE-13931 Move Unsafe based operations to UnsafeAccess.
> HBASE-12345 Unsafe based ByteBuffer Comparator.
> HBASE-13998 Remove CellComparator#compareRows(byte[] left, int loffset, int
> llength, byte[] right, int
[jira] [Commented] (HBASE-17138) Backport read-path offheap (HBASE-11425) to branch-1
[
https://issues.apache.org/jira/browse/HBASE-17138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15994227#comment-15994227
]
Jingcheng Du commented on HBASE-17138:
--
The changes in {{RegionScanner}} is only binary compatibility, and its
annotations are
{{@InterfaceAudience.LimitedPrivate(HBaseInterfaceAudience.COPROC)}} and
{{@InterfaceStability.Evolving}}.
According to the compatibility matrix in HBase
(https://hbase.apache.org/book.html#hbase.versioning), does it mean we allow
the changes of binary compatibility in branch-1 minor releases? [~busbey]
[~anoop.hbase] [~Apache9]
> Backport read-path offheap (HBASE-11425) to branch-1
>
>
> Key: HBASE-17138
> URL: https://issues.apache.org/jira/browse/HBASE-17138
> Project: HBase
> Issue Type: Improvement
>Reporter: Yu Li
>Assignee: Yu Sun
> Attachments:
> 0001-fix-EHB-511-Resolve-client-compatibility-issue-introduced-by-offheap-change.patch,
> 0001-to-EHB-446-offheap-hfile-format-should-keep-compatible-v3.patch,
> 0001-to-EHB-456-Cell-should-be-compatible-with-branch-1.1.2.patch
>
>
> From the
> [thread|http://mail-archives.apache.org/mod_mbox/hbase-user/201611.mbox/%3CCAM7-19%2Bn7cEiY4H9iLQ3N9V0NXppOPduZwk-hhgNLEaJfiV3kA%40mail.gmail.com%3E]
> of sharing our experience and performance data of read-path offheap usage in
> Alibaba search, we could see people are positive to have HBASE-11425 in
> branch-1, so I'd like to create a JIRA and move the discussion and decision
> making here.
> Echoing some comments from the mail thread:
> Bryan:
> Is the backported patch available anywhere? If it ends up not getting
> officially backported to branch-1 due to 2.0 around the corner, some of us
> who build our own deploy may want to integrate into our builds
> Andrew:
> Yes, please, the patches will be useful to the community even if we decide
> not to backport into an official 1.x release.
> Enis:
> I don't see any reason why we cannot backport to branch-1.
> Ted:
> Opening a JIRA would be fine. This makes it easier for people to obtain the
> patch(es)
> Nick:
> From the DISCUSS thread re: EOL of 1.1, it seems we'll continue to
> support 1.x releases for some time... I would guess these will be
> maintained until 2.2 at least. Therefore, offheap patches that have seen
> production exposure seem like a reasonable candidate for backport, perhaps in
> a 1.4 or 1.5 release timeframe.
> Anoop:
> Because of some compatibility issues, we decide that this will be done in 2.0
> only.. Ya as Andy said, it would be great to share the 1.x backported
> patches.
> The following is all the jira ids we have back ported:
> HBASE-10930 Change Filters and GetClosestRowBeforeTracker to work with Cells
> (Ram)
> HBASE-13373 Squash HFileReaderV3 together with HFileReaderV2 and
> AbstractHFileReader; ditto for Scanners and BlockReader, etc.
> HBASE-13429 Remove deprecated seek/reseek methods from HFileScanner.
> HBASE-13450 - Purge RawBytescomparator from the writers and readers for
> HBASE-10800 (Ram)
> HBASE-13501 - Deprecate/Remove getComparator() in HRegionInfo.
> HBASE-12048 Remove deprecated APIs from Filter.
> HBASE-10800 - Use CellComparator instead of KVComparator (Ram)
> HBASE-13679 Change ColumnTracker and SQM to deal with Cell instead of byte[],
> int, int.
> HBASE-13642 Deprecate RegionObserver#postScannerFilterRow CP hook with
> byte[],int,int args in favor of taking Cell arg.
> HBASE-13641 Deperecate Filter#filterRowKey(byte[] buffer, int offset, int
> length) in favor of filterRowKey(Cell firstRowCell).
> HBASE-13827 Delayed scanner close in KeyValueHeap and StoreScanner.
> HBASE-13871 Change RegionScannerImpl to deal with Cell instead of byte[],
> int, int.
> HBASE-11911 Break up tests into more fine grained categories (Alex Newman)
> HBASE-12059 Create hbase-annotations module
> HBASE-12106 Move test annotations to test artifact (Enis Soztutar)
> HBASE-13916 Create MultiByteBuffer an aggregation of ByteBuffers.
> HBASE-15679 Assertion on wrong variable in
> TestReplicationThrottler#testThrottling
> HBASE-13931 Move Unsafe based operations to UnsafeAccess.
> HBASE-12345 Unsafe based ByteBuffer Comparator.
> HBASE-13998 Remove CellComparator#compareRows(byte[] left, int loffset, int
> llength, byte[] right, int roffset, int rlength).
> HBASE-13998 Remove CellComparator#compareRows()- Addendum to fix javadoc warn
> HBASE-13579 Avoid isCellTTLExpired() for NO-TAG cases (partially backport
> this patch)
> HBASE-13448 New Cell implementation with cached component offsets/lengths.
> HBASE-13387 Add ByteBufferedCell an extension to Cell.
> HBASE-13387 Add ByteBufferedCell an extension to Cell - addendum.
> HBASE-12650 Move ServerName to hbase-common module (partially backport this
> patch)
> HBASE-12296 Filters should work with
[jira] [Updated] (HBASE-17875) Document why objects over 10MB are not well-suited for hbase
[ https://issues.apache.org/jira/browse/HBASE-17875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingcheng Du updated HBASE-17875: - Attachment: HBASE-17875-V2.patch Thanks a lot [~busbey] for the advices! The patch V2 is attached for them. > Document why objects over 10MB are not well-suited for hbase > > > Key: HBASE-17875 > URL: https://issues.apache.org/jira/browse/HBASE-17875 > Project: HBase > Issue Type: Bug > Components: documentation, mob >Reporter: stack >Assignee: Jingcheng Du > Fix For: 2.0.0 > > Attachments: HBASE-17875.patch, HBASE-17875-V2.patch > > > A new-user who has patently done their homework is unable to find in doc why > objects above 10MB are not recommended for hbase. Lets add explanation that > has a link from MOB on what happens when objects this size are requested form > HBase. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17947) Location of Examples.proto is wrong in comment of RowCountEndPoint.java
[
https://issues.apache.org/jira/browse/HBASE-17947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15980700#comment-15980700
]
Jingcheng Du commented on HBASE-17947:
--
Thanks for the patch [~water].
It is better only change the incorrect path not to change other statement.
+1 after the comment is addressed.
> Location of Examples.proto is wrong in comment of RowCountEndPoint.java
> ---
>
> Key: HBASE-17947
> URL: https://issues.apache.org/jira/browse/HBASE-17947
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors
>Reporter: Xiang Li
>Assignee: Xiang Li
>Priority: Trivial
> Labels: easyfix
> Fix For: 2.0.0
>
> Attachments: HBASE-17947.master.000.patch
>
>
> In the comment of RowCountEndpoint.java
> {code}
> /**
> * Sample coprocessor endpoint exposing a Service interface for counting rows
> and key values.
> *
> *
> * For the protocol buffer definition of the RowCountService, see the source
> file located under
> * hbase-server/src/main/protobuf/Examples.proto.
> *
> */
> {code}
> Examples.proto is not under hbase-server/src/main/protobuf/, but under
> hbase-examples/src/main/protobuf/
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17875) Document why objects over 10MB are not well-suited for hbase
[ https://issues.apache.org/jira/browse/HBASE-17875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingcheng Du updated HBASE-17875: - Status: Patch Available (was: Open) > Document why objects over 10MB are not well-suited for hbase > > > Key: HBASE-17875 > URL: https://issues.apache.org/jira/browse/HBASE-17875 > Project: HBase > Issue Type: Bug > Components: documentation >Reporter: stack >Assignee: Jingcheng Du > Attachments: HBASE-17875.patch > > > A new-user who has patently done their homework is unable to find in doc why > objects above 10MB are not recommended for hbase. Lets add explanation that > has a link from MOB on what happens when objects this size are requested form > HBase. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HBASE-17875) Document why objects over 10MB are not well-suited for hbase
[ https://issues.apache.org/jira/browse/HBASE-17875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingcheng Du reassigned HBASE-17875: Assignee: Jingcheng Du > Document why objects over 10MB are not well-suited for hbase > > > Key: HBASE-17875 > URL: https://issues.apache.org/jira/browse/HBASE-17875 > Project: HBase > Issue Type: Bug > Components: documentation >Reporter: stack >Assignee: Jingcheng Du > Attachments: HBASE-17875.patch > > > A new-user who has patently done their homework is unable to find in doc why > objects above 10MB are not recommended for hbase. Lets add explanation that > has a link from MOB on what happens when objects this size are requested form > HBase. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17875) Document why objects over 10MB are not well-suited for hbase
[ https://issues.apache.org/jira/browse/HBASE-17875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingcheng Du updated HBASE-17875: - Attachment: HBASE-17875.patch > Document why objects over 10MB are not well-suited for hbase > > > Key: HBASE-17875 > URL: https://issues.apache.org/jira/browse/HBASE-17875 > Project: HBase > Issue Type: Bug > Components: documentation >Reporter: stack > Attachments: HBASE-17875.patch > > > A new-user who has patently done their homework is unable to find in doc why > objects above 10MB are not recommended for hbase. Lets add explanation that > has a link from MOB on what happens when objects this size are requested form > HBase. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17875) Document why objects over 10MB are not well-suited for hbase
[ https://issues.apache.org/jira/browse/HBASE-17875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingcheng Du updated HBASE-17875: - Attachment: (was: HBASE-17875.diff) > Document why objects over 10MB are not well-suited for hbase > > > Key: HBASE-17875 > URL: https://issues.apache.org/jira/browse/HBASE-17875 > Project: HBase > Issue Type: Bug > Components: documentation >Reporter: stack > > A new-user who has patently done their homework is unable to find in doc why > objects above 10MB are not recommended for hbase. Lets add explanation that > has a link from MOB on what happens when objects this size are requested form > HBase. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17875) Document why objects over 10MB are not well-suited for hbase
[ https://issues.apache.org/jira/browse/HBASE-17875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingcheng Du updated HBASE-17875: - Attachment: HBASE-17875.diff Upload the patch for review. Thanks. > Document why objects over 10MB are not well-suited for hbase > > > Key: HBASE-17875 > URL: https://issues.apache.org/jira/browse/HBASE-17875 > Project: HBase > Issue Type: Bug > Components: documentation >Reporter: stack > > A new-user who has patently done their homework is unable to find in doc why > objects above 10MB are not recommended for hbase. Lets add explanation that > has a link from MOB on what happens when objects this size are requested form > HBase. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17172) Optimize mob compaction with _del files
[ https://issues.apache.org/jira/browse/HBASE-17172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15963773#comment-15963773 ] Jingcheng Du commented on HBASE-17172: -- Thanks [~huaxiang]. Yes, that is correct. But your implementation makes sense too, we can use this now. Thanks! > Optimize mob compaction with _del files > --- > > Key: HBASE-17172 > URL: https://issues.apache.org/jira/browse/HBASE-17172 > Project: HBase > Issue Type: Improvement > Components: mob >Affects Versions: 2.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun > Fix For: 2.0.0 > > Attachments: HBASE-17172-master-001.patch, > HBASE-17172.master.001.patch, HBASE-17172.master.002.patch, > HBASE-17172.master.003.patch > > > Today, when there is a _del file in mobdir, with major mob compaction, every > mob file will be recompacted, this causes lots of IO and slow down major mob > compaction (may take months to finish). This needs to be improved. A few > ideas are: > 1) Do not compact all _del files into one, instead, compact them based on > groups with startKey as the key. Then use firstKey/startKey to make each mob > file to see if the _del file needs to be included for this partition. > 2). Based on the timerange of the _del file, compaction for files after that > timerange does not need to include the _del file as these are newer files. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17875) Document why objects over 10MB are not well-suited for hbase
[ https://issues.apache.org/jira/browse/HBASE-17875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956654#comment-15956654 ] Jingcheng Du commented on HBASE-17875: -- I can add this. > Document why objects over 10MB are not well-suited for hbase > > > Key: HBASE-17875 > URL: https://issues.apache.org/jira/browse/HBASE-17875 > Project: HBase > Issue Type: Bug > Components: documentation >Reporter: stack > > A new-user who has patently done their homework is unable to find in doc why > objects above 10MB are not recommended for hbase. Lets add explanation that > has a link from MOB on what happens when objects this size are requested form > HBase. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HBASE-17688) MultiRowRangeFilter not working correctly if given same start and stop RowKey
[
https://issues.apache.org/jira/browse/HBASE-17688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15887279#comment-15887279
]
Jingcheng Du edited comment on HBASE-17688 at 2/28/17 5:10 AM:
---
Pushed to master and branch-1.1+ branch. Thanks [~ahujaravi1] for finding the
issue, and thanks [~yuzhih...@gmail.com] and [~anoop.hbase] for review.
was (Author: jingcheng.du):
Pushed to master and branch-1.1+ branch. Thanks for review
[~yuzhih...@gmail.com] and [~anoop.hbase].
> MultiRowRangeFilter not working correctly if given same start and stop RowKey
> -
>
> Key: HBASE-17688
> URL: https://issues.apache.org/jira/browse/HBASE-17688
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.0, 1.2.4, 0.98.24, 1.1.9
>Reporter: Ravi Ahuj
>Assignee: Jingcheng Du
>Priority: Minor
> Fix For: 2.0.0, 1.3.1, 1.2.5, 1.1.10
>
> Attachments: HBASE-17688.branch-1.x.patch, HBASE-17688.master.patch,
> HBASE-17688.master-v1.patch
>
>
>
>
> try (final Connection conn = ConnectionFactory.createConnection(conf);
>final Table scanTable = conn.getTable(table)){
> ArrayList rowRangesList = new
> ArrayList<>();
>
> String startRowkey="abc";
> String stopRowkey="abc";
> rowRangesList.add(new MultiRowRangeFilter.RowRange(startRowkey,
> true, stopRowkey, true));
> Scan scan = new Scan();
> scan.setFilter(new MultiRowRangeFilter(rowRangesList));
>
>ResultScanner scanner=scanTable.getScanner(scan);
>
> for (Result result : scanner) {
> String rowkey=new String(result.getRow());
>System.out.println(rowkey);
>
> }
> }
>
> In Hbase API of Java, we want to do multiple scans in the table using
> MultiRowRangeFilter.
> When we give multiple filters of startRowKey and stopRowKey, it is not
> working Properly with same StartRowKey and StopRowkey.
> Ideally, it should give only one Row with that Rowkey, but instead it is
> giving all the rows starting from that Rowkey in that Hbase Table
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17688) MultiRowRangeFilter not working correctly if given same start and stop RowKey
[
https://issues.apache.org/jira/browse/HBASE-17688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingcheng Du updated HBASE-17688:
-
Affects Version/s: (was: 2.0.0)
> MultiRowRangeFilter not working correctly if given same start and stop RowKey
> -
>
> Key: HBASE-17688
> URL: https://issues.apache.org/jira/browse/HBASE-17688
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.0, 1.2.4, 0.98.24, 1.1.9
>Reporter: Ravi Ahuj
>Assignee: Jingcheng Du
>Priority: Minor
> Fix For: 2.0.0, 1.3.1, 1.2.5, 1.1.10
>
> Attachments: HBASE-17688.branch-1.x.patch, HBASE-17688.master.patch,
> HBASE-17688.master-v1.patch
>
>
>
>
> try (final Connection conn = ConnectionFactory.createConnection(conf);
>final Table scanTable = conn.getTable(table)){
> ArrayList rowRangesList = new
> ArrayList<>();
>
> String startRowkey="abc";
> String stopRowkey="abc";
> rowRangesList.add(new MultiRowRangeFilter.RowRange(startRowkey,
> true, stopRowkey, true));
> Scan scan = new Scan();
> scan.setFilter(new MultiRowRangeFilter(rowRangesList));
>
>ResultScanner scanner=scanTable.getScanner(scan);
>
> for (Result result : scanner) {
> String rowkey=new String(result.getRow());
>System.out.println(rowkey);
>
> }
> }
>
> In Hbase API of Java, we want to do multiple scans in the table using
> MultiRowRangeFilter.
> When we give multiple filters of startRowKey and stopRowKey, it is not
> working Properly with same StartRowKey and StopRowkey.
> Ideally, it should give only one Row with that Rowkey, but instead it is
> giving all the rows starting from that Rowkey in that Hbase Table
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17688) MultiRowRangeFilter not working correctly if given same start and stop RowKey
[
https://issues.apache.org/jira/browse/HBASE-17688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingcheng Du updated HBASE-17688:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 1.1.10
1.2.5
1.3.1
2.0.0
Status: Resolved (was: Patch Available)
> MultiRowRangeFilter not working correctly if given same start and stop RowKey
> -
>
> Key: HBASE-17688
> URL: https://issues.apache.org/jira/browse/HBASE-17688
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.0, 1.2.4, 0.98.24, 1.1.9
>Reporter: Ravi Ahuj
>Assignee: Jingcheng Du
>Priority: Minor
> Fix For: 2.0.0, 1.3.1, 1.2.5, 1.1.10
>
> Attachments: HBASE-17688.branch-1.x.patch, HBASE-17688.master.patch,
> HBASE-17688.master-v1.patch
>
>
>
>
> try (final Connection conn = ConnectionFactory.createConnection(conf);
>final Table scanTable = conn.getTable(table)){
> ArrayList rowRangesList = new
> ArrayList<>();
>
> String startRowkey="abc";
> String stopRowkey="abc";
> rowRangesList.add(new MultiRowRangeFilter.RowRange(startRowkey,
> true, stopRowkey, true));
> Scan scan = new Scan();
> scan.setFilter(new MultiRowRangeFilter(rowRangesList));
>
>ResultScanner scanner=scanTable.getScanner(scan);
>
> for (Result result : scanner) {
> String rowkey=new String(result.getRow());
>System.out.println(rowkey);
>
> }
> }
>
> In Hbase API of Java, we want to do multiple scans in the table using
> MultiRowRangeFilter.
> When we give multiple filters of startRowKey and stopRowKey, it is not
> working Properly with same StartRowKey and StopRowkey.
> Ideally, it should give only one Row with that Rowkey, but instead it is
> giving all the rows starting from that Rowkey in that Hbase Table
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17688) MultiRowRangeFilter not working correctly if given same start and stop RowKey
[
https://issues.apache.org/jira/browse/HBASE-17688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15887279#comment-15887279
]
Jingcheng Du commented on HBASE-17688:
--
Pushed to master and branch-1.1+ branch. Thanks for review
[~yuzhih...@gmail.com] and [~anoop.hbase].
> MultiRowRangeFilter not working correctly if given same start and stop RowKey
> -
>
> Key: HBASE-17688
> URL: https://issues.apache.org/jira/browse/HBASE-17688
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.0, 1.2.4, 0.98.24, 1.1.9
>Reporter: Ravi Ahuj
>Assignee: Jingcheng Du
>Priority: Minor
> Attachments: HBASE-17688.branch-1.x.patch, HBASE-17688.master.patch,
> HBASE-17688.master-v1.patch
>
>
>
>
> try (final Connection conn = ConnectionFactory.createConnection(conf);
>final Table scanTable = conn.getTable(table)){
> ArrayList rowRangesList = new
> ArrayList<>();
>
> String startRowkey="abc";
> String stopRowkey="abc";
> rowRangesList.add(new MultiRowRangeFilter.RowRange(startRowkey,
> true, stopRowkey, true));
> Scan scan = new Scan();
> scan.setFilter(new MultiRowRangeFilter(rowRangesList));
>
>ResultScanner scanner=scanTable.getScanner(scan);
>
> for (Result result : scanner) {
> String rowkey=new String(result.getRow());
>System.out.println(rowkey);
>
> }
> }
>
> In Hbase API of Java, we want to do multiple scans in the table using
> MultiRowRangeFilter.
> When we give multiple filters of startRowKey and stopRowKey, it is not
> working Properly with same StartRowKey and StopRowkey.
> Ideally, it should give only one Row with that Rowkey, but instead it is
> giving all the rows starting from that Rowkey in that Hbase Table
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17688) MultiRowRangeFilter not working correctly if given same start and stop RowKey
[
https://issues.apache.org/jira/browse/HBASE-17688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingcheng Du updated HBASE-17688:
-
Affects Version/s: (was: 1.1.8)
1.1.9
> MultiRowRangeFilter not working correctly if given same start and stop RowKey
> -
>
> Key: HBASE-17688
> URL: https://issues.apache.org/jira/browse/HBASE-17688
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.0, 1.2.4, 0.98.24, 1.1.9
>Reporter: Ravi Ahuj
>Assignee: Jingcheng Du
>Priority: Minor
> Attachments: HBASE-17688.branch-1.x.patch, HBASE-17688.master.patch,
> HBASE-17688.master-v1.patch
>
>
>
>
> try (final Connection conn = ConnectionFactory.createConnection(conf);
>final Table scanTable = conn.getTable(table)){
> ArrayList rowRangesList = new
> ArrayList<>();
>
> String startRowkey="abc";
> String stopRowkey="abc";
> rowRangesList.add(new MultiRowRangeFilter.RowRange(startRowkey,
> true, stopRowkey, true));
> Scan scan = new Scan();
> scan.setFilter(new MultiRowRangeFilter(rowRangesList));
>
>ResultScanner scanner=scanTable.getScanner(scan);
>
> for (Result result : scanner) {
> String rowkey=new String(result.getRow());
>System.out.println(rowkey);
>
> }
> }
>
> In Hbase API of Java, we want to do multiple scans in the table using
> MultiRowRangeFilter.
> When we give multiple filters of startRowKey and stopRowKey, it is not
> working Properly with same StartRowKey and StopRowkey.
> Ideally, it should give only one Row with that Rowkey, but instead it is
> giving all the rows starting from that Rowkey in that Hbase Table
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17688) MultiRowRangeFilter not working correctly if given same start and stop RowKey
[
https://issues.apache.org/jira/browse/HBASE-17688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15887136#comment-15887136
]
Jingcheng Du commented on HBASE-17688:
--
bq. So if we have a Scan with includeStopRow is false and having this Filter
with some ranges, all with same start and stop keys, there will be NO results
returned?
You are right, no results are returned.
> MultiRowRangeFilter not working correctly if given same start and stop RowKey
> -
>
> Key: HBASE-17688
> URL: https://issues.apache.org/jira/browse/HBASE-17688
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.0, 1.2.4, 0.98.24, 1.1.8
>Reporter: Ravi Ahuj
>Assignee: Jingcheng Du
>Priority: Minor
> Attachments: HBASE-17688.branch-1.x.patch, HBASE-17688.master.patch,
> HBASE-17688.master-v1.patch
>
>
>
>
> try (final Connection conn = ConnectionFactory.createConnection(conf);
>final Table scanTable = conn.getTable(table)){
> ArrayList rowRangesList = new
> ArrayList<>();
>
> String startRowkey="abc";
> String stopRowkey="abc";
> rowRangesList.add(new MultiRowRangeFilter.RowRange(startRowkey,
> true, stopRowkey, true));
> Scan scan = new Scan();
> scan.setFilter(new MultiRowRangeFilter(rowRangesList));
>
>ResultScanner scanner=scanTable.getScanner(scan);
>
> for (Result result : scanner) {
> String rowkey=new String(result.getRow());
>System.out.println(rowkey);
>
> }
> }
>
> In Hbase API of Java, we want to do multiple scans in the table using
> MultiRowRangeFilter.
> When we give multiple filters of startRowKey and stopRowKey, it is not
> working Properly with same StartRowKey and StopRowkey.
> Ideally, it should give only one Row with that Rowkey, but instead it is
> giving all the rows starting from that Rowkey in that Hbase Table
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17690) Clean up MOB code
[ https://issues.apache.org/jira/browse/HBASE-17690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15887060#comment-15887060 ] Jingcheng Du commented on HBASE-17690: -- Pushed to master, thanks Ted for review! > Clean up MOB code > - > > Key: HBASE-17690 > URL: https://issues.apache.org/jira/browse/HBASE-17690 > Project: HBase > Issue Type: Improvement > Components: mob >Reporter: Jingcheng Du >Assignee: Jingcheng Du > Fix For: 2.0.0 > > Attachments: HBASE-17690.patch, HBASE-17690-v1.patch > > > Clean up the code in MOB. > # Fix the incorrect description in comments. > # Fix the warning and remove redundant import in code. > # Remove the references to the deprecated code. > # Add throughput controller for DefaultMobStoreFlusher and > DefaultMobStoreCompactor. > The RB link is https://reviews.apache.org/r/57078/. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17690) Clean up MOB code
[ https://issues.apache.org/jira/browse/HBASE-17690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingcheng Du updated HBASE-17690: - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.0.0 Status: Resolved (was: Patch Available) > Clean up MOB code > - > > Key: HBASE-17690 > URL: https://issues.apache.org/jira/browse/HBASE-17690 > Project: HBase > Issue Type: Improvement > Components: mob >Reporter: Jingcheng Du >Assignee: Jingcheng Du > Fix For: 2.0.0 > > Attachments: HBASE-17690.patch, HBASE-17690-v1.patch > > > Clean up the code in MOB. > # Fix the incorrect description in comments. > # Fix the warning and remove redundant import in code. > # Remove the references to the deprecated code. > # Add throughput controller for DefaultMobStoreFlusher and > DefaultMobStoreCompactor. > The RB link is https://reviews.apache.org/r/57078/. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17690) Clean up MOB code
[ https://issues.apache.org/jira/browse/HBASE-17690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15887041#comment-15887041 ] Jingcheng Du commented on HBASE-17690: -- The test failures should not be related with this patch. As [~yuzhih...@gmail.com]'s +1 from RB, I will push this patch to master soon. Thank you Ted for the review. > Clean up MOB code > - > > Key: HBASE-17690 > URL: https://issues.apache.org/jira/browse/HBASE-17690 > Project: HBase > Issue Type: Improvement > Components: mob >Reporter: Jingcheng Du >Assignee: Jingcheng Du > Attachments: HBASE-17690.patch, HBASE-17690-v1.patch > > > Clean up the code in MOB. > # Fix the incorrect description in comments. > # Fix the warning and remove redundant import in code. > # Remove the references to the deprecated code. > # Add throughput controller for DefaultMobStoreFlusher and > DefaultMobStoreCompactor. > The RB link is https://reviews.apache.org/r/57078/. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17690) Clean up MOB code
[ https://issues.apache.org/jira/browse/HBASE-17690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingcheng Du updated HBASE-17690: - Description: Clean up the code in MOB. # Fix the incorrect description in comments. # Fix the warning and remove redundant import in code. # Remove the references to the deprecated code. # Add throughput controller for DefaultMobStoreFlusher and DefaultMobStoreCompactor. The RB link is https://reviews.apache.org/r/57078/. was: Clean up the code in MOB. # Fix the incorrect description in comments. # Fix the warning and remove redundant import in code. # Remove the references to the deprecated code. # Add throughput controller for DefaultMobStoreFlusher and DefaultMobStoreCompactor. > Clean up MOB code > - > > Key: HBASE-17690 > URL: https://issues.apache.org/jira/browse/HBASE-17690 > Project: HBase > Issue Type: Improvement > Components: mob >Reporter: Jingcheng Du >Assignee: Jingcheng Du > Attachments: HBASE-17690.patch, HBASE-17690-v1.patch > > > Clean up the code in MOB. > # Fix the incorrect description in comments. > # Fix the warning and remove redundant import in code. > # Remove the references to the deprecated code. > # Add throughput controller for DefaultMobStoreFlusher and > DefaultMobStoreCompactor. > The RB link is https://reviews.apache.org/r/57078/. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17690) Clean up MOB code
[ https://issues.apache.org/jira/browse/HBASE-17690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingcheng Du updated HBASE-17690: - Attachment: HBASE-17690-v1.patch Upload a patch to address [~yuzhih...@gmail.com]'s comments. And upload it to RB https://reviews.apache.org/r/57078/. > Clean up MOB code > - > > Key: HBASE-17690 > URL: https://issues.apache.org/jira/browse/HBASE-17690 > Project: HBase > Issue Type: Improvement > Components: mob >Reporter: Jingcheng Du >Assignee: Jingcheng Du > Attachments: HBASE-17690.patch, HBASE-17690-v1.patch > > > Clean up the code in MOB. > # Fix the incorrect description in comments. > # Fix the warning and remove redundant import in code. > # Remove the references to the deprecated code. > # Add throughput controller for DefaultMobStoreFlusher and > DefaultMobStoreCompactor. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17688) MultiRowRangeFilter not working correctly if given same start and stop RowKey
[
https://issues.apache.org/jira/browse/HBASE-17688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingcheng Du updated HBASE-17688:
-
Attachment: HBASE-17688.master-v1.patch
> MultiRowRangeFilter not working correctly if given same start and stop RowKey
> -
>
> Key: HBASE-17688
> URL: https://issues.apache.org/jira/browse/HBASE-17688
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.0, 1.2.4, 0.98.24, 1.1.8
>Reporter: Ravi Ahuj
>Assignee: Jingcheng Du
>Priority: Minor
> Attachments: HBASE-17688.branch-1.x.patch, HBASE-17688.master.patch,
> HBASE-17688.master-v1.patch
>
>
>
>
> try (final Connection conn = ConnectionFactory.createConnection(conf);
>final Table scanTable = conn.getTable(table)){
> ArrayList rowRangesList = new
> ArrayList<>();
>
> String startRowkey="abc";
> String stopRowkey="abc";
> rowRangesList.add(new MultiRowRangeFilter.RowRange(startRowkey,
> true, stopRowkey, true));
> Scan scan = new Scan();
> scan.setFilter(new MultiRowRangeFilter(rowRangesList));
>
>ResultScanner scanner=scanTable.getScanner(scan);
>
> for (Result result : scanner) {
> String rowkey=new String(result.getRow());
>System.out.println(rowkey);
>
> }
> }
>
> In Hbase API of Java, we want to do multiple scans in the table using
> MultiRowRangeFilter.
> When we give multiple filters of startRowKey and stopRowKey, it is not
> working Properly with same StartRowKey and StopRowkey.
> Ideally, it should give only one Row with that Rowkey, but instead it is
> giving all the rows starting from that Rowkey in that Hbase Table
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17688) MultiRowRangeFilter not working correctly if given same start and stop RowKey
[
https://issues.apache.org/jira/browse/HBASE-17688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingcheng Du updated HBASE-17688:
-
Attachment: (was: HBASE-17688.master-v1.patch)
> MultiRowRangeFilter not working correctly if given same start and stop RowKey
> -
>
> Key: HBASE-17688
> URL: https://issues.apache.org/jira/browse/HBASE-17688
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.0, 1.2.4, 0.98.24, 1.1.8
>Reporter: Ravi Ahuj
>Assignee: Jingcheng Du
>Priority: Minor
> Attachments: HBASE-17688.branch-1.x.patch, HBASE-17688.master.patch
>
>
>
>
> try (final Connection conn = ConnectionFactory.createConnection(conf);
>final Table scanTable = conn.getTable(table)){
> ArrayList rowRangesList = new
> ArrayList<>();
>
> String startRowkey="abc";
> String stopRowkey="abc";
> rowRangesList.add(new MultiRowRangeFilter.RowRange(startRowkey,
> true, stopRowkey, true));
> Scan scan = new Scan();
> scan.setFilter(new MultiRowRangeFilter(rowRangesList));
>
>ResultScanner scanner=scanTable.getScanner(scan);
>
> for (Result result : scanner) {
> String rowkey=new String(result.getRow());
>System.out.println(rowkey);
>
> }
> }
>
> In Hbase API of Java, we want to do multiple scans in the table using
> MultiRowRangeFilter.
> When we give multiple filters of startRowKey and stopRowKey, it is not
> working Properly with same StartRowKey and StopRowkey.
> Ideally, it should give only one Row with that Rowkey, but instead it is
> giving all the rows starting from that Rowkey in that Hbase Table
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17688) MultiRowRangeFilter not working correctly if given same start and stop RowKey
[
https://issues.apache.org/jira/browse/HBASE-17688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885411#comment-15885411
]
Jingcheng Du commented on HBASE-17688:
--
Upload a patch for branch-1.x and another new patch for master branch.
Hi [~anoop.hbase], are you alright with these patches? Thanks.
> MultiRowRangeFilter not working correctly if given same start and stop RowKey
> -
>
> Key: HBASE-17688
> URL: https://issues.apache.org/jira/browse/HBASE-17688
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.0, 1.2.4, 0.98.24, 1.1.8
>Reporter: Ravi Ahuj
>Assignee: Jingcheng Du
>Priority: Minor
> Attachments: HBASE-17688.branch-1.x.patch, HBASE-17688.master.patch,
> HBASE-17688.master-v1.patch
>
>
>
>
> try (final Connection conn = ConnectionFactory.createConnection(conf);
>final Table scanTable = conn.getTable(table)){
> ArrayList rowRangesList = new
> ArrayList<>();
>
> String startRowkey="abc";
> String stopRowkey="abc";
> rowRangesList.add(new MultiRowRangeFilter.RowRange(startRowkey,
> true, stopRowkey, true));
> Scan scan = new Scan();
> scan.setFilter(new MultiRowRangeFilter(rowRangesList));
>
>ResultScanner scanner=scanTable.getScanner(scan);
>
> for (Result result : scanner) {
> String rowkey=new String(result.getRow());
>System.out.println(rowkey);
>
> }
> }
>
> In Hbase API of Java, we want to do multiple scans in the table using
> MultiRowRangeFilter.
> When we give multiple filters of startRowKey and stopRowKey, it is not
> working Properly with same StartRowKey and StopRowkey.
> Ideally, it should give only one Row with that Rowkey, but instead it is
> giving all the rows starting from that Rowkey in that Hbase Table
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17688) MultiRowRangeFilter not working correctly if given same start and stop RowKey
[
https://issues.apache.org/jira/browse/HBASE-17688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingcheng Du updated HBASE-17688:
-
Attachment: HBASE-17688.branch-1.x.patch
> MultiRowRangeFilter not working correctly if given same start and stop RowKey
> -
>
> Key: HBASE-17688
> URL: https://issues.apache.org/jira/browse/HBASE-17688
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.0, 1.2.4, 0.98.24, 1.1.8
>Reporter: Ravi Ahuj
>Assignee: Jingcheng Du
>Priority: Minor
> Attachments: HBASE-17688.branch-1.x.patch, HBASE-17688.master.patch,
> HBASE-17688.master-v1.patch
>
>
>
>
> try (final Connection conn = ConnectionFactory.createConnection(conf);
>final Table scanTable = conn.getTable(table)){
> ArrayList rowRangesList = new
> ArrayList<>();
>
> String startRowkey="abc";
> String stopRowkey="abc";
> rowRangesList.add(new MultiRowRangeFilter.RowRange(startRowkey,
> true, stopRowkey, true));
> Scan scan = new Scan();
> scan.setFilter(new MultiRowRangeFilter(rowRangesList));
>
>ResultScanner scanner=scanTable.getScanner(scan);
>
> for (Result result : scanner) {
> String rowkey=new String(result.getRow());
>System.out.println(rowkey);
>
> }
> }
>
> In Hbase API of Java, we want to do multiple scans in the table using
> MultiRowRangeFilter.
> When we give multiple filters of startRowKey and stopRowKey, it is not
> working Properly with same StartRowKey and StopRowkey.
> Ideally, it should give only one Row with that Rowkey, but instead it is
> giving all the rows starting from that Rowkey in that Hbase Table
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17688) MultiRowRangeFilter not working correctly if given same start and stop RowKey
[
https://issues.apache.org/jira/browse/HBASE-17688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingcheng Du updated HBASE-17688:
-
Attachment: HBASE-17688.master-v1.patch
> MultiRowRangeFilter not working correctly if given same start and stop RowKey
> -
>
> Key: HBASE-17688
> URL: https://issues.apache.org/jira/browse/HBASE-17688
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.0, 1.2.4, 0.98.24, 1.1.8
>Reporter: Ravi Ahuj
>Assignee: Jingcheng Du
>Priority: Minor
> Attachments: HBASE-17688.master.patch, HBASE-17688.master-v1.patch
>
>
>
>
> try (final Connection conn = ConnectionFactory.createConnection(conf);
>final Table scanTable = conn.getTable(table)){
> ArrayList rowRangesList = new
> ArrayList<>();
>
> String startRowkey="abc";
> String stopRowkey="abc";
> rowRangesList.add(new MultiRowRangeFilter.RowRange(startRowkey,
> true, stopRowkey, true));
> Scan scan = new Scan();
> scan.setFilter(new MultiRowRangeFilter(rowRangesList));
>
>ResultScanner scanner=scanTable.getScanner(scan);
>
> for (Result result : scanner) {
> String rowkey=new String(result.getRow());
>System.out.println(rowkey);
>
> }
> }
>
> In Hbase API of Java, we want to do multiple scans in the table using
> MultiRowRangeFilter.
> When we give multiple filters of startRowKey and stopRowKey, it is not
> working Properly with same StartRowKey and StopRowkey.
> Ideally, it should give only one Row with that Rowkey, but instead it is
> giving all the rows starting from that Rowkey in that Hbase Table
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17688) MultiRowRangeFilter not working correctly if given same start and stop RowKey
[
https://issues.apache.org/jira/browse/HBASE-17688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885404#comment-15885404
]
Jingcheng Du commented on HBASE-17688:
--
Right. They keep the same logic.
> MultiRowRangeFilter not working correctly if given same start and stop RowKey
> -
>
> Key: HBASE-17688
> URL: https://issues.apache.org/jira/browse/HBASE-17688
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.0, 1.2.4, 0.98.24, 1.1.8
>Reporter: Ravi Ahuj
>Assignee: Jingcheng Du
>Priority: Minor
> Attachments: HBASE-17688.master.patch
>
>
>
>
> try (final Connection conn = ConnectionFactory.createConnection(conf);
>final Table scanTable = conn.getTable(table)){
> ArrayList rowRangesList = new
> ArrayList<>();
>
> String startRowkey="abc";
> String stopRowkey="abc";
> rowRangesList.add(new MultiRowRangeFilter.RowRange(startRowkey,
> true, stopRowkey, true));
> Scan scan = new Scan();
> scan.setFilter(new MultiRowRangeFilter(rowRangesList));
>
>ResultScanner scanner=scanTable.getScanner(scan);
>
> for (Result result : scanner) {
> String rowkey=new String(result.getRow());
>System.out.println(rowkey);
>
> }
> }
>
> In Hbase API of Java, we want to do multiple scans in the table using
> MultiRowRangeFilter.
> When we give multiple filters of startRowKey and stopRowKey, it is not
> working Properly with same StartRowKey and StopRowkey.
> Ideally, it should give only one Row with that Rowkey, but instead it is
> giving all the rows starting from that Rowkey in that Hbase Table
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17688) MultiRowRangeFilter not working correctly if given same start and stop RowKey
[
https://issues.apache.org/jira/browse/HBASE-17688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885237#comment-15885237
]
Jingcheng Du commented on HBASE-17688:
--
bq. If make a Scan request (with no filters) and with same start and end keys,
what will be the behavior?
If start and stop keys are equal and the {{includeStopRow}} is true, it is a
get.
If {{includeStopRow}} is false, nothing returns.
> MultiRowRangeFilter not working correctly if given same start and stop RowKey
> -
>
> Key: HBASE-17688
> URL: https://issues.apache.org/jira/browse/HBASE-17688
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.0, 1.2.4, 0.98.24, 1.1.8
>Reporter: Ravi Ahuj
>Assignee: Jingcheng Du
>Priority: Minor
> Attachments: HBASE-17688.master.patch
>
>
>
>
> try (final Connection conn = ConnectionFactory.createConnection(conf);
>final Table scanTable = conn.getTable(table)){
> ArrayList rowRangesList = new
> ArrayList<>();
>
> String startRowkey="abc";
> String stopRowkey="abc";
> rowRangesList.add(new MultiRowRangeFilter.RowRange(startRowkey,
> true, stopRowkey, true));
> Scan scan = new Scan();
> scan.setFilter(new MultiRowRangeFilter(rowRangesList));
>
>ResultScanner scanner=scanTable.getScanner(scan);
>
> for (Result result : scanner) {
> String rowkey=new String(result.getRow());
>System.out.println(rowkey);
>
> }
> }
>
> In Hbase API of Java, we want to do multiple scans in the table using
> MultiRowRangeFilter.
> When we give multiple filters of startRowKey and stopRowKey, it is not
> working Properly with same StartRowKey and StopRowkey.
> Ideally, it should give only one Row with that Rowkey, but instead it is
> giving all the rows starting from that Rowkey in that Hbase Table
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17690) Clean up MOB code
[ https://issues.apache.org/jira/browse/HBASE-17690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885174#comment-15885174 ] Jingcheng Du commented on HBASE-17690: -- bq. Is the above just an estimate (not all store files have the same size) ? You are right, Ted. It is just an estimate, the same logic in Compactor#performCompaction. > Clean up MOB code > - > > Key: HBASE-17690 > URL: https://issues.apache.org/jira/browse/HBASE-17690 > Project: HBase > Issue Type: Improvement > Components: mob >Reporter: Jingcheng Du >Assignee: Jingcheng Du > Attachments: HBASE-17690.patch > > > Clean up the code in MOB. > # Fix the incorrect description in comments. > # Fix the warning and remove redundant import in code. > # Remove the references to the deprecated code. > # Add throughput controller for DefaultMobStoreFlusher and > DefaultMobStoreCompactor. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17690) Clean up MOB code
[ https://issues.apache.org/jira/browse/HBASE-17690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885168#comment-15885168 ] Jingcheng Du commented on HBASE-17690: -- Thanks [~yuzhih...@gmail.com] for the comments! The RB currently goes down, I will upload it as soon as it comes back. Thanks! > Clean up MOB code > - > > Key: HBASE-17690 > URL: https://issues.apache.org/jira/browse/HBASE-17690 > Project: HBase > Issue Type: Improvement > Components: mob >Reporter: Jingcheng Du >Assignee: Jingcheng Du > Attachments: HBASE-17690.patch > > > Clean up the code in MOB. > # Fix the incorrect description in comments. > # Fix the warning and remove redundant import in code. > # Remove the references to the deprecated code. > # Add throughput controller for DefaultMobStoreFlusher and > DefaultMobStoreCompactor. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17688) MultiRowRangeFilter not working correctly if given same start and stop RowKey
[
https://issues.apache.org/jira/browse/HBASE-17688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingcheng Du updated HBASE-17688:
-
Status: Patch Available (was: Open)
> MultiRowRangeFilter not working correctly if given same start and stop RowKey
> -
>
> Key: HBASE-17688
> URL: https://issues.apache.org/jira/browse/HBASE-17688
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.1.8, 0.98.24, 1.2.4, 1.3.0, 2.0.0
>Reporter: Ravi Ahuj
>Assignee: Jingcheng Du
>Priority: Minor
> Attachments: HBASE-17688.master.patch
>
>
>
>
> try (final Connection conn = ConnectionFactory.createConnection(conf);
>final Table scanTable = conn.getTable(table)){
> ArrayList rowRangesList = new
> ArrayList<>();
>
> String startRowkey="abc";
> String stopRowkey="abc";
> rowRangesList.add(new MultiRowRangeFilter.RowRange(startRowkey,
> true, stopRowkey, true));
> Scan scan = new Scan();
> scan.setFilter(new MultiRowRangeFilter(rowRangesList));
>
>ResultScanner scanner=scanTable.getScanner(scan);
>
> for (Result result : scanner) {
> String rowkey=new String(result.getRow());
>System.out.println(rowkey);
>
> }
> }
>
> In Hbase API of Java, we want to do multiple scans in the table using
> MultiRowRangeFilter.
> When we give multiple filters of startRowKey and stopRowKey, it is not
> working Properly with same StartRowKey and StopRowkey.
> Ideally, it should give only one Row with that Rowkey, but instead it is
> giving all the rows starting from that Rowkey in that Hbase Table
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17688) MultiRowRangeFilter not working correctly if given same start and stop RowKey
[
https://issues.apache.org/jira/browse/HBASE-17688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingcheng Du updated HBASE-17688:
-
Affects Version/s: (was: 1.1.2)
2.0.0
1.3.0
1.2.4
0.98.24
1.1.8
> MultiRowRangeFilter not working correctly if given same start and stop RowKey
> -
>
> Key: HBASE-17688
> URL: https://issues.apache.org/jira/browse/HBASE-17688
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.0, 1.2.4, 0.98.24, 1.1.8
>Reporter: Ravi Ahuj
>Assignee: Jingcheng Du
>Priority: Minor
> Attachments: HBASE-17688.master.patch
>
>
>
>
> try (final Connection conn = ConnectionFactory.createConnection(conf);
>final Table scanTable = conn.getTable(table)){
> ArrayList rowRangesList = new
> ArrayList<>();
>
> String startRowkey="abc";
> String stopRowkey="abc";
> rowRangesList.add(new MultiRowRangeFilter.RowRange(startRowkey,
> true, stopRowkey, true));
> Scan scan = new Scan();
> scan.setFilter(new MultiRowRangeFilter(rowRangesList));
>
>ResultScanner scanner=scanTable.getScanner(scan);
>
> for (Result result : scanner) {
> String rowkey=new String(result.getRow());
>System.out.println(rowkey);
>
> }
> }
>
> In Hbase API of Java, we want to do multiple scans in the table using
> MultiRowRangeFilter.
> When we give multiple filters of startRowKey and stopRowKey, it is not
> working Properly with same StartRowKey and StopRowkey.
> Ideally, it should give only one Row with that Rowkey, but instead it is
> giving all the rows starting from that Rowkey in that Hbase Table
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17688) MultiRowRangeFilter not working correctly if given same start and stop RowKey
[
https://issues.apache.org/jira/browse/HBASE-17688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingcheng Du updated HBASE-17688:
-
Attachment: HBASE-17688.master.patch
> MultiRowRangeFilter not working correctly if given same start and stop RowKey
> -
>
> Key: HBASE-17688
> URL: https://issues.apache.org/jira/browse/HBASE-17688
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.1.2
>Reporter: Ravi Ahuj
>Assignee: Jingcheng Du
>Priority: Minor
> Attachments: HBASE-17688.master.patch
>
>
>
>
> try (final Connection conn = ConnectionFactory.createConnection(conf);
>final Table scanTable = conn.getTable(table)){
> ArrayList rowRangesList = new
> ArrayList<>();
>
> String startRowkey="abc";
> String stopRowkey="abc";
> rowRangesList.add(new MultiRowRangeFilter.RowRange(startRowkey,
> true, stopRowkey, true));
> Scan scan = new Scan();
> scan.setFilter(new MultiRowRangeFilter(rowRangesList));
>
>ResultScanner scanner=scanTable.getScanner(scan);
>
> for (Result result : scanner) {
> String rowkey=new String(result.getRow());
>System.out.println(rowkey);
>
> }
> }
>
> In Hbase API of Java, we want to do multiple scans in the table using
> MultiRowRangeFilter.
> When we give multiple filters of startRowKey and stopRowKey, it is not
> working Properly with same StartRowKey and StopRowkey.
> Ideally, it should give only one Row with that Rowkey, but instead it is
> giving all the rows starting from that Rowkey in that Hbase Table
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17688) MultiRowRangeFilter not working correctly if given same start and stop RowKey
[
https://issues.apache.org/jira/browse/HBASE-17688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingcheng Du updated HBASE-17688:
-
Attachment: (was: HBASE-17688.master.patch)
> MultiRowRangeFilter not working correctly if given same start and stop RowKey
> -
>
> Key: HBASE-17688
> URL: https://issues.apache.org/jira/browse/HBASE-17688
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.1.2
>Reporter: Ravi Ahuj
>Assignee: Jingcheng Du
>Priority: Minor
>
>
>
> try (final Connection conn = ConnectionFactory.createConnection(conf);
>final Table scanTable = conn.getTable(table)){
> ArrayList rowRangesList = new
> ArrayList<>();
>
> String startRowkey="abc";
> String stopRowkey="abc";
> rowRangesList.add(new MultiRowRangeFilter.RowRange(startRowkey,
> true, stopRowkey, true));
> Scan scan = new Scan();
> scan.setFilter(new MultiRowRangeFilter(rowRangesList));
>
>ResultScanner scanner=scanTable.getScanner(scan);
>
> for (Result result : scanner) {
> String rowkey=new String(result.getRow());
>System.out.println(rowkey);
>
> }
> }
>
> In Hbase API of Java, we want to do multiple scans in the table using
> MultiRowRangeFilter.
> When we give multiple filters of startRowKey and stopRowKey, it is not
> working Properly with same StartRowKey and StopRowkey.
> Ideally, it should give only one Row with that Rowkey, but instead it is
> giving all the rows starting from that Rowkey in that Hbase Table
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17688) MultiRowRangeFilter not working correctly if given same start and stop RowKey
[
https://issues.apache.org/jira/browse/HBASE-17688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingcheng Du updated HBASE-17688:
-
Attachment: HBASE-17688.master.patch
Upload the patch for mater branch for review.
> MultiRowRangeFilter not working correctly if given same start and stop RowKey
> -
>
> Key: HBASE-17688
> URL: https://issues.apache.org/jira/browse/HBASE-17688
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.1.2
>Reporter: Ravi Ahuj
>Assignee: Jingcheng Du
>Priority: Minor
> Attachments: HBASE-17688.master.patch
>
>
>
>
> try (final Connection conn = ConnectionFactory.createConnection(conf);
>final Table scanTable = conn.getTable(table)){
> ArrayList rowRangesList = new
> ArrayList<>();
>
> String startRowkey="abc";
> String stopRowkey="abc";
> rowRangesList.add(new MultiRowRangeFilter.RowRange(startRowkey,
> true, stopRowkey, true));
> Scan scan = new Scan();
> scan.setFilter(new MultiRowRangeFilter(rowRangesList));
>
>ResultScanner scanner=scanTable.getScanner(scan);
>
> for (Result result : scanner) {
> String rowkey=new String(result.getRow());
>System.out.println(rowkey);
>
> }
> }
>
> In Hbase API of Java, we want to do multiple scans in the table using
> MultiRowRangeFilter.
> When we give multiple filters of startRowKey and stopRowKey, it is not
> working Properly with same StartRowKey and StopRowkey.
> Ideally, it should give only one Row with that Rowkey, but instead it is
> giving all the rows starting from that Rowkey in that Hbase Table
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HBASE-17688) MultiRowRangeFilter not working correctly if given same start and stop RowKey
[
https://issues.apache.org/jira/browse/HBASE-17688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingcheng Du reassigned HBASE-17688:
Assignee: Jingcheng Du
> MultiRowRangeFilter not working correctly if given same start and stop RowKey
> -
>
> Key: HBASE-17688
> URL: https://issues.apache.org/jira/browse/HBASE-17688
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.1.2
>Reporter: Ravi Ahuj
>Assignee: Jingcheng Du
>Priority: Minor
>
>
>
> try (final Connection conn = ConnectionFactory.createConnection(conf);
>final Table scanTable = conn.getTable(table)){
> ArrayList rowRangesList = new
> ArrayList<>();
>
> String startRowkey="abc";
> String stopRowkey="abc";
> rowRangesList.add(new MultiRowRangeFilter.RowRange(startRowkey,
> true, stopRowkey, true));
> Scan scan = new Scan();
> scan.setFilter(new MultiRowRangeFilter(rowRangesList));
>
>ResultScanner scanner=scanTable.getScanner(scan);
>
> for (Result result : scanner) {
> String rowkey=new String(result.getRow());
>System.out.println(rowkey);
>
> }
> }
>
> In Hbase API of Java, we want to do multiple scans in the table using
> MultiRowRangeFilter.
> When we give multiple filters of startRowKey and stopRowKey, it is not
> working Properly with same StartRowKey and StopRowkey.
> Ideally, it should give only one Row with that Rowkey, but instead it is
> giving all the rows starting from that Rowkey in that Hbase Table
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17688) MultiRowRangeFilter not working correctly if given same start and stop RowKey
[
https://issues.apache.org/jira/browse/HBASE-17688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15882171#comment-15882171
]
Jingcheng Du commented on HBASE-17688:
--
This issue exists in other branches too. It is caused by the issue in
{{RowRange.contains}} by wrongly using isScan.
Hi [~ahujaravi1], do you want provide the patch? Or I can do it as well. Thanks.
> MultiRowRangeFilter not working correctly if given same start and stop RowKey
> -
>
> Key: HBASE-17688
> URL: https://issues.apache.org/jira/browse/HBASE-17688
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.1.2
>Reporter: Ravi Ahuj
>Priority: Minor
>
>
>
> try (final Connection conn = ConnectionFactory.createConnection(conf);
>final Table scanTable = conn.getTable(table)){
> ArrayList rowRangesList = new
> ArrayList<>();
>
> String startRowkey="abc";
> String stopRowkey="abc";
> rowRangesList.add(new MultiRowRangeFilter.RowRange(startRowkey,
> true, stopRowkey, true));
> Scan scan = new Scan();
> scan.setFilter(new MultiRowRangeFilter(rowRangesList));
>
>ResultScanner scanner=scanTable.getScanner(scan);
>
> for (Result result : scanner) {
> String rowkey=new String(result.getRow());
>System.out.println(rowkey);
>
> }
> }
>
> In Hbase API of Java, we want to do multiple scans in the table using
> MultiRowRangeFilter.
> When we give multiple filters of startRowKey and stopRowKey, it is not
> working Properly with same StartRowKey and StopRowkey.
> Ideally, it should give only one Row with that Rowkey, but instead it is
> giving all the rows starting from that Rowkey in that Hbase Table
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17690) Clean up MOB code
[ https://issues.apache.org/jira/browse/HBASE-17690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingcheng Du updated HBASE-17690: - Status: Patch Available (was: Open) > Clean up MOB code > - > > Key: HBASE-17690 > URL: https://issues.apache.org/jira/browse/HBASE-17690 > Project: HBase > Issue Type: Improvement > Components: mob >Reporter: Jingcheng Du >Assignee: Jingcheng Du > Attachments: HBASE-17690.patch > > > Clean up the code in MOB. > # Fix the incorrect description in comments. > # Fix the warning and remove redundant import in code. > # Remove the references to the deprecated code. > # Add throughput controller for DefaultMobStoreFlusher and > DefaultMobStoreCompactor. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17690) Clean up MOB code
[ https://issues.apache.org/jira/browse/HBASE-17690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingcheng Du updated HBASE-17690: - Description: Clean up the code in MOB. # Fix the incorrect description in comments. # Fix the warning and remove redundant import in code. # Remove the references to the deprecated code. # Add throughput controller for DefaultMobStoreFlusher and DefaultMobStoreCompactor. was: Clean up the code in MOB. # Fix the incorrect description in comments. # Fix the warning and remove redundant reference in code. # Correct the code used in uni test. # Add throughput controller for DefaultMobStoreFlusher and DefaultMobStoreCompactor. > Clean up MOB code > - > > Key: HBASE-17690 > URL: https://issues.apache.org/jira/browse/HBASE-17690 > Project: HBase > Issue Type: Improvement > Components: mob >Reporter: Jingcheng Du >Assignee: Jingcheng Du > Attachments: HBASE-17690.patch > > > Clean up the code in MOB. > # Fix the incorrect description in comments. > # Fix the warning and remove redundant import in code. > # Remove the references to the deprecated code. > # Add throughput controller for DefaultMobStoreFlusher and > DefaultMobStoreCompactor. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17690) Clean up MOB code
[ https://issues.apache.org/jira/browse/HBASE-17690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingcheng Du updated HBASE-17690: - Attachment: HBASE-17690.patch Upload the first patch for review. > Clean up MOB code > - > > Key: HBASE-17690 > URL: https://issues.apache.org/jira/browse/HBASE-17690 > Project: HBase > Issue Type: Improvement > Components: mob >Reporter: Jingcheng Du >Assignee: Jingcheng Du > Attachments: HBASE-17690.patch > > > Clean up the code in MOB. > # Fix the incorrect description in comments. > # Fix the warning and remove redundant reference in code. > # Correct the code used in uni test. > # Add throughput controller for DefaultMobStoreFlusher and > DefaultMobStoreCompactor. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
