[jira] [Commented] (HBASE-19507) Get or Scan Mob by rowkey return error value when run compact_mob or major_compact_mob after change MOB_THRESHOLD bigger
[
https://issues.apache.org/jira/browse/HBASE-19507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16292108#comment-16292108
]
WangYuan commented on HBASE-19507:
--
[~jingcheng.du] Yes,thank you!
[~huaxiang] Add an empty mobfile is a good idea ,thank you!
> Get or Scan Mob by rowkey return error value when run compact_mob or
> major_compact_mob after change MOB_THRESHOLD bigger
>
>
> Key: HBASE-19507
> URL: https://issues.apache.org/jira/browse/HBASE-19507
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: WangYuan
>Assignee: huaxiang sun
>
> 1、
> create 'abc',{NAME => 'cf', MOB_THRESHOLD => '10', IS_MOB => 'true'}
> put 'abc','1','cf:a','1'
> put 'abc','2','cf:a','2'
> put 'abc','3','cf:a','3'
> put 'abc','4','cf:a','y1'
> put 'abc','5','cf:a','y2'
> put 'abc','6','cf:a','y3'
>
> hbase(main):011:0> scan 'abc'
> ROWCOLUMN+CELL
>
>
> 1 column=cf:a,
> timestamp=1513171753098, value=1
>
> 2 column=cf:a,
> timestamp=1513171753208, value=2
>
> 3 column=cf:a,
> timestamp=1513171753246, value=3
>
> 4 column=cf:a,
> timestamp=1513171753273, value=y1
>
> 5 column=cf:a,
> timestamp=1513171753301, value=y2
>
> 6 column=cf:a,
> timestamp=1513171754282, value=y3
>
> hbase(main):012:0> flush 'abc'
> hbase(main):012:0> major_compact 'abc'
> hbase(main):012:0> major_compact_mob 'abc'
> 2、
> [See Hfile]:
> hbase org.apache.hadoop.hbase.io.hfile.HFile -f
> /hbase/data/default/abc/a31b3146cba0d4569a7bf44e70e299c9/cf/22a432ba5c2c4802bedd947b99626f10
> -p
> K: 1/cf:a/1513172294864/Put/vlen=5/seqid=4 V: 1
> K: 2/cf:a/1513172294892/Put/vlen=5/seqid=5 V: 2
> K: 3/cf:a/1513172294914/Put/vlen=5/seqid=6 V: 3
> K: 4/cf:a/1513172294954/Put/vlen=76/seqid=7 V:
> \x00\x00\x00\x0Ed41d8cd98f00b204e9800998ecf8427e20171213ce022548c4c3498e864fda289b81e711
> T[0]: T[1]: abc
> K: 5/cf:a/1513172294982/Put/vlen=76/seqid=8 V:
> \x00\x00\x00\x0Ed41d8cd98f00b204e9800998ecf8427e20171213ce022548c4c3498e864fda289b81e711
> T[0]: T[1]: abc
> K: 6/cf:a/1513172296455/Put/vlen=76/seqid=9 V:
> \x00\x00\x00\x0Ed41d8cd98f00b204e9800998ecf8427e20171213ce022548c4c3498e864fda289b81e711
> T[0]: T[1]: abc
> Scanned kv count -> 6
> [See Mobfile]:
> hbase org.apache.hadoop.hbase.io.hfile.HFile -f
> /hbase/mobdir/data/default/abc/07aab825b62dd9111831839cc9039df9/cf/d41d8cd98f00b204e9800998ecf8427e20171213bd8cfaf146684d4096ebf7994f050e96
> -p
> K: 4/cf:a/1513172924196/Put/vlen=14/seqid=7 V: y1
> K: 5/cf:a/1513172924214/Put/vlen=14/seqid=8 V: y2
> K: 6/cf:a/1513172925768/Put/vlen=14/seqid=9 V: y3
> 3、
> alter 'abc',{NAME => 'cf', MOB_THRESHOLD => '10240' }
> put
>
[jira] [Comment Edited] (HBASE-19507) Get or Scan Mob by rowkey return error value when run compact_mob or major_compact_mob after change MOB_THRESHOLD bigger
[
https://issues.apache.org/jira/browse/HBASE-19507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16291981#comment-16291981
]
WangYuan edited comment on HBASE-19507 at 12/15/17 3:27 AM:
Thank U [~jingcheng.du] [~huaxiang]
I found the reason:
The hfile version need modify to V3,but the old conf is V2。
The configuration value in hbase-site.xml is
Then I change conf value from 2 to 3, then switch(stop-start) hmaster , then
major_mob ,it's OK.
But I need to do something else :
A、If more then 1 mobfile with same mobDate in 1 region ,it's OK after mob
majorcompact.
B、But， if only 1 mobfile(or 1 mobfile with only 1 single mobDate ) in 1 region
,it can't recovery becaues majorcompact need 2 files at least, So I have to do :
b1、 put 1 new record into region then flush
b2、modify new and old mobfile's mobDate to same mobDate
b3、majorcompact it to recovery data.
b4、delete the new record
C、notify mobfile's size and change it ,may be can't to majorcompact mob because
hbase.mob.file.compaction.mergeable.threshold is 192M.
was (Author: wangyuan):
Thank U [~jingcheng.du] [~huaxiang]
I found the reason:
The hfile version need modify to V3,but the old conf is V2。
The configuration value in hbase-site.xml is
Then I change conf value from 2 to 3, then switch(stop-start) hmaster , then
major_mob ,it's OK.
But I need to do something else :
A、If more then 1 mobfile with same mobDate in 1 region ,it's OK after mob
majorcompact.
B、But， if only 1 mobfile(or 1 mobfile with only 1 single mobDate ) in 1 region
,it can't recovery becaues majorcompact need 2 files at least, So I have to do :
b1、 put 1 new record into region then flush
b2、modify new and old mobfile's mobDate to same mobDate
b3、majorcompact it to recovery data.
C、notify mobfile's size and change it ,may be can't to majorcompact mob because
hbase.mob.file.compaction.mergeable.threshold is 192M.
> Get or Scan Mob by rowkey return error value when run compact_mob or
> major_compact_mob after change MOB_THRESHOLD bigger
>
>
> Key: HBASE-19507
> URL: https://issues.apache.org/jira/browse/HBASE-19507
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: WangYuan
>Assignee: huaxiang sun
>
> 1、
> create 'abc',{NAME => 'cf', MOB_THRESHOLD => '10', IS_MOB => 'true'}
> put 'abc','1','cf:a','1'
> put 'abc','2','cf:a','2'
> put 'abc','3','cf:a','3'
> put 'abc','4','cf:a','y1'
> put 'abc','5','cf:a','y2'
> put 'abc','6','cf:a','y3'
>
> hbase(main):011:0> scan 'abc'
> ROWCOLUMN+CELL
>
>
> 1 column=cf:a,
> timestamp=1513171753098, value=1
>
> 2 column=cf:a,
> timestamp=1513171753208, value=2
>
> 3 column=cf:a,
> timestamp=1513171753246, value=3
>
> 4 column=cf:a,
> timestamp=1513171753273, value=y1
>
> 5 column=cf:a,
> timestamp=1513171753301, value=y2
>
> 6 column=cf:a,
> timestamp=1513171754282, value=y3
>
> hbase(main):012:0> flush 'abc'
> hbase(main):012:0> major_compact 'abc'
> hbase(main):012:0> major_compact_mob 'abc'
> 2、
> [See Hfile]:
> hbase org.apache.hadoop.hbase.io.hfile.HFile -f
> /hbase/data/default/abc/a31b3146cba0d4569a7bf44e70e299c9/cf/22a432ba5c2c4802bedd947b99626f10
> -p
> K: 1/cf:a/1513172294864/Put/vlen=5/seqid=4 V: 1
> K: 2/cf:a/1513172294892/Put/vlen=5/seqid=5 V: 2
> K: 3/cf:a/1513172294914/Put/vlen=5/seqid=6 V: 3
> K: 4/cf:a/1513172294954/Put/vlen=76/seqid=7 V:
> \x00\x00\x00\x0Ed41d8cd98f00b204e9800998ecf8427e20171213ce022548c4c3498e864fda289b81e711
> T[0]: T[1]: abc
> K: 5/cf:a/1513172294982/Put/vlen=76/seqid=8 V:
> \x00\x00\x00\x0Ed41d8cd98f00b204e9800998ecf8427e20171213ce022548c4c3498e864fda289b81e711
> T[0]: T[1]: abc
> K: 6/cf:a/1513172296455/Put/vlen=76/seqid=9 V:
>
[jira] [Comment Edited] (HBASE-19507) Get or Scan Mob by rowkey return error value when run compact_mob or major_compact_mob after change MOB_THRESHOLD bigger
[
https://issues.apache.org/jira/browse/HBASE-19507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16291981#comment-16291981
]
WangYuan edited comment on HBASE-19507 at 12/15/17 3:26 AM:
Thank U [~jingcheng.du] [~huaxiang]
I found the reason:
The hfile version need modify to V3,but the old conf is V2。
The configuration value in hbase-site.xml is
Then I change conf value from 2 to 3, then switch(stop-start) hmaster , then
major_mob ,it's OK.
But I need to do something else :
A、If more then 1 mobfile with same mobDate in 1 region ,it's OK after mob
majorcompact.
B、But， if only 1 mobfile(or 1 mobfile with only 1 single mobDate ) in 1 region
,it can't recovery becaues majorcompact need 2 files at least, So I have to do :
b1、 put 1 new record into region then flush
b2、modify new and old mobfile's mobDate to same mobDate
b3、majorcompact it to recovery data.
C、notify mobfile's size and change it ,may be can't to majorcompact mob because
hbase.mob.file.compaction.mergeable.threshold is 192M.
was (Author: wangyuan):
Thank U [~jingcheng.du]
I found the reason:
The hfile version need modify to V3,but the old conf is V2。
The configuration value in hbase-site.xml is
Then I change conf value from 2 to 3, then switch(stop-start) hmaster , then
major_mob ,it's OK.
But I need to do something else :
A、If more then 1 mobfile with same mobDate in 1 region ,it's OK after mob
majorcompact.
B、But， if only 1 mobfile(or 1 mobfile with only 1 single mobDate ) in 1 region
,it can't recovery becaues majorcompact need 2 files at least, So I have to do :
b1、 put 1 new record into region then flush
b2、modify new and old mobfile's mobDate to same mobDate
b3、majorcompact it to recovery data.
C、notify mobfile's size and change it ,may be can't to majorcompact mob because
hbase.mob.file.compaction.mergeable.threshold is 192M.
> Get or Scan Mob by rowkey return error value when run compact_mob or
> major_compact_mob after change MOB_THRESHOLD bigger
>
>
> Key: HBASE-19507
> URL: https://issues.apache.org/jira/browse/HBASE-19507
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: WangYuan
>Assignee: huaxiang sun
>
> 1、
> create 'abc',{NAME => 'cf', MOB_THRESHOLD => '10', IS_MOB => 'true'}
> put 'abc','1','cf:a','1'
> put 'abc','2','cf:a','2'
> put 'abc','3','cf:a','3'
> put 'abc','4','cf:a','y1'
> put 'abc','5','cf:a','y2'
> put 'abc','6','cf:a','y3'
>
> hbase(main):011:0> scan 'abc'
> ROWCOLUMN+CELL
>
>
> 1 column=cf:a,
> timestamp=1513171753098, value=1
>
> 2 column=cf:a,
> timestamp=1513171753208, value=2
>
> 3 column=cf:a,
> timestamp=1513171753246, value=3
>
> 4 column=cf:a,
> timestamp=1513171753273, value=y1
>
> 5 column=cf:a,
> timestamp=1513171753301, value=y2
>
> 6 column=cf:a,
> timestamp=1513171754282, value=y3
>
> hbase(main):012:0> flush 'abc'
> hbase(main):012:0> major_compact 'abc'
> hbase(main):012:0> major_compact_mob 'abc'
> 2、
> [See Hfile]:
> hbase org.apache.hadoop.hbase.io.hfile.HFile -f
> /hbase/data/default/abc/a31b3146cba0d4569a7bf44e70e299c9/cf/22a432ba5c2c4802bedd947b99626f10
> -p
> K: 1/cf:a/1513172294864/Put/vlen=5/seqid=4 V: 1
> K: 2/cf:a/1513172294892/Put/vlen=5/seqid=5 V: 2
> K: 3/cf:a/1513172294914/Put/vlen=5/seqid=6 V: 3
> K: 4/cf:a/1513172294954/Put/vlen=76/seqid=7 V:
> \x00\x00\x00\x0Ed41d8cd98f00b204e9800998ecf8427e20171213ce022548c4c3498e864fda289b81e711
> T[0]: T[1]: abc
> K: 5/cf:a/1513172294982/Put/vlen=76/seqid=8 V:
> \x00\x00\x00\x0Ed41d8cd98f00b204e9800998ecf8427e20171213ce022548c4c3498e864fda289b81e711
> T[0]: T[1]: abc
> K: 6/cf:a/1513172296455/Put/vlen=76/seqid=9 V:
> \x00\x00\x00\x0Ed41d8cd98f00b204e9800998ecf8427e20171213ce022548c4c3498e864fda289b81e711
> T[0]: T[1]: abc
> Scanned kv
[jira] [Commented] (HBASE-19507) Get or Scan Mob by rowkey return error value when run compact_mob or major_compact_mob after change MOB_THRESHOLD bigger
[
https://issues.apache.org/jira/browse/HBASE-19507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16291981#comment-16291981
]
WangYuan commented on HBASE-19507:
--
Thank U [~jingcheng.du]
I found the reason:
The hfile version need modify to V3,but the old conf is V2。
The configuration value in hbase-site.xml is
Then I change conf value from 2 to 3, then switch(stop-start) hmaster , then
major_mob ,it's OK.
But I need to do something else :
A、If more then 1 mobfile with same mobDate in 1 region ,it's OK after mob
majorcompact.
B、But， if only 1 mobfile(or 1 mobfile with only 1 single mobDate ) in 1 region
,it can't recovery becaues majorcompact need 2 files at least, So I have to do :
b1、 put 1 new record into region then flush
b2、modify new and old mobfile's mobDate to same mobDate
b3、majorcompact it to recovery data.
C、notify mobfile's size and change it ,may be can't to majorcompact mob because
hbase.mob.file.compaction.mergeable.threshold is 192M.
> Get or Scan Mob by rowkey return error value when run compact_mob or
> major_compact_mob after change MOB_THRESHOLD bigger
>
>
> Key: HBASE-19507
> URL: https://issues.apache.org/jira/browse/HBASE-19507
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: WangYuan
>Assignee: huaxiang sun
>
> 1、
> create 'abc',{NAME => 'cf', MOB_THRESHOLD => '10', IS_MOB => 'true'}
> put 'abc','1','cf:a','1'
> put 'abc','2','cf:a','2'
> put 'abc','3','cf:a','3'
> put 'abc','4','cf:a','y1'
> put 'abc','5','cf:a','y2'
> put 'abc','6','cf:a','y3'
>
> hbase(main):011:0> scan 'abc'
> ROWCOLUMN+CELL
>
>
> 1 column=cf:a,
> timestamp=1513171753098, value=1
>
> 2 column=cf:a,
> timestamp=1513171753208, value=2
>
> 3 column=cf:a,
> timestamp=1513171753246, value=3
>
> 4 column=cf:a,
> timestamp=1513171753273, value=y1
>
> 5 column=cf:a,
> timestamp=1513171753301, value=y2
>
> 6 column=cf:a,
> timestamp=1513171754282, value=y3
>
> hbase(main):012:0> flush 'abc'
> hbase(main):012:0> major_compact 'abc'
> hbase(main):012:0> major_compact_mob 'abc'
> 2、
> [See Hfile]:
> hbase org.apache.hadoop.hbase.io.hfile.HFile -f
> /hbase/data/default/abc/a31b3146cba0d4569a7bf44e70e299c9/cf/22a432ba5c2c4802bedd947b99626f10
> -p
> K: 1/cf:a/1513172294864/Put/vlen=5/seqid=4 V: 1
> K: 2/cf:a/1513172294892/Put/vlen=5/seqid=5 V: 2
> K: 3/cf:a/1513172294914/Put/vlen=5/seqid=6 V: 3
> K: 4/cf:a/1513172294954/Put/vlen=76/seqid=7 V:
> \x00\x00\x00\x0Ed41d8cd98f00b204e9800998ecf8427e20171213ce022548c4c3498e864fda289b81e711
> T[0]: T[1]: abc
> K: 5/cf:a/1513172294982/Put/vlen=76/seqid=8 V:
> \x00\x00\x00\x0Ed41d8cd98f00b204e9800998ecf8427e20171213ce022548c4c3498e864fda289b81e711
> T[0]: T[1]: abc
> K: 6/cf:a/1513172296455/Put/vlen=76/seqid=9 V:
> \x00\x00\x00\x0Ed41d8cd98f00b204e9800998ecf8427e20171213ce022548c4c3498e864fda289b81e711
> T[0]: T[1]: abc
> Scanned kv count -> 6
> [See Mobfile]:
> hbase org.apache.hadoop.hbase.io.hfile.HFile -f
> /hbase/mobdir/data/default/abc/07aab825b62dd9111831839cc9039df9/cf/d41d8cd98f00b204e9800998ecf8427e20171213bd8cfaf146684d4096ebf7994f050e96
> -p
> K: 4/cf:a/1513172924196/Put/vlen=14/seqid=7 V: y1
> K: 5/cf:a/1513172924214/Put/vlen=14/seqid=8 V: y2
> K: 6/cf:a/1513172925768/Put/vlen=14/seqid=9 V: y3
> 3、
> alter 'abc',{NAME => 'cf', MOB_THRESHOLD => '10240' }
> put
>
[jira] [Commented] (HBASE-19507) Get or Scan Mob by rowkey return error value when run compact_mob or major_compact_mob after change MOB_THRESHOLD bigger
[
https://issues.apache.org/jira/browse/HBASE-19507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16290325#comment-16290325
]
WangYuan commented on HBASE-19507:
--
Thank you for your reply,Jingcheng Du.
This error I found for CDH5.7.1(HBase1.2.0) . I will download HBase
master-branch project and test it later.
> Get or Scan Mob by rowkey return error value when run compact_mob or
> major_compact_mob after change MOB_THRESHOLD bigger
>
>
> Key: HBASE-19507
> URL: https://issues.apache.org/jira/browse/HBASE-19507
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: WangYuan
>
> 1、
> create 'abc',{NAME => 'cf', MOB_THRESHOLD => '10', IS_MOB => 'true'}
> put 'abc','1','cf:a','1'
> put 'abc','2','cf:a','2'
> put 'abc','3','cf:a','3'
> put 'abc','4','cf:a','y1'
> put 'abc','5','cf:a','y2'
> put 'abc','6','cf:a','y3'
>
> hbase(main):011:0> scan 'abc'
> ROWCOLUMN+CELL
>
>
> 1 column=cf:a,
> timestamp=1513171753098, value=1
>
> 2 column=cf:a,
> timestamp=1513171753208, value=2
>
> 3 column=cf:a,
> timestamp=1513171753246, value=3
>
> 4 column=cf:a,
> timestamp=1513171753273, value=y1
>
> 5 column=cf:a,
> timestamp=1513171753301, value=y2
>
> 6 column=cf:a,
> timestamp=1513171754282, value=y3
>
> hbase(main):012:0> flush 'abc'
> hbase(main):012:0> major_compact 'abc'
> hbase(main):012:0> major_compact_mob 'abc'
> 2、
> [See Hfile]:
> hbase org.apache.hadoop.hbase.io.hfile.HFile -f
> /hbase/data/default/abc/a31b3146cba0d4569a7bf44e70e299c9/cf/22a432ba5c2c4802bedd947b99626f10
> -p
> K: 1/cf:a/1513172294864/Put/vlen=5/seqid=4 V: 1
> K: 2/cf:a/1513172294892/Put/vlen=5/seqid=5 V: 2
> K: 3/cf:a/1513172294914/Put/vlen=5/seqid=6 V: 3
> K: 4/cf:a/1513172294954/Put/vlen=76/seqid=7 V:
> \x00\x00\x00\x0Ed41d8cd98f00b204e9800998ecf8427e20171213ce022548c4c3498e864fda289b81e711
> T[0]: T[1]: abc
> K: 5/cf:a/1513172294982/Put/vlen=76/seqid=8 V:
> \x00\x00\x00\x0Ed41d8cd98f00b204e9800998ecf8427e20171213ce022548c4c3498e864fda289b81e711
> T[0]: T[1]: abc
> K: 6/cf:a/1513172296455/Put/vlen=76/seqid=9 V:
> \x00\x00\x00\x0Ed41d8cd98f00b204e9800998ecf8427e20171213ce022548c4c3498e864fda289b81e711
> T[0]: T[1]: abc
> Scanned kv count -> 6
> [See Mobfile]:
> hbase org.apache.hadoop.hbase.io.hfile.HFile -f
> /hbase/mobdir/data/default/abc/07aab825b62dd9111831839cc9039df9/cf/d41d8cd98f00b204e9800998ecf8427e20171213bd8cfaf146684d4096ebf7994f050e96
> -p
> K: 4/cf:a/1513172924196/Put/vlen=14/seqid=7 V: y1
> K: 5/cf:a/1513172924214/Put/vlen=14/seqid=8 V: y2
> K: 6/cf:a/1513172925768/Put/vlen=14/seqid=9 V: y3
> 3、
> alter 'abc',{NAME => 'cf', MOB_THRESHOLD => '10240' }
> put
>
[jira] [Updated] (HBASE-19507) Get or Scan Mob by rowkey return error value when run compact_mob or major_compact_mob after change MOB_THRESHOLD bigger
[
https://issues.apache.org/jira/browse/HBASE-19507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
WangYuan updated HBASE-19507:
-
Summary: Get or Scan Mob by rowkey return error value when run compact_mob
or major_compact_mob after change MOB_THRESHOLD bigger (was: Get Mob by rowkey
return error value when run compact_mob or major_compact_mob after change
MOB_THRESHOLD bigger)
> Get or Scan Mob by rowkey return error value when run compact_mob or
> major_compact_mob after change MOB_THRESHOLD bigger
>
>
> Key: HBASE-19507
> URL: https://issues.apache.org/jira/browse/HBASE-19507
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: WangYuan
>
> 1、
> create 'abc',{NAME => 'cf', MOB_THRESHOLD => '10', IS_MOB => 'true'}
> put 'abc','1','cf:a','1'
> put 'abc','2','cf:a','2'
> put 'abc','3','cf:a','3'
> put 'abc','4','cf:a','y1'
> put 'abc','5','cf:a','y2'
> put 'abc','6','cf:a','y3'
>
> hbase(main):011:0> scan 'abc'
> ROWCOLUMN+CELL
>
>
> 1 column=cf:a,
> timestamp=1513171753098, value=1
>
> 2 column=cf:a,
> timestamp=1513171753208, value=2
>
> 3 column=cf:a,
> timestamp=1513171753246, value=3
>
> 4 column=cf:a,
> timestamp=1513171753273, value=y1
>
> 5 column=cf:a,
> timestamp=1513171753301, value=y2
>
> 6 column=cf:a,
> timestamp=1513171754282, value=y3
>
> hbase(main):012:0> flush 'abc'
> hbase(main):012:0> major_compact 'abc'
> hbase(main):012:0> major_compact_mob 'abc'
> 2、
> [See Hfile]:
> hbase org.apache.hadoop.hbase.io.hfile.HFile -f
> /hbase/data/default/abc/a31b3146cba0d4569a7bf44e70e299c9/cf/22a432ba5c2c4802bedd947b99626f10
> -p
> K: 1/cf:a/1513172294864/Put/vlen=5/seqid=4 V: 1
> K: 2/cf:a/1513172294892/Put/vlen=5/seqid=5 V: 2
> K: 3/cf:a/1513172294914/Put/vlen=5/seqid=6 V: 3
> K: 4/cf:a/1513172294954/Put/vlen=76/seqid=7 V:
> \x00\x00\x00\x0Ed41d8cd98f00b204e9800998ecf8427e20171213ce022548c4c3498e864fda289b81e711
> T[0]: T[1]: abc
> K: 5/cf:a/1513172294982/Put/vlen=76/seqid=8 V:
> \x00\x00\x00\x0Ed41d8cd98f00b204e9800998ecf8427e20171213ce022548c4c3498e864fda289b81e711
> T[0]: T[1]: abc
> K: 6/cf:a/1513172296455/Put/vlen=76/seqid=9 V:
> \x00\x00\x00\x0Ed41d8cd98f00b204e9800998ecf8427e20171213ce022548c4c3498e864fda289b81e711
> T[0]: T[1]: abc
> Scanned kv count -> 6
> [See Mobfile]:
> hbase org.apache.hadoop.hbase.io.hfile.HFile -f
> /hbase/mobdir/data/default/abc/07aab825b62dd9111831839cc9039df9/cf/d41d8cd98f00b204e9800998ecf8427e20171213bd8cfaf146684d4096ebf7994f050e96
> -p
> K: 4/cf:a/1513172924196/Put/vlen=14/seqid=7 V: y1
> K: 5/cf:a/1513172924214/Put/vlen=14/seqid=8 V: y2
> K: 6/cf:a/1513172925768/Put/vlen=14/seqid=9 V: y3
> 3、
> alter 'abc',{NAME => 'cf', MOB_THRESHOLD => '10240' }
> put
>
[jira] [Comment Edited] (HBASE-19507) Get or Scan Mob by rowkey return error value when run compact_mob or major_compact_mob after change MOB_THRESHOLD bigger
[
https://issues.apache.org/jira/browse/HBASE-19507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16289308#comment-16289308
]
WangYuan edited comment on HBASE-19507 at 12/13/17 2:16 PM:
How to SCAN? If I want to scan/get a mob value instead of a reference mobpath .
Thanks.
was (Author: wangyuan):
How to SCAN if I want to scan/get a value instead of a reference mobpath ?
> Get or Scan Mob by rowkey return error value when run compact_mob or
> major_compact_mob after change MOB_THRESHOLD bigger
>
>
> Key: HBASE-19507
> URL: https://issues.apache.org/jira/browse/HBASE-19507
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: WangYuan
>
> 1、
> create 'abc',{NAME => 'cf', MOB_THRESHOLD => '10', IS_MOB => 'true'}
> put 'abc','1','cf:a','1'
> put 'abc','2','cf:a','2'
> put 'abc','3','cf:a','3'
> put 'abc','4','cf:a','y1'
> put 'abc','5','cf:a','y2'
> put 'abc','6','cf:a','y3'
>
> hbase(main):011:0> scan 'abc'
> ROWCOLUMN+CELL
>
>
> 1 column=cf:a,
> timestamp=1513171753098, value=1
>
> 2 column=cf:a,
> timestamp=1513171753208, value=2
>
> 3 column=cf:a,
> timestamp=1513171753246, value=3
>
> 4 column=cf:a,
> timestamp=1513171753273, value=y1
>
> 5 column=cf:a,
> timestamp=1513171753301, value=y2
>
> 6 column=cf:a,
> timestamp=1513171754282, value=y3
>
> hbase(main):012:0> flush 'abc'
> hbase(main):012:0> major_compact 'abc'
> hbase(main):012:0> major_compact_mob 'abc'
> 2、
> [See Hfile]:
> hbase org.apache.hadoop.hbase.io.hfile.HFile -f
> /hbase/data/default/abc/a31b3146cba0d4569a7bf44e70e299c9/cf/22a432ba5c2c4802bedd947b99626f10
> -p
> K: 1/cf:a/1513172294864/Put/vlen=5/seqid=4 V: 1
> K: 2/cf:a/1513172294892/Put/vlen=5/seqid=5 V: 2
> K: 3/cf:a/1513172294914/Put/vlen=5/seqid=6 V: 3
> K: 4/cf:a/1513172294954/Put/vlen=76/seqid=7 V:
> \x00\x00\x00\x0Ed41d8cd98f00b204e9800998ecf8427e20171213ce022548c4c3498e864fda289b81e711
> T[0]: T[1]: abc
> K: 5/cf:a/1513172294982/Put/vlen=76/seqid=8 V:
> \x00\x00\x00\x0Ed41d8cd98f00b204e9800998ecf8427e20171213ce022548c4c3498e864fda289b81e711
> T[0]: T[1]: abc
> K: 6/cf:a/1513172296455/Put/vlen=76/seqid=9 V:
> \x00\x00\x00\x0Ed41d8cd98f00b204e9800998ecf8427e20171213ce022548c4c3498e864fda289b81e711
> T[0]: T[1]: abc
> Scanned kv count -> 6
> [See Mobfile]:
> hbase org.apache.hadoop.hbase.io.hfile.HFile -f
> /hbase/mobdir/data/default/abc/07aab825b62dd9111831839cc9039df9/cf/d41d8cd98f00b204e9800998ecf8427e20171213bd8cfaf146684d4096ebf7994f050e96
> -p
> K: 4/cf:a/1513172924196/Put/vlen=14/seqid=7 V: y1
> K: 5/cf:a/1513172924214/Put/vlen=14/seqid=8 V: y2
> K: 6/cf:a/1513172925768/Put/vlen=14/seqid=9 V: y3
> 3、
> alter 'abc',{NAME => 'cf', MOB_THRESHOLD => '10240' }
> put
>
[jira] [Commented] (HBASE-19507) Get or Scan Mob by rowkey return error value when run compact_mob or major_compact_mob after change MOB_THRESHOLD bigger
[
https://issues.apache.org/jira/browse/HBASE-19507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16289308#comment-16289308
]
WangYuan commented on HBASE-19507:
--
How to SCAN if I want to scan/get a value instead of a reference mobpath ?
> Get or Scan Mob by rowkey return error value when run compact_mob or
> major_compact_mob after change MOB_THRESHOLD bigger
>
>
> Key: HBASE-19507
> URL: https://issues.apache.org/jira/browse/HBASE-19507
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: WangYuan
>
> 1、
> create 'abc',{NAME => 'cf', MOB_THRESHOLD => '10', IS_MOB => 'true'}
> put 'abc','1','cf:a','1'
> put 'abc','2','cf:a','2'
> put 'abc','3','cf:a','3'
> put 'abc','4','cf:a','y1'
> put 'abc','5','cf:a','y2'
> put 'abc','6','cf:a','y3'
>
> hbase(main):011:0> scan 'abc'
> ROWCOLUMN+CELL
>
>
> 1 column=cf:a,
> timestamp=1513171753098, value=1
>
> 2 column=cf:a,
> timestamp=1513171753208, value=2
>
> 3 column=cf:a,
> timestamp=1513171753246, value=3
>
> 4 column=cf:a,
> timestamp=1513171753273, value=y1
>
> 5 column=cf:a,
> timestamp=1513171753301, value=y2
>
> 6 column=cf:a,
> timestamp=1513171754282, value=y3
>
> hbase(main):012:0> flush 'abc'
> hbase(main):012:0> major_compact 'abc'
> hbase(main):012:0> major_compact_mob 'abc'
> 2、
> [See Hfile]:
> hbase org.apache.hadoop.hbase.io.hfile.HFile -f
> /hbase/data/default/abc/a31b3146cba0d4569a7bf44e70e299c9/cf/22a432ba5c2c4802bedd947b99626f10
> -p
> K: 1/cf:a/1513172294864/Put/vlen=5/seqid=4 V: 1
> K: 2/cf:a/1513172294892/Put/vlen=5/seqid=5 V: 2
> K: 3/cf:a/1513172294914/Put/vlen=5/seqid=6 V: 3
> K: 4/cf:a/1513172294954/Put/vlen=76/seqid=7 V:
> \x00\x00\x00\x0Ed41d8cd98f00b204e9800998ecf8427e20171213ce022548c4c3498e864fda289b81e711
> T[0]: T[1]: abc
> K: 5/cf:a/1513172294982/Put/vlen=76/seqid=8 V:
> \x00\x00\x00\x0Ed41d8cd98f00b204e9800998ecf8427e20171213ce022548c4c3498e864fda289b81e711
> T[0]: T[1]: abc
> K: 6/cf:a/1513172296455/Put/vlen=76/seqid=9 V:
> \x00\x00\x00\x0Ed41d8cd98f00b204e9800998ecf8427e20171213ce022548c4c3498e864fda289b81e711
> T[0]: T[1]: abc
> Scanned kv count -> 6
> [See Mobfile]:
> hbase org.apache.hadoop.hbase.io.hfile.HFile -f
> /hbase/mobdir/data/default/abc/07aab825b62dd9111831839cc9039df9/cf/d41d8cd98f00b204e9800998ecf8427e20171213bd8cfaf146684d4096ebf7994f050e96
> -p
> K: 4/cf:a/1513172924196/Put/vlen=14/seqid=7 V: y1
> K: 5/cf:a/1513172924214/Put/vlen=14/seqid=8 V: y2
> K: 6/cf:a/1513172925768/Put/vlen=14/seqid=9 V: y3
> 3、
> alter 'abc',{NAME => 'cf', MOB_THRESHOLD => '10240' }
> put
>
[jira] [Updated] (HBASE-19507) Get Mob by rowkey return error value when run compact_mob or major_compact_mob after change MOB_THRESHOLD bigger
[
https://issues.apache.org/jira/browse/HBASE-19507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
WangYuan updated HBASE-19507:
-
Description:
1、
create 'abc',{NAME => 'cf', MOB_THRESHOLD => '10', IS_MOB => 'true'}
put 'abc','1','cf:a','1'
put 'abc','2','cf:a','2'
put 'abc','3','cf:a','3'
put 'abc','4','cf:a','y1'
put 'abc','5','cf:a','y2'
put 'abc','6','cf:a','y3'
hbase(main):011:0> scan 'abc'
ROWCOLUMN+CELL
1 column=cf:a,
timestamp=1513171753098, value=1
2 column=cf:a,
timestamp=1513171753208, value=2
3 column=cf:a,
timestamp=1513171753246, value=3
4 column=cf:a,
timestamp=1513171753273, value=y1
5 column=cf:a,
timestamp=1513171753301, value=y2
6 column=cf:a,
timestamp=1513171754282, value=y3
hbase(main):012:0> flush 'abc'
hbase(main):012:0> major_compact 'abc'
hbase(main):012:0> major_compact_mob 'abc'
2、
[See Hfile]:
hbase org.apache.hadoop.hbase.io.hfile.HFile -f
/hbase/data/default/abc/a31b3146cba0d4569a7bf44e70e299c9/cf/22a432ba5c2c4802bedd947b99626f10
-p
K: 1/cf:a/1513172294864/Put/vlen=5/seqid=4 V: 1
K: 2/cf:a/1513172294892/Put/vlen=5/seqid=5 V: 2
K: 3/cf:a/1513172294914/Put/vlen=5/seqid=6 V: 3
K: 4/cf:a/1513172294954/Put/vlen=76/seqid=7 V:
\x00\x00\x00\x0Ed41d8cd98f00b204e9800998ecf8427e20171213ce022548c4c3498e864fda289b81e711
T[0]: T[1]: abc
K: 5/cf:a/1513172294982/Put/vlen=76/seqid=8 V:
\x00\x00\x00\x0Ed41d8cd98f00b204e9800998ecf8427e20171213ce022548c4c3498e864fda289b81e711
T[0]: T[1]: abc
K: 6/cf:a/1513172296455/Put/vlen=76/seqid=9 V:
\x00\x00\x00\x0Ed41d8cd98f00b204e9800998ecf8427e20171213ce022548c4c3498e864fda289b81e711
T[0]: T[1]: abc
Scanned kv count -> 6
[See Mobfile]:
hbase org.apache.hadoop.hbase.io.hfile.HFile -f
/hbase/mobdir/data/default/abc/07aab825b62dd9111831839cc9039df9/cf/d41d8cd98f00b204e9800998ecf8427e20171213bd8cfaf146684d4096ebf7994f050e96
-p
K: 4/cf:a/1513172924196/Put/vlen=14/seqid=7 V: y1
K: 5/cf:a/1513172924214/Put/vlen=14/seqid=8 V: y2
K: 6/cf:a/1513172925768/Put/vlen=14/seqid=9 V: y3
3、
alter 'abc',{NAME => 'cf', MOB_THRESHOLD => '10240' }
put
[jira] [Commented] (HBASE-19507) Get Mob by rowkey return error value when run compact_mob or major_compact_mob after change MOB_THRESHOLD bigger
[ https://issues.apache.org/jira/browse/HBASE-19507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16289290#comment-16289290 ] WangYuan commented on HBASE-19507: -- I don't know why? I guess it may be a bug because the scan result is changed. A、old value is : y1 、y2、y3 but now is only a ref value : =\x00\x00\x00\x0Ed41d8cd98f00b204e9800998ecf8427e2017121320a7c40681254d6a8a396e9db34529af B、old value is : zz .'(10304 Byte) but now is only a ref(mob path) value :\x00\x00(@d41d8cd98f00b204e9800998ecf8427e2017121320a7c40681254d6a8a396e9db34529af But ,the newest mobfile content is OK,like this : hbase org.apache.hadoop.hbase.io.hfile.HFile -f /hbase/mobdir/data/default/abc/07aab825b62dd9111831839cc9039df9/cf/d41d8cd98f00b204e9800998ecf8427e201712131b5d3a8734ac4c8eb41211e1bd065f3d -p K: 4/cf:a/1513172924196/Put/vlen=14/seqid=7 V: y1 K: 5/cf:a/1513172924214/Put/vlen=14/seqid=8 V: y2 K: 6/cf:a/1513172925768/Put/vlen=14/seqid=9 V: y3 K: 7/cf:a/1513173155842/Put/vlen=10304/seqid=18 V:
[jira] [Created] (HBASE-19507) get Mob by rowkey only return error value when run compact_mob or major_compact_mob after change MOB_THRESHOLD
WangYuan created HBASE-19507: Summary: get Mob by rowkey only return error value when run compact_mob or major_compact_mob after change MOB_THRESHOLD Key: HBASE-19507 URL: https://issues.apache.org/jira/browse/HBASE-19507 Project: HBase Issue Type: Bug Affects Versions: 1.2.0 Reporter: WangYuan -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19507) Get Mob by rowkey return error value when run compact_mob or major_compact_mob after change MOB_THRESHOLD bigger
[ https://issues.apache.org/jira/browse/HBASE-19507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WangYuan updated HBASE-19507: - Summary: Get Mob by rowkey return error value when run compact_mob or major_compact_mob after change MOB_THRESHOLD bigger (was: get Mob by rowkey only return error value when run compact_mob or major_compact_mob after change MOB_THRESHOLD) > Get Mob by rowkey return error value when run compact_mob or > major_compact_mob after change MOB_THRESHOLD bigger > > > Key: HBASE-19507 > URL: https://issues.apache.org/jira/browse/HBASE-19507 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: WangYuan > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-17983) control region numbers when create table to improve performance
[ https://issues.apache.org/jira/browse/HBASE-17983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15992778#comment-15992778 ] WangYuan commented on HBASE-17983: -- Sorry, my description is not very accurate. In fact, I hope to keep the performance of HBase put and get relatively stable.Create a new table then region (EG:100 partition) increase, then bulkload .So data will increase and meanwhile client's concurrent will also increase . You said that the reduction of handle number, I think it may block the requests, because the length of the RPC queue is the default number of handle multiplied by 10, so that the queue length becomes shorter. If we can limit the number of regions in each regionserver in a reasonable range, so that even if the number of concurrent become larger and regions become more and more, I only need to increase the size of the regionserver clusters by a rule: hbase.client.region.averageload.numbers . So whether concurrent or regions increases, the performance can remain stable. > control region numbers when create table to improve performance > --- > > Key: HBASE-17983 > URL: https://issues.apache.org/jira/browse/HBASE-17983 > Project: HBase > Issue Type: Improvement > Components: Admin, Client >Affects Versions: 2.0.0 >Reporter: WangYuan >Priority: Minor > Fix For: 2.0.0 > > Attachments: > HBASE-17983-control-region-numbers-when-create-table.patch > > > I found that with the increasing number of regions in every RegionServer , > HBase read and write performance decreased, and failed to achieve the desired > performance. Therefore, we hope to control the number of regions in every > RegionServer , and add the judgment before creating tables. > I can set up a region parameter in hbase-default.xml, > hbase.client.region.averageload.numbers, when the client builds a table that > exceeds the value of this parameter, throws an exception. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17983) control region numbers when create table to improve performance
[ https://issues.apache.org/jira/browse/HBASE-17983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15992632#comment-15992632 ] WangYuan commented on HBASE-17983: -- Thank you. I read HBASE-8410, It's a good issue,but I understand issue's point of view is different. One is the RegionServer level to control average Region number, one is the overall control of the NameSpace level. For example: in our cluster RegionServer average load of Region increased from 200 then 500 then 1000, single read and write performance will increase from 5ms then 10ms then 100ms. I don't want to go down. But HBASE-8410 should not explain the relationship between RegionServer and Region numbers in it . Because the number of RegionServer nodes can be increased (When data increases or table numbers increases), I want to control the average number of region load ,rather than total table numbers or region numbers. > control region numbers when create table to improve performance > --- > > Key: HBASE-17983 > URL: https://issues.apache.org/jira/browse/HBASE-17983 > Project: HBase > Issue Type: Improvement > Components: Admin, Client >Affects Versions: 2.0.0 >Reporter: WangYuan >Priority: Minor > Fix For: 2.0.0 > > Attachments: > HBASE-17983-control-region-numbers-when-create-table.patch > > > I found that with the increasing number of regions in every RegionServer , > HBase read and write performance decreased, and failed to achieve the desired > performance. Therefore, we hope to control the number of regions in every > RegionServer , and add the judgment before creating tables. > I can set up a region parameter in hbase-default.xml, > hbase.client.region.averageload.numbers, when the client builds a table that > exceeds the value of this parameter, throws an exception. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17983) control region numbers when create table to improve performance
[ https://issues.apache.org/jira/browse/HBASE-17983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WangYuan updated HBASE-17983: - Attachment: HBASE-17983-control-region-numbers-when-create-table.patch > control region numbers when create table to improve performance > --- > > Key: HBASE-17983 > URL: https://issues.apache.org/jira/browse/HBASE-17983 > Project: HBase > Issue Type: Improvement > Components: Admin, Client >Affects Versions: 2.0.0 >Reporter: WangYuan >Priority: Minor > Fix For: 2.0.0 > > Attachments: > HBASE-17983-control-region-numbers-when-create-table.patch > > > I found that with the increasing number of regions in every RegionServer , > HBase read and write performance decreased, and failed to achieve the desired > performance. Therefore, we hope to control the number of regions in every > RegionServer , and add the judgment before creating tables. > I can set up a region parameter in hbase-default.xml, > hbase.client.region.averageload.numbers, when the client builds a table that > exceeds the value of this parameter, throws an exception. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17983) control region numbers when create table to improve performance
WangYuan created HBASE-17983: Summary: control region numbers when create table to improve performance Key: HBASE-17983 URL: https://issues.apache.org/jira/browse/HBASE-17983 Project: HBase Issue Type: Improvement Components: Admin, Client Affects Versions: 2.0.0 Reporter: WangYuan Priority: Minor Fix For: 2.0.0 I found that with the increasing number of regions in every RegionServer , HBase read and write performance decreased, and failed to achieve the desired performance. Therefore, we hope to control the number of regions in every RegionServer , and add the judgment before creating tables. I can set up a region parameter in hbase-default.xml, hbase.client.region.averageload.numbers, when the client builds a table that exceeds the value of this parameter, throws an exception. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
