[jira] [Resolved] (HBASE-17099) Is there a plan to support auth connection by username/password like mysql or redis
[
https://issues.apache.org/jira/browse/HBASE-17099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liubangchen resolved HBASE-17099.
-
Resolution: Invalid
> Is there a plan to support auth connection by username/password like mysql or
> redis
> ---
>
> Key: HBASE-17099
> URL: https://issues.apache.org/jira/browse/HBASE-17099
> Project: HBase
> Issue Type: Brainstorming
> Components: security
>Reporter: liubangchen
>Priority: Trivial
>
> Product managers ask our hbase cluster to support auth connection by
> username/password.
> {code}
> private boolean authorizeConnection() throws IOException {
> try {
> // If auth method is DIGEST, the token was obtained by the
> // real user for the effective user, therefore not required to
> // authorize real user. doAs is allowed only for simple or kerberos
> // authentication
> if (user != null && user.getRealUser() != null
> && (authMethod != AuthMethod.DIGEST)) {
> ProxyUsers.authorize(user, this.getHostAddress(), conf);
> }
> authorize(user, connectionHeader, getHostInetAddress());
> metrics.authorizationSuccess();
> } catch (AuthorizationException ae) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("Connection authorization failed: " + ae.getMessage(),
> ae);
> }
> metrics.authorizationFailure();
> setupResponse(authFailedResponse, authFailedCall,
> new AccessDeniedException(ae), ae.getMessage());
> responder.doRespond(authFailedCall);
> return false;
> }
> return true;
> }
> {code}
> Whether can add a connectionAuthorrizer in method authorizeConnection of
> class RpcServer to auth connection by init the handler from conf.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (HBASE-18124) Make Hbase Communication Support Virtual Network
[
https://issues.apache.org/jira/browse/HBASE-18124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liubangchen resolved HBASE-18124.
-
Resolution: Duplicate
Release Note: supported by HBASE-12954
> Make Hbase Communication Support Virtual Network
>
>
> Key: HBASE-18124
> URL: https://issues.apache.org/jira/browse/HBASE-18124
> Project: HBase
> Issue Type: New Feature
> Components: Client, hbase, master
>Reporter: liubangchen
>Assignee: liubangchen
> Attachments: 1.jpg, HBASE-18124.pdf
>
>
> Hbase only have one way to locate hmaster or hregionserver not like hdfs has
> two way to locate datanode used by name or hostname.
> I'm a engineer of cloud computing , and I'm in charge of to make Hbase as a
> cloud service.
> Our Hbase cloud service architecture shown as follows: 1.jpg
> 1.VM
> User’s Hbase client work in vm and use virtual ip address to access hbase
> cluster.
> 2.NAT
> Network Address Translation, vip(Virtual Network Address) to pip
> (Physical Network Address)
> 3. HbaseCluster Service
> HbaseCluster Service work in physical network
> Problem
> Vm use vip to communicate with hbase cluster, but hbase have only one
> way to communicate with each server using host which is setting by parameters
> hbase.regionserver.hostname or hbase.master.hostname. When HMaster startup
> will publish master address and meta region server address in zookeeper,
> then the address is pip(Physical Network Address) . Because hbase cluster
> work in physical network. But the problem is that when vm get the address
> from zookeeper is pip, not vip. If I set host as vip, it will has problems
> for hbase cluster communication inside. so it needs two ways for
> communication setting by parameters.
> Solution
> 1. protocol extend
> change strcut ServerName as
> {code}
> message ServerName {
> required string host_name = 1;
> optional uint32 port = 2;
> optional uint64 start_code = 3;
> optional string name=4; //new field
> }
> {code}
> It will be publish in zookeeper. We can choose host_name or name
> configured by parameters hbase.client.use.hostname
> 2. metatable extend
>Add a column to hbase:meta named info:namelocation . So the original
> column info:server configured with hbase.regionserver.hostname, and the new
> column info:namelocation configured with hbase.regionserver.servername
> 3. hbase-server
>When regionserver startup, configured hbase.regionserver.hostname as
> pip and configured hbase.regionserver.servername as vip. Then
> hbase.regionserver.hostname will be writed in ServerName's host_name, and
> hbase.regionserver.servername will be writed in ServerName's name.When
> hmaster startup, configured hbase.hmaster.hostname as pip and configured
> hbase.hmaster.servername as vip. Then hbase.hmaster.hostname will be writed
> in ServerName's host_name, and hbase.hmaster.servername will be writed in
> ServerName's name.
> 4. hbase-client
> Add a parameters named hbase.client.use.hostname to use vip or pip.
> This patch is base on Hbase-1.3.0
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (HBASE-18124) Add Property name Of Strcut ServerName To Location HMaster Or HRegionServer
liubangchen created HBASE-18124: --- Summary: Add Property name Of Strcut ServerName To Location HMaster Or HRegionServer Key: HBASE-18124 URL: https://issues.apache.org/jira/browse/HBASE-18124 Project: HBase Issue Type: New Feature Components: Client, hbase, master Reporter: liubangchen Assignee: liubangchen -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17099) Is there a plan to support auth connection by username/password like mysql or redis
liubangchen created HBASE-17099:
---
Summary: Is there a plan to support auth connection by
username/password like mysql or redis
Key: HBASE-17099
URL: https://issues.apache.org/jira/browse/HBASE-17099
Project: HBase
Issue Type: Wish
Components: security
Reporter: liubangchen
Priority: Trivial
Product managers ask our hbase cluster to support auth connection by
username/password.
{code}
private boolean authorizeConnection() throws IOException {
try {
// If auth method is DIGEST, the token was obtained by the
// real user for the effective user, therefore not required to
// authorize real user. doAs is allowed only for simple or kerberos
// authentication
if (user != null && user.getRealUser() != null
&& (authMethod != AuthMethod.DIGEST)) {
ProxyUsers.authorize(user, this.getHostAddress(), conf);
}
authorize(user, connectionHeader, getHostInetAddress());
metrics.authorizationSuccess();
} catch (AuthorizationException ae) {
if (LOG.isDebugEnabled()) {
LOG.debug("Connection authorization failed: " + ae.getMessage(), ae);
}
metrics.authorizationFailure();
setupResponse(authFailedResponse, authFailedCall,
new AccessDeniedException(ae), ae.getMessage());
responder.doRespond(authFailedCall);
return false;
}
return true;
}
{code}
Whether can add a connectionAuthorrizer in method authorizeConnection of
class RpcServer to auth connection by init the handler from conf.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-16993) BucketCache throw java.io.IOException: Invalid HFile block magic when DATA_BLOCK_ENCODING set to DIFF
[
https://issues.apache.org/jira/browse/HBASE-16993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liubangchen resolved HBASE-16993.
-
Resolution: Not A Bug
Must be a multiple of 1024 else you will run into
'java.io.IOException: Invalid HFile block magic' when you go to read from
cache.
If you specify no values here, then you pick up the default bucketsizes set
in code (See BucketAllocator#DEFAULT_BUCKET_SIZES).
> BucketCache throw java.io.IOException: Invalid HFile block magic when
> DATA_BLOCK_ENCODING set to DIFF
> -
>
> Key: HBASE-16993
> URL: https://issues.apache.org/jira/browse/HBASE-16993
> Project: HBase
> Issue Type: Bug
> Components: io
>Affects Versions: 1.1.3
> Environment: hbase version 1.1.3
>Reporter: liubangchen
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> hbase-site.xml setting
>
> hbase.bucketcache.bucket.sizes
> 16384,32768,40960,
> 46000,49152,51200,65536,131072,524288
>
>
> hbase.bucketcache.size
> 16384
>
>
> hbase.bucketcache.ioengine
> offheap
>
>
> hfile.block.cache.size
> 0.3
>
>
> hfile.block.bloom.cacheonwrite
> true
>
>
> hbase.rs.cacheblocksonwrite
> true
>
>
> hfile.block.index.cacheonwrite
> true
> n_splits = 200
> create 'usertable',{NAME =>'family', COMPRESSION => 'snappy', VERSIONS =>
> 1,DATA_BLOCK_ENCODING => 'DIFF',CONFIGURATION =>
> {'hbase.hregion.memstore.block.multiplier' => 5}},{DURABILITY =>
> 'SKIP_WAL'},{SPLITS => (1..n_splits).map {|i|
> "user#{1000+i*(-1000)/n_splits}"}}
> load data
> bin/ycsb load hbase10 -P workloads/workloada -p table=usertable -p
> columnfamily=family -p fieldcount=10 -p fieldlength=100 -p
> recordcount=2 -p insertorder=hashed -p insertstart=0 -p
> clientbuffering=true -p durability=SKIP_WAL -threads 20 -s
> run
> bin/ycsb run hbase10 -P workloads/workloadb -p table=usertable -p
> columnfamily=family -p fieldcount=10 -p fieldlength=100 -p
> operationcount=2000 -p readallfields=true -p clientbuffering=true -p
> requestdistribution=zipfian -threads 10 -s
> log info
> 2016-11-02 20:20:20,261 ERROR
> [RW.default.readRpcServer.handler=36,queue=21,port=6020] bucket.BucketCache:
> Failed reading block fdcc7ed6f3b2498b9ef316cc8206c233_44819759 from bucket
> cache
> java.io.IOException: Invalid HFile block magic:
> \x00\x00\x00\x00\x00\x00\x00\x00
> at
> org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:154)
> at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:167)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.(HFileBlock.java:273)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$1.deserialize(HFileBlock.java:134)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$1.deserialize(HFileBlock.java:121)
> at
> org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.getBlock(BucketCache.java:427)
> at
> org.apache.hadoop.hbase.io.hfile.CombinedBlockCache.getBlock(CombinedBlockCache.java:85)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.getCachedBlock(HFileReaderV2.java:266)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:403)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:269)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:634)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:584)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:247)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:156)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:363)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:217)
> at
> org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2071)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:5369)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2546)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2532)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2514)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6558)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6537)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.g
[jira] [Reopened] (HBASE-16993) BucketCache throw java.io.IOException: Invalid HFile block magic when DATA_BLOCK_ENCODING set to DIFF
[
https://issues.apache.org/jira/browse/HBASE-16993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liubangchen reopened HBASE-16993:
-
use non-standard bucket.sizes server throws Invalid Hfile block magin Exception
> BucketCache throw java.io.IOException: Invalid HFile block magic when
> DATA_BLOCK_ENCODING set to DIFF
> -
>
> Key: HBASE-16993
> URL: https://issues.apache.org/jira/browse/HBASE-16993
> Project: HBase
> Issue Type: Bug
> Components: io
>Affects Versions: 1.1.3
> Environment: hbase version 1.1.3
>Reporter: liubangchen
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> hbase-site.xml setting
>
> hbase.bucketcache.bucket.sizes
> 16384,32768,40960,
> 46000,49152,51200,65536,131072,524288
>
>
> hbase.bucketcache.size
> 16384
>
>
> hbase.bucketcache.ioengine
> offheap
>
>
> hfile.block.cache.size
> 0.3
>
>
> hfile.block.bloom.cacheonwrite
> true
>
>
> hbase.rs.cacheblocksonwrite
> true
>
>
> hfile.block.index.cacheonwrite
> true
> n_splits = 200
> create 'usertable',{NAME =>'family', COMPRESSION => 'snappy', VERSIONS =>
> 1,DATA_BLOCK_ENCODING => 'DIFF',CONFIGURATION =>
> {'hbase.hregion.memstore.block.multiplier' => 5}},{DURABILITY =>
> 'SKIP_WAL'},{SPLITS => (1..n_splits).map {|i|
> "user#{1000+i*(-1000)/n_splits}"}}
> load data
> bin/ycsb load hbase10 -P workloads/workloada -p table=usertable -p
> columnfamily=family -p fieldcount=10 -p fieldlength=100 -p
> recordcount=2 -p insertorder=hashed -p insertstart=0 -p
> clientbuffering=true -p durability=SKIP_WAL -threads 20 -s
> run
> bin/ycsb run hbase10 -P workloads/workloadb -p table=usertable -p
> columnfamily=family -p fieldcount=10 -p fieldlength=100 -p
> operationcount=2000 -p readallfields=true -p clientbuffering=true -p
> requestdistribution=zipfian -threads 10 -s
> log info
> 2016-11-02 20:20:20,261 ERROR
> [RW.default.readRpcServer.handler=36,queue=21,port=6020] bucket.BucketCache:
> Failed reading block fdcc7ed6f3b2498b9ef316cc8206c233_44819759 from bucket
> cache
> java.io.IOException: Invalid HFile block magic:
> \x00\x00\x00\x00\x00\x00\x00\x00
> at
> org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:154)
> at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:167)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.(HFileBlock.java:273)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$1.deserialize(HFileBlock.java:134)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$1.deserialize(HFileBlock.java:121)
> at
> org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.getBlock(BucketCache.java:427)
> at
> org.apache.hadoop.hbase.io.hfile.CombinedBlockCache.getBlock(CombinedBlockCache.java:85)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.getCachedBlock(HFileReaderV2.java:266)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:403)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:269)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:634)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:584)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:247)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:156)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:363)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:217)
> at
> org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2071)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:5369)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2546)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2532)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2514)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6558)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6537)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:1935)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32381)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(Rp
[jira] [Resolved] (HBASE-16993) BucketCache throw java.io.IOException: Invalid HFile block magic when DATA_BLOCK_ENCODING set to DIFF
[
https://issues.apache.org/jira/browse/HBASE-16993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liubangchen resolved HBASE-16993.
-
Resolution: Not A Bug
caused by hbase.bucketcache.bucket.sizes incorrect value
> BucketCache throw java.io.IOException: Invalid HFile block magic when
> DATA_BLOCK_ENCODING set to DIFF
> -
>
> Key: HBASE-16993
> URL: https://issues.apache.org/jira/browse/HBASE-16993
> Project: HBase
> Issue Type: Bug
> Components: io
>Affects Versions: 1.1.3
> Environment: hbase version 1.1.3
>Reporter: liubangchen
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> hbase-site.xml setting
>
> hbase.bucketcache.bucket.sizes
> 16384,32768,40960,
> 46000,49152,51200,65536,131072,524288
>
>
> hbase.bucketcache.size
> 16384
>
>
> hbase.bucketcache.ioengine
> offheap
>
>
> hfile.block.cache.size
> 0.3
>
>
> hfile.block.bloom.cacheonwrite
> true
>
>
> hbase.rs.cacheblocksonwrite
> true
>
>
> hfile.block.index.cacheonwrite
> true
> n_splits = 200
> create 'usertable',{NAME =>'family', COMPRESSION => 'snappy', VERSIONS =>
> 1,DATA_BLOCK_ENCODING => 'DIFF',CONFIGURATION =>
> {'hbase.hregion.memstore.block.multiplier' => 5}},{DURABILITY =>
> 'SKIP_WAL'},{SPLITS => (1..n_splits).map {|i|
> "user#{1000+i*(-1000)/n_splits}"}}
> load data
> bin/ycsb load hbase10 -P workloads/workloada -p table=usertable -p
> columnfamily=family -p fieldcount=10 -p fieldlength=100 -p
> recordcount=2 -p insertorder=hashed -p insertstart=0 -p
> clientbuffering=true -p durability=SKIP_WAL -threads 20 -s
> run
> bin/ycsb run hbase10 -P workloads/workloadb -p table=usertable -p
> columnfamily=family -p fieldcount=10 -p fieldlength=100 -p
> operationcount=2000 -p readallfields=true -p clientbuffering=true -p
> requestdistribution=zipfian -threads 10 -s
> log info
> 2016-11-02 20:20:20,261 ERROR
> [RW.default.readRpcServer.handler=36,queue=21,port=6020] bucket.BucketCache:
> Failed reading block fdcc7ed6f3b2498b9ef316cc8206c233_44819759 from bucket
> cache
> java.io.IOException: Invalid HFile block magic:
> \x00\x00\x00\x00\x00\x00\x00\x00
> at
> org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:154)
> at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:167)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.(HFileBlock.java:273)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$1.deserialize(HFileBlock.java:134)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$1.deserialize(HFileBlock.java:121)
> at
> org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.getBlock(BucketCache.java:427)
> at
> org.apache.hadoop.hbase.io.hfile.CombinedBlockCache.getBlock(CombinedBlockCache.java:85)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.getCachedBlock(HFileReaderV2.java:266)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:403)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:269)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:634)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:584)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:247)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:156)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:363)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:217)
> at
> org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2071)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:5369)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2546)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2532)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2514)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6558)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6537)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:1935)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32381)
> at org.apache.hadoop.hbase.ipc.RpcServer.call
[jira] [Created] (HBASE-16993) BucketCache throw java.io.IOException: Invalid HFile block magic when DATA_BLOCK_ENCODING set to DIFF
liubangchen created HBASE-16993:
---
Summary: BucketCache throw java.io.IOException: Invalid HFile
block magic when DATA_BLOCK_ENCODING set to DIFF
Key: HBASE-16993
URL: https://issues.apache.org/jira/browse/HBASE-16993
Project: HBase
Issue Type: Bug
Components: io
Affects Versions: 1.1.3
Environment: hbase-site.xml setting
hbase.bucketcache.bucket.sizes
16384,32768,40960, 46000,49152,51200,65536,131072,524288
hbase.bucketcache.size
16384
hbase.bucketcache.ioengine
offheap
hfile.block.cache.size
0.3
hfile.block.bloom.cacheonwrite
true
hbase.rs.cacheblocksonwrite
true
hfile.block.index.cacheonwrite
true
'family', COMPRESSION => 'snappy', VERSIONS =>
1,DATA_BLOCK_ENCODING => 'DIFF',CONFIGURATION =>
{'hbase.hregion.memstore.block.multiplier' => 5}},{DURABILITY =>
'SKIP_WAL'},{SPLITS => (1..n_splits).map {|i|
"user#{1000+i*(-1000)/n_splits}"}}
load data
bin/ycsb load hbase10 -P workloads/workloada -p table=usertable -p
columnfamily=family -p fieldcount=10 -p fieldlength=100 -p
recordcount=2 -p insertorder=hashed -p insertstart=0 -p
clientbuffering=true -p durability=SKIP_WAL -threads 20 -s
run
bin/ycsb run hbase10 -P workloads/workloadb -p table=usertable -p
columnfamily=family -p fieldcount=10 -p fieldlength=100 -p
operationcount=2000 -p readallfields=true -p clientbuffering=true -p
requestdistribution=zipfian -threads 10 -s
log info
2016-11-02 20:20:20,261 ERROR
[RW.default.readRpcServer.handler=36,queue=21,port=6020] bucket.BucketCache:
Failed reading block fdcc7ed6f3b2498b9ef316cc8206c233_44819759 from bucket cache
java.io.IOException: Invalid HFile block magic: \x00\x00\x00\x00\x00\x00\x00\x00
at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:154)
at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:167)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock.(HFileBlock.java:273)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$1.deserialize(HFileBlock.java:134)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$1.deserialize(HFileBlock.java:121)
at
org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.getBlock(BucketCache.java:427)
at
org.apache.hadoop.hbase.io.hfile.CombinedBlockCache.getBlock(CombinedBlockCache.java:85)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2.getCachedBlock(HFileReaderV2.java:266)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:403)
at
org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:269)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:634)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:584)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:247)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:156)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:363)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:217)
at
org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2071)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:5369)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2546)
at
org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2532)
at
org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2514)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6558)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6537)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:1935)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32381)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2117)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:104)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
at java.lang.Thread.run(Thread.java:745)
2016-11-02 20:20:20,263 ERROR
[RW.default.readRpcServer.handler=50,queue=20,port=6020] bucket.BucketCache:
Failed reading block c45d6b14789546b785bae94c69c683d5_34198622 from bucket cache
java.io.IOException: Invalid HFile block magic: \x00\x00\x00\x00\x00\x00\x00\x00
at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:154)
