[jira] [Commented] (HBASE-16993) BucketCache throw java.io.IOException: Invalid HFile block magic when DATA_BLOCK_ENCODING set to DIFF
[
https://issues.apache.org/jira/browse/HBASE-16993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15650118#comment-15650118
]
liubangchen commented on HBASE-16993:
-
Thank you sir.
> BucketCache throw java.io.IOException: Invalid HFile block magic when
> DATA_BLOCK_ENCODING set to DIFF
> -
>
> Key: HBASE-16993
> URL: https://issues.apache.org/jira/browse/HBASE-16993
> Project: HBase
> Issue Type: Bug
> Components: io
>Affects Versions: 1.1.3
> Environment: hbase version 1.1.3
>Reporter: liubangchen
>Assignee: liubangchen
> Attachments: HBASE-16993.000.patch, HBASE-16993.001.patch
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> hbase-site.xml setting
>
> hbase.bucketcache.bucket.sizes
> 16384,32768,40960,
> 46000,49152,51200,65536,131072,524288
>
>
> hbase.bucketcache.size
> 16384
>
>
> hbase.bucketcache.ioengine
> offheap
>
>
> hfile.block.cache.size
> 0.3
>
>
> hfile.block.bloom.cacheonwrite
> true
>
>
> hbase.rs.cacheblocksonwrite
> true
>
>
> hfile.block.index.cacheonwrite
> true
> n_splits = 200
> create 'usertable',{NAME =>'family', COMPRESSION => 'snappy', VERSIONS =>
> 1,DATA_BLOCK_ENCODING => 'DIFF',CONFIGURATION =>
> {'hbase.hregion.memstore.block.multiplier' => 5}},{DURABILITY =>
> 'SKIP_WAL'},{SPLITS => (1..n_splits).map {|i|
> "user#{1000+i*(-1000)/n_splits}"}}
> load data
> bin/ycsb load hbase10 -P workloads/workloada -p table=usertable -p
> columnfamily=family -p fieldcount=10 -p fieldlength=100 -p
> recordcount=2 -p insertorder=hashed -p insertstart=0 -p
> clientbuffering=true -p durability=SKIP_WAL -threads 20 -s
> run
> bin/ycsb run hbase10 -P workloads/workloadb -p table=usertable -p
> columnfamily=family -p fieldcount=10 -p fieldlength=100 -p
> operationcount=2000 -p readallfields=true -p clientbuffering=true -p
> requestdistribution=zipfian -threads 10 -s
> log info
> 2016-11-02 20:20:20,261 ERROR
> [RW.default.readRpcServer.handler=36,queue=21,port=6020] bucket.BucketCache:
> Failed reading block fdcc7ed6f3b2498b9ef316cc8206c233_44819759 from bucket
> cache
> java.io.IOException: Invalid HFile block magic:
> \x00\x00\x00\x00\x00\x00\x00\x00
> at
> org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:154)
> at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:167)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.(HFileBlock.java:273)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$1.deserialize(HFileBlock.java:134)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$1.deserialize(HFileBlock.java:121)
> at
> org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.getBlock(BucketCache.java:427)
> at
> org.apache.hadoop.hbase.io.hfile.CombinedBlockCache.getBlock(CombinedBlockCache.java:85)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.getCachedBlock(HFileReaderV2.java:266)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:403)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:269)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:634)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:584)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:247)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:156)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:363)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:217)
> at
> org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2071)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:5369)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2546)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2532)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2514)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6558)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6537)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:1935)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBl
[jira] [Commented] (HBASE-16570) Compute region locality in parallel at startup
[ https://issues.apache.org/jira/browse/HBASE-16570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15650087#comment-15650087 ] binlijin commented on HBASE-16570: -- scheduleFullRefresh is only need when balance, so no need to call it at startup, we will call it when balance start. > Compute region locality in parallel at startup > -- > > Key: HBASE-16570 > URL: https://issues.apache.org/jira/browse/HBASE-16570 > Project: HBase > Issue Type: Sub-task >Reporter: binlijin >Assignee: binlijin > Fix For: 2.0.0, 1.4.0, 1.3.1 > > Attachments: HBASE-16570-master_V1.patch, > HBASE-16570-master_V2.patch, HBASE-16570-master_V3.patch, > HBASE-16570-master_V4.patch, HBASE-16570.branch-1.3-addendum.patch, > HBASE-16570_addnum.patch, HBASE-16570_addnum_v2.patch, > HBASE-16570_addnum_v3.patch, HBASE-16570_addnum_v4.patch, > HBASE-16570_addnum_v5.patch, HBASE-16570_addnum_v6.patch, > HBASE-16570_addnum_v7.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15324) Jitter may cause desiredMaxFileSize overflow in ConstantSizeRegionSplitPolicy and trigger unexpected split
[
https://issues.apache.org/jira/browse/HBASE-15324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15650083#comment-15650083
]
Hudson commented on HBASE-15324:
SUCCESS: Integrated in Jenkins build HBase-1.1-JDK8 #1899 (See
[https://builds.apache.org/job/HBase-1.1-JDK8/1899/])
HBASE-15324 Jitter may cause desiredMaxFileSize overflow in (esteban: rev
9fb5a8608dff761160e217dc3dd5b2dfa9b4875d)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java
> Jitter may cause desiredMaxFileSize overflow in ConstantSizeRegionSplitPolicy
> and trigger unexpected split
> --
>
> Key: HBASE-15324
> URL: https://issues.apache.org/jira/browse/HBASE-15324
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.1.3
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.5, 1.1.8
>
> Attachments: HBASE-15324.patch, HBASE-15324_v2.patch,
> HBASE-15324_v3.patch, HBASE-15324_v3.patch
>
>
> We introduce jitter for region split decision in HBASE-13412, but the
> following line in {{ConstantSizeRegionSplitPolicy}} may cause long value
> overflow if MAX_FILESIZE is specified to Long.MAX_VALUE:
> {code}
> this.desiredMaxFileSize += (long)(desiredMaxFileSize * (RANDOM.nextFloat() -
> 0.5D) * jitter);
> {code}
> In our case we specify MAX_FILESIZE to Long.MAX_VALUE to prevent target
> region to split.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16993) BucketCache throw java.io.IOException: Invalid HFile block magic when DATA_BLOCK_ENCODING set to DIFF
[
https://issues.apache.org/jira/browse/HBASE-16993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15650078#comment-15650078
]
Anoop Sam John commented on HBASE-16993:
Am not the original author of this. So sorry I also dont the real reason behind
this. I believe as we make offset as long, we wont get any overflow. 3 bytes
per entry saving is less compared to original heap size per BucketEntry what we
have now. So am ok with this change.
> BucketCache throw java.io.IOException: Invalid HFile block magic when
> DATA_BLOCK_ENCODING set to DIFF
> -
>
> Key: HBASE-16993
> URL: https://issues.apache.org/jira/browse/HBASE-16993
> Project: HBase
> Issue Type: Bug
> Components: io
>Affects Versions: 1.1.3
> Environment: hbase version 1.1.3
>Reporter: liubangchen
>Assignee: liubangchen
> Attachments: HBASE-16993.000.patch, HBASE-16993.001.patch
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> hbase-site.xml setting
>
> hbase.bucketcache.bucket.sizes
> 16384,32768,40960,
> 46000,49152,51200,65536,131072,524288
>
>
> hbase.bucketcache.size
> 16384
>
>
> hbase.bucketcache.ioengine
> offheap
>
>
> hfile.block.cache.size
> 0.3
>
>
> hfile.block.bloom.cacheonwrite
> true
>
>
> hbase.rs.cacheblocksonwrite
> true
>
>
> hfile.block.index.cacheonwrite
> true
> n_splits = 200
> create 'usertable',{NAME =>'family', COMPRESSION => 'snappy', VERSIONS =>
> 1,DATA_BLOCK_ENCODING => 'DIFF',CONFIGURATION =>
> {'hbase.hregion.memstore.block.multiplier' => 5}},{DURABILITY =>
> 'SKIP_WAL'},{SPLITS => (1..n_splits).map {|i|
> "user#{1000+i*(-1000)/n_splits}"}}
> load data
> bin/ycsb load hbase10 -P workloads/workloada -p table=usertable -p
> columnfamily=family -p fieldcount=10 -p fieldlength=100 -p
> recordcount=2 -p insertorder=hashed -p insertstart=0 -p
> clientbuffering=true -p durability=SKIP_WAL -threads 20 -s
> run
> bin/ycsb run hbase10 -P workloads/workloadb -p table=usertable -p
> columnfamily=family -p fieldcount=10 -p fieldlength=100 -p
> operationcount=2000 -p readallfields=true -p clientbuffering=true -p
> requestdistribution=zipfian -threads 10 -s
> log info
> 2016-11-02 20:20:20,261 ERROR
> [RW.default.readRpcServer.handler=36,queue=21,port=6020] bucket.BucketCache:
> Failed reading block fdcc7ed6f3b2498b9ef316cc8206c233_44819759 from bucket
> cache
> java.io.IOException: Invalid HFile block magic:
> \x00\x00\x00\x00\x00\x00\x00\x00
> at
> org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:154)
> at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:167)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.(HFileBlock.java:273)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$1.deserialize(HFileBlock.java:134)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$1.deserialize(HFileBlock.java:121)
> at
> org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.getBlock(BucketCache.java:427)
> at
> org.apache.hadoop.hbase.io.hfile.CombinedBlockCache.getBlock(CombinedBlockCache.java:85)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.getCachedBlock(HFileReaderV2.java:266)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:403)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:269)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:634)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:584)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:247)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:156)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:363)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:217)
> at
> org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2071)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:5369)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2546)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2532)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2514)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6558)
>
[jira] [Commented] (HBASE-16962) Add readPoint to preCompactScannerOpen() and preFlushScannerOpen() API
[
https://issues.apache.org/jira/browse/HBASE-16962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15650073#comment-15650073
]
Anoop Sam John commented on HBASE-16962:
Patch looks good overall.
Not directly on ur patch
{code}
protected InternalScanner preCreateCoprocScanner(final CompactionRequest
request,
final ScanType scanType, final long earliestPutTs, final
List scanners,
User user) throws IOException {
{code}
Seems the code flow dont use this at all.. Finally we pass null as User always
to the CP area. A miss? Or we dont mind the user here? ANy idea [~tedyu]?
> Add readPoint to preCompactScannerOpen() and preFlushScannerOpen() API
> --
>
> Key: HBASE-16962
> URL: https://issues.apache.org/jira/browse/HBASE-16962
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Thiruvel Thirumoolan
>Assignee: Thiruvel Thirumoolan
> Attachments: HBASE-16962.master.001.patch,
> HBASE-16962.master.002.patch, HBASE-16962.master.003.patch,
> HBASE-16962.master.004.patch, HBASE-16962.rough.patch
>
>
> Similar to HBASE-15759, I would like to add readPoint to the
> preCompactScannerOpen() API.
> I have a CP where I create a StoreScanner() as part of the
> preCompactScannerOpen() API. I need the readpoint which was obtained in
> Compactor.compact() method to be consistent.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16570) Compute region locality in parallel at startup
[ https://issues.apache.org/jira/browse/HBASE-16570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649981#comment-15649981 ] binlijin edited comment on HBASE-16570 at 11/9/16 7:05 AM: --- The new added UT case is to make sure after RegionLocationFinder#refreshAndWait, regions's HDFSBlocksDistribution have cached in master's memory. And change lastFullRefresh is to make HMaster start up faster when backup master becomes active master, we have encounter a longer startup(backup master becomes active master) which last 2.5 hours because namenode in high load, and after the change we only less than 5 minutes. - private long lastFullRefresh = 0; + // Do not scheduleFullRefresh at master startup + private long lastFullRefresh = EnvironmentEdgeManager.currentTime(); was (Author: aoxiang): The new added UT case is to make sure after RegionLocationFinder#refreshAndWait, regions's HDFSBlocksDistribution have cached in master's memory. And change lastFullRefresh is to make HMaster start up faster when become active master from backup master, we have encounter a longer startup(backup master become active master) which last 2.5 hours because namenode in high load. - private long lastFullRefresh = 0; + // Do not scheduleFullRefresh at master startup + private long lastFullRefresh = EnvironmentEdgeManager.currentTime(); > Compute region locality in parallel at startup > -- > > Key: HBASE-16570 > URL: https://issues.apache.org/jira/browse/HBASE-16570 > Project: HBase > Issue Type: Sub-task >Reporter: binlijin >Assignee: binlijin > Fix For: 2.0.0, 1.4.0, 1.3.1 > > Attachments: HBASE-16570-master_V1.patch, > HBASE-16570-master_V2.patch, HBASE-16570-master_V3.patch, > HBASE-16570-master_V4.patch, HBASE-16570.branch-1.3-addendum.patch, > HBASE-16570_addnum.patch, HBASE-16570_addnum_v2.patch, > HBASE-16570_addnum_v3.patch, HBASE-16570_addnum_v4.patch, > HBASE-16570_addnum_v5.patch, HBASE-16570_addnum_v6.patch, > HBASE-16570_addnum_v7.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16570) Compute region locality in parallel at startup
[ https://issues.apache.org/jira/browse/HBASE-16570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649981#comment-15649981 ] binlijin edited comment on HBASE-16570 at 11/9/16 7:05 AM: --- The new added UT case is to make sure after RegionLocationFinder#refreshAndWait, regions's HDFSBlocksDistribution have cached in master's memory. And change lastFullRefresh is to make HMaster start up faster when backup master becomes active master, we have encounter a longer startup(backup master becomes active master) which last 2.5 hours because namenode in high load, and after the change we only need less than 5 minutes. - private long lastFullRefresh = 0; + // Do not scheduleFullRefresh at master startup + private long lastFullRefresh = EnvironmentEdgeManager.currentTime(); was (Author: aoxiang): The new added UT case is to make sure after RegionLocationFinder#refreshAndWait, regions's HDFSBlocksDistribution have cached in master's memory. And change lastFullRefresh is to make HMaster start up faster when backup master becomes active master, we have encounter a longer startup(backup master becomes active master) which last 2.5 hours because namenode in high load, and after the change we only less than 5 minutes. - private long lastFullRefresh = 0; + // Do not scheduleFullRefresh at master startup + private long lastFullRefresh = EnvironmentEdgeManager.currentTime(); > Compute region locality in parallel at startup > -- > > Key: HBASE-16570 > URL: https://issues.apache.org/jira/browse/HBASE-16570 > Project: HBase > Issue Type: Sub-task >Reporter: binlijin >Assignee: binlijin > Fix For: 2.0.0, 1.4.0, 1.3.1 > > Attachments: HBASE-16570-master_V1.patch, > HBASE-16570-master_V2.patch, HBASE-16570-master_V3.patch, > HBASE-16570-master_V4.patch, HBASE-16570.branch-1.3-addendum.patch, > HBASE-16570_addnum.patch, HBASE-16570_addnum_v2.patch, > HBASE-16570_addnum_v3.patch, HBASE-16570_addnum_v4.patch, > HBASE-16570_addnum_v5.patch, HBASE-16570_addnum_v6.patch, > HBASE-16570_addnum_v7.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14141) HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits from backup tables
[ https://issues.apache.org/jira/browse/HBASE-14141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15650007#comment-15650007 ] Devaraj Das commented on HBASE-14141: - [~vrodionov] is there a way in which we can reliably identify the WALs that need to be kept track of, when a request for an incremental backup comes. Like, if there are multiple WALs, and some of them are part of a backup set or something, and some are not, can we make that distinction, and store the metadata about these. We could totally stay out of the business of doing anything WAL specific, but we should just be able to query some metadata state about which specific WALs we should back up when a request for an incremental backup comes... The default implementation could be returning all the WALs that are currently present in the WAL directory, and they are all backed up (this is what we have today). I guess it's an extension to what you say above, but am trying to see if things can be simplified. So am proposing we flip it around, and say that if someone wants to take backups, they need to plan for it, and set up WALs appropriately. > HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits > from backup tables > > > Key: HBASE-14141 > URL: https://issues.apache.org/jira/browse/HBASE-14141 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16398) optimize HRegion computeHDFSBlocksDistribution
[ https://issues.apache.org/jira/browse/HBASE-16398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15650003#comment-15650003 ] binlijin commented on HBASE-16398: -- Thanks very much for the perf benchmark, i run on one of our cluster(opentsdb cluster, tsdb table have 5690 regions) and i can get the flowing result: Batch mode of computation: 45 seconds Old mode of computation: 70 seconds . > optimize HRegion computeHDFSBlocksDistribution > -- > > Key: HBASE-16398 > URL: https://issues.apache.org/jira/browse/HBASE-16398 > Project: HBase > Issue Type: Sub-task >Reporter: binlijin >Assignee: binlijin > Attachments: HBASE-16398.patch, LocatedBlockStatusComparison.java > > > First i assume there is no reference and link in a region family's directory. > Without the patch to computeHDFSBlocksDistribution for a region family, there > is 1+2*N rpc call, N is hfile numbers, The first rpc call is to > DistributedFileSystem#listStatus to get hfiles, for every hfile there is two > rpc call DistributedFileSystem#getFileStatus(path) and then > DistributedFileSystem#getFileBlockLocations(status, start, length). > With the patch to computeHDFSBlocksDistribution for a region family, there is > 2 rpc call, they are DistributedFileSystem#getFileStatus(path) and > DistributedFileSystem#listLocatedStatus(final Path p, final PathFilter > filter). > So if there is at least one hfile, with the patch, the rpc call will less. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16570) Compute region locality in parallel at startup
[ https://issues.apache.org/jira/browse/HBASE-16570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649981#comment-15649981 ] binlijin commented on HBASE-16570: -- The new added UT case is to make sure after RegionLocationFinder#refreshAndWait, regions's HDFSBlocksDistribution have cached in master's memory. And change lastFullRefresh is to make HMaster start up faster when become active master from backup master, we have encounter a longer startup(backup master become active master) which last 2.5 hours because namenode in high load. - private long lastFullRefresh = 0; + // Do not scheduleFullRefresh at master startup + private long lastFullRefresh = EnvironmentEdgeManager.currentTime(); > Compute region locality in parallel at startup > -- > > Key: HBASE-16570 > URL: https://issues.apache.org/jira/browse/HBASE-16570 > Project: HBase > Issue Type: Sub-task >Reporter: binlijin >Assignee: binlijin > Fix For: 2.0.0, 1.4.0, 1.3.1 > > Attachments: HBASE-16570-master_V1.patch, > HBASE-16570-master_V2.patch, HBASE-16570-master_V3.patch, > HBASE-16570-master_V4.patch, HBASE-16570.branch-1.3-addendum.patch, > HBASE-16570_addnum.patch, HBASE-16570_addnum_v2.patch, > HBASE-16570_addnum_v3.patch, HBASE-16570_addnum_v4.patch, > HBASE-16570_addnum_v5.patch, HBASE-16570_addnum_v6.patch, > HBASE-16570_addnum_v7.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15324) Jitter may cause desiredMaxFileSize overflow in ConstantSizeRegionSplitPolicy and trigger unexpected split
[
https://issues.apache.org/jira/browse/HBASE-15324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649904#comment-15649904
]
Hudson commented on HBASE-15324:
SUCCESS: Integrated in Jenkins build HBase-1.2-JDK8 #61 (See
[https://builds.apache.org/job/HBase-1.2-JDK8/61/])
HBASE-15324 Jitter may cause desiredMaxFileSize overflow in (esteban: rev
6b8472038dd2632d20fdf8d3216c2fe86ee68580)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java
> Jitter may cause desiredMaxFileSize overflow in ConstantSizeRegionSplitPolicy
> and trigger unexpected split
> --
>
> Key: HBASE-15324
> URL: https://issues.apache.org/jira/browse/HBASE-15324
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.1.3
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.5, 1.1.8
>
> Attachments: HBASE-15324.patch, HBASE-15324_v2.patch,
> HBASE-15324_v3.patch, HBASE-15324_v3.patch
>
>
> We introduce jitter for region split decision in HBASE-13412, but the
> following line in {{ConstantSizeRegionSplitPolicy}} may cause long value
> overflow if MAX_FILESIZE is specified to Long.MAX_VALUE:
> {code}
> this.desiredMaxFileSize += (long)(desiredMaxFileSize * (RANDOM.nextFloat() -
> 0.5D) * jitter);
> {code}
> In our case we specify MAX_FILESIZE to Long.MAX_VALUE to prevent target
> region to split.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17039) SimpleLoadBalancer schedules large amount of invalid region moves
[
https://issues.apache.org/jira/browse/HBASE-17039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649893#comment-15649893
]
Yu Li commented on HBASE-17039:
---
Thanks [~tedyu] and [~enis] for review, will commit this one soon if no
objections.
> SimpleLoadBalancer schedules large amount of invalid region moves
> -
>
> Key: HBASE-17039
> URL: https://issues.apache.org/jira/browse/HBASE-17039
> Project: HBase
> Issue Type: Bug
> Components: Balancer
>Affects Versions: 2.0.0, 1.2.3, 1.1.7
>Reporter: Charlie Qiangeng Xu
>Assignee: Charlie Qiangeng Xu
> Attachments: HBASE-17039.patch
>
>
> After increasing one of our clusters to 1600 nodes, we observed a large
> amount of invalid region moves(more than 30k moves) fired by the balance
> chore. Thus we simulated the problem and printed out the balance plan, only
> to find out many servers that had two regions for a certain table(we use by
> table strategy), sent out both regions to other two servers that have zero
> region.
> In the SimpleLoadBalancer's balanceCluster function,
> the code block that determines the underLoadedServers might have a problem:
> {code}
> if (load >= min && load > 0) {
> continue; // look for other servers which haven't reached min
> }
> int regionsToPut = min - load;
> if (regionsToPut == 0)
> {
> regionsToPut = 1;
> }
> {code}
> if min is zero, some server that has load of zero, which equals to min would
> be marked as underloaded, which would cause the phenomenon mentioned above.
> Since we increased the cluster's size to 1600+, many tables that only have
> 1000 regions, now would encounter such issue.
> By fixing it up, the balance plan went back to normal.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16838) Implement basic scan
[ https://issues.apache.org/jira/browse/HBASE-16838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649884#comment-15649884 ] Duo Zhang commented on HBASE-16838: --- [~stack] [~carp84] Any concerns on the v3 patch? Thanks. > Implement basic scan > > > Key: HBASE-16838 > URL: https://issues.apache.org/jira/browse/HBASE-16838 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16838-v1.patch, HBASE-16838-v2.patch, > HBASE-16838-v3.patch, HBASE-16838.patch > > > Implement a scan works like the grpc streaming call that all returned results > will be passed to a ScanObserver. The methods of the observer will be called > directly in the rpc framework threads so it is not allowed to do time > consuming work in the methods. So in general only experts or the > implementation of other methods in AsyncTable can call this method directly, > that's why I call it 'basic scan'. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17053) Remove LogRollerExitedChecker
[ https://issues.apache.org/jira/browse/HBASE-17053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17053: -- Status: Patch Available (was: Open) > Remove LogRollerExitedChecker > - > > Key: HBASE-17053 > URL: https://issues.apache.org/jira/browse/HBASE-17053 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17053.patch > > > Now the LogRoll may exit before WAL exit but we will still write some close > event to WAL when shutting down RS. And for AsyncFSWAL, we will open a new > wal writer when error occur. If LogRoller has already exited then AsyncFSWAL > will wait for ever and cause RS to hang. > It does not make sense to quit LogRoller ahead of WAL, this jira aims to > change the shutdown order. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17053) Remove LogRollerExitedChecker
[ https://issues.apache.org/jira/browse/HBASE-17053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17053: -- Attachment: HBASE-17053.patch Add a running flag to LogRoller instead of server.isStopped. And also remove the metaLogRoller as now the LogRoller can handle rolling of multiple WALs. > Remove LogRollerExitedChecker > - > > Key: HBASE-17053 > URL: https://issues.apache.org/jira/browse/HBASE-17053 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17053.patch > > > Now the LogRoll may exit before WAL exit but we will still write some close > event to WAL when shutting down RS. And for AsyncFSWAL, we will open a new > wal writer when error occur. If LogRoller has already exited then AsyncFSWAL > will wait for ever and cause RS to hang. > It does not make sense to quit LogRoller ahead of WAL, this jira aims to > change the shutdown order. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16962) Add readPoint to preCompactScannerOpen() and preFlushScannerOpen() API
[
https://issues.apache.org/jira/browse/HBASE-16962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649872#comment-15649872
]
Hadoop QA commented on HBASE-16962:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 9s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
48s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
34s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 14s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 94m 29s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 130m 57s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hbase.util.TestHBaseFsckReplicas |
| | org.apache.hadoop.hbase.TestRegionRebalancing |
| | org.apache.hadoop.hbase.util.TestRegionSplitter |
| | org.apache.hadoop.hbase.util.TestMiniClusterLoadParallel |
| | org.apache.hadoop.hbase.mapred.TestTableSnapshotInputFormat |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12838111/HBASE-16962.master.004.patch
|
| JIRA Issue | HBASE-16962 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux d5d4d84b4a4d 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 03bc884 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/4394/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/4394/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4394/testR
[jira] [Commented] (HBASE-17047) Add an API to get HBase connection cache statistics
[ https://issues.apache.org/jira/browse/HBASE-17047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649860#comment-15649860 ] Weiqing Yang commented on HBASE-17047: -- Sorry for the misleading title/description. Our client has asked us to log information about the cache, but we’ve decide not to do so. It’s an API that gives statistics information of the cache (like you suggested). The refcount has nothing to do with this patch, but its decrement is here: https://github.com/apache/hbase/blob/master/hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/HBaseConnectionCache.scala#L131 > Add an API to get HBase connection cache statistics > --- > > Key: HBASE-17047 > URL: https://issues.apache.org/jira/browse/HBASE-17047 > Project: HBase > Issue Type: Improvement > Components: spark >Reporter: Weiqing Yang >Assignee: Weiqing Yang >Priority: Minor > Attachments: HBASE-17047_v1.patch > > > This patch will add a function "getStat" for the user to get the statistics > of the HBase connection cache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14123) HBase Backup/Restore Phase 2
[ https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649851#comment-15649851 ] stack commented on HBASE-14123: --- Is the patch rebased to master? > HBase Backup/Restore Phase 2 > > > Key: HBASE-14123 > URL: https://issues.apache.org/jira/browse/HBASE-14123 > Project: HBase > Issue Type: Umbrella >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: 14123-master.v14.txt, 14123-master.v15.txt, > 14123-master.v16.txt, 14123-master.v17.txt, 14123-master.v18.txt, > 14123-master.v19.txt, 14123-master.v2.txt, 14123-master.v20.txt, > 14123-master.v21.txt, 14123-master.v24.txt, 14123-master.v25.txt, > 14123-master.v27.txt, 14123-master.v28.txt, 14123-master.v29.full.txt, > 14123-master.v3.txt, 14123-master.v30.txt, 14123-master.v31.txt, > 14123-master.v32.txt, 14123-master.v33.txt, 14123-master.v34.txt, > 14123-master.v35.txt, 14123-master.v5.txt, 14123-master.v6.txt, > 14123-master.v7.txt, 14123-master.v8.txt, 14123-master.v9.txt, 14123-v14.txt, > HBASE-14123-for-7912-v1.patch, HBASE-14123-for-7912-v6.patch, > HBASE-14123-v1.patch, HBASE-14123-v10.patch, HBASE-14123-v11.patch, > HBASE-14123-v12.patch, HBASE-14123-v13.patch, HBASE-14123-v15.patch, > HBASE-14123-v16.patch, HBASE-14123-v2.patch, HBASE-14123-v3.patch, > HBASE-14123-v4.patch, HBASE-14123-v5.patch, HBASE-14123-v6.patch, > HBASE-14123-v7.patch, HBASE-14123-v9.patch > > > Phase 2 umbrella JIRA. See HBASE-7912 for design document and description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17047) Add an API to get HBase connection cache statistics
[ https://issues.apache.org/jira/browse/HBASE-17047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiqing Yang updated HBASE-17047: - Description: This patch will add a function "getStat" for the user to get the statistics of the HBase connection cache. (was: This patch will add a function "getStat" for the user to get the state of the HBase connection cache.) > Add an API to get HBase connection cache statistics > --- > > Key: HBASE-17047 > URL: https://issues.apache.org/jira/browse/HBASE-17047 > Project: HBase > Issue Type: Improvement > Components: spark >Reporter: Weiqing Yang >Assignee: Weiqing Yang >Priority: Minor > Attachments: HBASE-17047_v1.patch > > > This patch will add a function "getStat" for the user to get the statistics > of the HBase connection cache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17047) Add an API to get HBase connection cache statistics
[ https://issues.apache.org/jira/browse/HBASE-17047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiqing Yang updated HBASE-17047: - Summary: Add an API to get HBase connection cache statistics (was: Add an API to get the state of HBase connection cache) > Add an API to get HBase connection cache statistics > --- > > Key: HBASE-17047 > URL: https://issues.apache.org/jira/browse/HBASE-17047 > Project: HBase > Issue Type: Improvement > Components: spark >Reporter: Weiqing Yang >Assignee: Weiqing Yang >Priority: Minor > Attachments: HBASE-17047_v1.patch > > > This patch will add a function "getStat" for the user to get the state of the > HBase connection cache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17047) Add an API to get the state of HBase connection cache
[ https://issues.apache.org/jira/browse/HBASE-17047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiqing Yang updated HBASE-17047: - Summary: Add an API to get the state of HBase connection cache (was: Add an API to log the state of HBase connection cache) > Add an API to get the state of HBase connection cache > - > > Key: HBASE-17047 > URL: https://issues.apache.org/jira/browse/HBASE-17047 > Project: HBase > Issue Type: Improvement > Components: spark >Reporter: Weiqing Yang >Assignee: Weiqing Yang >Priority: Minor > Attachments: HBASE-17047_v1.patch > > > This patch will add a function "getStat" for the user to get the state of the > HBase connection cache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17039) SimpleLoadBalancer schedules large amount of invalid region moves
[
https://issues.apache.org/jira/browse/HBASE-17039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649678#comment-15649678
]
Charlie Qiangeng Xu commented on HBASE-17039:
-
Hi Enis, The reason that we don't use Stochastic one is because we used to find
the regions were moved erratically.
And the conf parameter that control the StochasticLB is a bit hard to
fine-tune.(the chance that we can play back and forth on large scale production
cluster is limited)
Another factor that might contribute to the decision would be that
simpleBalancer can guarantee table level region balance, which is very crucial
in our use case.
Those are old conclusions though, may be outdated, since StochasticLB has been
updated by several patches after
we observe those disadvantages.
We will definitely retry StochasticLB after these busy months and will give you
feed back after that :)
> SimpleLoadBalancer schedules large amount of invalid region moves
> -
>
> Key: HBASE-17039
> URL: https://issues.apache.org/jira/browse/HBASE-17039
> Project: HBase
> Issue Type: Bug
> Components: Balancer
>Affects Versions: 2.0.0, 1.2.3, 1.1.7
>Reporter: Charlie Qiangeng Xu
>Assignee: Charlie Qiangeng Xu
> Attachments: HBASE-17039.patch
>
>
> After increasing one of our clusters to 1600 nodes, we observed a large
> amount of invalid region moves(more than 30k moves) fired by the balance
> chore. Thus we simulated the problem and printed out the balance plan, only
> to find out many servers that had two regions for a certain table(we use by
> table strategy), sent out both regions to other two servers that have zero
> region.
> In the SimpleLoadBalancer's balanceCluster function,
> the code block that determines the underLoadedServers might have a problem:
> {code}
> if (load >= min && load > 0) {
> continue; // look for other servers which haven't reached min
> }
> int regionsToPut = min - load;
> if (regionsToPut == 0)
> {
> regionsToPut = 1;
> }
> {code}
> if min is zero, some server that has load of zero, which equals to min would
> be marked as underloaded, which would cause the phenomenon mentioned above.
> Since we increased the cluster's size to 1600+, many tables that only have
> 1000 regions, now would encounter such issue.
> By fixing it up, the balance plan went back to normal.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14955) OOME: cannot create native thread is back
[
https://issues.apache.org/jira/browse/HBASE-14955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649670#comment-15649670
]
Hadoop QA commented on HBASE-14955:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
10s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
53s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 50s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 55s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 91m 7s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 23s
{color} | {color:red} The patch generated 24 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 142m 43s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hbase.TestHBaseTestingUtility |
| | org.apache.hadoop.hbase.constraint.TestConstraint |
| |
org.apache.hadoop.hbase.master.normalizer.TestSimpleRegionNormalizerOnCluster |
| | org.apache.hadoop.hbase.TestJMXListener |
| | org.apache.hadoop.hbase.mapred.TestTableMapReduce |
| | org.apache.hadoop.hbase.client.TestFromClientSideWithCoprocessor |
| | org.apache.hadoop.hbase.TestPartialResultsFromClientSide |
| | org.apache.hadoop.hbase.mapred.TestTableSnapshotInputFormat |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12838050/14955.patch |
| JIRA Issue | HBASE-14955 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 242f1121d941 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 0998af0 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/4393/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/4393/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https:/
[jira] [Commented] (HBASE-16993) BucketCache throw java.io.IOException: Invalid HFile block magic when DATA_BLOCK_ENCODING set to DIFF
[
https://issues.apache.org/jira/browse/HBASE-16993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649657#comment-15649657
]
liubangchen commented on HBASE-16993:
-
I want to confirm that the bucket.sizes runs correctly with not multiple of
1024. So I insert some changes in this unit test. I am not sure if it is
necessary. I will remove it if not. Original author do the did the byte shift
gymnastics. Maybe he did not thinking about such large cache, and using int
can save some a few bytes. That is just my assumption. I wish the original
author [~anoop.hbase] would like to help me to review the patch is correct or
necessary.Thnaks
> BucketCache throw java.io.IOException: Invalid HFile block magic when
> DATA_BLOCK_ENCODING set to DIFF
> -
>
> Key: HBASE-16993
> URL: https://issues.apache.org/jira/browse/HBASE-16993
> Project: HBase
> Issue Type: Bug
> Components: io
>Affects Versions: 1.1.3
> Environment: hbase version 1.1.3
>Reporter: liubangchen
>Assignee: liubangchen
> Attachments: HBASE-16993.000.patch, HBASE-16993.001.patch
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> hbase-site.xml setting
>
> hbase.bucketcache.bucket.sizes
> 16384,32768,40960,
> 46000,49152,51200,65536,131072,524288
>
>
> hbase.bucketcache.size
> 16384
>
>
> hbase.bucketcache.ioengine
> offheap
>
>
> hfile.block.cache.size
> 0.3
>
>
> hfile.block.bloom.cacheonwrite
> true
>
>
> hbase.rs.cacheblocksonwrite
> true
>
>
> hfile.block.index.cacheonwrite
> true
> n_splits = 200
> create 'usertable',{NAME =>'family', COMPRESSION => 'snappy', VERSIONS =>
> 1,DATA_BLOCK_ENCODING => 'DIFF',CONFIGURATION =>
> {'hbase.hregion.memstore.block.multiplier' => 5}},{DURABILITY =>
> 'SKIP_WAL'},{SPLITS => (1..n_splits).map {|i|
> "user#{1000+i*(-1000)/n_splits}"}}
> load data
> bin/ycsb load hbase10 -P workloads/workloada -p table=usertable -p
> columnfamily=family -p fieldcount=10 -p fieldlength=100 -p
> recordcount=2 -p insertorder=hashed -p insertstart=0 -p
> clientbuffering=true -p durability=SKIP_WAL -threads 20 -s
> run
> bin/ycsb run hbase10 -P workloads/workloadb -p table=usertable -p
> columnfamily=family -p fieldcount=10 -p fieldlength=100 -p
> operationcount=2000 -p readallfields=true -p clientbuffering=true -p
> requestdistribution=zipfian -threads 10 -s
> log info
> 2016-11-02 20:20:20,261 ERROR
> [RW.default.readRpcServer.handler=36,queue=21,port=6020] bucket.BucketCache:
> Failed reading block fdcc7ed6f3b2498b9ef316cc8206c233_44819759 from bucket
> cache
> java.io.IOException: Invalid HFile block magic:
> \x00\x00\x00\x00\x00\x00\x00\x00
> at
> org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:154)
> at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:167)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.(HFileBlock.java:273)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$1.deserialize(HFileBlock.java:134)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$1.deserialize(HFileBlock.java:121)
> at
> org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.getBlock(BucketCache.java:427)
> at
> org.apache.hadoop.hbase.io.hfile.CombinedBlockCache.getBlock(CombinedBlockCache.java:85)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.getCachedBlock(HFileReaderV2.java:266)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:403)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:269)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:634)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:584)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:247)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:156)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:363)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:217)
> at
> org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2071)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:5369)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2546)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getSc
[jira] [Updated] (HBASE-16962) Add readPoint to preCompactScannerOpen() and preFlushScannerOpen() API
[ https://issues.apache.org/jira/browse/HBASE-16962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvel Thirumoolan updated HBASE-16962: - Release Note: The following RegionObserver methods are deprecated and would no longer be called in hbase 2.0: InternalScanner preFlushScannerOpen(final ObserverContext c, final Store store, final KeyValueScanner memstoreScanner, final InternalScanner s) throws IOException; InternalScanner preCompactScannerOpen(final ObserverContext c, final Store store, List scanners, final ScanType scanType, final long earliestPutTs, final InternalScanner s, CompactionRequest request) Instead, use the following methods: InternalScanner preFlushScannerOpen(final ObserverContext c, final Store store, final KeyValueScanner memstoreScanner, final InternalScanner s, final long readPoint) throws IOException; InternalScanner preCompactScannerOpen(final ObserverContext c, final Store store, List scanners, final ScanType scanType, final long earliestPutTs, final InternalScanner s, final CompactionRequest request, final long readPoint) throws IOException Status: Patch Available (was: Open) > Add readPoint to preCompactScannerOpen() and preFlushScannerOpen() API > -- > > Key: HBASE-16962 > URL: https://issues.apache.org/jira/browse/HBASE-16962 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Attachments: HBASE-16962.master.001.patch, > HBASE-16962.master.002.patch, HBASE-16962.master.003.patch, > HBASE-16962.master.004.patch, HBASE-16962.rough.patch > > > Similar to HBASE-15759, I would like to add readPoint to the > preCompactScannerOpen() API. > I have a CP where I create a StoreScanner() as part of the > preCompactScannerOpen() API. I need the readpoint which was obtained in > Compactor.compact() method to be consistent. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16962) Add readPoint to preCompactScannerOpen() and preFlushScannerOpen() API
[ https://issues.apache.org/jira/browse/HBASE-16962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvel Thirumoolan updated HBASE-16962: - Attachment: HBASE-16962.master.004.patch > Add readPoint to preCompactScannerOpen() and preFlushScannerOpen() API > -- > > Key: HBASE-16962 > URL: https://issues.apache.org/jira/browse/HBASE-16962 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Attachments: HBASE-16962.master.001.patch, > HBASE-16962.master.002.patch, HBASE-16962.master.003.patch, > HBASE-16962.master.004.patch, HBASE-16962.rough.patch > > > Similar to HBASE-15759, I would like to add readPoint to the > preCompactScannerOpen() API. > I have a CP where I create a StoreScanner() as part of the > preCompactScannerOpen() API. I need the readpoint which was obtained in > Compactor.compact() method to be consistent. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16962) Add readPoint to preCompactScannerOpen() and preFlushScannerOpen() API
[ https://issues.apache.org/jira/browse/HBASE-16962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvel Thirumoolan updated HBASE-16962: - Attachment: HBASE-16962.master.003.patch > Add readPoint to preCompactScannerOpen() and preFlushScannerOpen() API > -- > > Key: HBASE-16962 > URL: https://issues.apache.org/jira/browse/HBASE-16962 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Attachments: HBASE-16962.master.001.patch, > HBASE-16962.master.002.patch, HBASE-16962.master.003.patch, > HBASE-16962.rough.patch > > > Similar to HBASE-15759, I would like to add readPoint to the > preCompactScannerOpen() API. > I have a CP where I create a StoreScanner() as part of the > preCompactScannerOpen() API. I need the readpoint which was obtained in > Compactor.compact() method to be consistent. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16962) Add readPoint to preCompactScannerOpen() and preFlushScannerOpen() API
[ https://issues.apache.org/jira/browse/HBASE-16962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvel Thirumoolan updated HBASE-16962: - Attachment: HBASE-16962.master.002.patch > Add readPoint to preCompactScannerOpen() and preFlushScannerOpen() API > -- > > Key: HBASE-16962 > URL: https://issues.apache.org/jira/browse/HBASE-16962 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Attachments: HBASE-16962.master.001.patch, > HBASE-16962.master.002.patch, HBASE-16962.rough.patch > > > Similar to HBASE-15759, I would like to add readPoint to the > preCompactScannerOpen() API. > I have a CP where I create a StoreScanner() as part of the > preCompactScannerOpen() API. I need the readpoint which was obtained in > Compactor.compact() method to be consistent. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16962) Add readPoint to preCompactScannerOpen() and preFlushScannerOpen() API
[ https://issues.apache.org/jira/browse/HBASE-16962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvel Thirumoolan updated HBASE-16962: - Summary: Add readPoint to preCompactScannerOpen() and preFlushScannerOpen() API (was: Add readPoint to preCompactScannerOpen()/ API) > Add readPoint to preCompactScannerOpen() and preFlushScannerOpen() API > -- > > Key: HBASE-16962 > URL: https://issues.apache.org/jira/browse/HBASE-16962 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Attachments: HBASE-16962.master.001.patch, HBASE-16962.rough.patch > > > Similar to HBASE-15759, I would like to add readPoint to the > preCompactScannerOpen() API. > I have a CP where I create a StoreScanner() as part of the > preCompactScannerOpen() API. I need the readpoint which was obtained in > Compactor.compact() method to be consistent. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16962) Add readPoint to preCompactScannerOpen()/ API
[ https://issues.apache.org/jira/browse/HBASE-16962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvel Thirumoolan updated HBASE-16962: - Summary: Add readPoint to preCompactScannerOpen()/ API (was: Add readPoint to preCompactScannerOpen() API) > Add readPoint to preCompactScannerOpen()/ API > - > > Key: HBASE-16962 > URL: https://issues.apache.org/jira/browse/HBASE-16962 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Attachments: HBASE-16962.master.001.patch, HBASE-16962.rough.patch > > > Similar to HBASE-15759, I would like to add readPoint to the > preCompactScannerOpen() API. > I have a CP where I create a StoreScanner() as part of the > preCompactScannerOpen() API. I need the readpoint which was obtained in > Compactor.compact() method to be consistent. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-17053) Remove LogRollerExitedChecker
Duo Zhang created HBASE-17053: - Summary: Remove LogRollerExitedChecker Key: HBASE-17053 URL: https://issues.apache.org/jira/browse/HBASE-17053 Project: HBase Issue Type: Sub-task Components: wal Affects Versions: 2.0.0 Reporter: Duo Zhang Assignee: Duo Zhang Fix For: 2.0.0 Now the LogRoll may exit before WAL exit but we will still write some close event to WAL when shutting down RS. And for AsyncFSWAL, we will open a new wal writer when error occur. If LogRoller has already exited then AsyncFSWAL will wait for ever and cause RS to hang. It does not make sense to quit LogRoller ahead of WAL, this jira aims to change the shutdown order. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15788) Use Offheap ByteBuffers from BufferPool to read RPC requests.
[ https://issues.apache.org/jira/browse/HBASE-15788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649570#comment-15649570 ] Duo Zhang commented on HBASE-15788: --- Yeah you can just use Runnable instead of CallCleanup here. A method can now be assigned to a FunctionalInterface in java8. > Use Offheap ByteBuffers from BufferPool to read RPC requests. > - > > Key: HBASE-15788 > URL: https://issues.apache.org/jira/browse/HBASE-15788 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-15788.patch, HBASE-15788_V4.patch, > HBASE-15788_V5.patch, HBASE-15788_V6.patch > > > Right now, when an RPC request reaches RpcServer, we read the request into an > on demand created byte[]. When it is write request and including many > mutations, the request size will be some what larger and we end up creating > many temp on heap byte[] and causing more GCs. > We have a ByteBufferPool of fixed sized off heap BBs. This is used at > RpcServer while sending read response only. We can make use of the same while > reading reqs also. Instead of reading whole of the request bytes into a > single BB, we can read into N BBs (based on the req size). When BB is not > available from pool, we will fall back to old way of on demand on heap byte[] > creation. > Remember these are off heap BBs. We read many proto objects from this read > request bytes (like header, Mutation protos etc). Thanks to PB 3 and our > shading work as it supports off heap BB now. Also the payload cells are also > in these DBBs now. The codec decoder can work on these and create off heap > BBs. Whole of our write path work with Cells now. At the time of addition to > memstore, these cells are by default copied to MSLAB ( off heap based pooled > MSLAB issue to follow this one). If MSLAB copy is not possible, we will do a > copy to on heap byte[]. > One possible down side of this is : > Before adding to Memstore, we do write to WAL. So the Cells created out of > the offheap BBs (Codec#Decoder) will be used to write to WAL. The default > FSHLog works with an OS obtained from DFSClient. This will have only standard > OS write APIs which is byte[] based. So just to write to WAL, we will end up > in temp on heap copy for each of the Cell. The other WAL imp (ie. AsynWAL) > supports writing offheap Cells directly. We have work in progress to make > AsycnWAL as default. Also we can raise HDFS req to support BB based write > APIs in their client OS? Until then, will try for a temp workaround solution. > Patch to say more on this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17000) [RegionServer] Compute region filesystem space use
[
https://issues.apache.org/jira/browse/HBASE-17000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649558#comment-15649558
]
Hadoop QA commented on HBASE-17000:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
5s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 33s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 6m
10s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
36s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 58s
{color} | {color:red} hbase-protocol-shaded in master has 24 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 1s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
2s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 6m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
38s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 24 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
30m 1s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:red}-1{color} | {color:red} hbaseprotoc {color} | {color:red} 0m 24s
{color} | {color:red} Patch generated 1 new protoc errors in hbase-server.
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
53s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 19s
{color} | {color:red} hbase-client generated 1 new + 13 unchanged - 0 fixed =
14 total (was 13) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 31s
{color} | {color:red} hbase-server generated 2 new + 1 unchanged - 0 fixed = 3
total (was 1) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 21s
{color} | {color:green} hbase-protocol in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 34s
{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 16s {color}
| {color:red} hbase-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 27s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 72m 47s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12838074/HBASE-17000
[jira] [Commented] (HBASE-15324) Jitter may cause desiredMaxFileSize overflow in ConstantSizeRegionSplitPolicy and trigger unexpected split
[
https://issues.apache.org/jira/browse/HBASE-15324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649550#comment-15649550
]
Hudson commented on HBASE-15324:
SUCCESS: Integrated in Jenkins build HBase-1.2-JDK7 #67 (See
[https://builds.apache.org/job/HBase-1.2-JDK7/67/])
HBASE-15324 Jitter may cause desiredMaxFileSize overflow in (esteban: rev
6b8472038dd2632d20fdf8d3216c2fe86ee68580)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java
> Jitter may cause desiredMaxFileSize overflow in ConstantSizeRegionSplitPolicy
> and trigger unexpected split
> --
>
> Key: HBASE-15324
> URL: https://issues.apache.org/jira/browse/HBASE-15324
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.1.3
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.5, 1.1.8
>
> Attachments: HBASE-15324.patch, HBASE-15324_v2.patch,
> HBASE-15324_v3.patch, HBASE-15324_v3.patch
>
>
> We introduce jitter for region split decision in HBASE-13412, but the
> following line in {{ConstantSizeRegionSplitPolicy}} may cause long value
> overflow if MAX_FILESIZE is specified to Long.MAX_VALUE:
> {code}
> this.desiredMaxFileSize += (long)(desiredMaxFileSize * (RANDOM.nextFloat() -
> 0.5D) * jitter);
> {code}
> In our case we specify MAX_FILESIZE to Long.MAX_VALUE to prevent target
> region to split.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16890) Analyze the performance of AsyncWAL and fix the same
[ https://issues.apache.org/jira/browse/HBASE-16890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649510#comment-15649510 ] Duo Zhang commented on HBASE-16890: --- It is the actual perf. AsyncFSWAL can finish WALPE with less time. But for PE AsyncFSWAL is about twice times slower. Maybe the problem is we put more stress on DN with AsyncFSWAL. In PE we will also do flush and compaction which also put stress on DN. Maybe this could make the DN too busy and slow thus leads to a bad performance. > Analyze the performance of AsyncWAL and fix the same > > > Key: HBASE-16890 > URL: https://issues.apache.org/jira/browse/HBASE-16890 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: AsyncWAL_disruptor.patch, AsyncWAL_disruptor_1 > (2).patch, AsyncWAL_disruptor_3.patch, AsyncWAL_disruptor_3.patch, > AsyncWAL_disruptor_4.patch, AsyncWAL_disruptor_6.patch, > HBASE-16890-rc-v2.patch, HBASE-16890-rc-v3.patch, > HBASE-16890-remove-contention-v1.patch, HBASE-16890-remove-contention.patch, > Screen Shot 2016-10-25 at 7.34.47 PM.png, Screen Shot 2016-10-25 at 7.39.07 > PM.png, Screen Shot 2016-10-25 at 7.39.48 PM.png, Screen Shot 2016-11-04 at > 5.21.27 PM.png, Screen Shot 2016-11-04 at 5.30.18 PM.png, async.svg, > classic.svg, contention.png, contention_defaultWAL.png > > > Tests reveal that AsyncWAL under load in single node cluster performs slower > than the Default WAL. This task is to analyze and see if we could fix it. > See some discussions in the tail of JIRA HBASE-15536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14955) OOME: cannot create native thread is back
[ https://issues.apache.org/jira/browse/HBASE-14955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649498#comment-15649498 ] Guanghao Zhang commented on HBASE-14955: I thought we need revisit this on branch-1. Because the local file system is only needed for the mr job. So we can only change the job's configuration and do not change TEST_UTIL's configuration. > OOME: cannot create native thread is back > - > > Key: HBASE-14955 > URL: https://issues.apache.org/jira/browse/HBASE-14955 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack >Assignee: Heng Chen > Fix For: 1.3.0 > > Attachments: 14955.patch, HBASE-14955-branch-1.patch > > > This fail is OOME cannot create native thread. Two MR jobs fail: > > org.apache.hadoop.hbase.mapreduce.TestImportTSVWithVisibilityLabels.org.apache.hadoop.hbase.mapreduce.TestImportTSVWithVisibilityLabels46 > ms1 > org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan1.org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan142 > ms1 > Was running 1.3 tests on H0. > Could try and purge resources used by these tests. > Making issue in meantime to keep an eye on it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-17052) compile-protobuf profile does not compile protobufs in some modules anymore
Enis Soztutar created HBASE-17052:
-
Summary: compile-protobuf profile does not compile protobufs in
some modules anymore
Key: HBASE-17052
URL: https://issues.apache.org/jira/browse/HBASE-17052
Project: HBase
Issue Type: Bug
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 2.0.0
Due to recent changes, we are not compiling the protobuf files in
hbase-endpoint, hbase-rsgroup, etc anymore.
{code}
[INFO] --- protobuf-maven-plugin:0.5.0:compile (compile-protoc) @ hbase-rsgroup
---
[INFO]
/Users/enis/projects/hbase-sal/hbase-rsgroup/src/main/protobuf/,/Users/enis/projects/hbase-sal/hbase-rsgroup/../hbase-protocol/src/main/protobuf
does not exist. Review the configuration or consider disabling the plugin.
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17021) Use RingBuffer to reduce the contention in AsyncFSWAL
[ https://issues.apache.org/jira/browse/HBASE-17021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17021: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to master. Thanks all for reviewing. > Use RingBuffer to reduce the contention in AsyncFSWAL > - > > Key: HBASE-17021 > URL: https://issues.apache.org/jira/browse/HBASE-17021 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: AsyncWAL_disruptor_7.patch, HBASE-17021-v1.patch, > HBASE-17021-v2.patch, HBASE-17021-v3.patch, HBASE-17021.patch > > > The WALPE result in HBASE-16890 shows that with disruptor's RingBuffer we can > get a better performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-17035) Check why we roll a wal writer at 10MB when the configured roll size is 120M+ with AsyncFSWAL
[ https://issues.apache.org/jira/browse/HBASE-17035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-17035. --- Resolution: Fixed Fixed by HBASE-17021. > Check why we roll a wal writer at 10MB when the configured roll size is 120M+ > with AsyncFSWAL > - > > Key: HBASE-17035 > URL: https://issues.apache.org/jira/browse/HBASE-17035 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > > Found this when addressing HBASE-16890. It is one of the possible reason that > why AsyncFSWAL performs worse than FSHLog when running PE tool. > https://issues.apache.org/jira/browse/HBASE-16890?focusedCommentId=15636688&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15636688 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17021) Use RingBuffer to reduce the contention in AsyncFSWAL
[ https://issues.apache.org/jira/browse/HBASE-17021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649451#comment-15649451 ] Duo Zhang commented on HBASE-17021: --- Will commit shortly. > Use RingBuffer to reduce the contention in AsyncFSWAL > - > > Key: HBASE-17021 > URL: https://issues.apache.org/jira/browse/HBASE-17021 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: AsyncWAL_disruptor_7.patch, HBASE-17021-v1.patch, > HBASE-17021-v2.patch, HBASE-17021-v3.patch, HBASE-17021.patch > > > The WALPE result in HBASE-16890 shows that with disruptor's RingBuffer we can > get a better performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14955) OOME: cannot create native thread is back
[ https://issues.apache.org/jira/browse/HBASE-14955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649435#comment-15649435 ] Guanghao Zhang commented on HBASE-14955: So this doesn't need to fix on master branch. > OOME: cannot create native thread is back > - > > Key: HBASE-14955 > URL: https://issues.apache.org/jira/browse/HBASE-14955 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack >Assignee: Heng Chen > Fix For: 1.3.0 > > Attachments: 14955.patch, HBASE-14955-branch-1.patch > > > This fail is OOME cannot create native thread. Two MR jobs fail: > > org.apache.hadoop.hbase.mapreduce.TestImportTSVWithVisibilityLabels.org.apache.hadoop.hbase.mapreduce.TestImportTSVWithVisibilityLabels46 > ms1 > org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan1.org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan142 > ms1 > Was running 1.3 tests on H0. > Could try and purge resources used by these tests. > Making issue in meantime to keep an eye on it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14955) OOME: cannot create native thread is back
[ https://issues.apache.org/jira/browse/HBASE-14955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649432#comment-15649432 ] Guanghao Zhang commented on HBASE-14955: I try the test without this in branch-1. And found that the mr job can't found some jar. So the local file system is only needed for the mr job. But in master branch this is not a problem. I thought the root cause is we use different hadoop version in branch-1 and master. > OOME: cannot create native thread is back > - > > Key: HBASE-14955 > URL: https://issues.apache.org/jira/browse/HBASE-14955 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack >Assignee: Heng Chen > Fix For: 1.3.0 > > Attachments: 14955.patch, HBASE-14955-branch-1.patch > > > This fail is OOME cannot create native thread. Two MR jobs fail: > > org.apache.hadoop.hbase.mapreduce.TestImportTSVWithVisibilityLabels.org.apache.hadoop.hbase.mapreduce.TestImportTSVWithVisibilityLabels46 > ms1 > org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan1.org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan142 > ms1 > Was running 1.3 tests on H0. > Could try and purge resources used by these tests. > Making issue in meantime to keep an eye on it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17051) libhbase++: implement RPC connection management
[ https://issues.apache.org/jira/browse/HBASE-17051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649347#comment-15649347 ] Enis Soztutar commented on HBASE-17051: --- bq. I was thinking of trying to separate the stuff into two tickets to avoid a jumbo patch, if possible Sorry for the confusion. I was just giving some context on the follow up patches and the overall approach. This jira should only do the ipc client. Later jiras will do the client on top. > libhbase++: implement RPC connection management > --- > > Key: HBASE-17051 > URL: https://issues.apache.org/jira/browse/HBASE-17051 > Project: HBase > Issue Type: Sub-task >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > > This proposes building RPC connection management layer, which supports the > equivalent functions resides in RpcConnection.java. Specifically, > handler/pipeline concepts will be used for implementation, similar to > NettyRpcConnection in java side. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-17051) libhbase++: implement RPC connection management
[ https://issues.apache.org/jira/browse/HBASE-17051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649328#comment-15649328 ] Xiaobing Zhou edited comment on HBASE-17051 at 11/9/16 12:32 AM: - Thanks [~enis] for the comments. I was thinking of trying to separate the stuff into two tickets to avoid a jumbo patch, if possible, i.e. rpc connection and client. Client will provide async capabilities of send out requests. was (Author: xiaobingo): Thanks [~enis] for the comments. I was thinking of trying to separate the stuff into two tickets to avoid a jumbo patch, i.e. rpc connection and client. Client will provide async capabilities of send out requests. > libhbase++: implement RPC connection management > --- > > Key: HBASE-17051 > URL: https://issues.apache.org/jira/browse/HBASE-17051 > Project: HBase > Issue Type: Sub-task >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > > This proposes building RPC connection management layer, which supports the > equivalent functions resides in RpcConnection.java. Specifically, > handler/pipeline concepts will be used for implementation, similar to > NettyRpcConnection in java side. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17051) libhbase++: implement RPC connection management
[ https://issues.apache.org/jira/browse/HBASE-17051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649328#comment-15649328 ] Xiaobing Zhou commented on HBASE-17051: --- Thanks [~enis] for the comments. I was thinking of trying to separate the stuff into two tickets to avoid a jumbo patch, i.e. rpc connection and client. Client will provide async capabilities of send out requests. > libhbase++: implement RPC connection management > --- > > Key: HBASE-17051 > URL: https://issues.apache.org/jira/browse/HBASE-17051 > Project: HBase > Issue Type: Sub-task >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > > This proposes building RPC connection management layer, which supports the > equivalent functions resides in RpcConnection.java. Specifically, > handler/pipeline concepts will be used for implementation, similar to > NettyRpcConnection in java side. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17051) libhbase++: implement RPC connection management
[ https://issues.apache.org/jira/browse/HBASE-17051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649287#comment-15649287 ] Enis Soztutar commented on HBASE-17051: --- Indeed. The idea is to follow the NettyRpcClient/AbstractRpcClient model to build the corresponding interfaces and implementation in the C++ land. The RPC client will share the same kind of low level responsibilities in terms of handling sockets, tracking outgoing RPCs via Calls, timeouts and IPC / Codec encodings. This layer will not know anything about higher level retries, and user-level requests (Get, Put, etc). Those will be build on top of this layer later following the AsyncTable model. > libhbase++: implement RPC connection management > --- > > Key: HBASE-17051 > URL: https://issues.apache.org/jira/browse/HBASE-17051 > Project: HBase > Issue Type: Sub-task >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > > This proposes building RPC connection management layer, which supports the > equivalent functions resides in RpcConnection.java. Specifically, > handler/pipeline concepts will be used for implementation, similar to > NettyRpcConnection in java side. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17051) libhbase++: implement RPC connection management
[ https://issues.apache.org/jira/browse/HBASE-17051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HBASE-17051: -- Description: This proposes building RPC connection management layer, which supports the equivalent functions resides in RpcConnection.java. Specifically, handler/pipeline concepts will be used for implementation, similar to NettyRpcConnection in java side. was:This proposes building RPC connection management layer, which supports the equivalent functions resides in RpcConnection.java. In addition, async semantics should be also supported, i.e. sending request with callback, which is similar to NettyRpcConnection in java side. > libhbase++: implement RPC connection management > --- > > Key: HBASE-17051 > URL: https://issues.apache.org/jira/browse/HBASE-17051 > Project: HBase > Issue Type: Sub-task >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > > This proposes building RPC connection management layer, which supports the > equivalent functions resides in RpcConnection.java. Specifically, > handler/pipeline concepts will be used for implementation, similar to > NettyRpcConnection in java side. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15324) Jitter may cause desiredMaxFileSize overflow in ConstantSizeRegionSplitPolicy and trigger unexpected split
[
https://issues.apache.org/jira/browse/HBASE-15324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649258#comment-15649258
]
Esteban Gutierrez commented on HBASE-15324:
---
Not pushing to 0.98 since the jitter added by HBASE-13412 is not enabled by
default.
> Jitter may cause desiredMaxFileSize overflow in ConstantSizeRegionSplitPolicy
> and trigger unexpected split
> --
>
> Key: HBASE-15324
> URL: https://issues.apache.org/jira/browse/HBASE-15324
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.1.3
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.5, 1.1.8
>
> Attachments: HBASE-15324.patch, HBASE-15324_v2.patch,
> HBASE-15324_v3.patch, HBASE-15324_v3.patch
>
>
> We introduce jitter for region split decision in HBASE-13412, but the
> following line in {{ConstantSizeRegionSplitPolicy}} may cause long value
> overflow if MAX_FILESIZE is specified to Long.MAX_VALUE:
> {code}
> this.desiredMaxFileSize += (long)(desiredMaxFileSize * (RANDOM.nextFloat() -
> 0.5D) * jitter);
> {code}
> In our case we specify MAX_FILESIZE to Long.MAX_VALUE to prevent target
> region to split.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15324) Jitter may cause desiredMaxFileSize overflow in ConstantSizeRegionSplitPolicy and trigger unexpected split
[
https://issues.apache.org/jira/browse/HBASE-15324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Esteban Gutierrez updated HBASE-15324:
--
Fix Version/s: 1.1.8
> Jitter may cause desiredMaxFileSize overflow in ConstantSizeRegionSplitPolicy
> and trigger unexpected split
> --
>
> Key: HBASE-15324
> URL: https://issues.apache.org/jira/browse/HBASE-15324
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.1.3
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.5, 1.1.8
>
> Attachments: HBASE-15324.patch, HBASE-15324_v2.patch,
> HBASE-15324_v3.patch, HBASE-15324_v3.patch
>
>
> We introduce jitter for region split decision in HBASE-13412, but the
> following line in {{ConstantSizeRegionSplitPolicy}} may cause long value
> overflow if MAX_FILESIZE is specified to Long.MAX_VALUE:
> {code}
> this.desiredMaxFileSize += (long)(desiredMaxFileSize * (RANDOM.nextFloat() -
> 0.5D) * jitter);
> {code}
> In our case we specify MAX_FILESIZE to Long.MAX_VALUE to prevent target
> region to split.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17051) libhbase++: implement RPC connection management
[ https://issues.apache.org/jira/browse/HBASE-17051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HBASE-17051: -- Description: This proposes building RPC connection management layer, which supports the equivalent functions resides in RpcConnection.java. In addition, async semantics should be also supported, i.e. sending request with callback, which is similar to NettyRpcConnection in java side. > libhbase++: implement RPC connection management > --- > > Key: HBASE-17051 > URL: https://issues.apache.org/jira/browse/HBASE-17051 > Project: HBase > Issue Type: Sub-task >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > > This proposes building RPC connection management layer, which supports the > equivalent functions resides in RpcConnection.java. In addition, async > semantics should be also supported, i.e. sending request with callback, which > is similar to NettyRpcConnection in java side. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-17051) libhbase++: implement RPC connection management
Xiaobing Zhou created HBASE-17051: - Summary: libhbase++: implement RPC connection management Key: HBASE-17051 URL: https://issues.apache.org/jira/browse/HBASE-17051 Project: HBase Issue Type: Sub-task Reporter: Xiaobing Zhou Assignee: Xiaobing Zhou -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15324) Jitter may cause desiredMaxFileSize overflow in ConstantSizeRegionSplitPolicy and trigger unexpected split
[
https://issues.apache.org/jira/browse/HBASE-15324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Esteban Gutierrez updated HBASE-15324:
--
Fix Version/s: 1.2.5
> Jitter may cause desiredMaxFileSize overflow in ConstantSizeRegionSplitPolicy
> and trigger unexpected split
> --
>
> Key: HBASE-15324
> URL: https://issues.apache.org/jira/browse/HBASE-15324
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.1.3
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.5
>
> Attachments: HBASE-15324.patch, HBASE-15324_v2.patch,
> HBASE-15324_v3.patch, HBASE-15324_v3.patch
>
>
> We introduce jitter for region split decision in HBASE-13412, but the
> following line in {{ConstantSizeRegionSplitPolicy}} may cause long value
> overflow if MAX_FILESIZE is specified to Long.MAX_VALUE:
> {code}
> this.desiredMaxFileSize += (long)(desiredMaxFileSize * (RANDOM.nextFloat() -
> 0.5D) * jitter);
> {code}
> In our case we specify MAX_FILESIZE to Long.MAX_VALUE to prevent target
> region to split.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13412) Region split decisions should have jitter
[ https://issues.apache.org/jira/browse/HBASE-13412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esteban Gutierrez updated HBASE-13412: -- Fix Version/s: 1.2.5 > Region split decisions should have jitter > - > > Key: HBASE-13412 > URL: https://issues.apache.org/jira/browse/HBASE-13412 > Project: HBase > Issue Type: New Feature > Components: regionserver >Affects Versions: 1.0.0, 2.0.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.1.0, 0.98.13 > > Attachments: HBASE-13412-v1.patch, HBASE-13412-v2.patch, > HBASE-13412-v3.patch, HBASE-13412.addendum.0.98-2.patch, > HBASE-13412.addendum.0.98.patch, HBASE-13412.patch, hbase-13412.addendum.patch > > > Whenever a region splits it causes lots of IO (compactions are queued for a > while). Because of this it's important to make sure that well distributed > tables don't have all of their regions split at exactly the same time. > This is basically the same as our compaction jitter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13412) Region split decisions should have jitter
[ https://issues.apache.org/jira/browse/HBASE-13412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esteban Gutierrez updated HBASE-13412: -- Fix Version/s: (was: 1.2.5) > Region split decisions should have jitter > - > > Key: HBASE-13412 > URL: https://issues.apache.org/jira/browse/HBASE-13412 > Project: HBase > Issue Type: New Feature > Components: regionserver >Affects Versions: 1.0.0, 2.0.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.1.0, 0.98.13 > > Attachments: HBASE-13412-v1.patch, HBASE-13412-v2.patch, > HBASE-13412-v3.patch, HBASE-13412.addendum.0.98-2.patch, > HBASE-13412.addendum.0.98.patch, HBASE-13412.patch, hbase-13412.addendum.patch > > > Whenever a region splits it causes lots of IO (compactions are queued for a > while). Because of this it's important to make sure that well distributed > tables don't have all of their regions split at exactly the same time. > This is basically the same as our compaction jitter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16982) Better integrate Apache CLI in AbstractHBaseTool
[ https://issues.apache.org/jira/browse/HBASE-16982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649155#comment-15649155 ] Hudson commented on HBASE-16982: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1940 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1940/]) HBASE-16982 Better integrate Apache CLI in AbstractHBaseTool. - (appy: rev 52241c90e4eab19b6613278d2e8107ab56c8a57a) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/util/AbstractHBaseTool.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java * (add) hbase-common/src/test/java/org/apache/hadoop/hbase/util/AbstractHBaseToolTest.java > Better integrate Apache CLI in AbstractHBaseTool > > > Key: HBASE-16982 > URL: https://issues.apache.org/jira/browse/HBASE-16982 > Project: HBase > Issue Type: Sub-task >Reporter: Appy >Assignee: Appy > Fix For: 2.0.0 > > Attachments: HBASE-16982.master.001.patch, > HBASE-16982.master.002.patch, HBASE-16982.master.003.patch, > HBASE-16982.master.004.patch, HBASE-16982.master.005.patch > > > Problem > 1. Inconsistencies in our user facing tools. > - options: > start with single/double dash > words separated by dash or underscore or just joined together > someplace use '=' to separate value, other use space (ExportSnapshot vs > HashTable) > - Description > Manually formatting options and their descriptions in printUsage() > Inconsistant formatting, sometimes even weird. > Incomplete. Sometimes people forget to add new option to description > 2. Manual parsing of options (those huge if-else loops iterating over args) > Solution > Use Apache CLI > - It has various validations for option names which'll fix first set of > issues. > - using AbstractHBaseTool's print usage function will ensure consistent > formatting (although we loose the power to order the options) > - If we enforce the method of defining options as in patch, it's highly > unlikely to forget adding description. > - CLI parses the options for us. > Using Apache CLI when writing new tools is straight forward, but it's not > easy when porting exiting tools since some option names are not valid as per > CLI's validation. > New method, processOldArgs(), will allow to port these tools in a backward > compatible manner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17030) Procedure v2 - A couple of tweaks to the SplitTableRegionProcedure
[ https://issues.apache.org/jira/browse/HBASE-17030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649150#comment-15649150 ] Hudson commented on HBASE-17030: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1940 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1940/]) HBASE-17030 Procedure v2 - A couple of tweaks to the (matteo.bertozzi: rev b143f6e4a3b3b019623e33fc07561977cda5c90e) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/SplitTableRegionProcedure.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java * (edit) hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/generated/MasterProcedureProtos.java * (edit) hbase-protocol-shaded/src/main/protobuf/MasterProcedure.proto * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsckOneRS.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestSplitTableRegionProcedure.java > Procedure v2 - A couple of tweaks to the SplitTableRegionProcedure > -- > > Key: HBASE-17030 > URL: https://issues.apache.org/jira/browse/HBASE-17030 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-17030-v0.patch, HBASE-17030-v0.patch > > > Make a couple of tweaks to HBASE-14551 split procedure > - remove tableName from SplitTableRegionProcedure ctor since we have the > RegionInfo that contains the name already > - move the checkRow in the constructor of the SplitTableRegionProcedure, > since the splitRow will never change and we can avoid to start the proc if we > have a bad splitRow. > - use the base AbstractStateMachineTableProcedure for the "user" field > - remove protobuf fields that can be extrapolated from other info > (table_name, split_row) > - avoid htd lookup every family iteration of splitStoreFiles() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17050) Upgrade Apache CLI version from 1.2 to 1.3.1
[ https://issues.apache.org/jira/browse/HBASE-17050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649151#comment-15649151 ] Hudson commented on HBASE-17050: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1940 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1940/]) HBASE-17050 Upgrade Apache CLI version from 1.2 to 1.3.1 (appy: rev 5e361b87f298916904be0e4a2bcd86b14dc1c571) * (edit) pom.xml > Upgrade Apache CLI version from 1.2 to 1.3.1 > > > Key: HBASE-17050 > URL: https://issues.apache.org/jira/browse/HBASE-17050 > Project: HBase > Issue Type: Sub-task >Reporter: Appy >Assignee: Appy >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17050.master.001.patch > > > http://mail-archives.apache.org/mod_mbox/hbase-dev/201611.mbox/%3CCAGHyZ6KNjTH6qnVN%2B%3Dd_rC%3DtkLEjELUCQf9pmqQ-ixJj%2BfbrOA%40mail.gmail.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16955) Fixup precommit protoc check to do new distributed protos and pb 3.1.0 build
[ https://issues.apache.org/jira/browse/HBASE-16955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649152#comment-15649152 ] Hudson commented on HBASE-16955: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1940 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1940/]) HBASE-16955 Fixup precommit protoc check to do new distributed protos (stack: rev c3b98b87fb22bbb94c820edcabcb695f176337f2) * (edit) hbase-protocol-shaded/pom.xml * (edit) dev-support/hbase-personality.sh > Fixup precommit protoc check to do new distributed protos and pb 3.1.0 build > > > Key: HBASE-16955 > URL: https://issues.apache.org/jira/browse/HBASE-16955 > Project: HBase > Issue Type: Task > Components: build, Protobufs >Reporter: stack >Assignee: stack > Fix For: 2.0.0 > > Attachments: 16955.patch, nothing_change.txt, nothing_change2 > (1).txt, nothing_change2 (2).txt, nothing_change2.txt, nothing_change2.txt, > nothing_change2.txt, nothing_change2.txt, nothing_change2.txt, > nothing_change2.txt > > > HBASE-15638 Shade protobuf and a follow-ons changed how we do protobufs. > One, protobufs are in the module they pertain to so distributed throughout > the modules and secondly, we do 2.5.0 pb for externally consumed protobuf -- > e.g. Coprocessor Endpoints -- but internally we use protobuf 3.1.0. > A precommit check looks to see if any proto changes break protoc compile. > This task is about updating the precommit to accommodate the changes brought > about by HBASE-15638. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16983) TestMultiTableSnapshotInputFormat failing with Unable to create region directory: /tmp/...
[
https://issues.apache.org/jira/browse/HBASE-16983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649154#comment-15649154
]
Hudson commented on HBASE-16983:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1940 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/1940/])
HBASE-16983 TestMultiTableSnapshotInputFormat failing with Unable to (stack:
rev f9c23f5e9debe2b8e1f3f21fc1d4c60505605309)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMultiTableSnapshotInputFormat.java
Revert "HBASE-16983 TestMultiTableSnapshotInputFormat failing with (stack: rev
9afcb4366d0098c42f2ea7c5da5aed02464bda63)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMultiTableSnapshotInputFormat.java
Revert "Revert "HBASE-16983 TestMultiTableSnapshotInputFormat failing (stack:
rev 0998af01df4b221793f7799922a4b088d951d39a)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMultiTableSnapshotInputFormat.java
* (edit)
hbase-common/src/test/java/org/apache/hadoop/hbase/HBaseCommonTestingUtility.java
> TestMultiTableSnapshotInputFormat failing with Unable to create region
> directory: /tmp/...
> ---
>
> Key: HBASE-16983
> URL: https://issues.apache.org/jira/browse/HBASE-16983
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: stack
>Assignee: stack
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: 16983.txt, HBASE-16983-ADDENDUM.patch,
> HBASE-16983-ADDENDUM.patch, HBASE-16983-ADDENDUM.patch,
> HBASE-16983-branch-1-ADDENDUM.patch, HBASE-16983-branch-1-ADDENDUM.patch
>
>
> Test is using /tmp. We failed creating dir in /tmp in a few tests from this
> suite just now:
> https://builds.apache.org/job/PreCommit-HBASE-Build/4253/testReport/org.apache.hadoop.hbase.mapred/TestMultiTableSnapshotInputFormat/testScanOBBToOPP/
> {code}
> Caused by: java.io.IOException: Unable to create region directory:
> /tmp/scantest2_snapshot__953e2b2d-22aa-4c6a-a46a-272619f5436e/data/default/scantest2/5629158a49e010e21ac0bd16453b2d8c
> at
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.createRegionOnFileSystem(HRegionFileSystem.java:896)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.createHRegion(HRegion.java:6520)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils.createRegion(ModifyRegionUtils.java:205)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils$1.call(ModifyRegionUtils.java:173)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils$1.call(ModifyRegionUtils.java:170)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> ...
> {code}
> No more detail than this. Let me change it so creates stuff in the test dir
> that it for sure owns/can write to.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17000) [RegionServer] Compute region filesystem space use
[ https://issues.apache.org/jira/browse/HBASE-17000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-17000: --- Attachment: HBASE-17000.001.patch .001 Some basic collection of region sizes being reported to the Master. The Master has a bare-bones implementation to collect this info. > [RegionServer] Compute region filesystem space use > -- > > Key: HBASE-17000 > URL: https://issues.apache.org/jira/browse/HBASE-17000 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: 2.0.0 > > Attachments: HBASE-17000.001.patch > > > Each RegionServer needs to track how much space a Region takes up and roll > this up to the table level. > Aggregation of this information in the Master will be covered by HBASE-16997. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17000) [RegionServer] Compute region filesystem space use
[ https://issues.apache.org/jira/browse/HBASE-17000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-17000: --- Fix Version/s: 2.0.0 Status: Patch Available (was: Open) > [RegionServer] Compute region filesystem space use > -- > > Key: HBASE-17000 > URL: https://issues.apache.org/jira/browse/HBASE-17000 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: 2.0.0 > > Attachments: HBASE-17000.001.patch > > > Each RegionServer needs to track how much space a Region takes up and roll > this up to the table level. > Aggregation of this information in the Master will be covered by HBASE-16997. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-17000) [RegionServer] Compute region filesystem space use
[ https://issues.apache.org/jira/browse/HBASE-17000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser reassigned HBASE-17000: -- Assignee: Josh Elser > [RegionServer] Compute region filesystem space use > -- > > Key: HBASE-17000 > URL: https://issues.apache.org/jira/browse/HBASE-17000 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Josh Elser >Assignee: Josh Elser > > Each RegionServer needs to track how much space a Region takes up and roll > this up to the table level. > Aggregation of this information in the Master will be covered by HBASE-16997. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16995) Build client Java API and client protobuf messages
[ https://issues.apache.org/jira/browse/HBASE-16995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649113#comment-15649113 ] Josh Elser commented on HBASE-16995: bq. Please put on review board when the patch is ready for review. Will do, [~tedyu]. I wanted to at least get a first implementation of the spectrum before asking for reviewers' time. Getting close -- end of this week, I'm hoping. > Build client Java API and client protobuf messages > -- > > Key: HBASE-16995 > URL: https://issues.apache.org/jira/browse/HBASE-16995 > Project: HBase > Issue Type: Sub-task > Components: Client >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: 2.0.0 > > Attachments: HBASE-16995.001.patch, HBASE-16995.002.patch > > > Extend the existing Java API and protobuf messages to allow the client to set > filesystem-use quotas via the Master. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15324) Jitter may cause desiredMaxFileSize overflow in ConstantSizeRegionSplitPolicy and trigger unexpected split
[
https://issues.apache.org/jira/browse/HBASE-15324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649100#comment-15649100
]
Esteban Gutierrez commented on HBASE-15324:
---
We ran into this in 1.2 with a customer and that caused 10s of thousands of new
regions to be created in matter of hours. I'm going to push it from 0.98 to 1.2
> Jitter may cause desiredMaxFileSize overflow in ConstantSizeRegionSplitPolicy
> and trigger unexpected split
> --
>
> Key: HBASE-15324
> URL: https://issues.apache.org/jira/browse/HBASE-15324
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.1.3
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15324.patch, HBASE-15324_v2.patch,
> HBASE-15324_v3.patch, HBASE-15324_v3.patch
>
>
> We introduce jitter for region split decision in HBASE-13412, but the
> following line in {{ConstantSizeRegionSplitPolicy}} may cause long value
> overflow if MAX_FILESIZE is specified to Long.MAX_VALUE:
> {code}
> this.desiredMaxFileSize += (long)(desiredMaxFileSize * (RANDOM.nextFloat() -
> 0.5D) * jitter);
> {code}
> In our case we specify MAX_FILESIZE to Long.MAX_VALUE to prevent target
> region to split.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16955) Fixup precommit protoc check to do new distributed protos and pb 3.1.0 build
[ https://issues.apache.org/jira/browse/HBASE-16955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-16955: -- Resolution: Fixed Fix Version/s: 2.0.0 Release Note: Test that environment no longer has to have protoc (2.5 and 3.1) available. Needed small adjustment in yetus protoc build but otherwise all works. Status: Resolved (was: Patch Available) Hurray That was it. Resolving after small change in our yetus personality around protoc. > Fixup precommit protoc check to do new distributed protos and pb 3.1.0 build > > > Key: HBASE-16955 > URL: https://issues.apache.org/jira/browse/HBASE-16955 > Project: HBase > Issue Type: Task > Components: build, Protobufs >Reporter: stack >Assignee: stack > Fix For: 2.0.0 > > Attachments: 16955.patch, nothing_change.txt, nothing_change2 > (1).txt, nothing_change2 (2).txt, nothing_change2.txt, nothing_change2.txt, > nothing_change2.txt, nothing_change2.txt, nothing_change2.txt, > nothing_change2.txt > > > HBASE-15638 Shade protobuf and a follow-ons changed how we do protobufs. > One, protobufs are in the module they pertain to so distributed throughout > the modules and secondly, we do 2.5.0 pb for externally consumed protobuf -- > e.g. Coprocessor Endpoints -- but internally we use protobuf 3.1.0. > A precommit check looks to see if any proto changes break protoc compile. > This task is about updating the precommit to accommodate the changes brought > about by HBASE-15638. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16983) TestMultiTableSnapshotInputFormat failing with Unable to create region directory: /tmp/...
[
https://issues.apache.org/jira/browse/HBASE-16983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15648937#comment-15648937
]
Hudson commented on HBASE-16983:
FAILURE: Integrated in Jenkins build HBase-1.4 #526 (See
[https://builds.apache.org/job/HBase-1.4/526/])
HBASE-16983 TestMultiTableSnapshotInputFormat failing with Unable to (stack:
rev a70f73c1e1bbb2d79faab77f7f38bcb91cbe1717)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMultiTableSnapshotInputFormat.java
> TestMultiTableSnapshotInputFormat failing with Unable to create region
> directory: /tmp/...
> ---
>
> Key: HBASE-16983
> URL: https://issues.apache.org/jira/browse/HBASE-16983
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: stack
>Assignee: stack
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: 16983.txt, HBASE-16983-ADDENDUM.patch,
> HBASE-16983-ADDENDUM.patch, HBASE-16983-ADDENDUM.patch,
> HBASE-16983-branch-1-ADDENDUM.patch, HBASE-16983-branch-1-ADDENDUM.patch
>
>
> Test is using /tmp. We failed creating dir in /tmp in a few tests from this
> suite just now:
> https://builds.apache.org/job/PreCommit-HBASE-Build/4253/testReport/org.apache.hadoop.hbase.mapred/TestMultiTableSnapshotInputFormat/testScanOBBToOPP/
> {code}
> Caused by: java.io.IOException: Unable to create region directory:
> /tmp/scantest2_snapshot__953e2b2d-22aa-4c6a-a46a-272619f5436e/data/default/scantest2/5629158a49e010e21ac0bd16453b2d8c
> at
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.createRegionOnFileSystem(HRegionFileSystem.java:896)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.createHRegion(HRegion.java:6520)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils.createRegion(ModifyRegionUtils.java:205)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils$1.call(ModifyRegionUtils.java:173)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils$1.call(ModifyRegionUtils.java:170)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> ...
> {code}
> No more detail than this. Let me change it so creates stuff in the test dir
> that it for sure owns/can write to.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17039) SimpleLoadBalancer schedules large amount of invalid region moves
[
https://issues.apache.org/jira/browse/HBASE-17039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15648907#comment-15648907
]
Enis Soztutar commented on HBASE-17039:
---
Is there a specific reason that you are not using StochasticLB? Is it because
in case of 1600 nodes, it does not work? I know that with 300+ nodes, it takes
a long time to come up with balance plans. Just asking to understand whether
StochasticLB improvements are needed at this scale.
> SimpleLoadBalancer schedules large amount of invalid region moves
> -
>
> Key: HBASE-17039
> URL: https://issues.apache.org/jira/browse/HBASE-17039
> Project: HBase
> Issue Type: Bug
> Components: Balancer
>Affects Versions: 2.0.0, 1.2.3, 1.1.7
>Reporter: Charlie Qiangeng Xu
>Assignee: Charlie Qiangeng Xu
> Attachments: HBASE-17039.patch
>
>
> After increasing one of our clusters to 1600 nodes, we observed a large
> amount of invalid region moves(more than 30k moves) fired by the balance
> chore. Thus we simulated the problem and printed out the balance plan, only
> to find out many servers that had two regions for a certain table(we use by
> table strategy), sent out both regions to other two servers that have zero
> region.
> In the SimpleLoadBalancer's balanceCluster function,
> the code block that determines the underLoadedServers might have a problem:
> {code}
> if (load >= min && load > 0) {
> continue; // look for other servers which haven't reached min
> }
> int regionsToPut = min - load;
> if (regionsToPut == 0)
> {
> regionsToPut = 1;
> }
> {code}
> if min is zero, some server that has load of zero, which equals to min would
> be marked as underloaded, which would cause the phenomenon mentioned above.
> Since we increased the cluster's size to 1600+, many tables that only have
> 1000 regions, now would encounter such issue.
> By fixing it up, the balance plan went back to normal.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-14141) HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits from backup tables
[ https://issues.apache.org/jira/browse/HBASE-14141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15648875#comment-15648875 ] Vladimir Rodionov edited comment on HBASE-14141 at 11/8/16 9:42 PM: Continued h5. Two mode of deployments h6. Standard When all or majority of tables require backup/restore. This is what we have right now. h6. Limited Usually small subset of table will require backup/restore. This is subject of this JIRA. h6. Switching between modes Switching will delete all backup meta - basically all backup data will be lost. This is by design to not encourage users to switch back and forth and to simplify implementation. was (Author: vrodionov): Continued h5. Two mode of deployments h6. Standard - when all or majority of tables require backup/restore. This is what we have right now. h6. Limited - usually small subset of table will require backup/restore. This is subject of this JIRA. h6. Switching between modes will clear up all backup meta - basically all backup data will be lost. This is by design to not encourage users to switch back and forth and to simplify implementation. > HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits > from backup tables > > > Key: HBASE-14141 > URL: https://issues.apache.org/jira/browse/HBASE-14141 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14141) HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits from backup tables
[ https://issues.apache.org/jira/browse/HBASE-14141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15648875#comment-15648875 ] Vladimir Rodionov commented on HBASE-14141: --- Continued h5. Two mode of deployments h6. Standard - when all or majority of tables require backup/restore. This is what we have right now. h6. Limited - usually small subset of table will require backup/restore. This is subject of this JIRA. h6. Switching between modes will clear up all backup meta - basically all backup data will be lost. This is by design to not encourage users to switch back and forth and to simplify implementation. > HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits > from backup tables > > > Key: HBASE-14141 > URL: https://issues.apache.org/jira/browse/HBASE-14141 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16955) Fixup precommit protoc check to do new distributed protos and pb 3.1.0 build
[
https://issues.apache.org/jira/browse/HBASE-16955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15648873#comment-15648873
]
Hadoop QA commented on HBASE-16955:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 2s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for
instructions. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
44s {color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
33m 31s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 2m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 19s
{color} | {color:green} hbase-protocol in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 34s
{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 10s
{color} | {color:green} hbase-spark in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 43m 1s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12838044/nothing_change2.txt |
| JIRA Issue | HBASE-16955 |
| Optional Tests | asflicense cc unit hbaseprotoc |
| uname | Linux 1e72775da6e5 3.13.0-98-generic #145-Ubuntu SMP Sat Oct 8
20:13:07 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 0998af0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4389/testReport/ |
| modules | C: hbase-protocol hbase-protocol-shaded hbase-spark U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4389/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Fixup precommit protoc check to do new distributed protos and pb 3.1.0 build
>
>
> Key: HBASE-16955
> URL: https://issues.apache.org/jira/browse/HBASE-16955
> Project: HBase
> Issue Type: Task
> Components: build, Protobufs
>Reporter: stack
>Assignee: stack
> Attachments: 16955.patch, nothing_change.txt, nothing_change2
> (1).txt, nothing_change2 (2).txt, nothing_change2.txt, nothing_change2.txt,
> nothing_change2.txt, nothing_change2.txt, nothing_change2.txt,
> nothing_change2.txt
>
>
> HBASE-15638 Shade protobuf and a follow-ons changed how we do protobufs.
> One, protobufs are in the module they pertain to so distributed throughout
> the modules and secondly, we do 2.5.0 pb for externally consumed protobuf --
> e.g. Coprocessor Endpoints -- but internally we use protobuf 3.1.0.
> A precommit check looks to see if any proto changes break protoc compile.
> This task is about updating the precommit to accommodate the changes brought
> about by HBASE-15638.
--
This messa
[jira] [Comment Edited] (HBASE-14141) HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits from backup tables
[
https://issues.apache.org/jira/browse/HBASE-14141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15648714#comment-15648714
]
Vladimir Rodionov edited comment on HBASE-14141 at 11/8/16 9:27 PM:
Continued.
h5. RegionGroupingProvider, RegionGroupingStrategy API change
Currently both work on encoded region names, where it is impossible to extract
table names from. So we will need to modify API and add table names to
{code}
public WAL getWAL(final byte[] identifier, byte[] namespace)
{code}
calls.
h5. New RegionGroupingStrategy - BackupRegionGroupingStrategy
* There will be a dedicated group for tables in backups.
* It is a wrapper for default or a user specified RegionGroupingStrategy. It
returns backup group name for a table in a backup list (from hbase:backup),
otherwise it calls default strategy
* Group name for backup tables is well known (hardcoded)
h5. Changes to current implementation
h6. All backup operations only on backup sets
Users won't be able to run backup on table or list of tables - only on backup
sets, but users still be able to restore a single table.
h6. Backup set add / remove
After adding/removing table to/from backup set, system disables/enables table
to read new backup set content (for activating/deactivating backup WAL for a
table)
Upd. Instead of direct disable/enable we can implement procedure, which updates
WAL instance in HRegion (may be tricky)
was (Author: vrodionov):
Continued.
h5. RegionGroupingProvider, RegionGroupingStrategy API change
Currently both work on encoded region names, where it is impossible to extract
table names from. So we will need to modify API and add table names to
{code}
public WAL getWAL(final byte[] identifier, byte[] namespace)
{code}
calls.
h5. New RegionGroupingStrategy - BackupRegionGroupingStrategy
* There will be a dedicated group for tables in backups.
* It is a wrapper for default or a user specified RegionGroupingStrategy. It
returns backup group name for a table in a backup list (from hbase:backup),
otherwise it calls default strategy
* Group name for backup tables is well known (hardcoded)
h5. Changes to current implementation
h6. All backup operations only on backup sets
Users won't be able to run backup on table or list of tables - only on backup
sets, but users still be able to restore a single table.
h6. Backup set add / remove
After adding/removing table to/from backup set, system disables/enables table
to read new backup set content (for activating/deactivating backup WAL for a
table)
Upd. Instead of direct disable/enable we can implement procedure, which which
updates WAL instance in HRegion (may be tricky)
> HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits
> from backup tables
>
>
> Key: HBASE-14141
> URL: https://issues.apache.org/jira/browse/HBASE-14141
> Project: HBase
> Issue Type: New Feature
>Affects Versions: 2.0.0
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Labels: backup
> Fix For: 2.0.0
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14141) HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits from backup tables
[ https://issues.apache.org/jira/browse/HBASE-14141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15648837#comment-15648837 ] Vladimir Rodionov commented on HBASE-14141: --- WAL is updated on open region only. That is why we need disable/enable table after table being added to backup set. As for hardcoded group name - there is no need to make it configurable I think. That is internal id used by backup exclusively. > HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits > from backup tables > > > Key: HBASE-14141 > URL: https://issues.apache.org/jira/browse/HBASE-14141 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-14141) HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits from backup tables
[
https://issues.apache.org/jira/browse/HBASE-14141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15648714#comment-15648714
]
Vladimir Rodionov edited comment on HBASE-14141 at 11/8/16 9:24 PM:
Continued.
h5. RegionGroupingProvider, RegionGroupingStrategy API change
Currently both work on encoded region names, where it is impossible to extract
table names from. So we will need to modify API and add table names to
{code}
public WAL getWAL(final byte[] identifier, byte[] namespace)
{code}
calls.
h5. New RegionGroupingStrategy - BackupRegionGroupingStrategy
* There will be a dedicated group for tables in backups.
* It is a wrapper for default or a user specified RegionGroupingStrategy. It
returns backup group name for a table in a backup list (from hbase:backup),
otherwise it calls default strategy
* Group name for backup tables is well known (hardcoded)
h5. Changes to current implementation
h6. All backup operations only on backup sets
Users won't be able to run backup on table or list of tables - only on backup
sets, but users still be able to restore a single table.
h6. Backup set add / remove
After adding/removing table to/from backup set, system disables/enables table
to read new backup set content (for activating/deactivating backup WAL for a
table)
Upd. Instead of direct disable/enable we can implement procedure, which which
updates WAL instance in HRegion (may be tricky)
was (Author: vrodionov):
Continued.
h5. RegionGroupingProvider, RegionGroupingStrategy API change
Currently both work on encoded region names, where it is impossible to extract
table names from. So we will need to modify API and add table names to
{code}
public WAL getWAL(final byte[] identifier, byte[] namespace)
{code}
calls.
h5. New RegionGroupingStrategy - BackupRegionGroupingStrategy
* There will be a dedicated group for tables in backups.
* It is a wrapper for default or a user specified RegionGroupingStrategy. It
returns backup group name for a table in a backup list (from hbase:backup),
otherwise it calls default strategy
* Group name for backup tables is well known (hardcoded)
h5. Changes to current implementation
h6. All backup operations only on backup sets
Users won't be able to run backup on table or list of tables - only on backup
sets, but users still be able to restore a single table.
h6. Backup set add / remove
After adding/removing table to/from backup set, system disables/enables table
to read new backup set content (for activating/deactivating backup WAL for a
table)
> HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits
> from backup tables
>
>
> Key: HBASE-14141
> URL: https://issues.apache.org/jira/browse/HBASE-14141
> Project: HBase
> Issue Type: New Feature
>Affects Versions: 2.0.0
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Labels: backup
> Fix For: 2.0.0
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14955) OOME: cannot create native thread is back
[ https://issues.apache.org/jira/browse/HBASE-14955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14955: -- Attachment: 14955.patch Patch for master. > OOME: cannot create native thread is back > - > > Key: HBASE-14955 > URL: https://issues.apache.org/jira/browse/HBASE-14955 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack >Assignee: Heng Chen > Fix For: 1.3.0 > > Attachments: 14955.patch, HBASE-14955-branch-1.patch > > > This fail is OOME cannot create native thread. Two MR jobs fail: > > org.apache.hadoop.hbase.mapreduce.TestImportTSVWithVisibilityLabels.org.apache.hadoop.hbase.mapreduce.TestImportTSVWithVisibilityLabels46 > ms1 > org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan1.org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan142 > ms1 > Was running 1.3 tests on H0. > Could try and purge resources used by these tests. > Making issue in meantime to keep an eye on it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14955) OOME: cannot create native thread is back
[ https://issues.apache.org/jira/browse/HBASE-14955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14955: -- Status: Patch Available (was: Reopened) > OOME: cannot create native thread is back > - > > Key: HBASE-14955 > URL: https://issues.apache.org/jira/browse/HBASE-14955 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack >Assignee: Heng Chen > Fix For: 1.3.0 > > Attachments: 14955.patch, HBASE-14955-branch-1.patch > > > This fail is OOME cannot create native thread. Two MR jobs fail: > > org.apache.hadoop.hbase.mapreduce.TestImportTSVWithVisibilityLabels.org.apache.hadoop.hbase.mapreduce.TestImportTSVWithVisibilityLabels46 > ms1 > org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan1.org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan142 > ms1 > Was running 1.3 tests on H0. > Could try and purge resources used by these tests. > Making issue in meantime to keep an eye on it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15788) Use Offheap ByteBuffers from BufferPool to read RPC requests.
[
https://issues.apache.org/jira/browse/HBASE-15788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15648788#comment-15648788
]
Hadoop QA commented on HBASE-15788:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 19s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
39s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 14s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
9s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
36s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
35s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
5s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
35s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
34m 15s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 47s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 0s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 96m 25s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 156m 51s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestRegionReplicaFailover |
| Timed out junit tests |
org.apache.hadoop.hbase.master.procedure.TestCreateTableProcedure |
| | org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures
|
| | org.apache.hadoop.hbase.master.snapshot.TestSnapshotFileCache |
| | org.apache.hadoop.hbase.master.procedure.TestMasterProcedureWalLease |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12838024/HBASE-15788_V6.patch |
| JIRA Issue | HBASE-15788 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 3b397aae5e0f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
[jira] [Updated] (HBASE-17050) Upgrade Apache CLI version from 1.2 to 1.3.1
[ https://issues.apache.org/jira/browse/HBASE-17050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-17050: Release Note: Upgrade Apache CLI version from 1.2 to 1.3.1. These are few good/important changes included in this update: - HelpFormatter now prints command-line options in the same order as they have been added. Fixes CLI-212. - Standard help text now shows mandatory arguments also for the first option. Fixes CLI-186. - A new parser is available: DefaultParser. It combines the features of the GnuParser and the PosixParser. It also provides additional features like partial matching for the long options, and long options without separator (i.e like the JVM memory settings: -Xmx512m). This new parser deprecates the previous ones. Fixes CLI-161,CLI-167,CLI-181. For full list of changes: https://commons.apache.org/proper/commons-cli/changes-report.html#a1.3 was:Upgrade Apache CLI version from 1.2 to 1.3.1. > Upgrade Apache CLI version from 1.2 to 1.3.1 > > > Key: HBASE-17050 > URL: https://issues.apache.org/jira/browse/HBASE-17050 > Project: HBase > Issue Type: Sub-task >Reporter: Appy >Assignee: Appy >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17050.master.001.patch > > > http://mail-archives.apache.org/mod_mbox/hbase-dev/201611.mbox/%3CCAGHyZ6KNjTH6qnVN%2B%3Dd_rC%3DtkLEjELUCQf9pmqQ-ixJj%2BfbrOA%40mail.gmail.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16983) TestMultiTableSnapshotInputFormat failing with Unable to create region directory: /tmp/...
[
https://issues.apache.org/jira/browse/HBASE-16983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-16983:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
Re-resolving now. Thanks [~zghaobac] for the fix. Will take up forward-port of
local MR running from HBASE-14955 to master over in the reopened HBASE-14955.
> TestMultiTableSnapshotInputFormat failing with Unable to create region
> directory: /tmp/...
> ---
>
> Key: HBASE-16983
> URL: https://issues.apache.org/jira/browse/HBASE-16983
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: stack
>Assignee: stack
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: 16983.txt, HBASE-16983-ADDENDUM.patch,
> HBASE-16983-ADDENDUM.patch, HBASE-16983-ADDENDUM.patch,
> HBASE-16983-branch-1-ADDENDUM.patch, HBASE-16983-branch-1-ADDENDUM.patch
>
>
> Test is using /tmp. We failed creating dir in /tmp in a few tests from this
> suite just now:
> https://builds.apache.org/job/PreCommit-HBASE-Build/4253/testReport/org.apache.hadoop.hbase.mapred/TestMultiTableSnapshotInputFormat/testScanOBBToOPP/
> {code}
> Caused by: java.io.IOException: Unable to create region directory:
> /tmp/scantest2_snapshot__953e2b2d-22aa-4c6a-a46a-272619f5436e/data/default/scantest2/5629158a49e010e21ac0bd16453b2d8c
> at
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.createRegionOnFileSystem(HRegionFileSystem.java:896)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.createHRegion(HRegion.java:6520)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils.createRegion(ModifyRegionUtils.java:205)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils$1.call(ModifyRegionUtils.java:173)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils$1.call(ModifyRegionUtils.java:170)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> ...
> {code}
> No more detail than this. Let me change it so creates stuff in the test dir
> that it for sure owns/can write to.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14141) HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits from backup tables
[ https://issues.apache.org/jira/browse/HBASE-14141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15648765#comment-15648765 ] Ted Yu commented on HBASE-14141: bq. Group name for backup tables is well known (hardcoded) Can you elaborate on the above hardcode ? bq. system disables/enables table to read new backup set content Why is the above needed ? Is this for the BACKUPREGIONGROUPINGSTRATEGY to reflect the change ? > HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits > from backup tables > > > Key: HBASE-14141 > URL: https://issues.apache.org/jira/browse/HBASE-14141 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16983) TestMultiTableSnapshotInputFormat failing with Unable to create region directory: /tmp/...
[
https://issues.apache.org/jira/browse/HBASE-16983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-16983:
--
Attachment: HBASE-16983-ADDENDUM.patch
Here is the addendum I applied to master branch. Includes forward port of the
getRandomDir from HBaseCommonTestingUtility
> TestMultiTableSnapshotInputFormat failing with Unable to create region
> directory: /tmp/...
> ---
>
> Key: HBASE-16983
> URL: https://issues.apache.org/jira/browse/HBASE-16983
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: stack
>Assignee: stack
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: 16983.txt, HBASE-16983-ADDENDUM.patch,
> HBASE-16983-ADDENDUM.patch, HBASE-16983-ADDENDUM.patch,
> HBASE-16983-branch-1-ADDENDUM.patch, HBASE-16983-branch-1-ADDENDUM.patch
>
>
> Test is using /tmp. We failed creating dir in /tmp in a few tests from this
> suite just now:
> https://builds.apache.org/job/PreCommit-HBASE-Build/4253/testReport/org.apache.hadoop.hbase.mapred/TestMultiTableSnapshotInputFormat/testScanOBBToOPP/
> {code}
> Caused by: java.io.IOException: Unable to create region directory:
> /tmp/scantest2_snapshot__953e2b2d-22aa-4c6a-a46a-272619f5436e/data/default/scantest2/5629158a49e010e21ac0bd16453b2d8c
> at
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.createRegionOnFileSystem(HRegionFileSystem.java:896)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.createHRegion(HRegion.java:6520)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils.createRegion(ModifyRegionUtils.java:205)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils$1.call(ModifyRegionUtils.java:173)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils$1.call(ModifyRegionUtils.java:170)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> ...
> {code}
> No more detail than this. Let me change it so creates stuff in the test dir
> that it for sure owns/can write to.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16955) Fixup precommit protoc check to do new distributed protos and pb 3.1.0 build
[ https://issues.apache.org/jira/browse/HBASE-16955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-16955: -- Attachment: nothing_change2.txt Lets see how we do now. > Fixup precommit protoc check to do new distributed protos and pb 3.1.0 build > > > Key: HBASE-16955 > URL: https://issues.apache.org/jira/browse/HBASE-16955 > Project: HBase > Issue Type: Task > Components: build, Protobufs >Reporter: stack >Assignee: stack > Attachments: 16955.patch, nothing_change.txt, nothing_change2 > (1).txt, nothing_change2 (2).txt, nothing_change2.txt, nothing_change2.txt, > nothing_change2.txt, nothing_change2.txt, nothing_change2.txt, > nothing_change2.txt > > > HBASE-15638 Shade protobuf and a follow-ons changed how we do protobufs. > One, protobufs are in the module they pertain to so distributed throughout > the modules and secondly, we do 2.5.0 pb for externally consumed protobuf -- > e.g. Coprocessor Endpoints -- but internally we use protobuf 3.1.0. > A precommit check looks to see if any proto changes break protoc compile. > This task is about updating the precommit to accommodate the changes brought > about by HBASE-15638. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16983) TestMultiTableSnapshotInputFormat failing with Unable to create region directory: /tmp/...
[
https://issues.apache.org/jira/browse/HBASE-16983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15648759#comment-15648759
]
stack commented on HBASE-16983:
---
Whoops. Reverted from master because cherry-pick broke master compile. No
getRandomDir in master. Let me add.
> TestMultiTableSnapshotInputFormat failing with Unable to create region
> directory: /tmp/...
> ---
>
> Key: HBASE-16983
> URL: https://issues.apache.org/jira/browse/HBASE-16983
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: stack
>Assignee: stack
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: 16983.txt, HBASE-16983-ADDENDUM.patch,
> HBASE-16983-ADDENDUM.patch, HBASE-16983-branch-1-ADDENDUM.patch,
> HBASE-16983-branch-1-ADDENDUM.patch
>
>
> Test is using /tmp. We failed creating dir in /tmp in a few tests from this
> suite just now:
> https://builds.apache.org/job/PreCommit-HBASE-Build/4253/testReport/org.apache.hadoop.hbase.mapred/TestMultiTableSnapshotInputFormat/testScanOBBToOPP/
> {code}
> Caused by: java.io.IOException: Unable to create region directory:
> /tmp/scantest2_snapshot__953e2b2d-22aa-4c6a-a46a-272619f5436e/data/default/scantest2/5629158a49e010e21ac0bd16453b2d8c
> at
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.createRegionOnFileSystem(HRegionFileSystem.java:896)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.createHRegion(HRegion.java:6520)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils.createRegion(ModifyRegionUtils.java:205)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils$1.call(ModifyRegionUtils.java:173)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils$1.call(ModifyRegionUtils.java:170)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> ...
> {code}
> No more detail than this. Let me change it so creates stuff in the test dir
> that it for sure owns/can write to.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16955) Fixup precommit protoc check to do new distributed protos and pb 3.1.0 build
[ https://issues.apache.org/jira/browse/HBASE-16955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-16955: -- Attachment: 16955.patch Interesting one. So, failure was in the step before only it wasn't a failure. At the protoc step, we were doing compile goal. hbase-protocol-shaded needs the install goal to run (shading is hooked up to the install step). Without it, the protoc generation was left in a half-way state. Committed this patch. Let me try my nothing change again now. > Fixup precommit protoc check to do new distributed protos and pb 3.1.0 build > > > Key: HBASE-16955 > URL: https://issues.apache.org/jira/browse/HBASE-16955 > Project: HBase > Issue Type: Task > Components: build, Protobufs >Reporter: stack >Assignee: stack > Attachments: 16955.patch, nothing_change.txt, nothing_change2 > (1).txt, nothing_change2 (2).txt, nothing_change2.txt, nothing_change2.txt, > nothing_change2.txt, nothing_change2.txt, nothing_change2.txt, > nothing_change2.txt > > > HBASE-15638 Shade protobuf and a follow-ons changed how we do protobufs. > One, protobufs are in the module they pertain to so distributed throughout > the modules and secondly, we do 2.5.0 pb for externally consumed protobuf -- > e.g. Coprocessor Endpoints -- but internally we use protobuf 3.1.0. > A precommit check looks to see if any proto changes break protoc compile. > This task is about updating the precommit to accommodate the changes brought > about by HBASE-15638. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14141) HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits from backup tables
[ https://issues.apache.org/jira/browse/HBASE-14141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15648716#comment-15648716 ] Vladimir Rodionov commented on HBASE-14141: --- [~devaraj], [~tedyu], what do you think? > HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits > from backup tables > > > Key: HBASE-14141 > URL: https://issues.apache.org/jira/browse/HBASE-14141 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14141) HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits from backup tables
[
https://issues.apache.org/jira/browse/HBASE-14141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15648714#comment-15648714
]
Vladimir Rodionov commented on HBASE-14141:
---
Continued.
h5. RegionGroupingProvider, RegionGroupingStrategy API change
Currently both work on encoded region names, where it is impossible to extract
table names from. So we will need to modify API and add table names to
{code}
public WAL getWAL(final byte[] identifier, byte[] namespace)
{code}
calls.
h5. New RegionGroupingStrategy - BackupRegionGroupingStrategy
* There will be a dedicated group for tables in backups.
* It is a wrapper for default or a user specified RegionGroupingStrategy. It
returns backup group name for a table in a backup list (from hbase:backup),
otherwise it calls default strategy
* Group name for backup tables is well known (hardcoded)
h5. Changes to current implementation
h6. All backup operations only on backup sets
Users won't be able to run backup on table or list of tables - only on backup
sets, but users still be able to restore a single table.
h6. Backup set add / remove
After adding/removing table to/from backup set, system disables/enables table
to read new backup set content (for activating/deactivating backup WAL for a
table)
> HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits
> from backup tables
>
>
> Key: HBASE-14141
> URL: https://issues.apache.org/jira/browse/HBASE-14141
> Project: HBase
> Issue Type: New Feature
>Affects Versions: 2.0.0
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Labels: backup
> Fix For: 2.0.0
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16982) Better integrate Apache CLI in AbstractHBaseTool
[ https://issues.apache.org/jira/browse/HBASE-16982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-16982: - Resolution: Fixed Status: Resolved (was: Patch Available) > Better integrate Apache CLI in AbstractHBaseTool > > > Key: HBASE-16982 > URL: https://issues.apache.org/jira/browse/HBASE-16982 > Project: HBase > Issue Type: Sub-task >Reporter: Appy >Assignee: Appy > Fix For: 2.0.0 > > Attachments: HBASE-16982.master.001.patch, > HBASE-16982.master.002.patch, HBASE-16982.master.003.patch, > HBASE-16982.master.004.patch, HBASE-16982.master.005.patch > > > Problem > 1. Inconsistencies in our user facing tools. > - options: > start with single/double dash > words separated by dash or underscore or just joined together > someplace use '=' to separate value, other use space (ExportSnapshot vs > HashTable) > - Description > Manually formatting options and their descriptions in printUsage() > Inconsistant formatting, sometimes even weird. > Incomplete. Sometimes people forget to add new option to description > 2. Manual parsing of options (those huge if-else loops iterating over args) > Solution > Use Apache CLI > - It has various validations for option names which'll fix first set of > issues. > - using AbstractHBaseTool's print usage function will ensure consistent > formatting (although we loose the power to order the options) > - If we enforce the method of defining options as in patch, it's highly > unlikely to forget adding description. > - CLI parses the options for us. > Using Apache CLI when writing new tools is straight forward, but it's not > easy when porting exiting tools since some option names are not valid as per > CLI's validation. > New method, processOldArgs(), will allow to port these tools in a backward > compatible manner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16982) Better integrate Apache CLI in AbstractHBaseTool
[ https://issues.apache.org/jira/browse/HBASE-16982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-16982: - Fix Version/s: 2.0.0 > Better integrate Apache CLI in AbstractHBaseTool > > > Key: HBASE-16982 > URL: https://issues.apache.org/jira/browse/HBASE-16982 > Project: HBase > Issue Type: Sub-task >Reporter: Appy >Assignee: Appy > Fix For: 2.0.0 > > Attachments: HBASE-16982.master.001.patch, > HBASE-16982.master.002.patch, HBASE-16982.master.003.patch, > HBASE-16982.master.004.patch, HBASE-16982.master.005.patch > > > Problem > 1. Inconsistencies in our user facing tools. > - options: > start with single/double dash > words separated by dash or underscore or just joined together > someplace use '=' to separate value, other use space (ExportSnapshot vs > HashTable) > - Description > Manually formatting options and their descriptions in printUsage() > Inconsistant formatting, sometimes even weird. > Incomplete. Sometimes people forget to add new option to description > 2. Manual parsing of options (those huge if-else loops iterating over args) > Solution > Use Apache CLI > - It has various validations for option names which'll fix first set of > issues. > - using AbstractHBaseTool's print usage function will ensure consistent > formatting (although we loose the power to order the options) > - If we enforce the method of defining options as in patch, it's highly > unlikely to forget adding description. > - CLI parses the options for us. > Using Apache CLI when writing new tools is straight forward, but it's not > easy when porting exiting tools since some option names are not valid as per > CLI's validation. > New method, processOldArgs(), will allow to port these tools in a backward > compatible manner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17050) Upgrade Apache CLI version from 1.2 to 1.3.1
[ https://issues.apache.org/jira/browse/HBASE-17050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-17050: - Fix Version/s: 2.0.0 > Upgrade Apache CLI version from 1.2 to 1.3.1 > > > Key: HBASE-17050 > URL: https://issues.apache.org/jira/browse/HBASE-17050 > Project: HBase > Issue Type: Sub-task >Reporter: Appy >Assignee: Appy >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17050.master.001.patch > > > http://mail-archives.apache.org/mod_mbox/hbase-dev/201611.mbox/%3CCAGHyZ6KNjTH6qnVN%2B%3Dd_rC%3DtkLEjELUCQf9pmqQ-ixJj%2BfbrOA%40mail.gmail.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17050) Upgrade Apache CLI version from 1.2 to 1.3.1
[ https://issues.apache.org/jira/browse/HBASE-17050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-17050: - Resolution: Fixed Status: Resolved (was: Patch Available) > Upgrade Apache CLI version from 1.2 to 1.3.1 > > > Key: HBASE-17050 > URL: https://issues.apache.org/jira/browse/HBASE-17050 > Project: HBase > Issue Type: Sub-task >Reporter: Appy >Assignee: Appy >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17050.master.001.patch > > > http://mail-archives.apache.org/mod_mbox/hbase-dev/201611.mbox/%3CCAGHyZ6KNjTH6qnVN%2B%3Dd_rC%3DtkLEjELUCQf9pmqQ-ixJj%2BfbrOA%40mail.gmail.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17050) Upgrade Apache CLI version from 1.2 to 1.3.1
[ https://issues.apache.org/jira/browse/HBASE-17050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-17050: - Release Note: Upgrade Apache CLI version from 1.2 to 1.3.1. > Upgrade Apache CLI version from 1.2 to 1.3.1 > > > Key: HBASE-17050 > URL: https://issues.apache.org/jira/browse/HBASE-17050 > Project: HBase > Issue Type: Sub-task >Reporter: Appy >Assignee: Appy >Priority: Minor > Attachments: HBASE-17050.master.001.patch > > > http://mail-archives.apache.org/mod_mbox/hbase-dev/201611.mbox/%3CCAGHyZ6KNjTH6qnVN%2B%3Dd_rC%3DtkLEjELUCQf9pmqQ-ixJj%2BfbrOA%40mail.gmail.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17050) Upgrade Apache CLI version from 1.2 to 1.3.1
[ https://issues.apache.org/jira/browse/HBASE-17050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15648656#comment-15648656 ] Sean Busbey commented on HBASE-17050: - +1 please draft a release note letting downstream know about the upgrade. > Upgrade Apache CLI version from 1.2 to 1.3.1 > > > Key: HBASE-17050 > URL: https://issues.apache.org/jira/browse/HBASE-17050 > Project: HBase > Issue Type: Sub-task >Reporter: Appy >Assignee: Appy >Priority: Minor > Attachments: HBASE-17050.master.001.patch > > > http://mail-archives.apache.org/mod_mbox/hbase-dev/201611.mbox/%3CCAGHyZ6KNjTH6qnVN%2B%3Dd_rC%3DtkLEjELUCQf9pmqQ-ixJj%2BfbrOA%40mail.gmail.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HBASE-14955) OOME: cannot create native thread is back
[ https://issues.apache.org/jira/browse/HBASE-14955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack reopened HBASE-14955: --- Reopening to try against master branch... try and keep the two branches the same (as suggested by [~Apache9]) > OOME: cannot create native thread is back > - > > Key: HBASE-14955 > URL: https://issues.apache.org/jira/browse/HBASE-14955 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack >Assignee: Heng Chen > Fix For: 1.3.0 > > Attachments: HBASE-14955-branch-1.patch > > > This fail is OOME cannot create native thread. Two MR jobs fail: > > org.apache.hadoop.hbase.mapreduce.TestImportTSVWithVisibilityLabels.org.apache.hadoop.hbase.mapreduce.TestImportTSVWithVisibilityLabels46 > ms1 > org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan1.org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan142 > ms1 > Was running 1.3 tests on H0. > Could try and purge resources used by these tests. > Making issue in meantime to keep an eye on it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16983) TestMultiTableSnapshotInputFormat failing with Unable to create region directory: /tmp/...
[
https://issues.apache.org/jira/browse/HBASE-16983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15648646#comment-15648646
]
stack commented on HBASE-16983:
---
Should be the same as you suggest. Let me try it. Reopening HBASE-14955. Thanks
[~Apache9]
> TestMultiTableSnapshotInputFormat failing with Unable to create region
> directory: /tmp/...
> ---
>
> Key: HBASE-16983
> URL: https://issues.apache.org/jira/browse/HBASE-16983
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: stack
>Assignee: stack
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: 16983.txt, HBASE-16983-ADDENDUM.patch,
> HBASE-16983-ADDENDUM.patch, HBASE-16983-branch-1-ADDENDUM.patch,
> HBASE-16983-branch-1-ADDENDUM.patch
>
>
> Test is using /tmp. We failed creating dir in /tmp in a few tests from this
> suite just now:
> https://builds.apache.org/job/PreCommit-HBASE-Build/4253/testReport/org.apache.hadoop.hbase.mapred/TestMultiTableSnapshotInputFormat/testScanOBBToOPP/
> {code}
> Caused by: java.io.IOException: Unable to create region directory:
> /tmp/scantest2_snapshot__953e2b2d-22aa-4c6a-a46a-272619f5436e/data/default/scantest2/5629158a49e010e21ac0bd16453b2d8c
> at
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.createRegionOnFileSystem(HRegionFileSystem.java:896)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.createHRegion(HRegion.java:6520)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils.createRegion(ModifyRegionUtils.java:205)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils$1.call(ModifyRegionUtils.java:173)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils$1.call(ModifyRegionUtils.java:170)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> ...
> {code}
> No more detail than this. Let me change it so creates stuff in the test dir
> that it for sure owns/can write to.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17010) Serial replication should handle daughter regions being assigned to another RS
[ https://issues.apache.org/jira/browse/HBASE-17010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15648645#comment-15648645 ] Hudson commented on HBASE-17010: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1939 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1939/]) HBASE-17010 Serial replication should handle daughter regions being (tedyu: rev 76814e84512ec3cf34e4844b07dc5accb8e92f11) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/ReplicationMetaCleaner.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestSerialReplication.java > Serial replication should handle daughter regions being assigned to another RS > -- > > Key: HBASE-17010 > URL: https://issues.apache.org/jira/browse/HBASE-17010 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Phil Yang > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17010-branch-1-v1.patch, HBASE-17010-v1.patch, > HBASE-17010-v2.patch, HBASE-17010-v3.patch > > > testRegionSplit and testRegionMerge were temporarily disabled by HBASE-16975. > HBASE-9465 has an assumption that when we split a region, two daughter > regions are in same RS with the parent region. But after HBASE-14551 went in, > daughter regions may be assigned to other RS directly. > This issue is to handle the new behavior and reenable the tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16983) TestMultiTableSnapshotInputFormat failing with Unable to create region directory: /tmp/...
[
https://issues.apache.org/jira/browse/HBASE-16983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15648641#comment-15648641
]
stack commented on HBASE-16983:
---
I pushed addendum to master and to branch-1. Thanks for the patch [~zghaobac]
and sorry for my neglect (thanks to [~carp84] for the kick).
> TestMultiTableSnapshotInputFormat failing with Unable to create region
> directory: /tmp/...
> ---
>
> Key: HBASE-16983
> URL: https://issues.apache.org/jira/browse/HBASE-16983
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: stack
>Assignee: stack
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: 16983.txt, HBASE-16983-ADDENDUM.patch,
> HBASE-16983-ADDENDUM.patch, HBASE-16983-branch-1-ADDENDUM.patch,
> HBASE-16983-branch-1-ADDENDUM.patch
>
>
> Test is using /tmp. We failed creating dir in /tmp in a few tests from this
> suite just now:
> https://builds.apache.org/job/PreCommit-HBASE-Build/4253/testReport/org.apache.hadoop.hbase.mapred/TestMultiTableSnapshotInputFormat/testScanOBBToOPP/
> {code}
> Caused by: java.io.IOException: Unable to create region directory:
> /tmp/scantest2_snapshot__953e2b2d-22aa-4c6a-a46a-272619f5436e/data/default/scantest2/5629158a49e010e21ac0bd16453b2d8c
> at
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.createRegionOnFileSystem(HRegionFileSystem.java:896)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.createHRegion(HRegion.java:6520)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils.createRegion(ModifyRegionUtils.java:205)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils$1.call(ModifyRegionUtils.java:173)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils$1.call(ModifyRegionUtils.java:170)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> ...
> {code}
> No more detail than this. Let me change it so creates stuff in the test dir
> that it for sure owns/can write to.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17047) Add an API to log the state of HBase connection cache
[
https://issues.apache.org/jira/browse/HBASE-17047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15648642#comment-15648642
]
Hadoop QA commented on HBASE-17047:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
6s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 6s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} scaladoc {color} | {color:green} 0m
27s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 4s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} scalac {color} | {color:green} 1m 4s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
33m 58s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:red}-1{color} | {color:red} scaladoc {color} | {color:red} 0m 29s
{color} | {color:red} hbase-spark generated 1 new + 0 unchanged - 0 fixed = 1
total (was 0) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 8s
{color} | {color:green} hbase-spark in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
9s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 43m 6s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12837924/HBASE-17047_v1.patch |
| JIRA Issue | HBASE-17047 |
| Optional Tests | asflicense scalac scaladoc unit compile |
| uname | Linux c7bbf212b263 3.13.0-98-generic #145-Ubuntu SMP Sat Oct 8
20:13:07 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / b143f6e |
| scaladoc |
https://builds.apache.org/job/PreCommit-HBASE-Build/4387/artifact/patchprocess/diff-scaladoc-scaladoc-hbase-spark.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4387/testReport/ |
| modules | C: hbase-spark U: hbase-spark |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4387/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Add an API to log the state of HBase connection cache
> -
>
> Key: HBASE-17047
> URL: https://issues.apache.org/jira/browse/HBASE-17047
> Project: HBase
> Issue Type: Improvement
> Components: spark
>Reporter: Weiqing Yang
>Assignee: Weiqing Yang
>Priority: Minor
> Attachments: HBASE-17047_v1.patch
>
>
> This patch will add a function "getStat" for the user to get the state of the
> HBase connection cache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16983) TestMultiTableSnapshotInputFormat failing with Unable to create region directory: /tmp/...
[
https://issues.apache.org/jira/browse/HBASE-16983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15648634#comment-15648634
]
stack commented on HBASE-16983:
---
Sorry for leaving this aside [~zghaobac]. Just tried it and indeed my commit
broke this test and your addendum does fix. Let me commit.
> TestMultiTableSnapshotInputFormat failing with Unable to create region
> directory: /tmp/...
> ---
>
> Key: HBASE-16983
> URL: https://issues.apache.org/jira/browse/HBASE-16983
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: stack
>Assignee: stack
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: 16983.txt, HBASE-16983-ADDENDUM.patch,
> HBASE-16983-ADDENDUM.patch, HBASE-16983-branch-1-ADDENDUM.patch,
> HBASE-16983-branch-1-ADDENDUM.patch
>
>
> Test is using /tmp. We failed creating dir in /tmp in a few tests from this
> suite just now:
> https://builds.apache.org/job/PreCommit-HBASE-Build/4253/testReport/org.apache.hadoop.hbase.mapred/TestMultiTableSnapshotInputFormat/testScanOBBToOPP/
> {code}
> Caused by: java.io.IOException: Unable to create region directory:
> /tmp/scantest2_snapshot__953e2b2d-22aa-4c6a-a46a-272619f5436e/data/default/scantest2/5629158a49e010e21ac0bd16453b2d8c
> at
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.createRegionOnFileSystem(HRegionFileSystem.java:896)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.createHRegion(HRegion.java:6520)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils.createRegion(ModifyRegionUtils.java:205)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils$1.call(ModifyRegionUtils.java:173)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils$1.call(ModifyRegionUtils.java:170)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> ...
> {code}
> No more detail than this. Let me change it so creates stuff in the test dir
> that it for sure owns/can write to.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16962) Add readPoint to preCompactScannerOpen() API
[ https://issues.apache.org/jira/browse/HBASE-16962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15648580#comment-15648580 ] stack commented on HBASE-16962: --- That'd be a big, though helpful, change. Should probably be coordinated w/ redo of CP hook points (Our [~mbertozzi] has a good argument on CP points need replumbing...) > Add readPoint to preCompactScannerOpen() API > > > Key: HBASE-16962 > URL: https://issues.apache.org/jira/browse/HBASE-16962 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Attachments: HBASE-16962.master.001.patch, HBASE-16962.rough.patch > > > Similar to HBASE-15759, I would like to add readPoint to the > preCompactScannerOpen() API. > I have a CP where I create a StoreScanner() as part of the > preCompactScannerOpen() API. I need the readpoint which was obtained in > Compactor.compact() method to be consistent. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17047) Add an API to log the state of HBase connection cache
[ https://issues.apache.org/jira/browse/HBASE-17047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15648547#comment-15648547 ] stack commented on HBASE-17047: --- An API to write a LOG message? Whats it for? Why not log add current count to a TRACE level message? Why not expose a metric w/ count? I see increment of refcount but not the decrement. > Add an API to log the state of HBase connection cache > - > > Key: HBASE-17047 > URL: https://issues.apache.org/jira/browse/HBASE-17047 > Project: HBase > Issue Type: Improvement > Components: spark >Reporter: Weiqing Yang >Assignee: Weiqing Yang >Priority: Minor > Attachments: HBASE-17047_v1.patch > > > This patch will add a function "getStat" for the user to get the state of the > HBase connection cache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17047) Add an API to log the state of HBase connection cache
[ https://issues.apache.org/jira/browse/HBASE-17047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-17047: --- Status: Patch Available (was: Open) > Add an API to log the state of HBase connection cache > - > > Key: HBASE-17047 > URL: https://issues.apache.org/jira/browse/HBASE-17047 > Project: HBase > Issue Type: Improvement > Components: spark >Reporter: Weiqing Yang >Assignee: Weiqing Yang >Priority: Minor > Attachments: HBASE-17047_v1.patch > > > This patch will add a function "getStat" for the user to get the state of the > HBase connection cache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
