[jira] [Commented] (HBASE-6852) SchemaMetrics.updateOnCacheHit costs too much while full scanning a table with all of its fields
[ https://issues.apache.org/jira/browse/HBASE-6852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484742#comment-13484742 ] Hudson commented on HBASE-6852: --- Integrated in HBase-0.94 #556 (See [https://builds.apache.org/job/HBase-0.94/556/]) HBASE-6852 SchemaMetrics.updateOnCacheHit costs too much while full scanning a table with all of its fields (Cheng Hao and LarsH) (Revision 1402392) Result = FAILURE larsh : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/metrics/TestSchemaMetrics.java > SchemaMetrics.updateOnCacheHit costs too much while full scanning a table > with all of its fields > > > Key: HBASE-6852 > URL: https://issues.apache.org/jira/browse/HBASE-6852 > Project: HBase > Issue Type: Improvement > Components: metrics >Affects Versions: 0.94.0 >Reporter: Cheng Hao >Assignee: Cheng Hao >Priority: Minor > Labels: performance > Fix For: 0.94.3 > > Attachments: 6852-0.94.txt, metrics_hotspots.png, > onhitcache-trunk.patch > > > The SchemaMetrics.updateOnCacheHit costs too much while I am doing the full > table scanning. > Here is the top 5 hotspots within regionserver while full scanning a table: > (Sorry for the less-well-format) > CPU: Intel Westmere microarchitecture, speed 2.262e+06 MHz (estimated) > Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit > mask of 0x00 (No unit mask) count 500 > samples %image name symbol name > --- > 9844713.4324 14033.jo void > org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics.updateOnCacheHit(org.apache.hadoop.hbase.io.hfile.BlockType$BlockCategory, > boolean) > 98447100.000 14033.jo void > org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics.updateOnCacheHit(org.apache.hadoop.hbase.io.hfile.BlockType$BlockCategory, > boolean) [self] > --- > 45814 6.2510 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(byte[], int, int, > byte[], int, int) > 45814100.000 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(byte[], int, int, > byte[], int, int) [self] > --- > 43523 5.9384 14033.jo boolean > org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(org.apache.hadoop.hbase.KeyValue) > 43523100.000 14033.jo boolean > org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(org.apache.hadoop.hbase.KeyValue) > [self] > --- > 42548 5.8054 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(byte[], int, int, > byte[], int, int) > 42548100.000 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(byte[], int, int, > byte[], int, int) [self] > --- > 40572 5.5358 14033.jo int > org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.binarySearchNonRootIndex(byte[], > int, int, java.nio.ByteBuffer, org.apache.hadoop.io.RawComparator)~1 > 40572100.000 14033.jo int > org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.binarySearchNonRootIndex(byte[], > int, int, java.nio.ByteBuffer, org.apache.hadoop.io.RawComparator)~1 [self] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for block cache lock
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484739#comment-13484739 ] stack commented on HBASE-5898: -- [~ram_krish] How you mean Ram? It was stuck where in particular? Was it a bunch of threads getting same block? What did the thread dump look like? There is some issue in here around the wait/notify it seems as implemented. The double-checked is probably better anyways but could the issue come back just less frequently after this patch goes in? > Consider double-checked locking for block cache lock > > > Key: HBASE-5898 > URL: https://issues.apache.org/jira/browse/HBASE-5898 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 0.94.1 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Critical > Fix For: 0.94.3, 0.96.0 > > Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, > HBASE-5898-1.patch, hbase-5898.txt > > > Running a workload with a high query rate against a dataset that fits in > cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by > HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a > lot of CPU doing lock management here. I wrote a quick patch to switch to a > double-checked locking and it improved throughput substantially for this > workload. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6410) Move RegionServer Metrics to metrics2
[ https://issues.apache.org/jira/browse/HBASE-6410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484738#comment-13484738 ] stack commented on HBASE-6410: -- I started some review here https://reviews.apache.org/r/7747/ Elliott > Move RegionServer Metrics to metrics2 > - > > Key: HBASE-6410 > URL: https://issues.apache.org/jira/browse/HBASE-6410 > Project: HBase > Issue Type: Sub-task > Components: metrics >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Blocker > Attachments: HBASE-6410-1.patch, HBASE-6410-2.patch, > HBASE-6410-3.patch, HBASE-6410-4.patch, HBASE-6410-5.patch, HBASE-6410.patch > > > Move RegionServer Metrics to metrics2 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6651) Thread safety of HTablePool is doubtful
[
https://issues.apache.org/jira/browse/HBASE-6651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484735#comment-13484735
]
Hadoop QA commented on HBASE-6651:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12550927/HBASE-6651-V5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 11 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
95 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestFromClientSideWithCoprocessor
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3154//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3154//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3154//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3154//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3154//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3154//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3154//console
This message is automatically generated.
> Thread safety of HTablePool is doubtful
> ---
>
> Key: HBASE-6651
> URL: https://issues.apache.org/jira/browse/HBASE-6651
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 0.94.1
>Reporter: Hiroshi Ikeda
>Priority: Minor
> Attachments: HBASE-6651.patch, HBASE-6651-V2.patch,
> HBASE-6651-V3.patch, HBASE-6651-V4.patch, HBASE-6651-V5.patch, sample.zip,
> sample.zip, sharedmap_for_hbaseclient.zip
>
>
> There are some operations in HTablePool to access to PoolMap in multiple
> times without any explict synchronization.
> For example HTablePool.closeTablePool() calles PoolMap.values(), and calles
> PoolMap.remove(). If other threads add new instances to the pool in the
> middle of the calls, the new added instances might be dropped.
> (HTablePool.closeTablePool() also has another problem that calling it by
> multple threads causes accessing HTable by multiple threads.)
> Moreover, PoolMap is not thread safe for the same reason.
> For example PoolMap.put() calles ConcurrentMap.get() and calles
> ConcurrentMap.put(). If other threads add a new instance to the concurent map
> in the middle of the calls, the new instance might be dropped.
> And also implementations of Pool have the same problems.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for block cache lock
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484734#comment-13484734 ] ramkrishna.s.vasudevan commented on HBASE-5898: --- We have recently hit this issue.. My major concern here is there is contention only happening right? But here in our case the scan itself did not happen for almost 10 mins? The thread dump clearly says what was found over in this JIRA. > Consider double-checked locking for block cache lock > > > Key: HBASE-5898 > URL: https://issues.apache.org/jira/browse/HBASE-5898 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 0.94.1 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Critical > Fix For: 0.94.3, 0.96.0 > > Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, > HBASE-5898-1.patch, hbase-5898.txt > > > Running a workload with a high query rate against a dataset that fits in > cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by > HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a > lot of CPU doing lock management here. I wrote a quick patch to switch to a > double-checked locking and it improved throughput substantially for this > workload. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7008) Set scanner caching to a better default
[ https://issues.apache.org/jira/browse/HBASE-7008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484732#comment-13484732 ] stack commented on HBASE-7008: -- I could go for #1. I just added to the doc. a link your fancy graphs above into the nagle's section so we have #3 now I'd argue (look in src -- haven't pushed the site yet) > Set scanner caching to a better default > --- > > Key: HBASE-7008 > URL: https://issues.apache.org/jira/browse/HBASE-7008 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: liang xie >Assignee: liang xie > Fix For: 0.94.3, 0.96.0 > > Attachments: 7008-0.94.txt, 7008-0.94-v2.txt, 7008-v3.txt, > 7008-v4.txt, HBASE-7008.patch, HBASE-7008-v2.patch > > > per > http://search-hadoop.com/m/qaRu9iM2f02/Set+scanner+caching+to+a+better+default%253F&subj=Set+scanner+caching+to+a+better+default+ > let's set to 100 by default -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-6852) SchemaMetrics.updateOnCacheHit costs too much while full scanning a table with all of its fields
[ https://issues.apache.org/jira/browse/HBASE-6852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl resolved HBASE-6852. -- Resolution: Fixed Hadoop Flags: Reviewed Committed to 0.94. Thanks for the profiling work and the patch, Cheng! > SchemaMetrics.updateOnCacheHit costs too much while full scanning a table > with all of its fields > > > Key: HBASE-6852 > URL: https://issues.apache.org/jira/browse/HBASE-6852 > Project: HBase > Issue Type: Improvement > Components: metrics >Affects Versions: 0.94.0 >Reporter: Cheng Hao >Assignee: Cheng Hao >Priority: Minor > Labels: performance > Fix For: 0.94.3 > > Attachments: 6852-0.94.txt, metrics_hotspots.png, > onhitcache-trunk.patch > > > The SchemaMetrics.updateOnCacheHit costs too much while I am doing the full > table scanning. > Here is the top 5 hotspots within regionserver while full scanning a table: > (Sorry for the less-well-format) > CPU: Intel Westmere microarchitecture, speed 2.262e+06 MHz (estimated) > Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit > mask of 0x00 (No unit mask) count 500 > samples %image name symbol name > --- > 9844713.4324 14033.jo void > org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics.updateOnCacheHit(org.apache.hadoop.hbase.io.hfile.BlockType$BlockCategory, > boolean) > 98447100.000 14033.jo void > org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics.updateOnCacheHit(org.apache.hadoop.hbase.io.hfile.BlockType$BlockCategory, > boolean) [self] > --- > 45814 6.2510 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(byte[], int, int, > byte[], int, int) > 45814100.000 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(byte[], int, int, > byte[], int, int) [self] > --- > 43523 5.9384 14033.jo boolean > org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(org.apache.hadoop.hbase.KeyValue) > 43523100.000 14033.jo boolean > org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(org.apache.hadoop.hbase.KeyValue) > [self] > --- > 42548 5.8054 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(byte[], int, int, > byte[], int, int) > 42548100.000 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(byte[], int, int, > byte[], int, int) [self] > --- > 40572 5.5358 14033.jo int > org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.binarySearchNonRootIndex(byte[], > int, int, java.nio.ByteBuffer, org.apache.hadoop.io.RawComparator)~1 > 40572100.000 14033.jo int > org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.binarySearchNonRootIndex(byte[], > int, int, java.nio.ByteBuffer, org.apache.hadoop.io.RawComparator)~1 [self] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-6852) SchemaMetrics.updateOnCacheHit costs too much while full scanning a table with all of its fields
[ https://issues.apache.org/jira/browse/HBASE-6852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl reassigned HBASE-6852: Assignee: Cheng Hao Added you as a contributor, Cheng, and assigned to you. Going to commit this. Thanks for your perseverance Cheng :) > SchemaMetrics.updateOnCacheHit costs too much while full scanning a table > with all of its fields > > > Key: HBASE-6852 > URL: https://issues.apache.org/jira/browse/HBASE-6852 > Project: HBase > Issue Type: Improvement > Components: metrics >Affects Versions: 0.94.0 >Reporter: Cheng Hao >Assignee: Cheng Hao >Priority: Minor > Labels: performance > Fix For: 0.94.3 > > Attachments: 6852-0.94.txt, metrics_hotspots.png, > onhitcache-trunk.patch > > > The SchemaMetrics.updateOnCacheHit costs too much while I am doing the full > table scanning. > Here is the top 5 hotspots within regionserver while full scanning a table: > (Sorry for the less-well-format) > CPU: Intel Westmere microarchitecture, speed 2.262e+06 MHz (estimated) > Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit > mask of 0x00 (No unit mask) count 500 > samples %image name symbol name > --- > 9844713.4324 14033.jo void > org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics.updateOnCacheHit(org.apache.hadoop.hbase.io.hfile.BlockType$BlockCategory, > boolean) > 98447100.000 14033.jo void > org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics.updateOnCacheHit(org.apache.hadoop.hbase.io.hfile.BlockType$BlockCategory, > boolean) [self] > --- > 45814 6.2510 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(byte[], int, int, > byte[], int, int) > 45814100.000 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(byte[], int, int, > byte[], int, int) [self] > --- > 43523 5.9384 14033.jo boolean > org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(org.apache.hadoop.hbase.KeyValue) > 43523100.000 14033.jo boolean > org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(org.apache.hadoop.hbase.KeyValue) > [self] > --- > 42548 5.8054 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(byte[], int, int, > byte[], int, int) > 42548100.000 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(byte[], int, int, > byte[], int, int) [self] > --- > 40572 5.5358 14033.jo int > org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.binarySearchNonRootIndex(byte[], > int, int, java.nio.ByteBuffer, org.apache.hadoop.io.RawComparator)~1 > 40572100.000 14033.jo int > org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.binarySearchNonRootIndex(byte[], > int, int, java.nio.ByteBuffer, org.apache.hadoop.io.RawComparator)~1 [self] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5943) Set tcpnodelay as default
[
https://issues.apache.org/jira/browse/HBASE-5943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484719#comment-13484719
]
Lars Hofhansl commented on HBASE-5943:
--
I did some of that in HBASE-7008 :)
> Set tcpnodelay as default
> -
>
> Key: HBASE-5943
> URL: https://issues.apache.org/jira/browse/HBASE-5943
> Project: HBase
> Issue Type: Improvement
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Labels: noob
> Fix For: 0.96.0
>
> Attachments: nodelay.txt
>
>
> From IRC:
> {code}
> 14:46 < tlipcon> i turned off the FirstKeyOnlyFilter on the shell count
> command
> 14:46 < tlipcon> thgouhtput dropped like carzy... looked at strace, and saw
> 40ms delays
> 14:46 < tlipcon> turned on ipc.server.nodelay or whaever and it got fast again
> 14:46 < tlipcon> we should really ship that by default on
> 14:46 < tlipcon> there are probably people getting screwed by it
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-2611) Handle RS that fails while processing the failure of another one
[ https://issues.apache.org/jira/browse/HBASE-2611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484717#comment-13484717 ] Lars Hofhansl edited comment on HBASE-2611 at 10/26/12 5:08 AM: [~ctrezzo] Wanna work on this? Have a look at the discussion in HBASE-6695. If we accept a bit of herding initially we can fix this with a simple change (I think)... And ZK herding is still better than data loss. was (Author: lhofhansl): [~ctrezzo] Wanna work on this. Have a look at the discussion in HBASE-6695. If we accept a bit of herding initially we can fix this with a simple change (I think)... And ZK herding is still better than data loss. > Handle RS that fails while processing the failure of another one > > > Key: HBASE-2611 > URL: https://issues.apache.org/jira/browse/HBASE-2611 > Project: HBase > Issue Type: Sub-task > Components: Replication >Reporter: Jean-Daniel Cryans >Assignee: Chris Trezzo > Fix For: 0.94.3 > > Attachments: HBase-2611-upstream-v1.patch > > > HBASE-2223 doesn't manage region servers that fail while doing the transfer > of HLogs queues from other region servers that failed. Devise a reliable way > to do it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-2611) Handle RS that fails while processing the failure of another one
[ https://issues.apache.org/jira/browse/HBASE-2611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl reassigned HBASE-2611: Assignee: Chris Trezzo (was: Jean-Daniel Cryans) [~ctrezzo] Wanna work on this. Have a look at the discussion in HBASE-6695. If we accept a bit of herding initially we can fix this with a simple change (I think)... And ZK herding is still better than data loss. > Handle RS that fails while processing the failure of another one > > > Key: HBASE-2611 > URL: https://issues.apache.org/jira/browse/HBASE-2611 > Project: HBase > Issue Type: Sub-task > Components: Replication >Reporter: Jean-Daniel Cryans >Assignee: Chris Trezzo > Fix For: 0.94.3 > > Attachments: HBase-2611-upstream-v1.patch > > > HBASE-2223 doesn't manage region servers that fail while doing the transfer > of HLogs queues from other region servers that failed. Devise a reliable way > to do it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5323) Need to handle assertion error while splitting log through ServerShutDownHandler by shutting down the master
[
https://issues.apache.org/jira/browse/HBASE-5323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-5323:
-
Fix Version/s: (was: 0.94.3)
0.94.4
On to the greener pastures of 0.94.4
> Need to handle assertion error while splitting log through
> ServerShutDownHandler by shutting down the master
>
>
> Key: HBASE-5323
> URL: https://issues.apache.org/jira/browse/HBASE-5323
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.90.5
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 0.90.8, 0.94.4
>
> Attachments: HBASE-5323.patch, HBASE-5323.patch
>
>
> We know that while parsing the HLog we expect the proper length from HDFS.
> In WALReaderFSDataInputStream
> {code}
> assert(realLength >= this.length);
> {code}
> We are trying to come out if the above condition is not satisfied. But if
> SSH.splitLog() gets this problem then it lands in the run method of
> EventHandler. This kills the SSH thread and so further assignment does not
> happen. If ROOT and META are to be assigned they cannot be.
> I think in this condition we abort the master by catching such exceptions.
> Please do suggest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3852) ThriftServer leaks scanners
[ https://issues.apache.org/jira/browse/HBASE-3852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-3852: - Fix Version/s: (was: 0.94.3) 0.94.4 And on to 0.94.4 > ThriftServer leaks scanners > --- > > Key: HBASE-3852 > URL: https://issues.apache.org/jira/browse/HBASE-3852 > Project: HBase > Issue Type: Bug > Components: Thrift >Affects Versions: 0.90.2 >Reporter: Jean-Daniel Cryans > Fix For: 0.94.4 > > Attachments: 3852.txt, thrift2-scanner.patch > > > The scannerMap in ThriftServer relies on the user to clean it by closing the > scanner. If that doesn't happen, the ResultScanner will stay in the thrift > server's memory and if any pre-fetching was done, it will also start > accumulating Results (with all their data). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6324) Direct API calls from embedded Thrift server to regionserver
[ https://issues.apache.org/jira/browse/HBASE-6324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484714#comment-13484714 ] stack commented on HBASE-6324: -- [~sershe] The 'direct api calls from embedded thrift server to regionserver' seems to have been a failed or abandoned experiment Sergey. See http://search-hadoop.com/m/GAFt6B9CRZ Might not be worth expending effort on. > Direct API calls from embedded Thrift server to regionserver > > > Key: HBASE-6324 > URL: https://issues.apache.org/jira/browse/HBASE-6324 > Project: HBase > Issue Type: Improvement >Reporter: Mikhail Bautin >Assignee: Mikhail Bautin > > When handling Thrift calls in the regionserver we should not go through RPC > to talk to the local regionserver. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6651) Thread safety of HTablePool is doubtful
[ https://issues.apache.org/jira/browse/HBASE-6651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hiroshi Ikeda updated HBASE-6651: - Attachment: HBASE-6651-V5.patch Patch v5 from review board. > Thread safety of HTablePool is doubtful > --- > > Key: HBASE-6651 > URL: https://issues.apache.org/jira/browse/HBASE-6651 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.94.1 >Reporter: Hiroshi Ikeda >Priority: Minor > Attachments: HBASE-6651.patch, HBASE-6651-V2.patch, > HBASE-6651-V3.patch, HBASE-6651-V4.patch, HBASE-6651-V5.patch, sample.zip, > sample.zip, sharedmap_for_hbaseclient.zip > > > There are some operations in HTablePool to access to PoolMap in multiple > times without any explict synchronization. > For example HTablePool.closeTablePool() calles PoolMap.values(), and calles > PoolMap.remove(). If other threads add new instances to the pool in the > middle of the calls, the new added instances might be dropped. > (HTablePool.closeTablePool() also has another problem that calling it by > multple threads causes accessing HTable by multiple threads.) > Moreover, PoolMap is not thread safe for the same reason. > For example PoolMap.put() calles ConcurrentMap.get() and calles > ConcurrentMap.put(). If other threads add a new instance to the concurent map > in the middle of the calls, the new instance might be dropped. > And also implementations of Pool have the same problems. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6312) Make BlockCache eviction thresholds configurable
[ https://issues.apache.org/jira/browse/HBASE-6312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484710#comment-13484710 ] Hudson commented on HBASE-6312: --- Integrated in HBase-0.94 #555 (See [https://builds.apache.org/job/HBase-0.94/555/]) HBASE-7053 port blockcache configurability (part of HBASE-6312, and HBASE-7033) to 0.94 (Sergey Shelukhin) (Revision 1402385) Result = SUCCESS larsh : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileDataBlockEncoder.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java > Make BlockCache eviction thresholds configurable > > > Key: HBASE-6312 > URL: https://issues.apache.org/jira/browse/HBASE-6312 > Project: HBase > Issue Type: Improvement > Components: io >Affects Versions: 0.94.0 >Reporter: Jie Huang >Assignee: Jie Huang >Priority: Minor > Fix For: 0.96.0 > > Attachments: hbase-6312.patch, hbase-6312_v2.patch, > hbase-6312_v3.patch > > > Some of our customers found that tuning the BlockCache eviction thresholds > made test results different in their test environment. However, those > thresholds are not configurable in the current implementation. The only way > to change those values is to re-compile the HBase source code. We wonder if > it is possible to make them configurable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7033) Add hbase.lru.blockcache.acceptable.factor to configuration, akin to the min.factor added by HBASE-6312
[ https://issues.apache.org/jira/browse/HBASE-7033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484708#comment-13484708 ] Hudson commented on HBASE-7033: --- Integrated in HBase-0.94 #555 (See [https://builds.apache.org/job/HBase-0.94/555/]) HBASE-7053 port blockcache configurability (part of HBASE-6312, and HBASE-7033) to 0.94 (Sergey Shelukhin) (Revision 1402385) Result = SUCCESS larsh : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileDataBlockEncoder.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java > Add hbase.lru.blockcache.acceptable.factor to configuration, akin to the > min.factor added by HBASE-6312 > --- > > Key: HBASE-7033 > URL: https://issues.apache.org/jira/browse/HBASE-7033 > Project: HBase > Issue Type: Improvement > Components: io >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Minor > Fix For: 0.96.0 > > Attachments: HBASE-7033.patch > > > Background: we want to make the change to block cache setting available on > 0.94 without actually changing the defaults as was done in HBASE-6312, as > this can be destabilizing. > Thus, both of these would be configurable instead of just one, and the user > would be able to switch to new values. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7053) port blockcache configurability (part of HBASE-6312, and HBASE-7033) to 0.94
[ https://issues.apache.org/jira/browse/HBASE-7053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484709#comment-13484709 ] Hudson commented on HBASE-7053: --- Integrated in HBase-0.94 #555 (See [https://builds.apache.org/job/HBase-0.94/555/]) HBASE-7053 port blockcache configurability (part of HBASE-6312, and HBASE-7033) to 0.94 (Sergey Shelukhin) (Revision 1402385) Result = SUCCESS larsh : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileDataBlockEncoder.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java > port blockcache configurability (part of HBASE-6312, and HBASE-7033) to 0.94 > - > > Key: HBASE-7053 > URL: https://issues.apache.org/jira/browse/HBASE-7053 > Project: HBase > Issue Type: Task >Affects Versions: 0.94.2 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Minor > Fix For: 0.94.3 > > Attachments: HBASE-7053.patch, HBASE-7053-v2-squashed.patch > > > Add an option to get the improvement from 6312 w/o changing the defaults in > stable release. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6223) Document hbck improvements: HBASE-6173, HBASE-5360
[ https://issues.apache.org/jira/browse/HBASE-6223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484707#comment-13484707 ] stack commented on HBASE-6223: -- +1 on commit. On commit add in a 'the' between 'in' and 'right'. Should that be 'or not deployed' (add 'or') and 'so you can' -- add a 'so'? Maybe break up the big sentence. Its hard to pass. Good on you Jimmy. +Once a region is split, the offline parent will be cleaned up automatically. Sometimes, daughter regions +are split again before their parents are cleaned up. HBase can clean up parents in right order. However, +if there is a lingering offline split parent, i.e. it's in META, in HDFS, not deployed, HBase can't +clean it up, you can use the -fixSplitParents option to reset the parent in META to be +online and not split so that hbck can merge it with other regions with options to fix +overlapping regions. This option should not normally be used, and it is not in -fixAll. > Document hbck improvements: HBASE-6173, HBASE-5360 > --- > > Key: HBASE-6223 > URL: https://issues.apache.org/jira/browse/HBASE-6223 > Project: HBase > Issue Type: Task > Components: documentation, hbck >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Attachments: trunk-6223.patch, trunk-6223_v2.patch > > > We had a couple hbck improvements recently: HBASE-6173 and HBASE-5360. > We should document them. Especially, for HBASE-5360, it's something > one normally doesn't do. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6060) Regions's in OPENING state from failed regionservers takes a long time to recover
[ https://issues.apache.org/jira/browse/HBASE-6060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484706#comment-13484706 ] Lars Hofhansl commented on HBASE-6060: -- Looking at the patches and trying to read through the comments here, I have to admit... I have NO idea what is going on. I think in 0.96 Jimmy is ripping apart AM, which would hopefully cover this. Should we attempt this at all on 0.94, or just let it go? Please let me know. > Regions's in OPENING state from failed regionservers takes a long time to > recover > - > > Key: HBASE-6060 > URL: https://issues.apache.org/jira/browse/HBASE-6060 > Project: HBase > Issue Type: Bug > Components: master, regionserver >Reporter: Enis Soztutar >Assignee: rajeshbabu > Fix For: 0.92.3, 0.94.3, 0.96.0 > > Attachments: 6060-94-v3.patch, 6060-94-v4_1.patch, > 6060-94-v4_1.patch, 6060-94-v4.patch, 6060_alternative_suggestion.txt, > 6060_suggestion2_based_off_v3.patch, 6060_suggestion_based_off_v3.patch, > 6060_suggestion_toassign_rs_wentdown_beforerequest.patch, 6060-trunk_2.patch, > 6060-trunk_3.patch, 6060-trunk.patch, 6060-trunk.patch, HBASE-6060-92.patch, > HBASE-6060-94.patch, HBASE-6060_latest.patch, HBASE-6060_latest.patch, > HBASE-6060_latest.patch, HBASE-6060-trunk_4.patch, HBASE-6060_trunk_5.patch > > > we have seen a pattern in tests, that the regions are stuck in OPENING state > for a very long time when the region server who is opening the region fails. > My understanding of the process: > > - master calls rs to open the region. If rs is offline, a new plan is > generated (a new rs is chosen). RegionState is set to PENDING_OPEN (only in > master memory, zk still shows OFFLINE). See HRegionServer.openRegion(), > HMaster.assign() > - RegionServer, starts opening a region, changes the state in znode. But > that znode is not ephemeral. (see ZkAssign) > - Rs transitions zk node from OFFLINE to OPENING. See > OpenRegionHandler.process() > - rs then opens the region, and changes znode from OPENING to OPENED > - when rs is killed between OPENING and OPENED states, then zk shows OPENING > state, and the master just waits for rs to change the region state, but since > rs is down, that wont happen. > - There is a AssignmentManager.TimeoutMonitor, which does exactly guard > against these kind of conditions. It periodically checks (every 10 sec by > default) the regions in transition to see whether they timedout > (hbase.master.assignment.timeoutmonitor.timeout). Default timeout is 30 min, > which explains what you and I are seeing. > - ServerShutdownHandler in Master does not reassign regions in OPENING > state, although it handles other states. > Lowering that threshold from the configuration is one option, but still I > think we can do better. > Will investigate more. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6852) SchemaMetrics.updateOnCacheHit costs too much while full scanning a table with all of its fields
[ https://issues.apache.org/jira/browse/HBASE-6852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484705#comment-13484705 ] stack commented on HBASE-6852: -- Patch looks good to me for 0.94 > SchemaMetrics.updateOnCacheHit costs too much while full scanning a table > with all of its fields > > > Key: HBASE-6852 > URL: https://issues.apache.org/jira/browse/HBASE-6852 > Project: HBase > Issue Type: Improvement > Components: metrics >Affects Versions: 0.94.0 >Reporter: Cheng Hao >Priority: Minor > Labels: performance > Fix For: 0.94.3 > > Attachments: 6852-0.94.txt, metrics_hotspots.png, > onhitcache-trunk.patch > > > The SchemaMetrics.updateOnCacheHit costs too much while I am doing the full > table scanning. > Here is the top 5 hotspots within regionserver while full scanning a table: > (Sorry for the less-well-format) > CPU: Intel Westmere microarchitecture, speed 2.262e+06 MHz (estimated) > Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit > mask of 0x00 (No unit mask) count 500 > samples %image name symbol name > --- > 9844713.4324 14033.jo void > org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics.updateOnCacheHit(org.apache.hadoop.hbase.io.hfile.BlockType$BlockCategory, > boolean) > 98447100.000 14033.jo void > org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics.updateOnCacheHit(org.apache.hadoop.hbase.io.hfile.BlockType$BlockCategory, > boolean) [self] > --- > 45814 6.2510 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(byte[], int, int, > byte[], int, int) > 45814100.000 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(byte[], int, int, > byte[], int, int) [self] > --- > 43523 5.9384 14033.jo boolean > org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(org.apache.hadoop.hbase.KeyValue) > 43523100.000 14033.jo boolean > org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(org.apache.hadoop.hbase.KeyValue) > [self] > --- > 42548 5.8054 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(byte[], int, int, > byte[], int, int) > 42548100.000 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(byte[], int, int, > byte[], int, int) [self] > --- > 40572 5.5358 14033.jo int > org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.binarySearchNonRootIndex(byte[], > int, int, java.nio.ByteBuffer, org.apache.hadoop.io.RawComparator)~1 > 40572100.000 14033.jo int > org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.binarySearchNonRootIndex(byte[], > int, int, java.nio.ByteBuffer, org.apache.hadoop.io.RawComparator)~1 [self] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5943) Set tcpnodelay as default
[
https://issues.apache.org/jira/browse/HBASE-5943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-5943:
-
Priority: Critical (was: Major)
Tags: noob
Fix Version/s: 0.96.0
Labels: noob (was: )
This is one that a noob could get stuck into. Mr or Mrs noob could try
different data sizes and measure and make a recommendation. This would be a
great help to the project.
> Set tcpnodelay as default
> -
>
> Key: HBASE-5943
> URL: https://issues.apache.org/jira/browse/HBASE-5943
> Project: HBase
> Issue Type: Improvement
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Labels: noob
> Fix For: 0.96.0
>
> Attachments: nodelay.txt
>
>
> From IRC:
> {code}
> 14:46 < tlipcon> i turned off the FirstKeyOnlyFilter on the shell count
> command
> 14:46 < tlipcon> thgouhtput dropped like carzy... looked at strace, and saw
> 40ms delays
> 14:46 < tlipcon> turned on ipc.server.nodelay or whaever and it got fast again
> 14:46 < tlipcon> we should really ship that by default on
> 14:46 < tlipcon> there are probably people getting screwed by it
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7056) TestMetaMigrationConvertingToPB failed in HBase-TRUNK-on-Hadoop-2.0.0 two times consecutively
[
https://issues.apache.org/jira/browse/HBASE-7056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484702#comment-13484702
]
stack commented on HBASE-7056:
--
Any chance of a patch mighty Ted?
> TestMetaMigrationConvertingToPB failed in HBase-TRUNK-on-Hadoop-2.0.0 two
> times consecutively
> -
>
> Key: HBASE-7056
> URL: https://issues.apache.org/jira/browse/HBASE-7056
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Critical
> Fix For: 0.96.0
>
>
> Building against hadoop 2.0.2-alpha, TestMetaMigrationConvertingToPB failed
> twice - builds 236 and 237.
> See
> https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/236/testReport/org.apache.hadoop.hbase.catalog/TestMetaMigrationConvertingToPB/org_apache_hadoop_hbase_catalog_TestMetaMigrationConvertingToPB/
> {code}
> java.io.IOException: Failed put; errcode=1
> at
> org.apache.hadoop.hbase.catalog.TestMetaMigrationConvertingToPB.doFsCommand(TestMetaMigrationConvertingToPB.java:155)
> at
> org.apache.hadoop.hbase.catalog.TestMetaMigrationConvertingToPB.setUpBeforeClass(TestMetaMigrationConvertingToPB.java:108)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5943) Set tcpnodelay as default
[
https://issues.apache.org/jira/browse/HBASE-5943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-5943:
-
Fix Version/s: (was: 0.94.3)
(was: 0.96.0)
Alright, unassigning from specific releases again... Sorry for the noise.
> Set tcpnodelay as default
> -
>
> Key: HBASE-5943
> URL: https://issues.apache.org/jira/browse/HBASE-5943
> Project: HBase
> Issue Type: Improvement
>Reporter: stack
>Assignee: stack
> Attachments: nodelay.txt
>
>
> From IRC:
> {code}
> 14:46 < tlipcon> i turned off the FirstKeyOnlyFilter on the shell count
> command
> 14:46 < tlipcon> thgouhtput dropped like carzy... looked at strace, and saw
> 40ms delays
> 14:46 < tlipcon> turned on ipc.server.nodelay or whaever and it got fast again
> 14:46 < tlipcon> we should really ship that by default on
> 14:46 < tlipcon> there are probably people getting screwed by it
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6417) hbck merges .META. regions if there's an old leftover
[ https://issues.apache.org/jira/browse/HBASE-6417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6417: - Fix Version/s: (was: 0.94.3) 0.94.4 > hbck merges .META. regions if there's an old leftover > - > > Key: HBASE-6417 > URL: https://issues.apache.org/jira/browse/HBASE-6417 > Project: HBase > Issue Type: Bug >Reporter: Jean-Daniel Cryans > Fix For: 0.96.0, 0.94.4 > > Attachments: hbck.log > > > Trying to see what caused HBASE-6310, one of the things I figured is that the > bad .META. row is actually one from the time that we were permitting meta > splitting and that folder had just been staying there for a while. > So I tried to recreate the issue with -repair and it merged my good .META. > region with the one that's 3 years old that also has the same start key. I > ended up with a brand new .META. region! > I'll be attaching the full log in a separate file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6470) SingleColumnValueFilter with private fields and methods
[ https://issues.apache.org/jira/browse/HBASE-6470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6470: - Fix Version/s: (was: 0.94.3) Removing from 0.94 until there is a patch. > SingleColumnValueFilter with private fields and methods > --- > > Key: HBASE-6470 > URL: https://issues.apache.org/jira/browse/HBASE-6470 > Project: HBase > Issue Type: Improvement > Components: Filters >Affects Versions: 0.94.0 >Reporter: Benjamin Kim > Labels: patch > > Why are most fields and methods declared private in SingleColumnValueFilter? > I'm trying to extend the functions of the SingleColumnValueFilter to support > complex column types such as JSON, Array, CSV, etc. > But inheriting the SingleColumnValueFilter doesn't give any benefits for I > have to rewrite the codes. > I think all private fields and methods could turn into protected mode. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6563) s.isMajorCompaction() throws npe will cause current major Compaction checking abort
[ https://issues.apache.org/jira/browse/HBASE-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6563: - Fix Version/s: (was: 0.94.3) > s.isMajorCompaction() throws npe will cause current major Compaction checking > abort > --- > > Key: HBASE-6563 > URL: https://issues.apache.org/jira/browse/HBASE-6563 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Zhou wenjian >Assignee: Zhou wenjian > Attachments: HBASE-6563-trunk.patch, HBASE-6563-trunk-v2.patch, > HBASE-6563-trunk-v3.patch > > > 2012-05-05 00:49:43,265 ERROR > org.apache.hadoop.hbase.regionserver.HRegionServer$MajorCompactionChecker: > Caught exception > java.lang.NullPointerException > at > org.apache.hadoop.hbase.regionserver.Store.isMajorCompaction(Store.java:938) > at > org.apache.hadoop.hbase.regionserver.Store.isMajorCompaction(Store.java:917) > at > org.apache.hadoop.hbase.regionserver.HRegion.isMajorCompaction(HRegion.java:3250) > at > org.apache.hadoop.hbase.regionserver.HRegionServer$MajorCompactionChecker.chore(HRegionServer.java:1222) > at org.apache.hadoop.hbase.Chore.run(Chore.java:66) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6628) Add HBASE-6059 to 0.94 branch
[ https://issues.apache.org/jira/browse/HBASE-6628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484699#comment-13484699 ] stack commented on HBASE-6628: -- Looks like a good find (Did I volunteer to do the backport? If so, you can stick me for it Lars -- assign me the issue). > Add HBASE-6059 to 0.94 branch > - > > Key: HBASE-6628 > URL: https://issues.apache.org/jira/browse/HBASE-6628 > Project: HBase > Issue Type: Task >Reporter: stack > Fix For: 0.94.3 > > > Look at adding HBASE-6059 to 0.94. Its in trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6563) s.isMajorCompaction() throws npe will cause current major Compaction checking abort
[ https://issues.apache.org/jira/browse/HBASE-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6563: - Resolution: Not A Problem Status: Resolved (was: Patch Available) Closing, since this was only 0.94. > s.isMajorCompaction() throws npe will cause current major Compaction checking > abort > --- > > Key: HBASE-6563 > URL: https://issues.apache.org/jira/browse/HBASE-6563 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Zhou wenjian >Assignee: Zhou wenjian > Fix For: 0.94.3 > > Attachments: HBASE-6563-trunk.patch, HBASE-6563-trunk-v2.patch, > HBASE-6563-trunk-v3.patch > > > 2012-05-05 00:49:43,265 ERROR > org.apache.hadoop.hbase.regionserver.HRegionServer$MajorCompactionChecker: > Caught exception > java.lang.NullPointerException > at > org.apache.hadoop.hbase.regionserver.Store.isMajorCompaction(Store.java:938) > at > org.apache.hadoop.hbase.regionserver.Store.isMajorCompaction(Store.java:917) > at > org.apache.hadoop.hbase.regionserver.HRegion.isMajorCompaction(HRegion.java:3250) > at > org.apache.hadoop.hbase.regionserver.HRegionServer$MajorCompactionChecker.chore(HRegionServer.java:1222) > at org.apache.hadoop.hbase.Chore.run(Chore.java:66) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6563) s.isMajorCompaction() throws npe will cause current major Compaction checking abort
[ https://issues.apache.org/jira/browse/HBASE-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484697#comment-13484697 ] Lars Hofhansl commented on HBASE-6563: -- I'm not a fan of catching NPE. They should be addressed by null checks. Also, I see in Store.isMajorCompaction (in 0.94+) that there already is such a null check. > s.isMajorCompaction() throws npe will cause current major Compaction checking > abort > --- > > Key: HBASE-6563 > URL: https://issues.apache.org/jira/browse/HBASE-6563 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Zhou wenjian >Assignee: Zhou wenjian > Fix For: 0.94.3 > > Attachments: HBASE-6563-trunk.patch, HBASE-6563-trunk-v2.patch, > HBASE-6563-trunk-v3.patch > > > 2012-05-05 00:49:43,265 ERROR > org.apache.hadoop.hbase.regionserver.HRegionServer$MajorCompactionChecker: > Caught exception > java.lang.NullPointerException > at > org.apache.hadoop.hbase.regionserver.Store.isMajorCompaction(Store.java:938) > at > org.apache.hadoop.hbase.regionserver.Store.isMajorCompaction(Store.java:917) > at > org.apache.hadoop.hbase.regionserver.HRegion.isMajorCompaction(HRegion.java:3250) > at > org.apache.hadoop.hbase.regionserver.HRegionServer$MajorCompactionChecker.chore(HRegionServer.java:1222) > at org.apache.hadoop.hbase.Chore.run(Chore.java:66) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6628) Add HBASE-6059 to 0.94 branch
[ https://issues.apache.org/jira/browse/HBASE-6628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484695#comment-13484695 ] Lars Hofhansl commented on HBASE-6628: -- [~saint@gmail.com] How important do you think this is? > Add HBASE-6059 to 0.94 branch > - > > Key: HBASE-6628 > URL: https://issues.apache.org/jira/browse/HBASE-6628 > Project: HBase > Issue Type: Task >Reporter: stack > Fix For: 0.94.3 > > > Look at adding HBASE-6059 to 0.94. Its in trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-7053) port blockcache configurability (part of HBASE-6312, and HBASE-7033) to 0.94
[ https://issues.apache.org/jira/browse/HBASE-7053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl resolved HBASE-7053. -- Resolution: Fixed Hadoop Flags: Reviewed Never mind. I see. Committed to 0.94. > port blockcache configurability (part of HBASE-6312, and HBASE-7033) to 0.94 > - > > Key: HBASE-7053 > URL: https://issues.apache.org/jira/browse/HBASE-7053 > Project: HBase > Issue Type: Task >Affects Versions: 0.94.2 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Minor > Fix For: 0.94.3 > > Attachments: HBASE-7053.patch, HBASE-7053-v2-squashed.patch > > > Add an option to get the improvement from 6312 w/o changing the defaults in > stable release. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7053) port blockcache configurability (part of HBASE-6312, and HBASE-7033) to 0.94
[ https://issues.apache.org/jira/browse/HBASE-7053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484682#comment-13484682 ] Lars Hofhansl commented on HBASE-7053: -- This only ports HBASE-6312, right? (not HBASE-7033) > port blockcache configurability (part of HBASE-6312, and HBASE-7033) to 0.94 > - > > Key: HBASE-7053 > URL: https://issues.apache.org/jira/browse/HBASE-7053 > Project: HBase > Issue Type: Task >Affects Versions: 0.94.2 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Minor > Fix For: 0.94.3 > > Attachments: HBASE-7053.patch, HBASE-7053-v2-squashed.patch > > > Add an option to get the improvement from 6312 w/o changing the defaults in > stable release. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6651) Thread safety of HTablePool is doubtful
[ https://issues.apache.org/jira/browse/HBASE-6651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484681#comment-13484681 ] Lars Hofhansl commented on HBASE-6651: -- [~karthick] Do you want to have a look? > Thread safety of HTablePool is doubtful > --- > > Key: HBASE-6651 > URL: https://issues.apache.org/jira/browse/HBASE-6651 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.94.1 >Reporter: Hiroshi Ikeda >Priority: Minor > Attachments: HBASE-6651.patch, HBASE-6651-V2.patch, > HBASE-6651-V3.patch, HBASE-6651-V4.patch, sample.zip, sample.zip, > sharedmap_for_hbaseclient.zip > > > There are some operations in HTablePool to access to PoolMap in multiple > times without any explict synchronization. > For example HTablePool.closeTablePool() calles PoolMap.values(), and calles > PoolMap.remove(). If other threads add new instances to the pool in the > middle of the calls, the new added instances might be dropped. > (HTablePool.closeTablePool() also has another problem that calling it by > multple threads causes accessing HTable by multiple threads.) > Moreover, PoolMap is not thread safe for the same reason. > For example PoolMap.put() calles ConcurrentMap.get() and calles > ConcurrentMap.put(). If other threads add a new instance to the concurent map > in the middle of the calls, the new instance might be dropped. > And also implementations of Pool have the same problems. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7051) Read/Updates (increment,checkAndPut) should properly read MVCC
[
https://issues.apache.org/jira/browse/HBASE-7051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484678#comment-13484678
]
Lars Hofhansl commented on HBASE-7051:
--
I don't see how that can happen. checkAndMutate takes a row lock, reads the row
(following MVCC), and updates the row (which will also follow the MVCC
protocol).
> Read/Updates (increment,checkAndPut) should properly read MVCC
> --
>
> Key: HBASE-7051
> URL: https://issues.apache.org/jira/browse/HBASE-7051
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.96.0
>Reporter: Gregory Chanan
>
> See, for example:
> {code}
> // TODO: Use MVCC to make this set of increments atomic to reads
> {code}
> Here's an example of what I can happen (would probably be good to write up a
> test case for each read/update):
> Concurrent update via increment and put.
> The put grabs the row lock first and updates the memstore, but releases the
> row lock before the MVCC is advanced. Then, the increment grabs the row lock
> and reads right away, reading the old value and incrementing based on that.
> There are a few options here:
> 1) Waiting for the MVCC to advance for read/updates: the downside is that you
> have to wait for updates on other rows.
> 2) Have an MVCC per-row (table configuration): this avoids the unnecessary
> contention of 1)
> 3) Transform the read/updates to write-only with rollup on read.. E.g. an
> increment would just have the number of values to increment.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7053) port blockcache configurability (part of HBASE-6312, and HBASE-7033) to 0.94
[ https://issues.apache.org/jira/browse/HBASE-7053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484679#comment-13484679 ] Lars Hofhansl commented on HBASE-7053: -- This is a low risk fix, I don't think anybody would object to this. Committing now. > port blockcache configurability (part of HBASE-6312, and HBASE-7033) to 0.94 > - > > Key: HBASE-7053 > URL: https://issues.apache.org/jira/browse/HBASE-7053 > Project: HBase > Issue Type: Task >Affects Versions: 0.94.2 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Minor > Fix For: 0.94.3 > > Attachments: HBASE-7053.patch, HBASE-7053-v2-squashed.patch > > > Add an option to get the improvement from 6312 w/o changing the defaults in > stable release. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7056) TestMetaMigrationConvertingToPB failed in HBase-TRUNK-on-Hadoop-2.0.0 two times consecutively
Ted Yu created HBASE-7056:
-
Summary: TestMetaMigrationConvertingToPB failed in
HBase-TRUNK-on-Hadoop-2.0.0 two times consecutively
Key: HBASE-7056
URL: https://issues.apache.org/jira/browse/HBASE-7056
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Priority: Critical
Fix For: 0.96.0
Building against hadoop 2.0.2-alpha, TestMetaMigrationConvertingToPB failed
twice - builds 236 and 237.
See
https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/236/testReport/org.apache.hadoop.hbase.catalog/TestMetaMigrationConvertingToPB/org_apache_hadoop_hbase_catalog_TestMetaMigrationConvertingToPB/
{code}
java.io.IOException: Failed put; errcode=1
at
org.apache.hadoop.hbase.catalog.TestMetaMigrationConvertingToPB.doFsCommand(TestMetaMigrationConvertingToPB.java:155)
at
org.apache.hadoop.hbase.catalog.TestMetaMigrationConvertingToPB.setUpBeforeClass(TestMetaMigrationConvertingToPB.java:108)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7053) port blockcache configurability (part of HBASE-6312, and HBASE-7033) to 0.94
[ https://issues.apache.org/jira/browse/HBASE-7053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484643#comment-13484643 ] Sergey Shelukhin commented on HBASE-7053: - Thanks! Is anything more needed to commit? > port blockcache configurability (part of HBASE-6312, and HBASE-7033) to 0.94 > - > > Key: HBASE-7053 > URL: https://issues.apache.org/jira/browse/HBASE-7053 > Project: HBase > Issue Type: Task >Affects Versions: 0.94.2 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Minor > Fix For: 0.94.3 > > Attachments: HBASE-7053.patch, HBASE-7053-v2-squashed.patch > > > Add an option to get the improvement from 6312 w/o changing the defaults in > stable release. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7051) Read/Updates (increment,checkAndPut) should properly read MVCC
[
https://issues.apache.org/jira/browse/HBASE-7051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484634#comment-13484634
]
Gregory Chanan commented on HBASE-7051:
---
Great. My surface-level reading of the code was that it could be 11, but I'll
work on an actual test case :).
> Read/Updates (increment,checkAndPut) should properly read MVCC
> --
>
> Key: HBASE-7051
> URL: https://issues.apache.org/jira/browse/HBASE-7051
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.96.0
>Reporter: Gregory Chanan
>
> See, for example:
> {code}
> // TODO: Use MVCC to make this set of increments atomic to reads
> {code}
> Here's an example of what I can happen (would probably be good to write up a
> test case for each read/update):
> Concurrent update via increment and put.
> The put grabs the row lock first and updates the memstore, but releases the
> row lock before the MVCC is advanced. Then, the increment grabs the row lock
> and reads right away, reading the old value and incrementing based on that.
> There are a few options here:
> 1) Waiting for the MVCC to advance for read/updates: the downside is that you
> have to wait for updates on other rows.
> 2) Have an MVCC per-row (table configuration): this avoids the unnecessary
> contention of 1)
> 3) Transform the read/updates to write-only with rollup on read.. E.g. an
> increment would just have the number of values to increment.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7051) Read/Updates (increment,checkAndPut) should properly read MVCC
[
https://issues.apache.org/jira/browse/HBASE-7051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484632#comment-13484632
]
Lars Hofhansl commented on HBASE-7051:
--
"Yes" was agreeing that 50 should always be the outcome. Is that ever not the
case?
> Read/Updates (increment,checkAndPut) should properly read MVCC
> --
>
> Key: HBASE-7051
> URL: https://issues.apache.org/jira/browse/HBASE-7051
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.96.0
>Reporter: Gregory Chanan
>
> See, for example:
> {code}
> // TODO: Use MVCC to make this set of increments atomic to reads
> {code}
> Here's an example of what I can happen (would probably be good to write up a
> test case for each read/update):
> Concurrent update via increment and put.
> The put grabs the row lock first and updates the memstore, but releases the
> row lock before the MVCC is advanced. Then, the increment grabs the row lock
> and reads right away, reading the old value and incrementing based on that.
> There are a few options here:
> 1) Waiting for the MVCC to advance for read/updates: the downside is that you
> have to wait for updates on other rows.
> 2) Have an MVCC per-row (table configuration): this avoids the unnecessary
> contention of 1)
> 3) Transform the read/updates to write-only with rollup on read.. E.g. an
> increment would just have the number of values to increment.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6852) SchemaMetrics.updateOnCacheHit costs too much while full scanning a table with all of its fields
[ https://issues.apache.org/jira/browse/HBASE-6852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6852: - Attachment: 6852-0.94.txt 0.94 patch. Also uses AtomicLongArray instead of AtomicLong[]. > SchemaMetrics.updateOnCacheHit costs too much while full scanning a table > with all of its fields > > > Key: HBASE-6852 > URL: https://issues.apache.org/jira/browse/HBASE-6852 > Project: HBase > Issue Type: Improvement > Components: metrics >Affects Versions: 0.94.0 >Reporter: Cheng Hao >Priority: Minor > Labels: performance > Fix For: 0.94.3 > > Attachments: 6852-0.94.txt, metrics_hotspots.png, > onhitcache-trunk.patch > > > The SchemaMetrics.updateOnCacheHit costs too much while I am doing the full > table scanning. > Here is the top 5 hotspots within regionserver while full scanning a table: > (Sorry for the less-well-format) > CPU: Intel Westmere microarchitecture, speed 2.262e+06 MHz (estimated) > Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit > mask of 0x00 (No unit mask) count 500 > samples %image name symbol name > --- > 9844713.4324 14033.jo void > org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics.updateOnCacheHit(org.apache.hadoop.hbase.io.hfile.BlockType$BlockCategory, > boolean) > 98447100.000 14033.jo void > org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics.updateOnCacheHit(org.apache.hadoop.hbase.io.hfile.BlockType$BlockCategory, > boolean) [self] > --- > 45814 6.2510 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(byte[], int, int, > byte[], int, int) > 45814100.000 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(byte[], int, int, > byte[], int, int) [self] > --- > 43523 5.9384 14033.jo boolean > org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(org.apache.hadoop.hbase.KeyValue) > 43523100.000 14033.jo boolean > org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(org.apache.hadoop.hbase.KeyValue) > [self] > --- > 42548 5.8054 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(byte[], int, int, > byte[], int, int) > 42548100.000 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(byte[], int, int, > byte[], int, int) [self] > --- > 40572 5.5358 14033.jo int > org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.binarySearchNonRootIndex(byte[], > int, int, java.nio.ByteBuffer, org.apache.hadoop.io.RawComparator)~1 > 40572100.000 14033.jo int > org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.binarySearchNonRootIndex(byte[], > int, int, java.nio.ByteBuffer, org.apache.hadoop.io.RawComparator)~1 [self] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6852) SchemaMetrics.updateOnCacheHit costs too much while full scanning a table with all of its fields
[ https://issues.apache.org/jira/browse/HBASE-6852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6852: - Status: Open (was: Patch Available) > SchemaMetrics.updateOnCacheHit costs too much while full scanning a table > with all of its fields > > > Key: HBASE-6852 > URL: https://issues.apache.org/jira/browse/HBASE-6852 > Project: HBase > Issue Type: Improvement > Components: metrics >Affects Versions: 0.94.0 >Reporter: Cheng Hao >Priority: Minor > Labels: performance > Fix For: 0.94.3 > > Attachments: 6852-0.94.txt, metrics_hotspots.png, > onhitcache-trunk.patch > > > The SchemaMetrics.updateOnCacheHit costs too much while I am doing the full > table scanning. > Here is the top 5 hotspots within regionserver while full scanning a table: > (Sorry for the less-well-format) > CPU: Intel Westmere microarchitecture, speed 2.262e+06 MHz (estimated) > Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit > mask of 0x00 (No unit mask) count 500 > samples %image name symbol name > --- > 9844713.4324 14033.jo void > org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics.updateOnCacheHit(org.apache.hadoop.hbase.io.hfile.BlockType$BlockCategory, > boolean) > 98447100.000 14033.jo void > org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics.updateOnCacheHit(org.apache.hadoop.hbase.io.hfile.BlockType$BlockCategory, > boolean) [self] > --- > 45814 6.2510 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(byte[], int, int, > byte[], int, int) > 45814100.000 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(byte[], int, int, > byte[], int, int) [self] > --- > 43523 5.9384 14033.jo boolean > org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(org.apache.hadoop.hbase.KeyValue) > 43523100.000 14033.jo boolean > org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(org.apache.hadoop.hbase.KeyValue) > [self] > --- > 42548 5.8054 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(byte[], int, int, > byte[], int, int) > 42548100.000 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(byte[], int, int, > byte[], int, int) [self] > --- > 40572 5.5358 14033.jo int > org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.binarySearchNonRootIndex(byte[], > int, int, java.nio.ByteBuffer, org.apache.hadoop.io.RawComparator)~1 > 40572100.000 14033.jo int > org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.binarySearchNonRootIndex(byte[], > int, int, java.nio.ByteBuffer, org.apache.hadoop.io.RawComparator)~1 [self] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7051) Read/Updates (increment,checkAndPut) should properly read MVCC
[
https://issues.apache.org/jira/browse/HBASE-7051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484618#comment-13484618
]
Gregory Chanan commented on HBASE-7051:
---
What does "Yes" refer to?
I'm suggesting that the result of the above two operations should always be 50.
Having the checkAndPut begin and complete a memstore insert before doing the
read (option 1 above), is one example of how to get those semantics.
> Read/Updates (increment,checkAndPut) should properly read MVCC
> --
>
> Key: HBASE-7051
> URL: https://issues.apache.org/jira/browse/HBASE-7051
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.96.0
>Reporter: Gregory Chanan
>
> See, for example:
> {code}
> // TODO: Use MVCC to make this set of increments atomic to reads
> {code}
> Here's an example of what I can happen (would probably be good to write up a
> test case for each read/update):
> Concurrent update via increment and put.
> The put grabs the row lock first and updates the memstore, but releases the
> row lock before the MVCC is advanced. Then, the increment grabs the row lock
> and reads right away, reading the old value and incrementing based on that.
> There are a few options here:
> 1) Waiting for the MVCC to advance for read/updates: the downside is that you
> have to wait for updates on other rows.
> 2) Have an MVCC per-row (table configuration): this avoids the unnecessary
> contention of 1)
> 3) Transform the read/updates to write-only with rollup on read.. E.g. an
> increment would just have the number of values to increment.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6852) SchemaMetrics.updateOnCacheHit costs too much while full scanning a table with all of its fields
[ https://issues.apache.org/jira/browse/HBASE-6852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6852: - Fix Version/s: (was: 0.96.0) > SchemaMetrics.updateOnCacheHit costs too much while full scanning a table > with all of its fields > > > Key: HBASE-6852 > URL: https://issues.apache.org/jira/browse/HBASE-6852 > Project: HBase > Issue Type: Improvement > Components: metrics >Affects Versions: 0.94.0 >Reporter: Cheng Hao >Priority: Minor > Labels: performance > Fix For: 0.94.3 > > Attachments: metrics_hotspots.png, onhitcache-trunk.patch > > > The SchemaMetrics.updateOnCacheHit costs too much while I am doing the full > table scanning. > Here is the top 5 hotspots within regionserver while full scanning a table: > (Sorry for the less-well-format) > CPU: Intel Westmere microarchitecture, speed 2.262e+06 MHz (estimated) > Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit > mask of 0x00 (No unit mask) count 500 > samples %image name symbol name > --- > 9844713.4324 14033.jo void > org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics.updateOnCacheHit(org.apache.hadoop.hbase.io.hfile.BlockType$BlockCategory, > boolean) > 98447100.000 14033.jo void > org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics.updateOnCacheHit(org.apache.hadoop.hbase.io.hfile.BlockType$BlockCategory, > boolean) [self] > --- > 45814 6.2510 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(byte[], int, int, > byte[], int, int) > 45814100.000 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(byte[], int, int, > byte[], int, int) [self] > --- > 43523 5.9384 14033.jo boolean > org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(org.apache.hadoop.hbase.KeyValue) > 43523100.000 14033.jo boolean > org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(org.apache.hadoop.hbase.KeyValue) > [self] > --- > 42548 5.8054 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(byte[], int, int, > byte[], int, int) > 42548100.000 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(byte[], int, int, > byte[], int, int) [self] > --- > 40572 5.5358 14033.jo int > org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.binarySearchNonRootIndex(byte[], > int, int, java.nio.ByteBuffer, org.apache.hadoop.io.RawComparator)~1 > 40572100.000 14033.jo int > org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.binarySearchNonRootIndex(byte[], > int, int, java.nio.ByteBuffer, org.apache.hadoop.io.RawComparator)~1 [self] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7053) port blockcache configurability (part of HBASE-6312, and HBASE-7033) to 0.94
[ https://issues.apache.org/jira/browse/HBASE-7053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484613#comment-13484613 ] Lars Hofhansl commented on HBASE-7053: -- Nice... Thanks Sergey... +1 on v2 > port blockcache configurability (part of HBASE-6312, and HBASE-7033) to 0.94 > - > > Key: HBASE-7053 > URL: https://issues.apache.org/jira/browse/HBASE-7053 > Project: HBase > Issue Type: Task >Affects Versions: 0.94.2 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Minor > Fix For: 0.94.3 > > Attachments: HBASE-7053.patch, HBASE-7053-v2-squashed.patch > > > Add an option to get the improvement from 6312 w/o changing the defaults in > stable release. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7051) Read/Updates (increment,checkAndPut) should properly read MVCC
[
https://issues.apache.org/jira/browse/HBASE-7051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484610#comment-13484610
]
Lars Hofhansl commented on HBASE-7051:
--
Yes. There is nothing special about checkAndPut.
Maybe I am confused... What specifically do you want to fix w.r.t. checkAndPut?
(I think the only weird things we have that do not honor MVCC are Increment and
Append)
> Read/Updates (increment,checkAndPut) should properly read MVCC
> --
>
> Key: HBASE-7051
> URL: https://issues.apache.org/jira/browse/HBASE-7051
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.96.0
>Reporter: Gregory Chanan
>
> See, for example:
> {code}
> // TODO: Use MVCC to make this set of increments atomic to reads
> {code}
> Here's an example of what I can happen (would probably be good to write up a
> test case for each read/update):
> Concurrent update via increment and put.
> The put grabs the row lock first and updates the memstore, but releases the
> row lock before the MVCC is advanced. Then, the increment grabs the row lock
> and reads right away, reading the old value and incrementing based on that.
> There are a few options here:
> 1) Waiting for the MVCC to advance for read/updates: the downside is that you
> have to wait for updates on other rows.
> 2) Have an MVCC per-row (table configuration): this avoids the unnecessary
> contention of 1)
> 3) Transform the read/updates to write-only with rollup on read.. E.g. an
> increment would just have the number of values to increment.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6223) Document hbck improvements: HBASE-6173, HBASE-5360
[
https://issues.apache.org/jira/browse/HBASE-6223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484604#comment-13484604
]
Hadoop QA commented on HBASE-6223:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12550888/trunk-6223_v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation patch that doesn't require tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
85 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 3 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3153//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3153//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3153//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3153//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3153//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3153//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3153//console
This message is automatically generated.
> Document hbck improvements: HBASE-6173, HBASE-5360
> ---
>
> Key: HBASE-6223
> URL: https://issues.apache.org/jira/browse/HBASE-6223
> Project: HBase
> Issue Type: Task
> Components: documentation, hbck
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
> Attachments: trunk-6223.patch, trunk-6223_v2.patch
>
>
> We had a couple hbck improvements recently: HBASE-6173 and HBASE-5360.
> We should document them. Especially, for HBASE-5360, it's something
> one normally doesn't do.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7051) Read/Updates (increment,checkAndPut) should properly read MVCC
[
https://issues.apache.org/jira/browse/HBASE-7051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484602#comment-13484602
]
Gregory Chanan commented on HBASE-7051:
---
Sorry last sentence should be:
"Then A goes, reads 10 and puts 11"
> Read/Updates (increment,checkAndPut) should properly read MVCC
> --
>
> Key: HBASE-7051
> URL: https://issues.apache.org/jira/browse/HBASE-7051
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.96.0
>Reporter: Gregory Chanan
>
> See, for example:
> {code}
> // TODO: Use MVCC to make this set of increments atomic to reads
> {code}
> Here's an example of what I can happen (would probably be good to write up a
> test case for each read/update):
> Concurrent update via increment and put.
> The put grabs the row lock first and updates the memstore, but releases the
> row lock before the MVCC is advanced. Then, the increment grabs the row lock
> and reads right away, reading the old value and incrementing based on that.
> There are a few options here:
> 1) Waiting for the MVCC to advance for read/updates: the downside is that you
> have to wait for updates on other rows.
> 2) Have an MVCC per-row (table configuration): this avoids the unnecessary
> contention of 1)
> 3) Transform the read/updates to write-only with rollup on read.. E.g. an
> increment would just have the number of values to increment.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7051) Read/Updates (increment,checkAndPut) should properly read MVCC
[
https://issues.apache.org/jira/browse/HBASE-7051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484600#comment-13484600
]
Gregory Chanan commented on HBASE-7051:
---
I think the special memstoreTS logic in Increment/Append is a separate issue
than what I'm bringing up here. I could be wrong (I've just really started
looking into this code). Could you explain what is wrong about the following:
The current value of some cell is 10.
I issue two concurrent requests:
A) a check and put where check value = 10, put value = 11
B) a put where put value = 50
The only result at the end of these operations that seems reasonable to me is
the value of the cell being 50. If A occurred first (ACID wise), then our
values go 10->11->50. If B occurred first, then our values go 10->50 (and the
checkAndPut fails). Or do we just not consider checkAndPuts to be transactions?
My reading of the code is that we could get 11. B goes first but releases the
rowLock before completing his MVCC memstore insert. Then A goes, reads 10 and
puts 50.
> Read/Updates (increment,checkAndPut) should properly read MVCC
> --
>
> Key: HBASE-7051
> URL: https://issues.apache.org/jira/browse/HBASE-7051
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.96.0
>Reporter: Gregory Chanan
>
> See, for example:
> {code}
> // TODO: Use MVCC to make this set of increments atomic to reads
> {code}
> Here's an example of what I can happen (would probably be good to write up a
> test case for each read/update):
> Concurrent update via increment and put.
> The put grabs the row lock first and updates the memstore, but releases the
> row lock before the MVCC is advanced. Then, the increment grabs the row lock
> and reads right away, reading the old value and incrementing based on that.
> There are a few options here:
> 1) Waiting for the MVCC to advance for read/updates: the downside is that you
> have to wait for updates on other rows.
> 2) Have an MVCC per-row (table configuration): this avoids the unnecessary
> contention of 1)
> 3) Transform the read/updates to write-only with rollup on read.. E.g. an
> increment would just have the number of values to increment.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7054) Have our website conform to apache branding requirements
[ https://issues.apache.org/jira/browse/HBASE-7054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484598#comment-13484598 ] Hudson commented on HBASE-7054: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #237 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/237/]) HBASE-7054 Have our website conform to apache branding requirements (Revision 1402324) Result = FAILURE stack : Files : * /hbase/trunk/pom.xml * /hbase/trunk/src/docbkx/book.xml * /hbase/trunk/src/docbkx/case_studies.xml * /hbase/trunk/src/docbkx/community.xml * /hbase/trunk/src/docbkx/configuration.xml * /hbase/trunk/src/docbkx/developer.xml * /hbase/trunk/src/docbkx/external_apis.xml * /hbase/trunk/src/docbkx/ops_mgt.xml * /hbase/trunk/src/docbkx/performance.xml * /hbase/trunk/src/docbkx/security.xml * /hbase/trunk/src/docbkx/shell.xml * /hbase/trunk/src/docbkx/troubleshooting.xml * /hbase/trunk/src/docbkx/zookeeper.xml * /hbase/trunk/src/site/site.vm * /hbase/trunk/src/site/site.xml * /hbase/trunk/src/site/xdoc/acid-semantics.xml * /hbase/trunk/src/site/xdoc/bulk-loads.xml * /hbase/trunk/src/site/xdoc/cygwin.xml * /hbase/trunk/src/site/xdoc/index.xml * /hbase/trunk/src/site/xdoc/metrics.xml * /hbase/trunk/src/site/xdoc/old_news.xml * /hbase/trunk/src/site/xdoc/pseudo-distributed.xml * /hbase/trunk/src/site/xdoc/replication.xml * /hbase/trunk/src/site/xdoc/resources.xml * /hbase/trunk/src/site/xdoc/sponsors.xml > Have our website conform to apache branding requirements > > > Key: HBASE-7054 > URL: https://issues.apache.org/jira/browse/HBASE-7054 > Project: HBase > Issue Type: Task > Components: documentation >Reporter: stack >Assignee: stack > Attachments: 7054.txt > > > Make hbase.apache.org align with the apache branding requirements listed > here: http://www.apache.org/foundation/marks/pmcs -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7033) Add hbase.lru.blockcache.acceptable.factor to configuration, akin to the min.factor added by HBASE-6312
[ https://issues.apache.org/jira/browse/HBASE-7033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484597#comment-13484597 ] Hudson commented on HBASE-7033: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #237 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/237/]) HBASE-7033. Add hbase.lru.blockcache.acceptable.factor to configuration (Sergey Shelukhin) (Revision 1402314) Result = FAILURE apurtell : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java > Add hbase.lru.blockcache.acceptable.factor to configuration, akin to the > min.factor added by HBASE-6312 > --- > > Key: HBASE-7033 > URL: https://issues.apache.org/jira/browse/HBASE-7033 > Project: HBase > Issue Type: Improvement > Components: io >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Minor > Fix For: 0.96.0 > > Attachments: HBASE-7033.patch > > > Background: we want to make the change to block cache setting available on > 0.94 without actually changing the defaults as was done in HBASE-6312, as > this can be destabilizing. > Thus, both of these would be configurable instead of just one, and the user > would be able to switch to new values. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6324) Direct API calls from embedded Thrift server to regionserver
[ https://issues.apache.org/jira/browse/HBASE-6324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484580#comment-13484580 ] Sergey Shelukhin commented on HBASE-6324: - does it make sense to port this to 0.94? trunk uses protocol buffers so that'd require a brand new patch as far as I see, and with additional layer of indirection, unless protobufs allows some sort of "local" proxy. > Direct API calls from embedded Thrift server to regionserver > > > Key: HBASE-6324 > URL: https://issues.apache.org/jira/browse/HBASE-6324 > Project: HBase > Issue Type: Improvement >Reporter: Mikhail Bautin >Assignee: Mikhail Bautin > > When handling Thrift calls in the regionserver we should not go through RPC > to talk to the local regionserver. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7053) port blockcache configurability (part of HBASE-6312, and HBASE-7033) to 0.94
[ https://issues.apache.org/jira/browse/HBASE-7053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-7053: Attachment: HBASE-7053-v2-squashed.patch Updating patch > port blockcache configurability (part of HBASE-6312, and HBASE-7033) to 0.94 > - > > Key: HBASE-7053 > URL: https://issues.apache.org/jira/browse/HBASE-7053 > Project: HBase > Issue Type: Task >Affects Versions: 0.94.2 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Minor > Fix For: 0.94.3 > > Attachments: HBASE-7053.patch, HBASE-7053-v2-squashed.patch > > > Add an option to get the improvement from 6312 w/o changing the defaults in > stable release. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6651) Thread safety of HTablePool is doubtful
[ https://issues.apache.org/jira/browse/HBASE-6651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484577#comment-13484577 ] Ted Yu commented on HBASE-6651: --- It would be nice for another committer to check out this nice patch. > Thread safety of HTablePool is doubtful > --- > > Key: HBASE-6651 > URL: https://issues.apache.org/jira/browse/HBASE-6651 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.94.1 >Reporter: Hiroshi Ikeda >Priority: Minor > Attachments: HBASE-6651.patch, HBASE-6651-V2.patch, > HBASE-6651-V3.patch, HBASE-6651-V4.patch, sample.zip, sample.zip, > sharedmap_for_hbaseclient.zip > > > There are some operations in HTablePool to access to PoolMap in multiple > times without any explict synchronization. > For example HTablePool.closeTablePool() calles PoolMap.values(), and calles > PoolMap.remove(). If other threads add new instances to the pool in the > middle of the calls, the new added instances might be dropped. > (HTablePool.closeTablePool() also has another problem that calling it by > multple threads causes accessing HTable by multiple threads.) > Moreover, PoolMap is not thread safe for the same reason. > For example PoolMap.put() calles ConcurrentMap.get() and calles > ConcurrentMap.put(). If other threads add a new instance to the concurent map > in the middle of the calls, the new instance might be dropped. > And also implementations of Pool have the same problems. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6223) Document hbck improvements: HBASE-6173, HBASE-5360
[ https://issues.apache.org/jira/browse/HBASE-6223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-6223: --- Attachment: trunk-6223_v2.patch > Document hbck improvements: HBASE-6173, HBASE-5360 > --- > > Key: HBASE-6223 > URL: https://issues.apache.org/jira/browse/HBASE-6223 > Project: HBase > Issue Type: Task > Components: documentation, hbck >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Attachments: trunk-6223.patch, trunk-6223_v2.patch > > > We had a couple hbck improvements recently: HBASE-6173 and HBASE-5360. > We should document them. Especially, for HBASE-5360, it's something > one normally doesn't do. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6852) SchemaMetrics.updateOnCacheHit costs too much while full scanning a table with all of its fields
[ https://issues.apache.org/jira/browse/HBASE-6852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484551#comment-13484551 ] Elliott Clark commented on HBASE-6852: -- For 0.96 it's (hopefully) a non-issue. For 0.94 I think the perf gain might be worth applying this patch. > SchemaMetrics.updateOnCacheHit costs too much while full scanning a table > with all of its fields > > > Key: HBASE-6852 > URL: https://issues.apache.org/jira/browse/HBASE-6852 > Project: HBase > Issue Type: Improvement > Components: metrics >Affects Versions: 0.94.0 >Reporter: Cheng Hao >Priority: Minor > Labels: performance > Fix For: 0.94.3, 0.96.0 > > Attachments: metrics_hotspots.png, onhitcache-trunk.patch > > > The SchemaMetrics.updateOnCacheHit costs too much while I am doing the full > table scanning. > Here is the top 5 hotspots within regionserver while full scanning a table: > (Sorry for the less-well-format) > CPU: Intel Westmere microarchitecture, speed 2.262e+06 MHz (estimated) > Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit > mask of 0x00 (No unit mask) count 500 > samples %image name symbol name > --- > 9844713.4324 14033.jo void > org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics.updateOnCacheHit(org.apache.hadoop.hbase.io.hfile.BlockType$BlockCategory, > boolean) > 98447100.000 14033.jo void > org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics.updateOnCacheHit(org.apache.hadoop.hbase.io.hfile.BlockType$BlockCategory, > boolean) [self] > --- > 45814 6.2510 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(byte[], int, int, > byte[], int, int) > 45814100.000 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(byte[], int, int, > byte[], int, int) [self] > --- > 43523 5.9384 14033.jo boolean > org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(org.apache.hadoop.hbase.KeyValue) > 43523100.000 14033.jo boolean > org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(org.apache.hadoop.hbase.KeyValue) > [self] > --- > 42548 5.8054 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(byte[], int, int, > byte[], int, int) > 42548100.000 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(byte[], int, int, > byte[], int, int) [self] > --- > 40572 5.5358 14033.jo int > org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.binarySearchNonRootIndex(byte[], > int, int, java.nio.ByteBuffer, org.apache.hadoop.io.RawComparator)~1 > 40572100.000 14033.jo int > org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.binarySearchNonRootIndex(byte[], > int, int, java.nio.ByteBuffer, org.apache.hadoop.io.RawComparator)~1 [self] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for block cache lock
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484549#comment-13484549 ] Lars Hofhansl commented on HBASE-5898: -- Test failure is suspicious :) > Consider double-checked locking for block cache lock > > > Key: HBASE-5898 > URL: https://issues.apache.org/jira/browse/HBASE-5898 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 0.94.1 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Critical > Fix For: 0.94.3, 0.96.0 > > Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, > HBASE-5898-1.patch, hbase-5898.txt > > > Running a workload with a high query rate against a dataset that fits in > cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by > HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a > lot of CPU doing lock management here. I wrote a quick patch to switch to a > double-checked locking and it improved throughput substantially for this > workload. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7053) port blockcache configurability (part of HBASE-6312, and HBASE-7033) to 0.94
[
https://issues.apache.org/jira/browse/HBASE-7053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484547#comment-13484547
]
Lars Hofhansl commented on HBASE-7053:
--
Patch looks good. Thanks Sergey.
Can we get rid of the {{LruBlockCache(long maxSize, long blockSize)}} and
{{LruBlockCache(long maxSize, long blockSize, boolean evictionThread)}}
constructors? They are now only used from tests where we could just pass an
empty Configuration object.
> port blockcache configurability (part of HBASE-6312, and HBASE-7033) to 0.94
> -
>
> Key: HBASE-7053
> URL: https://issues.apache.org/jira/browse/HBASE-7053
> Project: HBase
> Issue Type: Task
>Affects Versions: 0.94.2
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Minor
> Fix For: 0.94.3
>
> Attachments: HBASE-7053.patch
>
>
> Add an option to get the improvement from 6312 w/o changing the defaults in
> stable release.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7053) port blockcache configurability (part of HBASE-6312, and HBASE-7033) to 0.94
[ https://issues.apache.org/jira/browse/HBASE-7053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-7053: - Status: Open (was: Patch Available) > port blockcache configurability (part of HBASE-6312, and HBASE-7033) to 0.94 > - > > Key: HBASE-7053 > URL: https://issues.apache.org/jira/browse/HBASE-7053 > Project: HBase > Issue Type: Task >Affects Versions: 0.94.2 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Minor > Fix For: 0.94.3 > > Attachments: HBASE-7053.patch > > > Add an option to get the improvement from 6312 w/o changing the defaults in > stable release. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for block cache lock
[
https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484544#comment-13484544
]
Hadoop QA commented on HBASE-5898:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12550877/HBASE-5898-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
85 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 3 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestStoreFile
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3152//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3152//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3152//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3152//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3152//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3152//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3152//console
This message is automatically generated.
> Consider double-checked locking for block cache lock
>
>
> Key: HBASE-5898
> URL: https://issues.apache.org/jira/browse/HBASE-5898
> Project: HBase
> Issue Type: Improvement
> Components: Performance
>Affects Versions: 0.94.1
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch,
> HBASE-5898-1.patch, hbase-5898.txt
>
>
> Running a workload with a high query rate against a dataset that fits in
> cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by
> HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a
> lot of CPU doing lock management here. I wrote a quick patch to switch to a
> double-checked locking and it improved throughput substantially for this
> workload.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6852) SchemaMetrics.updateOnCacheHit costs too much while full scanning a table with all of its fields
[ https://issues.apache.org/jira/browse/HBASE-6852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484543#comment-13484543 ] Lars Hofhansl commented on HBASE-6852: -- So we should probably not consider this for 0.94 then. And for 0.96 this is a non-issue, right? > SchemaMetrics.updateOnCacheHit costs too much while full scanning a table > with all of its fields > > > Key: HBASE-6852 > URL: https://issues.apache.org/jira/browse/HBASE-6852 > Project: HBase > Issue Type: Improvement > Components: metrics >Affects Versions: 0.94.0 >Reporter: Cheng Hao >Priority: Minor > Labels: performance > Fix For: 0.94.3, 0.96.0 > > Attachments: metrics_hotspots.png, onhitcache-trunk.patch > > > The SchemaMetrics.updateOnCacheHit costs too much while I am doing the full > table scanning. > Here is the top 5 hotspots within regionserver while full scanning a table: > (Sorry for the less-well-format) > CPU: Intel Westmere microarchitecture, speed 2.262e+06 MHz (estimated) > Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit > mask of 0x00 (No unit mask) count 500 > samples %image name symbol name > --- > 9844713.4324 14033.jo void > org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics.updateOnCacheHit(org.apache.hadoop.hbase.io.hfile.BlockType$BlockCategory, > boolean) > 98447100.000 14033.jo void > org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics.updateOnCacheHit(org.apache.hadoop.hbase.io.hfile.BlockType$BlockCategory, > boolean) [self] > --- > 45814 6.2510 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(byte[], int, int, > byte[], int, int) > 45814100.000 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(byte[], int, int, > byte[], int, int) [self] > --- > 43523 5.9384 14033.jo boolean > org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(org.apache.hadoop.hbase.KeyValue) > 43523100.000 14033.jo boolean > org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(org.apache.hadoop.hbase.KeyValue) > [self] > --- > 42548 5.8054 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(byte[], int, int, > byte[], int, int) > 42548100.000 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(byte[], int, int, > byte[], int, int) [self] > --- > 40572 5.5358 14033.jo int > org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.binarySearchNonRootIndex(byte[], > int, int, java.nio.ByteBuffer, org.apache.hadoop.io.RawComparator)~1 > 40572100.000 14033.jo int > org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.binarySearchNonRootIndex(byte[], > int, int, java.nio.ByteBuffer, org.apache.hadoop.io.RawComparator)~1 [self] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-5136) Redundant MonitoredTask instances in case of distributed log splitting retry
[
https://issues.apache.org/jira/browse/HBASE-5136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu resolved HBASE-5136.
---
Resolution: Duplicate
Manage Attachments dialog reverses the order of the two patches, compared to
the order on JIRA page :-)
> Redundant MonitoredTask instances in case of distributed log splitting retry
>
>
> Key: HBASE-5136
> URL: https://issues.apache.org/jira/browse/HBASE-5136
> Project: HBase
> Issue Type: Task
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 5136-trunk.patch, 5136.txt
>
>
> In case of log splitting retry, the following code would be executed multiple
> times:
> {code}
> public long splitLogDistributed(final List logDirs) throws
> IOException {
> MonitoredTask status = TaskMonitor.get().createStatus(
> "Doing distributed log split in " + logDirs);
> {code}
> leading to multiple MonitoredTask instances.
> User may get confused by multiple distributed log splitting entries for the
> same region server on master UI
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5136) Redundant MonitoredTask instances in case of distributed log splitting retry
[
https://issues.apache.org/jira/browse/HBASE-5136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484537#comment-13484537
]
Jimmy Xiang commented on HBASE-5136:
@Ted, were you referring to 5136.txt? That's your patch. My patch is
5136-trunk.patch, which is already committed in HBASE-6357.
I think we can close this one as a duplicate.
> Redundant MonitoredTask instances in case of distributed log splitting retry
>
>
> Key: HBASE-5136
> URL: https://issues.apache.org/jira/browse/HBASE-5136
> Project: HBase
> Issue Type: Task
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 5136-trunk.patch, 5136.txt
>
>
> In case of log splitting retry, the following code would be executed multiple
> times:
> {code}
> public long splitLogDistributed(final List logDirs) throws
> IOException {
> MonitoredTask status = TaskMonitor.get().createStatus(
> "Doing distributed log split in " + logDirs);
> {code}
> leading to multiple MonitoredTask instances.
> User may get confused by multiple distributed log splitting entries for the
> same region server on master UI
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5898) Consider double-checked locking for block cache lock
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-5898: - Status: Patch Available (was: Open) patch without the counter. > Consider double-checked locking for block cache lock > > > Key: HBASE-5898 > URL: https://issues.apache.org/jira/browse/HBASE-5898 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 0.94.1 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Critical > Fix For: 0.94.3, 0.96.0 > > Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, > HBASE-5898-1.patch, hbase-5898.txt > > > Running a workload with a high query rate against a dataset that fits in > cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by > HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a > lot of CPU doing lock management here. I wrote a quick patch to switch to a > double-checked locking and it improved throughput substantially for this > workload. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6223) Document hbck improvements: HBASE-6173, HBASE-5360
[
https://issues.apache.org/jira/browse/HBASE-6223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484529#comment-13484529
]
Hadoop QA commented on HBASE-6223:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12550864/trunk-6223.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation patch that doesn't require tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
85 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 3 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3150//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3150//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3150//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3150//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3150//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3150//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3150//console
This message is automatically generated.
> Document hbck improvements: HBASE-6173, HBASE-5360
> ---
>
> Key: HBASE-6223
> URL: https://issues.apache.org/jira/browse/HBASE-6223
> Project: HBase
> Issue Type: Task
> Components: documentation, hbck
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
> Attachments: trunk-6223.patch
>
>
> We had a couple hbck improvements recently: HBASE-6173 and HBASE-5360.
> We should document them. Especially, for HBASE-5360, it's something
> one normally doesn't do.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7033) Add hbase.lru.blockcache.acceptable.factor to configuration, akin to the min.factor added by HBASE-6312
[ https://issues.apache.org/jira/browse/HBASE-7033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484525#comment-13484525 ] Hudson commented on HBASE-7033: --- Integrated in HBase-TRUNK #3487 (See [https://builds.apache.org/job/HBase-TRUNK/3487/]) HBASE-7033. Add hbase.lru.blockcache.acceptable.factor to configuration (Sergey Shelukhin) (Revision 1402314) Result = FAILURE apurtell : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java > Add hbase.lru.blockcache.acceptable.factor to configuration, akin to the > min.factor added by HBASE-6312 > --- > > Key: HBASE-7033 > URL: https://issues.apache.org/jira/browse/HBASE-7033 > Project: HBase > Issue Type: Improvement > Components: io >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Minor > Fix For: 0.96.0 > > Attachments: HBASE-7033.patch > > > Background: we want to make the change to block cache setting available on > 0.94 without actually changing the defaults as was done in HBASE-6312, as > this can be destabilizing. > Thus, both of these would be configurable instead of just one, and the user > would be able to switch to new values. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7054) Have our website conform to apache branding requirements
[ https://issues.apache.org/jira/browse/HBASE-7054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484526#comment-13484526 ] Hudson commented on HBASE-7054: --- Integrated in HBase-TRUNK #3487 (See [https://builds.apache.org/job/HBase-TRUNK/3487/]) HBASE-7054 Have our website conform to apache branding requirements (Revision 1402324) Result = FAILURE stack : Files : * /hbase/trunk/pom.xml * /hbase/trunk/src/docbkx/book.xml * /hbase/trunk/src/docbkx/case_studies.xml * /hbase/trunk/src/docbkx/community.xml * /hbase/trunk/src/docbkx/configuration.xml * /hbase/trunk/src/docbkx/developer.xml * /hbase/trunk/src/docbkx/external_apis.xml * /hbase/trunk/src/docbkx/ops_mgt.xml * /hbase/trunk/src/docbkx/performance.xml * /hbase/trunk/src/docbkx/security.xml * /hbase/trunk/src/docbkx/shell.xml * /hbase/trunk/src/docbkx/troubleshooting.xml * /hbase/trunk/src/docbkx/zookeeper.xml * /hbase/trunk/src/site/site.vm * /hbase/trunk/src/site/site.xml * /hbase/trunk/src/site/xdoc/acid-semantics.xml * /hbase/trunk/src/site/xdoc/bulk-loads.xml * /hbase/trunk/src/site/xdoc/cygwin.xml * /hbase/trunk/src/site/xdoc/index.xml * /hbase/trunk/src/site/xdoc/metrics.xml * /hbase/trunk/src/site/xdoc/old_news.xml * /hbase/trunk/src/site/xdoc/pseudo-distributed.xml * /hbase/trunk/src/site/xdoc/replication.xml * /hbase/trunk/src/site/xdoc/resources.xml * /hbase/trunk/src/site/xdoc/sponsors.xml > Have our website conform to apache branding requirements > > > Key: HBASE-7054 > URL: https://issues.apache.org/jira/browse/HBASE-7054 > Project: HBase > Issue Type: Task > Components: documentation >Reporter: stack >Assignee: stack > Attachments: 7054.txt > > > Make hbase.apache.org align with the apache branding requirements listed > here: http://www.apache.org/foundation/marks/pmcs -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5898) Consider double-checked locking for block cache lock
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-5898: - Attachment: HBASE-5898-1.patch Good catch on that. Here's a patch without the counting. > Consider double-checked locking for block cache lock > > > Key: HBASE-5898 > URL: https://issues.apache.org/jira/browse/HBASE-5898 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 0.94.1 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Critical > Fix For: 0.94.3, 0.96.0 > > Attachments: 5898-TestBlocksRead.txt, HBASE-5898-0.patch, > HBASE-5898-1.patch, hbase-5898.txt > > > Running a workload with a high query rate against a dataset that fits in > cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by > HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a > lot of CPU doing lock management here. I wrote a quick patch to switch to a > double-checked locking and it improved throughput substantially for this > workload. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5136) Redundant MonitoredTask instances in case of distributed log splitting retry
[
https://issues.apache.org/jira/browse/HBASE-5136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484518#comment-13484518
]
Ted Yu commented on HBASE-5136:
---
For Jimmy's patch:
{code}
+Integer logDirsHash = logDirs.hashCode();
{code}
What if there is hash collision between two log directories ?
> Redundant MonitoredTask instances in case of distributed log splitting retry
>
>
> Key: HBASE-5136
> URL: https://issues.apache.org/jira/browse/HBASE-5136
> Project: HBase
> Issue Type: Task
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 5136-trunk.patch, 5136.txt
>
>
> In case of log splitting retry, the following code would be executed multiple
> times:
> {code}
> public long splitLogDistributed(final List logDirs) throws
> IOException {
> MonitoredTask status = TaskMonitor.get().createStatus(
> "Doing distributed log split in " + logDirs);
> {code}
> leading to multiple MonitoredTask instances.
> User may get confused by multiple distributed log splitting entries for the
> same region server on master UI
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7055) port HBASE-6371 tier-based compaction from 0.89-fb to trunk
Sergey Shelukhin created HBASE-7055: --- Summary: port HBASE-6371 tier-based compaction from 0.89-fb to trunk Key: HBASE-7055 URL: https://issues.apache.org/jira/browse/HBASE-7055 Project: HBase Issue Type: Task Affects Versions: 0.96.0 Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin There's divergence in the code :( See HBASE-6371 for details. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6597) Block Encoding Size Estimation
[ https://issues.apache.org/jira/browse/HBASE-6597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-6597: -- Attachment: 6597-trunk.txt Patch that compiles against trunk. There're test failures. > Block Encoding Size Estimation > -- > > Key: HBASE-6597 > URL: https://issues.apache.org/jira/browse/HBASE-6597 > Project: HBase > Issue Type: Improvement > Components: io >Affects Versions: 0.89-fb >Reporter: Brian Nixon >Assignee: Mikhail Bautin >Priority: Minor > Attachments: 6597-trunk.txt, D5895.1.patch, D5895.2.patch, > D5895.3.patch, D5895.4.patch, D5895.5.patch > > > Blocks boundaries as created by current writers are determined by the size of > the unencoded data. However, blocks in memory are kept encoded. By using an > estimate for the encoded size of the block, we can get greater consistency in > size. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6597) Block Encoding Size Estimation
[ https://issues.apache.org/jira/browse/HBASE-6597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-6597: -- Attachment: (was: 6597-trunk.txt) > Block Encoding Size Estimation > -- > > Key: HBASE-6597 > URL: https://issues.apache.org/jira/browse/HBASE-6597 > Project: HBase > Issue Type: Improvement > Components: io >Affects Versions: 0.89-fb >Reporter: Brian Nixon >Assignee: Mikhail Bautin >Priority: Minor > Attachments: D5895.1.patch, D5895.2.patch, D5895.3.patch, > D5895.4.patch, D5895.5.patch > > > Blocks boundaries as created by current writers are determined by the size of > the unencoded data. However, blocks in memory are kept encoded. By using an > estimate for the encoded size of the block, we can get greater consistency in > size. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7053) port blockcache configurability (part of HBASE-6312, and HBASE-7033) to 0.94
[
https://issues.apache.org/jira/browse/HBASE-7053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484500#comment-13484500
]
Hadoop QA commented on HBASE-7053:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12550865/HBASE-7053.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3151//console
This message is automatically generated.
> port blockcache configurability (part of HBASE-6312, and HBASE-7033) to 0.94
> -
>
> Key: HBASE-7053
> URL: https://issues.apache.org/jira/browse/HBASE-7053
> Project: HBase
> Issue Type: Task
>Affects Versions: 0.94.2
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Minor
> Fix For: 0.94.3
>
> Attachments: HBASE-7053.patch
>
>
> Add an option to get the improvement from 6312 w/o changing the defaults in
> stable release.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5416) Improve performance of scans with some kind of filters.
[ https://issues.apache.org/jira/browse/HBASE-5416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484497#comment-13484497 ] Sergey Shelukhin commented on HBASE-5416: - Hi. Should this be ok to commit to trunk? Thanks. > Improve performance of scans with some kind of filters. > --- > > Key: HBASE-5416 > URL: https://issues.apache.org/jira/browse/HBASE-5416 > Project: HBase > Issue Type: Improvement > Components: Filters, Performance, regionserver >Affects Versions: 0.90.4 >Reporter: Max Lapan >Assignee: Max Lapan > Fix For: 0.96.0 > > Attachments: 5416-Filtered_scans_v6.patch, 5416-v5.txt, 5416-v6.txt, > Filtered_scans.patch, Filtered_scans_v2.patch, Filtered_scans_v3.patch, > Filtered_scans_v4.patch, Filtered_scans_v5.1.patch, Filtered_scans_v5.patch, > Filtered_scans_v7.patch > > > When the scan is performed, whole row is loaded into result list, after that > filter (if exists) is applied to detect that row is needed. > But when scan is performed on several CFs and filter checks only data from > the subset of these CFs, data from CFs, not checked by a filter is not needed > on a filter stage. Only when we decided to include current row. And in such > case we can significantly reduce amount of IO performed by a scan, by loading > only values, actually checked by a filter. > For example, we have two CFs: flags and snap. Flags is quite small (bunch of > megabytes) and is used to filter large entries from snap. Snap is very large > (10s of GB) and it is quite costly to scan it. If we needed only rows with > some flag specified, we use SingleColumnValueFilter to limit result to only > small subset of region. But current implementation is loading both CFs to > perform scan, when only small subset is needed. > Attached patch adds one routine to Filter interface to allow filter to > specify which CF is needed to it's operation. In HRegion, we separate all > scanners into two groups: needed for filter and the rest (joined). When new > row is considered, only needed data is loaded, filter applied, and only if > filter accepts the row, rest of data is loaded. At our data, this speeds up > such kind of scans 30-50 times. Also, this gives us the way to better > normalize the data into separate columns by optimizing the scans performed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5136) Redundant MonitoredTask instances in case of distributed log splitting retry
[
https://issues.apache.org/jira/browse/HBASE-5136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484496#comment-13484496
]
Sergey Shelukhin commented on HBASE-5136:
-
Hi. Should this be ok to commit?
> Redundant MonitoredTask instances in case of distributed log splitting retry
>
>
> Key: HBASE-5136
> URL: https://issues.apache.org/jira/browse/HBASE-5136
> Project: HBase
> Issue Type: Task
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 5136-trunk.patch, 5136.txt
>
>
> In case of log splitting retry, the following code would be executed multiple
> times:
> {code}
> public long splitLogDistributed(final List logDirs) throws
> IOException {
> MonitoredTask status = TaskMonitor.get().createStatus(
> "Doing distributed log split in " + logDirs);
> {code}
> leading to multiple MonitoredTask instances.
> User may get confused by multiple distributed log splitting entries for the
> same region server on master UI
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7053) port blockcache configurability (part of HBASE-6312, and HBASE-7033) to 0.94
[ https://issues.apache.org/jira/browse/HBASE-7053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-7053: Attachment: HBASE-7053.patch > port blockcache configurability (part of HBASE-6312, and HBASE-7033) to 0.94 > - > > Key: HBASE-7053 > URL: https://issues.apache.org/jira/browse/HBASE-7053 > Project: HBase > Issue Type: Task >Affects Versions: 0.94.2 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Minor > Fix For: 0.94.3 > > Attachments: HBASE-7053.patch > > > Add an option to get the improvement from 6312 w/o changing the defaults in > stable release. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7053) port blockcache configurability (part of HBASE-6312, and HBASE-7033) to 0.94
[ https://issues.apache.org/jira/browse/HBASE-7053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-7053: Fix Version/s: 0.94.3 Affects Version/s: 0.94.2 Status: Patch Available (was: Open) > port blockcache configurability (part of HBASE-6312, and HBASE-7033) to 0.94 > - > > Key: HBASE-7053 > URL: https://issues.apache.org/jira/browse/HBASE-7053 > Project: HBase > Issue Type: Task >Affects Versions: 0.94.2 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Minor > Fix For: 0.94.3 > > Attachments: HBASE-7053.patch > > > Add an option to get the improvement from 6312 w/o changing the defaults in > stable release. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6852) SchemaMetrics.updateOnCacheHit costs too much while full scanning a table with all of its fields
[ https://issues.apache.org/jira/browse/HBASE-6852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484491#comment-13484491 ] Elliott Clark commented on HBASE-6852: -- HBASE-6410 removes most of the calls to concurrent hash maps. And it starts using the high scalability counter class. > SchemaMetrics.updateOnCacheHit costs too much while full scanning a table > with all of its fields > > > Key: HBASE-6852 > URL: https://issues.apache.org/jira/browse/HBASE-6852 > Project: HBase > Issue Type: Improvement > Components: metrics >Affects Versions: 0.94.0 >Reporter: Cheng Hao >Priority: Minor > Labels: performance > Fix For: 0.94.3, 0.96.0 > > Attachments: metrics_hotspots.png, onhitcache-trunk.patch > > > The SchemaMetrics.updateOnCacheHit costs too much while I am doing the full > table scanning. > Here is the top 5 hotspots within regionserver while full scanning a table: > (Sorry for the less-well-format) > CPU: Intel Westmere microarchitecture, speed 2.262e+06 MHz (estimated) > Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit > mask of 0x00 (No unit mask) count 500 > samples %image name symbol name > --- > 9844713.4324 14033.jo void > org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics.updateOnCacheHit(org.apache.hadoop.hbase.io.hfile.BlockType$BlockCategory, > boolean) > 98447100.000 14033.jo void > org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics.updateOnCacheHit(org.apache.hadoop.hbase.io.hfile.BlockType$BlockCategory, > boolean) [self] > --- > 45814 6.2510 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(byte[], int, int, > byte[], int, int) > 45814100.000 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(byte[], int, int, > byte[], int, int) [self] > --- > 43523 5.9384 14033.jo boolean > org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(org.apache.hadoop.hbase.KeyValue) > 43523100.000 14033.jo boolean > org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(org.apache.hadoop.hbase.KeyValue) > [self] > --- > 42548 5.8054 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(byte[], int, int, > byte[], int, int) > 42548100.000 14033.jo int > org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(byte[], int, int, > byte[], int, int) [self] > --- > 40572 5.5358 14033.jo int > org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.binarySearchNonRootIndex(byte[], > int, int, java.nio.ByteBuffer, org.apache.hadoop.io.RawComparator)~1 > 40572100.000 14033.jo int > org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.binarySearchNonRootIndex(byte[], > int, int, java.nio.ByteBuffer, org.apache.hadoop.io.RawComparator)~1 [self] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6223) Document hbck improvements: HBASE-6173, HBASE-5360
[ https://issues.apache.org/jira/browse/HBASE-6223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-6223: --- Status: Patch Available (was: Open) > Document hbck improvements: HBASE-6173, HBASE-5360 > --- > > Key: HBASE-6223 > URL: https://issues.apache.org/jira/browse/HBASE-6223 > Project: HBase > Issue Type: Task > Components: documentation, hbck >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Attachments: trunk-6223.patch > > > We had a couple hbck improvements recently: HBASE-6173 and HBASE-5360. > We should document them. Especially, for HBASE-5360, it's something > one normally doesn't do. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6223) Document hbck improvements: HBASE-6173, HBASE-5360
[ https://issues.apache.org/jira/browse/HBASE-6223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-6223: --- Attachment: trunk-6223.patch > Document hbck improvements: HBASE-6173, HBASE-5360 > --- > > Key: HBASE-6223 > URL: https://issues.apache.org/jira/browse/HBASE-6223 > Project: HBase > Issue Type: Task > Components: documentation, hbck >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Attachments: trunk-6223.patch > > > We had a couple hbck improvements recently: HBASE-6173 and HBASE-5360. > We should document them. Especially, for HBASE-5360, it's something > one normally doesn't do. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-6223) Document hbck improvements: HBASE-6173, HBASE-5360
[ https://issues.apache.org/jira/browse/HBASE-6223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang reassigned HBASE-6223: -- Assignee: Jimmy Xiang > Document hbck improvements: HBASE-6173, HBASE-5360 > --- > > Key: HBASE-6223 > URL: https://issues.apache.org/jira/browse/HBASE-6223 > Project: HBase > Issue Type: Task > Components: documentation, hbck >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Attachments: trunk-6223.patch > > > We had a couple hbck improvements recently: HBASE-6173 and HBASE-5360. > We should document them. Especially, for HBASE-5360, it's something > one normally doesn't do. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7053) port blockcache configurability (part of HBASE-6312, and HBASE-7033) to 0.94
[ https://issues.apache.org/jira/browse/HBASE-7053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-7053: Description: Add an option to get the improvement from 6312 w/o changing the defaults in stable release. > port blockcache configurability (part of HBASE-6312, and HBASE-7033) to 0.94 > - > > Key: HBASE-7053 > URL: https://issues.apache.org/jira/browse/HBASE-7053 > Project: HBase > Issue Type: Task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Minor > > Add an option to get the improvement from 6312 w/o changing the defaults in > stable release. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7054) Have our website conform to apache branding requirements
[ https://issues.apache.org/jira/browse/HBASE-7054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-7054: - Attachment: 7054.txt Patch applied. Based on work Andrew passed me. > Have our website conform to apache branding requirements > > > Key: HBASE-7054 > URL: https://issues.apache.org/jira/browse/HBASE-7054 > Project: HBase > Issue Type: Task > Components: documentation >Reporter: stack >Assignee: stack > Attachments: 7054.txt > > > Make hbase.apache.org align with the apache branding requirements listed > here: http://www.apache.org/foundation/marks/pmcs -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-7054) Have our website conform to apache branding requirements
[ https://issues.apache.org/jira/browse/HBASE-7054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-7054. -- Resolution: Fixed Committed. Will push up the website now. > Have our website conform to apache branding requirements > > > Key: HBASE-7054 > URL: https://issues.apache.org/jira/browse/HBASE-7054 > Project: HBase > Issue Type: Task > Components: documentation >Reporter: stack >Assignee: stack > Attachments: 7054.txt > > > Make hbase.apache.org align with the apache branding requirements listed > here: http://www.apache.org/foundation/marks/pmcs -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5257) Allow INCLUDE_AND_NEXT_COL in filters and use it in ColumnPaginationFilter
[
https://issues.apache.org/jira/browse/HBASE-5257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484467#comment-13484467
]
Hudson commented on HBASE-5257:
---
Integrated in HBase-0.92 #600 (See
[https://builds.apache.org/job/HBase-0.92/600/])
HBASE-5257 Addendum fixes typo in ColumnCountGetFilter.java (Revision
1402225)
Result = FAILURE
tedyu :
Files :
*
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/filter/ColumnCountGetFilter.java
> Allow INCLUDE_AND_NEXT_COL in filters and use it in ColumnPaginationFilter
> --
>
> Key: HBASE-5257
> URL: https://issues.apache.org/jira/browse/HBASE-5257
> Project: HBase
> Issue Type: Improvement
>Reporter: Lars Hofhansl
>Assignee: Varun Sharma
> Fix For: 0.92.3, 0.94.3, 0.96.0
>
> Attachments: 5257-0.92.addendum, 5257-trunk.txt, 5257-trunk-v2.txt,
> HBASE-5257-0.92.txt, HBASE-5257-0.94.txt
>
>
> There are various usecases and filter types where evaluating the filter
> before version are handled either do not make sense, or make filter handling
> more complicated.
> Also see this comment in ScanQueryMatcher:
> {code}
> /**
> * Filters should be checked before checking column trackers. If we do
> * otherwise, as was previously being done, ColumnTracker may increment
> its
> * counter for even that KV which may be discarded later on by Filter.
> This
> * would lead to incorrect results in certain cases.
> */
> {code}
> So we had Filters after the column trackers (which do the version checking),
> and then moved it.
> Should be at the discretion of the Filter.
> Could either add a new method to FilterBase (maybe excludeVersions() or
> something). Or have a new Filter wrapper (like WhileMatchFilter), that should
> only be used as outmost filter and indicates the same (maybe
> ExcludeVersionsFilter).
> See latest comments on HBASE-5229 for motivation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7054) Have our website conform to apache branding requirements
stack created HBASE-7054: Summary: Have our website conform to apache branding requirements Key: HBASE-7054 URL: https://issues.apache.org/jira/browse/HBASE-7054 Project: HBase Issue Type: Task Components: documentation Reporter: stack Assignee: stack Make hbase.apache.org align with the apache branding requirements listed here: http://www.apache.org/foundation/marks/pmcs -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7053) port blockcache configurability (part of HBASE-6312, and HBASE-7033) to 0.94
Sergey Shelukhin created HBASE-7053: --- Summary: port blockcache configurability (part of HBASE-6312, and HBASE-7033) to 0.94 Key: HBASE-7053 URL: https://issues.apache.org/jira/browse/HBASE-7053 Project: HBase Issue Type: Task Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Priority: Minor -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7033) Add hbase.lru.blockcache.acceptable.factor to configuration, akin to the min.factor added by HBASE-6312
[ https://issues.apache.org/jira/browse/HBASE-7033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-7033: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to trunk. Thanks for the patch Sergey! > Add hbase.lru.blockcache.acceptable.factor to configuration, akin to the > min.factor added by HBASE-6312 > --- > > Key: HBASE-7033 > URL: https://issues.apache.org/jira/browse/HBASE-7033 > Project: HBase > Issue Type: Improvement > Components: io >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Minor > Fix For: 0.96.0 > > Attachments: HBASE-7033.patch > > > Background: we want to make the change to block cache setting available on > 0.94 without actually changing the defaults as was done in HBASE-6312, as > this can be destabilizing. > Thus, both of these would be configurable instead of just one, and the user > would be able to switch to new values. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6410) Move RegionServer Metrics to metrics2
[
https://issues.apache.org/jira/browse/HBASE-6410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484447#comment-13484447
]
Hadoop QA commented on HBASE-6410:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12550834/HBASE-6410-5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 84 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
87 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 11 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.io.hfile.TestHFileBlock
org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3149//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3149//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3149//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3149//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3149//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3149//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3149//console
This message is automatically generated.
> Move RegionServer Metrics to metrics2
> -
>
> Key: HBASE-6410
> URL: https://issues.apache.org/jira/browse/HBASE-6410
> Project: HBase
> Issue Type: Sub-task
> Components: metrics
>Affects Versions: 0.96.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
>Priority: Blocker
> Attachments: HBASE-6410-1.patch, HBASE-6410-2.patch,
> HBASE-6410-3.patch, HBASE-6410-4.patch, HBASE-6410-5.patch, HBASE-6410.patch
>
>
> Move RegionServer Metrics to metrics2
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6597) Block Encoding Size Estimation
[ https://issues.apache.org/jira/browse/HBASE-6597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484441#comment-13484441 ] Ted Yu commented on HBASE-6597: --- I am planning to continue the work. > Block Encoding Size Estimation > -- > > Key: HBASE-6597 > URL: https://issues.apache.org/jira/browse/HBASE-6597 > Project: HBase > Issue Type: Improvement > Components: io >Affects Versions: 0.89-fb >Reporter: Brian Nixon >Assignee: Mikhail Bautin >Priority: Minor > Attachments: 6597-trunk.txt, D5895.1.patch, D5895.2.patch, > D5895.3.patch, D5895.4.patch, D5895.5.patch > > > Blocks boundaries as created by current writers are determined by the size of > the unencoded data. However, blocks in memory are kept encoded. By using an > estimate for the encoded size of the block, we can get greater consistency in > size. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7033) Add hbase.lru.blockcache.acceptable.factor to configuration, akin to the min.factor added by HBASE-6312
[ https://issues.apache.org/jira/browse/HBASE-7033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484439#comment-13484439 ] Andrew Purtell commented on HBASE-7033: --- +1 for commit. Trivial change, looks clean. Going to just go ahead and do this in a few minutes. > Add hbase.lru.blockcache.acceptable.factor to configuration, akin to the > min.factor added by HBASE-6312 > --- > > Key: HBASE-7033 > URL: https://issues.apache.org/jira/browse/HBASE-7033 > Project: HBase > Issue Type: Improvement > Components: io >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Minor > Fix For: 0.96.0 > > Attachments: HBASE-7033.patch > > > Background: we want to make the change to block cache setting available on > 0.94 without actually changing the defaults as was done in HBASE-6312, as > this can be destabilizing. > Thus, both of these would be configurable instead of just one, and the user > would be able to switch to new values. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6945) Compilation errors when using non-Sun JDKs to build HBase-0.94
[ https://issues.apache.org/jira/browse/HBASE-6945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484438#comment-13484438 ] Kumar Ravi commented on HBASE-6945: --- Stack, Hope my last comment addressed your questions and concerns. If you have further questions, please let me know. We would really like to be in a position to build and support HBase-0.94 built and run with IBM JDK, and we believe this solution supports multiple JDKs and JVMs. Thanks! > Compilation errors when using non-Sun JDKs to build HBase-0.94 > -- > > Key: HBASE-6945 > URL: https://issues.apache.org/jira/browse/HBASE-6945 > Project: HBase > Issue Type: Bug > Components: build >Affects Versions: 0.94.1 > Environment: RHEL 6.3, IBM Java 7 >Reporter: Kumar Ravi >Assignee: Kumar Ravi > Labels: patch > Fix For: 0.94.4 > > Attachments: ResourceCheckerJUnitListener_HBASE_6945-trunk.patch > > > When using IBM Java 7 to build HBase-0.94.1, the following comilation error > is seen. > [INFO] - > [ERROR] COMPILATION ERROR : > [INFO] - > [ERROR] > /home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[23,25] > error: package com.sun.management does not exist > [ERROR] > /home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[46,25] > error: cannot find symbol > [ERROR] symbol: class UnixOperatingSystemMXBean > location: class ResourceAnalyzer > /home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[75,29] > error: cannot find symbol > [ERROR] symbol: class UnixOperatingSystemMXBean > location: class ResourceAnalyzer > /home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[76,23] > error: cannot find symbol > [INFO] 4 errors > [INFO] - > [INFO] > > [INFO] BUILD FAILURE > [INFO] > > I have a patch available which should work for all JDKs including Sun. > I am in the process of testing this patch. Preliminary tests indicate the > build is working fine with this patch. I will post this patch when I am done > testing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6597) Block Encoding Size Estimation
[ https://issues.apache.org/jira/browse/HBASE-6597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484436#comment-13484436 ] Andrew Purtell commented on HBASE-6597: --- [~ted_yu] Are you planning to fix the compilation errors? > Block Encoding Size Estimation > -- > > Key: HBASE-6597 > URL: https://issues.apache.org/jira/browse/HBASE-6597 > Project: HBase > Issue Type: Improvement > Components: io >Affects Versions: 0.89-fb >Reporter: Brian Nixon >Assignee: Mikhail Bautin >Priority: Minor > Attachments: 6597-trunk.txt, D5895.1.patch, D5895.2.patch, > D5895.3.patch, D5895.4.patch, D5895.5.patch > > > Blocks boundaries as created by current writers are determined by the size of > the unencoded data. However, blocks in memory are kept encoded. By using an > estimate for the encoded size of the block, we can get greater consistency in > size. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6945) Compilation errors when using non-Sun JDKs to build HBase-0.94
[ https://issues.apache.org/jira/browse/HBASE-6945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484432#comment-13484432 ] Kumar Ravi commented on HBASE-6945: --- Thanks Lars, It would really be good to have this for 0.94. Without this patch, we cannot build HBASE-0.94 on IBM JDKs as the com.sun.management package is not available on the IBM JDK. > Compilation errors when using non-Sun JDKs to build HBase-0.94 > -- > > Key: HBASE-6945 > URL: https://issues.apache.org/jira/browse/HBASE-6945 > Project: HBase > Issue Type: Bug > Components: build >Affects Versions: 0.94.1 > Environment: RHEL 6.3, IBM Java 7 >Reporter: Kumar Ravi >Assignee: Kumar Ravi > Labels: patch > Fix For: 0.94.4 > > Attachments: ResourceCheckerJUnitListener_HBASE_6945-trunk.patch > > > When using IBM Java 7 to build HBase-0.94.1, the following comilation error > is seen. > [INFO] - > [ERROR] COMPILATION ERROR : > [INFO] - > [ERROR] > /home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[23,25] > error: package com.sun.management does not exist > [ERROR] > /home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[46,25] > error: cannot find symbol > [ERROR] symbol: class UnixOperatingSystemMXBean > location: class ResourceAnalyzer > /home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[75,29] > error: cannot find symbol > [ERROR] symbol: class UnixOperatingSystemMXBean > location: class ResourceAnalyzer > /home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[76,23] > error: cannot find symbol > [INFO] 4 errors > [INFO] - > [INFO] > > [INFO] BUILD FAILURE > [INFO] > > I have a patch available which should work for all JDKs including Sun. > I am in the process of testing this patch. Preliminary tests indicate the > build is working fine with this patch. I will post this patch when I am done > testing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6597) Block Encoding Size Estimation
[ https://issues.apache.org/jira/browse/HBASE-6597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-6597: -- Attachment: 6597-trunk.txt Patch for trunk. There are some compilation errors in tests. > Block Encoding Size Estimation > -- > > Key: HBASE-6597 > URL: https://issues.apache.org/jira/browse/HBASE-6597 > Project: HBase > Issue Type: Improvement > Components: io >Affects Versions: 0.89-fb >Reporter: Brian Nixon >Assignee: Mikhail Bautin >Priority: Minor > Attachments: 6597-trunk.txt, D5895.1.patch, D5895.2.patch, > D5895.3.patch, D5895.4.patch, D5895.5.patch > > > Blocks boundaries as created by current writers are determined by the size of > the unencoded data. However, blocks in memory are kept encoded. By using an > estimate for the encoded size of the block, we can get greater consistency in > size. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5974) Scanner retry behavior with RPC timeout on next() seems incorrect
[ https://issues.apache.org/jira/browse/HBASE-5974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484420#comment-13484420 ] Hudson commented on HBASE-5974: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #236 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/236/]) HBASE-5974 Scanner retry behavior with RPC timeout on next() seems incorrect (Revision 1402214) Result = FAILURE stack : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/OutOfOrderScannerNextException.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClientProtos.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/JVMClusterUtil.java * /hbase/trunk/hbase-server/src/main/protobuf/Client.proto * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestClientScannerRPCTimeout.java > Scanner retry behavior with RPC timeout on next() seems incorrect > - > > Key: HBASE-5974 > URL: https://issues.apache.org/jira/browse/HBASE-5974 > Project: HBase > Issue Type: Bug > Components: Client, regionserver >Affects Versions: 0.90.7, 0.92.1, 0.94.0, 0.96.0 >Reporter: Todd Lipcon >Assignee: Anoop Sam John >Priority: Critical > Fix For: 0.96.0 > > Attachments: 5974_94-V4.patch, 5974_trunk.patch, 5974_trunk-V2.patch, > 5974_trunk-V3.patch, HBASE-5974_0.94.patch, HBASE-5974_94-V2.patch, > HBASE-5974_94-V3.patch > > > I'm seeing the following behavior: > - set RPC timeout to a short value > - call next() for some batch of rows, big enough so the client times out > before the result is returned > - the HConnectionManager stuff will retry the next() call to the same server. > At this point, one of two things can happen: 1) the previous next() call will > still be processing, in which case you get a LeaseException, because it was > removed from the map during the processing, or 2) the next() call will > succeed but skip the prior batch of rows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7048) Regionsplitter requires the hadoop config path to be in hbase classpath
[
https://issues.apache.org/jira/browse/HBASE-7048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484421#comment-13484421
]
Hudson commented on HBASE-7048:
---
Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #236 (See
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/236/])
HBASE-7048 Regionsplitter requires the hadoop config path to be in hbase
classpath (Ming Ma) (Revision 1402197)
Result = FAILURE
tedyu :
Files :
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java
> Regionsplitter requires the hadoop config path to be in hbase classpath
> ---
>
> Key: HBASE-7048
> URL: https://issues.apache.org/jira/browse/HBASE-7048
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.92.2
>Reporter: Ted Yu
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 7048-0.94.patch, 7048-trunk.patch
>
>
> When hadoop config path isn't included in hbase classpath, you will get the
> following:
> {code}
> Exception in thread "main" java.lang.IllegalArgumentException: Wrong FS:
> hdfs://t3.e.com/hbase/usertable/_balancedSplit, expected: file:///
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:454)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:67)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:431)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:301)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1005)
> at
> org.apache.hadoop.hbase.util.RegionSplitter.getSplits(RegionSplitter.java:643)
> at
> org.apache.hadoop.hbase.util.RegionSplitter.rollingSplit(RegionSplitter.java:367)
> at org.apache.hadoop.hbase.util.RegionSplitter.main(RegionSplitter.java:295)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5257) Allow INCLUDE_AND_NEXT_COL in filters and use it in ColumnPaginationFilter
[
https://issues.apache.org/jira/browse/HBASE-5257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484418#comment-13484418
]
Hudson commented on HBASE-5257:
---
Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #236 (See
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/236/])
HBASE-5257 Allow INCLUDE_AND_NEXT_COL in filters and use it in
ColumnPaginationFilter (Varun) (Revision 1402209)
Result = FAILURE
tedyu :
Files :
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/filter/ColumnCountGetFilter.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/filter/ColumnPaginationFilter.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/filter/Filter.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/filter/FilterList.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestColumnPaginationFilter.java
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestFilter.java
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestFilterList.java
> Allow INCLUDE_AND_NEXT_COL in filters and use it in ColumnPaginationFilter
> --
>
> Key: HBASE-5257
> URL: https://issues.apache.org/jira/browse/HBASE-5257
> Project: HBase
> Issue Type: Improvement
>Reporter: Lars Hofhansl
>Assignee: Varun Sharma
> Fix For: 0.92.3, 0.94.3, 0.96.0
>
> Attachments: 5257-0.92.addendum, 5257-trunk.txt, 5257-trunk-v2.txt,
> HBASE-5257-0.92.txt, HBASE-5257-0.94.txt
>
>
> There are various usecases and filter types where evaluating the filter
> before version are handled either do not make sense, or make filter handling
> more complicated.
> Also see this comment in ScanQueryMatcher:
> {code}
> /**
> * Filters should be checked before checking column trackers. If we do
> * otherwise, as was previously being done, ColumnTracker may increment
> its
> * counter for even that KV which may be discarded later on by Filter.
> This
> * would lead to incorrect results in certain cases.
> */
> {code}
> So we had Filters after the column trackers (which do the version checking),
> and then moved it.
> Should be at the discretion of the Filter.
> Could either add a new method to FilterBase (maybe excludeVersions() or
> something). Or have a new Filter wrapper (like WhileMatchFilter), that should
> only be used as outmost filter and indicates the same (maybe
> ExcludeVersionsFilter).
> See latest comments on HBASE-5229 for motivation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6977) Multithread processing ZK assignment events
[ https://issues.apache.org/jira/browse/HBASE-6977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484419#comment-13484419 ] Hudson commented on HBASE-6977: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #236 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/236/]) HBASE-6977 Multithread processing ZK assignment events (Revision 1402226) Result = FAILURE jxiang : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/OfflineCallback.java > Multithread processing ZK assignment events > --- > > Key: HBASE-6977 > URL: https://issues.apache.org/jira/browse/HBASE-6977 > Project: HBase > Issue Type: Improvement > Components: Region Assignment >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Fix For: 0.96.0 > > Attachments: trunk-6977_v1.patch, trunk-6977_v2-1.patch, > trunk-6977_v3.patch > > > Related to HBASE-6976 and HBASE-6611. ZK events processing is a bottle neck > for assignments, since there is only one ZK event thread. If we can use > multiple threads, it should be better. > With multiple threads, the order of events could be messed up. However, if we > pass all events related to one region always to the same worker thread, the > order should be kept. > We need to play with it and find out how much performance imrovement we can > get. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6957) TestRowCounter consistently fails against hadoop-2.0
[
https://issues.apache.org/jira/browse/HBASE-6957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484417#comment-13484417
]
Hudson commented on HBASE-6957:
---
Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #236 (See
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/236/])
HBASE-6957 Upgrade to hadoop 2.0.2-alpha to see if TestRowCounter fails due
to division by zero (Revision 1402208)
Result = FAILURE
tedyu :
Files :
* /hbase/trunk/pom.xml
> TestRowCounter consistently fails against hadoop-2.0
>
>
> Key: HBASE-6957
> URL: https://issues.apache.org/jira/browse/HBASE-6957
> Project: HBase
> Issue Type: Sub-task
>Reporter: Ted Yu
>Priority: Critical
> Fix For: 0.96.0
>
> Attachments: 6957.txt
>
>
> In
> https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/210/testReport/org.apache.hadoop.hbase.mapreduce/TestRowCounter/testRowCounterHiddenColumn/
> , we can see:
> {code}
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:92)
> at org.junit.Assert.assertTrue(Assert.java:43)
> at org.junit.Assert.assertTrue(Assert.java:54)
> at
> org.apache.hadoop.hbase.mapreduce.TestRowCounter.runRowCount(TestRowCounter.java:135)
> at
> org.apache.hadoop.hbase.mapreduce.TestRowCounter.testRowCounterHiddenColumn(TestRowCounter.java:118)
> ...
> 2012-10-05 11:24:17,355 WARN [ContainersLauncher #1]
> launcher.ContainerLaunch(246): Failed to launch container.
> java.lang.ArithmeticException: / by zero
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:355)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
> at
> org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLogPathForWrite(LocalDirsHandlerService.java:268)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:126)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:68)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 2012-10-05 11:24:17,356 WARN [DeletionService #1]
> nodemanager.DefaultContainerExecutor(276): delete returned false for path:
> [/home/jenkins/jenkins-slave/workspace/HBase-TRUNK-on-Hadoop-2.0.0/trunk/hbase-server/target/org.apache.hadoop.mapred.MiniMRCluster/org.apache.hadoop.mapred.MiniMRCluster-localDir-nm-1_0/usercache/jenkins/appcache/application_1349436189156_0003/container_1349436189156_0003_01_02]
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
