[jira] [Updated] (HBASE-14479) Apply the Leader/Followers pattern to RpcServer's Reader
[
https://issues.apache.org/jira/browse/HBASE-14479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-14479:
--
Attachment: HBASE-14479-V2 (1).patch
Retry
> Apply the Leader/Followers pattern to RpcServer's Reader
>
>
> Key: HBASE-14479
> URL: https://issues.apache.org/jira/browse/HBASE-14479
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC, Performance
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Attachments: HBASE-14479-V2 (1).patch, HBASE-14479-V2.patch,
> HBASE-14479-V2.patch, HBASE-14479.patch
>
>
> {{RpcServer}} uses multiple selectors to read data for load distribution, but
> the distribution is just done by round-robin. It is uncertain, especially for
> long run, whether load is equally divided and resources are used without
> being wasted.
> Moreover, multiple selectors may cause excessive context switches which give
> priority to low latency (while we just add the requests to queues), and it is
> possible to reduce throughput of the whole server.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-14572) TestImportExport#testImport94Table can't find its src data file
[ https://issues.apache.org/jira/browse/HBASE-14572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-14572. --- Resolution: Fixed Assignee: stack Fix Version/s: 1.3.0 1.2.0 2.0.0 Pushed to branch-1.2+. > TestImportExport#testImport94Table can't find its src data file > --- > > Key: HBASE-14572 > URL: https://issues.apache.org/jira/browse/HBASE-14572 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack >Assignee: stack > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: 14572.txt > > > This test fails pretty frequently with this: > File > /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build%402/hbase/hbase-server/target/test-classes/org/apache/hadoop/hbase/mapreduce/exportedTableIn94Format > does not exist > ... as in here: > https://builds.apache.org/job/PreCommit-HBASE-Build/15890//testReport/org.apache.hadoop.hbase.mapreduce/TestImportExport/testImport94Table/ > It happened twice this evening. > The conversion from classpath URL to Hadoop local path is messing up, > probably stumbling over the character encoding above. > Let me change test so it doesn't fail if file can't be found... usually it > passes (it passes locally for instance). Unless... someone wants to work on > why we can't find the file -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14557) MapReduce WALPlayer issue with NoTagsKeyValue
[
https://issues.apache.org/jira/browse/HBASE-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946352#comment-14946352
]
ramkrishna.s.vasudevan commented on HBASE-14557:
bq.So, we back out the above?
NoTagsKeyValue can be used in the HFileReader path but in the WAL side may be
we not use it.
bq.To every KV? Lets not if we can avoid it.
Then on the WAL side we should allow the tag parsing to happening per KV.
> MapReduce WALPlayer issue with NoTagsKeyValue
> -
>
> Key: HBASE-14557
> URL: https://issues.apache.org/jira/browse/HBASE-14557
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Jerry He
>
> Running MapReduce WALPlayer to convert WAL info HFiles:
> {noformat}
> 15/10/05 20:28:08 INFO mapred.JobClient: Task Id :
> attempt_201508031611_0029_m_00_0, Status : FAILED
> java.io.IOException: Type mismatch in value from map: expected
> org.apache.hadoop.hbase.KeyValue, recieved
> org.apache.hadoop.hbase.NoTagsKeyValue
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:997)
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:689)
> at
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
> at
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALKeyValueMapper.map(WALPlayer.java:111)
> at
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALKeyValueMapper.map(WALPlayer.java:96)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:751)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:368)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at
> java.security.AccessController.doPrivileged(AccessController.java:369)
> at javax.security.auth.Subject.doAs(Subject.java:572)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1502)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14451) Move on to htrace-4.0.1 (from htrace-3.2.0)
[
https://issues.apache.org/jira/browse/HBASE-14451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946351#comment-14946351
]
Hadoop QA commented on HBASE-14451:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12765321/14451.v10.txt
against master branch at commit 8fd2d6507019f157427c388928d850a65076b9c0.
ATTACHMENT ID: 12765321
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 29 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15897//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15897//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15897//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15897//console
This message is automatically generated.
> Move on to htrace-4.0.1 (from htrace-3.2.0)
> ---
>
> Key: HBASE-14451
> URL: https://issues.apache.org/jira/browse/HBASE-14451
> Project: HBase
> Issue Type: Task
>Reporter: stack
>Assignee: stack
> Attachments: 14451.txt, 14451.v10.txt, 14451.v10.txt, 14451v2.txt,
> 14451v3.txt, 14451v4.txt, 14451v5.txt, 14451v6.txt, 14451v7.txt, 14451v8.txt,
> 14451v9.txt
>
>
> htrace-4.0.0 was just release with a new API. Get up on it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14572) TestImportExport#testImport94Table can't find its src data file
[ https://issues.apache.org/jira/browse/HBASE-14572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14572: -- Attachment: 14572.txt Just check if file can be found... the data file... and if not, just log and keep going. > TestImportExport#testImport94Table can't find its src data file > --- > > Key: HBASE-14572 > URL: https://issues.apache.org/jira/browse/HBASE-14572 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > Attachments: 14572.txt > > > This test fails pretty frequently with this: > File > /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build%402/hbase/hbase-server/target/test-classes/org/apache/hadoop/hbase/mapreduce/exportedTableIn94Format > does not exist > ... as in here: > https://builds.apache.org/job/PreCommit-HBASE-Build/15890//testReport/org.apache.hadoop.hbase.mapreduce/TestImportExport/testImport94Table/ > It happened twice this evening. > The conversion from classpath URL to Hadoop local path is messing up, > probably stumbling over the character encoding above. > Let me change test so it doesn't fail if file can't be found... usually it > passes (it passes locally for instance). Unless... someone wants to work on > why we can't find the file -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14572) TestImportExport#testImport94Table can't find its src data file
stack created HBASE-14572: - Summary: TestImportExport#testImport94Table can't find its src data file Key: HBASE-14572 URL: https://issues.apache.org/jira/browse/HBASE-14572 Project: HBase Issue Type: Sub-task Reporter: stack Attachments: 14572.txt This test fails pretty frequently with this: File /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build%402/hbase/hbase-server/target/test-classes/org/apache/hadoop/hbase/mapreduce/exportedTableIn94Format does not exist ... as in here: https://builds.apache.org/job/PreCommit-HBASE-Build/15890//testReport/org.apache.hadoop.hbase.mapreduce/TestImportExport/testImport94Table/ It happened twice this evening. The conversion from classpath URL to Hadoop local path is messing up, probably stumbling over the character encoding above. Let me change test so it doesn't fail if file can't be found... usually it passes (it passes locally for instance). Unless... someone wants to work on why we can't find the file -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14221) Reduce the number of time row comparison is done in a Scan
[
https://issues.apache.org/jira/browse/HBASE-14221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946344#comment-14946344
]
Hadoop QA commented on HBASE-14221:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12765329/14221-0.98-takeALook.txt
against 0.98 branch at commit 298721b259cc63ca13c35c1eb0cffe36fd553ce0.
ATTACHMENT ID: 12765329
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 javac{color}. The patch appears to cause mvn compile goal to
fail with Hadoop version 2.4.0.
Compilation errors resume:
[ERROR] COMPILATION ERROR :
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestQueryMatcher.java:[112,6]
error: setRow(byte[],int,short) has private access in ScanQueryMatcher
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestQueryMatcher.java:[175,6]
error: setRow(byte[],int,short) has private access in ScanQueryMatcher
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestQueryMatcher.java:[228,6]
error: setRow(byte[],int,short) has private access in ScanQueryMatcher
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestQueryMatcher.java:[281,6]
error: setRow(byte[],int,short) has private access in ScanQueryMatcher
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestQueryMatcher.java:[334,10]
error: setRow(byte[],int,short) has private access in ScanQueryMatcher
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile
(default-testCompile) on project hbase-server: Compilation failure: Compilation
failure:
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestQueryMatcher.java:[112,6]
error: setRow(byte[],int,short) has private access in ScanQueryMatcher
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestQueryMatcher.java:[175,6]
error: setRow(byte[],int,short) has private access in ScanQueryMatcher
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestQueryMatcher.java:[228,6]
error: setRow(byte[],int,short) has private access in ScanQueryMatcher
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestQueryMatcher.java:[281,6]
error: setRow(byte[],int,short) has private access in ScanQueryMatcher
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestQueryMatcher.java:[334,10]
error: setRow(byte[],int,short) has private access in ScanQueryMatcher
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :hbase-server
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15898//console
This message is automatically generated.
> Reduce the number of time row comparison is done in a Scan
> --
>
> Key: HBASE-14221
> URL: https://issues.apache.org/jira/browse/HBASE-14221
> Project: HBase
> Issue Type: Sub-task
> Components: Scanners
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: 14221-0.98-takeALook.txt, HBASE-14221.patch,
> HBASE-14221_1.patch, HBASE-14221_1.patch, HBASE-14221_6.patch,
> withmatchingRowspatch.png, withoutmatchingRowspatch.png
>
>
> When we tried to do some profiling
[jira] [Commented] (HBASE-14398) Create the fake keys required in the scan path to avoid copy to byte[]
[ https://issues.apache.org/jira/browse/HBASE-14398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946337#comment-14946337 ] stack commented on HBASE-14398: --- bq. When the cell is backed by DBB or byte[], we have to support the getFamilyArray() API. We have to support both? That is the bit I don't get. Where are there instances of this? I was thinking we'd go one path or the other (byte [] or BB). > Create the fake keys required in the scan path to avoid copy to byte[] > -- > > Key: HBASE-14398 > URL: https://issues.apache.org/jira/browse/HBASE-14398 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-14398.patch, HBASE-14398_1.patch > > > Already we have created some fake keys for the ByteBufferedCells so that we > can avoid the copy requried to create fake keys. This JIRA aims to fill up > all such places so that the Offheap BBs are not copied to onheap byte[]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14502) Purge use of jmock and remove as dependency
[ https://issues.apache.org/jira/browse/HBASE-14502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14502: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 1.3.0 1.2.0 2.0.0 Status: Resolved (was: Patch Available) Pushed to branch-1.2+. Thanks for nice patch [~gliptak] > Purge use of jmock and remove as dependency > --- > > Key: HBASE-14502 > URL: https://issues.apache.org/jira/browse/HBASE-14502 > Project: HBase > Issue Type: Task > Components: test >Reporter: stack >Assignee: Gabor Liptak > Labels: beginner > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14502.1.patch > > > jmock is a dependency used by one test only, TestBulkLoad. It looks like you > can do anything in mockito that can be done in jmock. Lets purge it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-14221) Reduce the number of time row comparison is done in a Scan
[
https://issues.apache.org/jira/browse/HBASE-14221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946327#comment-14946327
]
Lars Hofhansl edited comment on HBASE-14221 at 10/7/15 6:08 AM:
[~ram_krish], take a look at the "-takeALook" sample. That's what I mean.
I let the SQM decide when a new row is found (it's better encapsulation, and
it's doing the comparison there anyway).
Haven't tested in beyond running TestScanner and TestAtomicOperation, which
both still pass.
(I am not suggesting we use my patch, it's just easier to explain what I mean
by having it in a patch rather then describing it in words).
was (Author: lhofhansl):
[~ram_krish], take a look at the "-takeALook" sample. That's what I mean.
I let the SQM decide when a new row is found (it's better encapsulation, and
it's doing the comparison there anyway).
Haven't tested in beyond running TestScanner and TestAtomicOperation, which
both still pass.
> Reduce the number of time row comparison is done in a Scan
> --
>
> Key: HBASE-14221
> URL: https://issues.apache.org/jira/browse/HBASE-14221
> Project: HBase
> Issue Type: Sub-task
> Components: Scanners
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: 14221-0.98-takeALook.txt, HBASE-14221.patch,
> HBASE-14221_1.patch, HBASE-14221_1.patch, HBASE-14221_6.patch,
> withmatchingRowspatch.png, withoutmatchingRowspatch.png
>
>
> When we tried to do some profiling with the PE tool found this.
> Currently we do row comparisons in 3 places in a simple Scan case.
> 1) ScanQueryMatcher
> {code}
>int ret = this.rowComparator.compareRows(curCell, cell);
> if (!this.isReversed) {
> if (ret <= -1) {
> return MatchCode.DONE;
> } else if (ret >= 1) {
> // could optimize this, if necessary?
> // Could also be called SEEK_TO_CURRENT_ROW, but this
> // should be rare/never happens.
> return MatchCode.SEEK_NEXT_ROW;
> }
> } else {
> if (ret <= -1) {
> return MatchCode.SEEK_NEXT_ROW;
> } else if (ret >= 1) {
> return MatchCode.DONE;
> }
> }
> {code}
> 2) In StoreScanner next() while starting to scan the row
> {code}
> if (!scannerContext.hasAnyLimit(LimitScope.BETWEEN_CELLS) ||
> matcher.curCell == null ||
> isNewRow || !CellUtil.matchingRow(peeked, matcher.curCell)) {
> this.countPerRow = 0;
> matcher.setToNewRow(peeked);
> }
> {code}
> Particularly to see if we are in a new row.
> 3) In HRegion
> {code}
> scannerContext.setKeepProgress(true);
> heap.next(results, scannerContext);
> scannerContext.setKeepProgress(tmpKeepProgress);
> nextKv = heap.peek();
> moreCellsInRow = moreCellsInRow(nextKv, currentRowCell);
> {code}
> Here again there are cases where we need to careful for a MultiCF case. Was
> trying to solve this for the MultiCF case but is having lot of cases to
> solve. But atleast for a single CF case I think these comparison can be
> reduced.
> So for a single CF case in the SQM we are able to find if we have crossed a
> row using the code pasted above in SQM. That comparison is definitely needed.
> Now in case of a single CF the HRegion is going to have only one element in
> the heap and so the 3rd comparison can surely be avoided if the
> StoreScanner.next() was over due to MatchCode.DONE caused by SQM.
> Coming to the 2nd compareRows that we do in StoreScanner. next() - even that
> can be avoided if we know that the previous next() call was over due to a new
> row. Doing all this I found that the compareRows in the profiler which was
> 19% got reduced to 13%. Initially we can solve for single CF case which can
> be extended to MultiCF cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14221) Reduce the number of time row comparison is done in a Scan
[
https://issues.apache.org/jira/browse/HBASE-14221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-14221:
--
Attachment: 14221-0.98-takeALook.txt
[~ram_krish], take a look at the "-takeALook" sample. That's what I mean.
I let the SQM decide when a new row is found (it's better encapsulation, and
it's doing the comparison there anyway).
Haven't tested in beyond running TestScanner and TestAtomicOperation, which
both still pass.
> Reduce the number of time row comparison is done in a Scan
> --
>
> Key: HBASE-14221
> URL: https://issues.apache.org/jira/browse/HBASE-14221
> Project: HBase
> Issue Type: Sub-task
> Components: Scanners
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: 14221-0.98-takeALook.txt, HBASE-14221.patch,
> HBASE-14221_1.patch, HBASE-14221_1.patch, HBASE-14221_6.patch,
> withmatchingRowspatch.png, withoutmatchingRowspatch.png
>
>
> When we tried to do some profiling with the PE tool found this.
> Currently we do row comparisons in 3 places in a simple Scan case.
> 1) ScanQueryMatcher
> {code}
>int ret = this.rowComparator.compareRows(curCell, cell);
> if (!this.isReversed) {
> if (ret <= -1) {
> return MatchCode.DONE;
> } else if (ret >= 1) {
> // could optimize this, if necessary?
> // Could also be called SEEK_TO_CURRENT_ROW, but this
> // should be rare/never happens.
> return MatchCode.SEEK_NEXT_ROW;
> }
> } else {
> if (ret <= -1) {
> return MatchCode.SEEK_NEXT_ROW;
> } else if (ret >= 1) {
> return MatchCode.DONE;
> }
> }
> {code}
> 2) In StoreScanner next() while starting to scan the row
> {code}
> if (!scannerContext.hasAnyLimit(LimitScope.BETWEEN_CELLS) ||
> matcher.curCell == null ||
> isNewRow || !CellUtil.matchingRow(peeked, matcher.curCell)) {
> this.countPerRow = 0;
> matcher.setToNewRow(peeked);
> }
> {code}
> Particularly to see if we are in a new row.
> 3) In HRegion
> {code}
> scannerContext.setKeepProgress(true);
> heap.next(results, scannerContext);
> scannerContext.setKeepProgress(tmpKeepProgress);
> nextKv = heap.peek();
> moreCellsInRow = moreCellsInRow(nextKv, currentRowCell);
> {code}
> Here again there are cases where we need to careful for a MultiCF case. Was
> trying to solve this for the MultiCF case but is having lot of cases to
> solve. But atleast for a single CF case I think these comparison can be
> reduced.
> So for a single CF case in the SQM we are able to find if we have crossed a
> row using the code pasted above in SQM. That comparison is definitely needed.
> Now in case of a single CF the HRegion is going to have only one element in
> the heap and so the 3rd comparison can surely be avoided if the
> StoreScanner.next() was over due to MatchCode.DONE caused by SQM.
> Coming to the 2nd compareRows that we do in StoreScanner. next() - even that
> can be avoided if we know that the previous next() call was over due to a new
> row. Doing all this I found that the compareRows in the profiler which was
> 19% got reduced to 13%. Initially we can solve for single CF case which can
> be extended to MultiCF cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-14571) Purge TestProcessBasedCluster; it does nothing and then fails
[
https://issues.apache.org/jira/browse/HBASE-14571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack resolved HBASE-14571.
---
Resolution: Fixed
Fix Version/s: 0.98.16
1.1.3
1.0.3
1.3.0
1.2.0
2.0.0
> Purge TestProcessBasedCluster; it does nothing and then fails
> -
>
> Key: HBASE-14571
> URL: https://issues.apache.org/jira/browse/HBASE-14571
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: 14571.txt
>
>
> Remove TestProcessBasedCluster. Its an old test... It does nothing currently.
> It was supposed to do this:
> {code}
> A basic unit test that spins up a local HBase cluster.
> {code}
> ... but main test got disabled:
> // DISABLED BECAUSE FLAKEY @Test(timeout=300 * 1000)
> public void testProcessBasedCluster() throws Exception {
> ... now all that is left is this:
> @Test
> public void testHomePath() {
> File pom = new File(HBaseHomePath.getHomePath(), "pom.xml");
> assertTrue(pom.getPath() + " does not exist", pom.exists());
> }
> ... i.e. assert a pom is present.
> It is flakey too...
> Failed twice tonight.
> Removing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14571) Purge TestProcessBasedCluster; it does nothing and then fails
[
https://issues.apache.org/jira/browse/HBASE-14571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-14571:
--
Attachment: 14571.txt
What I pushed to 0.98+
> Purge TestProcessBasedCluster; it does nothing and then fails
> -
>
> Key: HBASE-14571
> URL: https://issues.apache.org/jira/browse/HBASE-14571
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: 14571.txt
>
>
> Remove TestProcessBasedCluster. Its an old test... It does nothing currently.
> It was supposed to do this:
> {code}
> A basic unit test that spins up a local HBase cluster.
> {code}
> ... but main test got disabled:
> // DISABLED BECAUSE FLAKEY @Test(timeout=300 * 1000)
> public void testProcessBasedCluster() throws Exception {
> ... now all that is left is this:
> @Test
> public void testHomePath() {
> File pom = new File(HBaseHomePath.getHomePath(), "pom.xml");
> assertTrue(pom.getPath() + " does not exist", pom.exists());
> }
> ... i.e. assert a pom is present.
> It is flakey too...
> Failed twice tonight.
> Removing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-14571) Purge TestProcessBasedCluster; it does nothing and then fails
stack created HBASE-14571:
-
Summary: Purge TestProcessBasedCluster; it does nothing and then
fails
Key: HBASE-14571
URL: https://issues.apache.org/jira/browse/HBASE-14571
Project: HBase
Issue Type: Sub-task
Components: test
Reporter: stack
Assignee: stack
Remove TestProcessBasedCluster. Its an old test... It does nothing currently.
It was supposed to do this:
{code}
A basic unit test that spins up a local HBase cluster.
{code}
... but main test got disabled:
// DISABLED BECAUSE FLAKEY @Test(timeout=300 * 1000)
public void testProcessBasedCluster() throws Exception {
... now all that is left is this:
@Test
public void testHomePath() {
File pom = new File(HBaseHomePath.getHomePath(), "pom.xml");
assertTrue(pom.getPath() + " does not exist", pom.exists());
}
... i.e. assert a pom is present.
It is flakey too...
Failed twice tonight.
Removing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14420) Zombie Stomping Session
[
https://issues.apache.org/jira/browse/HBASE-14420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946308#comment-14946308
]
Hadoop QA commented on HBASE-14420:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12765316/none_fix.txt
against master branch at commit 8fd2d6507019f157427c388928d850a65076b9c0.
ATTACHMENT ID: 12765316
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation, build,
or dev-support patch that doesn't require tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15895//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15895//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15895//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15895//console
This message is automatically generated.
> Zombie Stomping Session
> ---
>
> Key: HBASE-14420
> URL: https://issues.apache.org/jira/browse/HBASE-14420
> Project: HBase
> Issue Type: Umbrella
> Components: test
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: hangers.txt, none_fix.txt, none_fix.txt
>
>
> Patch build are now failing most of the time because we are dropping zombies.
> I confirm we are doing this on non-apache build boxes too.
> Left-over zombies consume resources on build boxes (OOME cannot create native
> threads). Having to do multiple test runs in the hope that we can get a
> non-zombie-making build or making (arbitrary) rulings that the zombies are
> 'not related' is a productivity sink. And so on...
> This is an umbrella issue for a zombie stomping session that started earlier
> this week. Will hang sub-issues of this one. Am running builds back-to-back
> on little cluster to turn out the monsters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14502) Purge use of jmock and remove as dependency
[ https://issues.apache.org/jira/browse/HBASE-14502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946306#comment-14946306 ] stack commented on HBASE-14502: --- The two test failures are unrelated. TestProcessBasedCluster does notthing but assert a pom is present. TestImportExport is trying to find a 0.94 type table but the URL to the resource is getting mangled. Will address the above two in other issues. Meantime let me commit this nice cleanup. > Purge use of jmock and remove as dependency > --- > > Key: HBASE-14502 > URL: https://issues.apache.org/jira/browse/HBASE-14502 > Project: HBase > Issue Type: Task > Components: test >Reporter: stack >Assignee: Gabor Liptak > Labels: beginner > Attachments: HBASE-14502.1.patch > > > jmock is a dependency used by one test only, TestBulkLoad. It looks like you > can do anything in mockito that can be done in jmock. Lets purge it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14221) Reduce the number of time row comparison is done in a Scan
[
https://issues.apache.org/jira/browse/HBASE-14221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946298#comment-14946298
]
Lars Hofhansl commented on HBASE-14221:
---
Good find.
Although, isn't there a simpler way to do this, without extending
KeyValueScanner and adding a new enum of return codes, row state to be
maintained, etc?
I always thought we can get rid of case #2 above, by piggy packing on the
comparison of case #1 (and then doing the reset there). Even made a patch for
that some point; like many things didn't finish it.
> Reduce the number of time row comparison is done in a Scan
> --
>
> Key: HBASE-14221
> URL: https://issues.apache.org/jira/browse/HBASE-14221
> Project: HBase
> Issue Type: Sub-task
> Components: Scanners
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: HBASE-14221.patch, HBASE-14221_1.patch,
> HBASE-14221_1.patch, HBASE-14221_6.patch, withmatchingRowspatch.png,
> withoutmatchingRowspatch.png
>
>
> When we tried to do some profiling with the PE tool found this.
> Currently we do row comparisons in 3 places in a simple Scan case.
> 1) ScanQueryMatcher
> {code}
>int ret = this.rowComparator.compareRows(curCell, cell);
> if (!this.isReversed) {
> if (ret <= -1) {
> return MatchCode.DONE;
> } else if (ret >= 1) {
> // could optimize this, if necessary?
> // Could also be called SEEK_TO_CURRENT_ROW, but this
> // should be rare/never happens.
> return MatchCode.SEEK_NEXT_ROW;
> }
> } else {
> if (ret <= -1) {
> return MatchCode.SEEK_NEXT_ROW;
> } else if (ret >= 1) {
> return MatchCode.DONE;
> }
> }
> {code}
> 2) In StoreScanner next() while starting to scan the row
> {code}
> if (!scannerContext.hasAnyLimit(LimitScope.BETWEEN_CELLS) ||
> matcher.curCell == null ||
> isNewRow || !CellUtil.matchingRow(peeked, matcher.curCell)) {
> this.countPerRow = 0;
> matcher.setToNewRow(peeked);
> }
> {code}
> Particularly to see if we are in a new row.
> 3) In HRegion
> {code}
> scannerContext.setKeepProgress(true);
> heap.next(results, scannerContext);
> scannerContext.setKeepProgress(tmpKeepProgress);
> nextKv = heap.peek();
> moreCellsInRow = moreCellsInRow(nextKv, currentRowCell);
> {code}
> Here again there are cases where we need to careful for a MultiCF case. Was
> trying to solve this for the MultiCF case but is having lot of cases to
> solve. But atleast for a single CF case I think these comparison can be
> reduced.
> So for a single CF case in the SQM we are able to find if we have crossed a
> row using the code pasted above in SQM. That comparison is definitely needed.
> Now in case of a single CF the HRegion is going to have only one element in
> the heap and so the 3rd comparison can surely be avoided if the
> StoreScanner.next() was over due to MatchCode.DONE caused by SQM.
> Coming to the 2nd compareRows that we do in StoreScanner. next() - even that
> can be avoided if we know that the previous next() call was over due to a new
> row. Doing all this I found that the compareRows in the profiler which was
> 19% got reduced to 13%. Initially we can solve for single CF case which can
> be extended to MultiCF cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14520) Optimize the number of calls for tags creation in bulk load
[ https://issues.apache.org/jira/browse/HBASE-14520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946289#comment-14946289 ] Bhupendra Kumar Jain commented on HBASE-14520: -- Thanks Ted and Anoop for review and commit. > Optimize the number of calls for tags creation in bulk load > --- > > Key: HBASE-14520 > URL: https://issues.apache.org/jira/browse/HBASE-14520 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Bhupendra Kumar Jain >Assignee: Bhupendra Kumar Jain > Fix For: 2.0.0 > > Attachments: HBASE-14520.patch > > > At present, ttl and Visibility expr is one per tsv line i.e. the values and > the tags remain same for all the columns present in that line. As per the > code, List of tags are created for each cell, Instead of creating new tags > for each cell, tags created once for the line can be reused by other cells. > Assume 1Million rows and 1000 columns. Currently tags creation will happen > for 1M * 1000 times. If reuse the tags, the tags creation can reduce to 1M > times. (i.e. one per tsv line). > This is applicable in both TsvImporterMapper and TextSortReducer logic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14366) NPE in case visibility expression is not present in labels table during importtsv run
[
https://issues.apache.org/jira/browse/HBASE-14366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946287#comment-14946287
]
Bhupendra Kumar Jain commented on HBASE-14366:
--
javadoc warnings are not related to the patch.
> NPE in case visibility expression is not present in labels table during
> importtsv run
> -
>
> Key: HBASE-14366
> URL: https://issues.apache.org/jira/browse/HBASE-14366
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Y. SREENIVASULU REDDY
>Assignee: Bhupendra Kumar Jain
>Priority: Minor
> Attachments: 0001-HBASE-14366.patch, 0001-HBASE-14366_1.patch,
> HBASE-14366-0.98.patch, HBASE-14366-branch-1.patch, HBASE-14366_2(1).patch,
> HBASE-14366_2.patch
>
>
> Below exception is shown in logs if visibility expression is not present in
> labels table during importtsv run. Appropriate exception / message should be
> logged for the user to take further action.
> {code}
> WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child :
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver$1.getLabelOrdinal(DefaultVisibilityExpressionResolver.java:127)
> at
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.getLabelOrdinals(VisibilityUtils.java:358)
> at
> org.apache.hadoop.hbase.security.visibility.VisibilityUtils.createVisibilityExpTags(VisibilityUtils.java:323)
> at
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver.createVisibilityExpTags(DefaultVisibilityExpressionResolver.java:137)
> at
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.populatePut(TsvImporterMapper.java:205)
> at
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:165)
> at
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:1)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-14570) Cleanup hanging TestHBaseFsck
stack created HBASE-14570: - Summary: Cleanup hanging TestHBaseFsck Key: HBASE-14570 URL: https://issues.apache.org/jira/browse/HBASE-14570 Project: HBase Issue Type: Sub-task Components: test Reporter: stack This one hangs regularly. Let me at least add timeouts. Looking in log, a bunch of tests are potentially hanging tests since they don't see to clean up after themselves. Will start watching and just disable likely candidates unless someone wants to have a go at fixing this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-9260) Timestamp Compactions
[
https://issues.apache.org/jira/browse/HBASE-9260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946284#comment-14946284
]
Lars Hofhansl commented on HBASE-9260:
--
Sounds very similar to me.
> Timestamp Compactions
> -
>
> Key: HBASE-9260
> URL: https://issues.apache.org/jira/browse/HBASE-9260
> Project: HBase
> Issue Type: New Feature
> Components: Compaction
>Affects Versions: 0.94.10
>Reporter: Adrien Mogenet
>Priority: Minor
> Labels: features, performance
>
> h1.TSCompactions
> h2.The issue
> One of the biggest issue I currently deal with is compacting big
> stores, i.e. when HBase cluster is 80% full on 4 TB nodes (let say
> with a single big table), compactions might take several hours (from
> 15 to 20 in my case).
> In 'time series' workloads, we could avoid compacting everything
> everytime. Think about OpenTSDB-like systems, or write-heavy,
> TTL based workloads where you want to free space everyday, deleting
> oldest data, and you're not concerned about read latency (i.e. read
> into a single bigger StoreFile).
> > Note: in this draft, I currently consider that we get free space from
> > the TTL behavior only, not really from the Delete operations.
> h2.Proposal and benefits
> For such cases, StoreFiles could be organized and managed in a way
> that would compact:
> * recent StoreFiles with recent data
> * oldest StoreFiles that are concerned by TTL eviction
> By the way, it would help when scanning with a timestamp criterion.
> h2.Configuration
> * {{hbase.hstore.compaction.sortByTS}} (boolean, default=false)
> This indicates if new behavior is enabled or not. Set it to
> {{false}} and compactions will remain the same than current ones.
> * {{hbase.hstore.compaction.ts.bucketSize}} (integer)
> If `sortByTS` is enabled, tells to HBase the target size of
> buckets. The lower, the more StoreFiles you'll get, but you should
> save more IO's. Higher values will generate less StoreFiles, but
> theses will be bigger and thus compactions could generate more
> IO's.
> h2.Examples
> Here is how a common store could look like after some flushes and
> perhaps some minor compactions:
> {noformat}
>,---, ,---, ,---,
>| | | | ,---, | |
>| | | | | | | |
>`---' `---' `---' `---'
> SF1 SF2 SF3 SF4
>\__ __/
> V
>for all of these Storefiles,
>let say minimum TS is 01/01/2013
>and maximum TS is 31/03/2013
> {noformat}
> Set the bucket size to 1 month, and that's what we have after
> compaction:
> {noformat}
> ,---, ,---,
> | | | |
> ,---, | | | |
> | | | | | |
> `---' `---' `---'
>SF1 SF2 SF3
>,-,
>| minimum TS | maximum TS |
> ,---'
> | SF1 | 03/03/2013 | 31/03/2013 | most recent, growing
> | SF2 | 31/01/2013 | 02/03/2013 | old data, "sealed"
> | SF3 | 01/01/2013 | 30/01/2013 | oldest data, "sealed"
> '---'
> {noformat}
> h2.StoreFile selection
> * for minor compactions, current algorithm should already do the
> right job. Pick up `n` eldest files that are small enough, and
> write a bigger file. Remember, TSCompaction are designed for time
> series, so this 'minor selection' should leave "sealed" big old
> files as they are.
> * for major compactions, when all the StoreFiles have been selected,
> apply the TTL first. StoreFiles that are entirely out of time just
> don't need to be rewritten. They'll be deleted in one time,
> avoiding lots of IO's.
> h2.New issues and trade-offs
> 1. In that case ({{bucketSize=1 month}}), after 1+ year, we'll have lots
> of StoreFiles (and more generally after `n * bucketSize` seconds) if
> there is no TTL eviction. In any case, a clever threshold should be
> implemented to limit the maximum number of StoreFiles.
> 2. If we later add old data that matches timerange of a StoreFile
> which has already been compacted, this could generate lots of IO's
> to reconstruct a single StoreFile for this time bucket, perhaps just
> to merge a few lines.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14549) Simplify scanner stack reset logic
[ https://issues.apache.org/jira/browse/HBASE-14549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946279#comment-14946279 ] Lars Hofhansl commented on HBASE-14549: --- Looking more at HBASE-5121, I think I do not understand what the issue is. We're recreating the scanner heap, how can there be state left over from the prior scan. I think as soon as I understand that, I can fix this one and make it simpler. > Simplify scanner stack reset logic > -- > > Key: HBASE-14549 > URL: https://issues.apache.org/jira/browse/HBASE-14549 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Attachments: 14549-0.98.txt > > > Looking at the code, I find that the logic is unnecessarily complex. > We indicate in updateReaders that the scanner stack needs to be reset. Then > almost all store scanner (and derived classes) methods need to check and > actually reset the scanner stack. > Compaction are rare, we should reset the scanner stack in update readers, and > hence avoid needing to check in all methods. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14271) Improve Nexus staging instructions
[ https://issues.apache.org/jira/browse/HBASE-14271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946274#comment-14946274 ] Hudson commented on HBASE-14271: FAILURE: Integrated in HBase-TRUNK #6880 (See [https://builds.apache.org/job/HBase-TRUNK/6880/]) HBASE-14271 Improve Nexus Staging Instructions (mstanleyjones: rev 8fd2d6507019f157427c388928d850a65076b9c0) * src/main/asciidoc/_chapters/developer.adoc > Improve Nexus staging instructions > -- > > Key: HBASE-14271 > URL: https://issues.apache.org/jira/browse/HBASE-14271 > Project: HBase > Issue Type: Task > Components: build, documentation >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-14271.patch > > > Refine the Nexus staging instructions a bit. (A promise I made a long time > ago.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14502) Purge use of jmock and remove as dependency
[
https://issues.apache.org/jira/browse/HBASE-14502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946273#comment-14946273
]
Hadoop QA commented on HBASE-14502:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12765317/HBASE-14502.1.patch
against master branch at commit 8fd2d6507019f157427c388928d850a65076b9c0.
ATTACHMENT ID: 12765317
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.util.TestProcessBasedCluster
org.apache.hadoop.hbase.mapreduce.TestImportExport
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15896//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15896//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15896//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15896//console
This message is automatically generated.
> Purge use of jmock and remove as dependency
> ---
>
> Key: HBASE-14502
> URL: https://issues.apache.org/jira/browse/HBASE-14502
> Project: HBase
> Issue Type: Task
> Components: test
>Reporter: stack
>Assignee: Gabor Liptak
> Labels: beginner
> Attachments: HBASE-14502.1.patch
>
>
> jmock is a dependency used by one test only, TestBulkLoad. It looks like you
> can do anything in mockito that can be done in jmock. Lets purge it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14424) Document that DisabledRegionSplitPolicy blocks manual splits
[ https://issues.apache.org/jira/browse/HBASE-14424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946277#comment-14946277 ] Hudson commented on HBASE-14424: FAILURE: Integrated in HBase-TRUNK #6880 (See [https://builds.apache.org/job/HBase-TRUNK/6880/]) HBASE-14424 Document that DisabledRegionSplitPolicy blocks manual splits (mstanleyjones: rev 5e60166eac9b607d719ae8a084312e13d65cf074) * src/main/asciidoc/_chapters/architecture.adoc * hbase-common/src/main/resources/hbase-default.xml > Document that DisabledRegionSplitPolicy blocks manual splits > > > Key: HBASE-14424 > URL: https://issues.apache.org/jira/browse/HBASE-14424 > Project: HBase > Issue Type: Task > Components: documentation >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-14424.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12615) Document GC conserving guidelines for contributors
[ https://issues.apache.org/jira/browse/HBASE-12615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946275#comment-14946275 ] Hudson commented on HBASE-12615: FAILURE: Integrated in HBase-TRUNK #6880 (See [https://builds.apache.org/job/HBase-TRUNK/6880/]) HBASE-12615 Document GC conserving guidelines for contributors (mstanleyjones: rev d55f4aee4ff7e952eedbd04565e1b5f7b67379f5) * src/main/asciidoc/_chapters/developer.adoc > Document GC conserving guidelines for contributors > -- > > Key: HBASE-12615 > URL: https://issues.apache.org/jira/browse/HBASE-12615 > Project: HBase > Issue Type: Bug > Components: documentation >Reporter: Andrew Purtell >Assignee: Misty Stanley-Jones > Fix For: 2.0.0 > > Attachments: HBASE-12615.patch > > > LinkedIn put up a blog post with a nice concise list of GC conserving > techniques we should document for contributors. Additionally, when we're at a > point our build supports custom error-prone plugins, we can develop warnings > for some of them. > Source: > http://engineering.linkedin.com/performance/linkedin-feed-faster-less-jvm-garbage > - Be careful with Iterators > - Estimate the size of a collection when initializing > - Defer expression evaluation > - Compile the regex patterns in advance > - Cache it if you can > - String Interns are useful but dangerous > All good advice and practice that I know we aim for. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12983) HBase book mentions hadoop.ssl.enabled when it should be hbase.ssl.enabled
[
https://issues.apache.org/jira/browse/HBASE-12983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946276#comment-14946276
]
Hudson commented on HBASE-12983:
FAILURE: Integrated in HBase-TRUNK #6880 (See
[https://builds.apache.org/job/HBase-TRUNK/6880/])
HBASE-12983 HBase book mentions hadoop.ssl.enabled when it should be
(mstanleyjones: rev bd9a41a3685ec3f42776b89b756e121c02640b93)
* src/main/asciidoc/_chapters/security.adoc
> HBase book mentions hadoop.ssl.enabled when it should be hbase.ssl.enabled
> --
>
> Key: HBASE-12983
> URL: https://issues.apache.org/jira/browse/HBASE-12983
> Project: HBase
> Issue Type: Bug
> Components: documentation
>Reporter: Esteban Gutierrez
>Assignee: Misty Stanley-Jones
> Fix For: 2.0.0
>
> Attachments: HBASE-12983.patch
>
>
> In the HBase book we say the following:
> {quote}
> A default HBase install uses insecure HTTP connections for web UIs for the
> master and region servers. To enable secure HTTP (HTTPS) connections instead,
> set *hadoop.ssl.enabled* to true in hbase-site.xml. This does not change the
> port used by the Web UI. To change the port for the web UI for a given HBase
> component, configure that port’s setting in hbase-site.xml. These settings
> are:
> {quote}
> The property should be *hbase.ssl.enabled* instead.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14568) Disable hanging test TestStochasticLoadBalancer2
[ https://issues.apache.org/jira/browse/HBASE-14568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14568: -- Assignee: Ted Yu > Disable hanging test TestStochasticLoadBalancer2 > > > Key: HBASE-14568 > URL: https://issues.apache.org/jira/browse/HBASE-14568 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack >Assignee: Ted Yu > > Please address the hanging tests TestStochasticLoadBalancer2 [~ted_yu] You > introduced it here: > commit ce72ce998f2e9ad23329b48ab8e85912d642fef1 > Author: tedyu > Date: Mon Aug 10 07:35:19 2015 -0700 > HBASE-14200 Separate RegionReplica subtests of TestStochasticLoadBalancer > into TestStochasticLoadBalancer2 > Otherwise, I'd like to just disable it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14569) Disable hanging test TestNamespaceAuditor
stack created HBASE-14569: - Summary: Disable hanging test TestNamespaceAuditor Key: HBASE-14569 URL: https://issues.apache.org/jira/browse/HBASE-14569 Project: HBase Issue Type: Sub-task Components: test Reporter: stack Assignee: Vandana Ayyalasomayajula The test hung here: https://builds.apache.org/job/PreCommit-HBASE-Build/15893//console It hangs quite regularly. Any chance of taking a look [~avandana]? Else, I'll just disable it so we can get clean builds again. Thanks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14568) Disable hanging test TestStochasticLoadBalancer2
stack created HBASE-14568: - Summary: Disable hanging test TestStochasticLoadBalancer2 Key: HBASE-14568 URL: https://issues.apache.org/jira/browse/HBASE-14568 Project: HBase Issue Type: Sub-task Reporter: stack Please address the hanging tests TestStochasticLoadBalancer2 [~ted_yu] You introduced it here: commit ce72ce998f2e9ad23329b48ab8e85912d642fef1 Author: tedyu Date: Mon Aug 10 07:35:19 2015 -0700 HBASE-14200 Separate RegionReplica subtests of TestStochasticLoadBalancer into TestStochasticLoadBalancer2 Otherwise, I'd like to just disable it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14567) Disable hanging test TestMobCompactor and TestMobSweeper
stack created HBASE-14567: - Summary: Disable hanging test TestMobCompactor and TestMobSweeper Key: HBASE-14567 URL: https://issues.apache.org/jira/browse/HBASE-14567 Project: HBase Issue Type: Sub-task Components: test Reporter: stack Assignee: Jingcheng Du These tests hang with some regularity. Please take a look [~jingcheng...@intel.com] since I believe they are yours. If not, let me know and i'll assign elsewhere. Otherwise, I'll just disable them. Thanks. They hung here most recently: https://builds.apache.org/job/PreCommit-HBASE-Build/15893//console -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14479) Apply the Leader/Followers pattern to RpcServer's Reader
[
https://issues.apache.org/jira/browse/HBASE-14479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946256#comment-14946256
]
stack commented on HBASE-14479:
---
{code}
kalashnikov:hbase.git.commit stack$ python ./dev-support/findHangingTests.py
https://builds.apache.org/job/PreCommit-HBASE-Build/15893//consoleText
Fetching https://builds.apache.org/job/PreCommit-HBASE-Build/15893//consoleText
Building remotely on ubuntu-2 (docker Ubuntu ubuntu) in workspace
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build
Testing patch for HBASE-14479.
Testing patch on branch master.
Printing hanging tests
Hanging test : org.apache.hadoop.hbase.util.TestHBaseFsck
Hanging test : org.apache.hadoop.hbase.namespace.TestNamespaceAuditor
Hanging test :
org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancer2
Hanging test : org.apache.hadoop.hbase.mob.mapreduce.TestMobSweeper
Hanging test : org.apache.hadoop.hbase.mob.compactions.TestMobCompactor
Printing Failing tests
{code}
These failed.
I'm just going to disable them all... they fail regularly. Let me make issues.
> Apply the Leader/Followers pattern to RpcServer's Reader
>
>
> Key: HBASE-14479
> URL: https://issues.apache.org/jira/browse/HBASE-14479
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC, Performance
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Attachments: HBASE-14479-V2.patch, HBASE-14479-V2.patch,
> HBASE-14479.patch
>
>
> {{RpcServer}} uses multiple selectors to read data for load distribution, but
> the distribution is just done by round-robin. It is uncertain, especially for
> long run, whether load is equally divided and resources are used without
> being wasted.
> Moreover, multiple selectors may cause excessive context switches which give
> priority to low latency (while we just add the requests to queues), and it is
> possible to reduce throughput of the whole server.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14557) MapReduce WALPlayer issue with NoTagsKeyValue
[
https://issues.apache.org/jira/browse/HBASE-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946252#comment-14946252
]
stack commented on HBASE-14557:
---
bq. It looks like to be related to the recent optimization with NoTagsKeyValue.
So, we back out the above?
> MapReduce WALPlayer issue with NoTagsKeyValue
> -
>
> Key: HBASE-14557
> URL: https://issues.apache.org/jira/browse/HBASE-14557
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Jerry He
>
> Running MapReduce WALPlayer to convert WAL info HFiles:
> {noformat}
> 15/10/05 20:28:08 INFO mapred.JobClient: Task Id :
> attempt_201508031611_0029_m_00_0, Status : FAILED
> java.io.IOException: Type mismatch in value from map: expected
> org.apache.hadoop.hbase.KeyValue, recieved
> org.apache.hadoop.hbase.NoTagsKeyValue
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:997)
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:689)
> at
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
> at
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALKeyValueMapper.map(WALPlayer.java:111)
> at
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALKeyValueMapper.map(WALPlayer.java:96)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:751)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:368)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at
> java.security.AccessController.doPrivileged(AccessController.java:369)
> at javax.security.auth.Subject.doAs(Subject.java:572)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1502)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14557) MapReduce WALPlayer issue with NoTagsKeyValue
[
https://issues.apache.org/jira/browse/HBASE-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946251#comment-14946251
]
stack commented on HBASE-14557:
---
bq. ... we cache tagsLen or add a boolean to indicate if tags is present or not
To every KV? Lets not if we can avoid it.
> MapReduce WALPlayer issue with NoTagsKeyValue
> -
>
> Key: HBASE-14557
> URL: https://issues.apache.org/jira/browse/HBASE-14557
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Jerry He
>
> Running MapReduce WALPlayer to convert WAL info HFiles:
> {noformat}
> 15/10/05 20:28:08 INFO mapred.JobClient: Task Id :
> attempt_201508031611_0029_m_00_0, Status : FAILED
> java.io.IOException: Type mismatch in value from map: expected
> org.apache.hadoop.hbase.KeyValue, recieved
> org.apache.hadoop.hbase.NoTagsKeyValue
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:997)
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:689)
> at
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
> at
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALKeyValueMapper.map(WALPlayer.java:111)
> at
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALKeyValueMapper.map(WALPlayer.java:96)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:751)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:368)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at
> java.security.AccessController.doPrivileged(AccessController.java:369)
> at javax.security.auth.Subject.doAs(Subject.java:572)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1502)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14557) MapReduce WALPlayer issue with NoTagsKeyValue
[
https://issues.apache.org/jira/browse/HBASE-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946223#comment-14946223
]
ramkrishna.s.vasudevan commented on HBASE-14557:
Or we should remove NoTagsKeyValue and always go with KeyValue provided we
cache tagsLen or add a boolean to indicate if tags is present or not. If we
don't do it then caching the taglen is a costly operation. But this will have
concerns on increasing the heap size occupied by per KV and little more GC (not
sure on the impact of GC though).
> MapReduce WALPlayer issue with NoTagsKeyValue
> -
>
> Key: HBASE-14557
> URL: https://issues.apache.org/jira/browse/HBASE-14557
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Jerry He
>
> Running MapReduce WALPlayer to convert WAL info HFiles:
> {noformat}
> 15/10/05 20:28:08 INFO mapred.JobClient: Task Id :
> attempt_201508031611_0029_m_00_0, Status : FAILED
> java.io.IOException: Type mismatch in value from map: expected
> org.apache.hadoop.hbase.KeyValue, recieved
> org.apache.hadoop.hbase.NoTagsKeyValue
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:997)
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:689)
> at
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
> at
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALKeyValueMapper.map(WALPlayer.java:111)
> at
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALKeyValueMapper.map(WALPlayer.java:96)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:751)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:368)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at
> java.security.AccessController.doPrivileged(AccessController.java:369)
> at javax.security.auth.Subject.doAs(Subject.java:572)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1502)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14557) MapReduce WALPlayer issue with NoTagsKeyValue
[
https://issues.apache.org/jira/browse/HBASE-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946220#comment-14946220
]
ramkrishna.s.vasudevan commented on HBASE-14557:
bq.Then we will have problem setting the OutputValueClass to either of the two
classes.
Ok. The problem is that WALCellCodec using KeyValueDecoder that does not write
tags. So it will always be NoTagsKeyValue and NoTagsKeyValue is a private
interface now.
I think this is a critical issue.
> MapReduce WALPlayer issue with NoTagsKeyValue
> -
>
> Key: HBASE-14557
> URL: https://issues.apache.org/jira/browse/HBASE-14557
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Jerry He
>
> Running MapReduce WALPlayer to convert WAL info HFiles:
> {noformat}
> 15/10/05 20:28:08 INFO mapred.JobClient: Task Id :
> attempt_201508031611_0029_m_00_0, Status : FAILED
> java.io.IOException: Type mismatch in value from map: expected
> org.apache.hadoop.hbase.KeyValue, recieved
> org.apache.hadoop.hbase.NoTagsKeyValue
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:997)
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:689)
> at
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
> at
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALKeyValueMapper.map(WALPlayer.java:111)
> at
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALKeyValueMapper.map(WALPlayer.java:96)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:751)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:368)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at
> java.security.AccessController.doPrivileged(AccessController.java:369)
> at javax.security.auth.Subject.doAs(Subject.java:572)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1502)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14458) AsyncRpcClient#createRpcChannel() should check and remove dead channel before creating new one to same server
[
https://issues.apache.org/jira/browse/HBASE-14458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946218#comment-14946218
]
Hadoop QA commented on HBASE-14458:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12765299/HBASE-14458%20%281%29.patch
against master branch at commit 0ea1f8122709302ee19279aaa438b37dac30c25b.
ATTACHMENT ID: 12765299
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:red}-1 findbugs{color}. The patch appears to cause Findbugs
(version 2.0.3) to fail.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.util.TestProcessBasedCluster
org.apache.hadoop.hbase.mapreduce.TestImportExport
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15894//testReport/
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15894//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15894//console
This message is automatically generated.
> AsyncRpcClient#createRpcChannel() should check and remove dead channel before
> creating new one to same server
> -
>
> Key: HBASE-14458
> URL: https://issues.apache.org/jira/browse/HBASE-14458
> Project: HBase
> Issue Type: Bug
> Components: IPC/RPC
>Affects Versions: 2.0.0, 1.2.0, 1.3.0, 1.1.3
>Reporter: Samir Ahmic
>Assignee: Samir Ahmic
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-14458 (1).patch, HBASE-14458.patch,
> HBASE-14458.patch
>
>
> I have notice this issue while testing master branch in distributed mode.
> Reproduction steps:
> 1. Write some data with hbase ltt
> 2. While ltt is writing execute $graceful_stop.sh --restart --reload [rs]
> 3. Wait until script start to reload regions to restarted server. In that
> moment ltt will stop writing and eventually fail.
> After some digging i have notice that while ltt is working correctly there is
> single connection per regionserver (lsof for single connection, 27109 is ltt
> PID )
> {code}
> java 27109 hbase 143u210579579 0t0TCP
> hnode1:40423->hnode5:16020 (ESTABLISHED)
> {code}
> and when in this example hnode5 server is restarted and script starts to
> reload regions on this server ltt start creating thousands of new tcp
> connections to this server:
> {code}
> java 27109 hbase *623u 210674415 0t0TCP
> hnode1:52948->hnode5:16020 (ESTABLISHED)
> java 27109 hbase *624u 210674416 0t0TCP
> hnode1:52949->hnode5:16020 (ESTABLISHED)
> java 27109 hbase *625u 210674417 0t0TCP
> hnode1:52950->hnode5:16020 (ESTABLISHED)
> java 27109 hbase *627u 210674419 0t0TCP
> hnode1:52952->hnode5:16020 (ESTABLISHED)
> java 27109 hbase *628u 210674420 0t0TCP
> hnode1:52953->hnode5:16020 (ESTABLISHED)
> java 27109 hbase *633u 210674425 0t0TCP
> hnode1:52958->hnode5:16020 (ESTABLISHED)
> ...
> {code}
> So here is what happened based on some additional logging and debugging:
> - AsyncRpcClient never detected that regionserver is restarted because
> regions were moved and there was no write/read requests to this server and
> there is no some sort of heart-bit mecha
[jira] [Commented] (HBASE-14522) Document and/or disable hsperfdata GC pauses
[
https://issues.apache.org/jira/browse/HBASE-14522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946211#comment-14946211
]
Vladimir Rodionov commented on HBASE-14522:
---
The work to do is HBase book update. I posted the link to a corresponding
CASSANDRA JIRA, which discusses all the issues, related to this JVM setting.
Several JVM performance tools won't work if the flag is enabled, all of them
are listed in the CASSANDRA JIRA. This JIRA should be committed and new follow
up documentation JIRA should be created.
> Document and/or disable hsperfdata GC pauses
>
>
> Key: HBASE-14522
> URL: https://issues.apache.org/jira/browse/HBASE-14522
> Project: HBase
> Issue Type: Task
> Components: documentation, Performance
>Reporter: Nick Dimiduk
>Assignee: Lars Francke
> Attachments: HBASE-14522.patch
>
>
> {quote}
> The JVM by default exports statistics by mmap-ing a file in /tmp
> (hsperfdata). On Linux, modifying a memory mapped file can block until disk
> I/O completes, which can be hundreds of milliseconds. Since the JVM modifies
> these statistics during garbage collection and safepoints, this causes pauses
> that are hundreds of milliseconds long.
> {quote}
> Via [JVM mmap pause|http://www.evanjones.ca/jvm-mmap-pause.html].
> We should add {{-XX:+PerfDisableSharedMem}} to our default options as was
> apparently done in CASSANDRA-9242 and/or document the presence of this bug so
> operators know to use tmpfs.
> Hat-tip [~vrodionov]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13082) Coarsen StoreScanner locks to RegionScanner
[ https://issues.apache.org/jira/browse/HBASE-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946198#comment-14946198 ] stack commented on HBASE-13082: --- bq. New files added by bulk load during an on going scan won't be included in the current scan. The new files will be included only when a new scan is started. Currently during a course of a scan the bulk loaded files can be included too. That makes more sense than our current implementation IMO. bq. Yes we can. I have a working version of a patch. Some issues with Merge regions and splits. Working on them. All the locks will be removed that are currently in StoreScanner and also the notifyReaders() will also go away totally. Just because we are updating the readers during a course of a scan we needed all those locks. Sweet. > Coarsen StoreScanner locks to RegionScanner > --- > > Key: HBASE-13082 > URL: https://issues.apache.org/jira/browse/HBASE-13082 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: ramkrishna.s.vasudevan > Attachments: 13082-test.txt, 13082-v2.txt, 13082-v3.txt, > 13082-v4.txt, 13082.txt, 13082.txt, gc.png, gc.png, gc.png, hits.png, > next.png, next.png > > > Continuing where HBASE-10015 left of. > We can avoid locking (and memory fencing) inside StoreScanner by deferring to > the lock already held by the RegionScanner. > In tests this shows quite a scan improvement and reduced CPU (the fences make > the cores wait for memory fetches). > There are some drawbacks too: > * All calls to RegionScanner need to be remain synchronized > * Implementors of coprocessors need to be diligent in following the locking > contract. For example Phoenix does not lock RegionScanner.nextRaw() and > required in the documentation (not picking on Phoenix, this one is my fault > as I told them it's OK) > * possible starving of flushes and compaction with heavy read load. > RegionScanner operations would keep getting the locks and the > flushes/compactions would not be able finalize the set of files. > I'll have a patch soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14283) Reverse scan doesn’t work with HFile inline index/bloom blocks
[ https://issues.apache.org/jira/browse/HBASE-14283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946197#comment-14946197 ] Ben Lau commented on HBASE-14283: - Hey guys, sorry, I should be able to get back to this soon. Finishing up an unrelated project right now. I didn't know that minor versions in HFiles were also non-backwards compatible. That's one less reason then to make this a major version bump. If anyone has a strong preference for this fix to go into a V3.X I can change the patch to use minor version (eg for header size calculation) when I have time to do it. If not I'll leave it as V4 since it's a little simpler in the code as a major version bump. My original intention btw if it wasn't clear was that this wouldn't be the only change in a V4, just the first change that would go into a V4, whose format/contents is not yet meant to be final even when this patch is committed, i.e. V4 would be essentially a WIP with more changes suggested and implemented in other tickets and eventually released in HBase 2.0. [~anoop.hbase] I'm down for committing a short-term read-the-header-always fix for now and then discussing the longer term solution second. Which branches do you want the patch for? > Reverse scan doesn’t work with HFile inline index/bloom blocks > -- > > Key: HBASE-14283 > URL: https://issues.apache.org/jira/browse/HBASE-14283 > Project: HBase > Issue Type: Bug >Reporter: Ben Lau >Assignee: Ben Lau > Attachments: HBASE-14283-v2.patch, HBASE-14283.patch, > hfile-seek-before.patch > > > Reverse scans do not work if an HFile contains inline bloom blocks or leaf > level index blocks. The reason is because the seekBefore() call calculates > the previous data block’s size by assuming data blocks are contiguous which > is not the case in HFile V2 and beyond. > Attached is a first cut patch (targeting > bcef28eefaf192b0ad48c8011f98b8e944340da5 on trunk) which includes: > (1) a unit test which exposes the bug and demonstrates failures for both > inline bloom blocks and inline index blocks > (2) a proposed fix for inline index blocks that does not require a new HFile > version change, but is only performant for 1 and 2-level indexes and not 3+. > 3+ requires an HFile format update for optimal performance. > This patch does not fix the bloom filter blocks bug. But the fix should be > similar to the case of inline index blocks. The reason I haven’t made the > change yet is I want to confirm that you guys would be fine with me revising > the HFile.Reader interface. > Specifically, these 2 functions (getGeneralBloomFilterMetadata and > getDeleteBloomFilterMetadata) need to return the BloomFilter. Right now the > HFileReader class doesn’t have a reference to the bloom filters (and hence > their indices) and only constructs the IO streams and hence has no way to > know where the bloom blocks are in the HFile. It seems that the HFile.Reader > bloom method comments state that they “know nothing about how that metadata > is structured” but I do not know if that is a requirement of the abstraction > (why?) or just an incidental current property. > We would like to do 3 things with community approval: > (1) Update the HFile.Reader interface and implementation to contain and > return BloomFilters directly rather than unstructured IO streams > (2) Merge the fixes for index blocks and bloom blocks into open source > (3) Create a new Jira ticket for open source HBase to add a ‘prevBlockSize’ > field in the block header in the next HFile version, so that seekBefore() > calls can not only be correct but performant in all cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14117) Check DBEs where fields are being read from Bytebuffers but unused.
[
https://issues.apache.org/jira/browse/HBASE-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-14117:
---
Issue Type: Bug (was: Sub-task)
Parent: (was: HBASE-11425)
> Check DBEs where fields are being read from Bytebuffers but unused.
> ---
>
> Key: HBASE-14117
> URL: https://issues.apache.org/jira/browse/HBASE-14117
> Project: HBase
> Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: Jingcheng Du
>
> {code}
> public Cell getFirstKeyCellInBlock(ByteBuff block) {
> block.mark();
> block.position(Bytes.SIZEOF_INT);
> int keyLength = ByteBuff.readCompressedInt(block);
> // TODO : See if we can avoid these reads as the read values are not
> getting used
> ByteBuff.readCompressedInt(block);
> {code}
> In DBEs many a places we read the integers just to skip them. This JIRA is to
> see if we can avoid this and rather go position based, as per a review
> comment in HBASE-12213.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14117) Check DBEs where fields are being read from Bytebuffers but unused.
[
https://issues.apache.org/jira/browse/HBASE-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946196#comment-14946196
]
ramkrishna.s.vasudevan commented on HBASE-14117:
Am able to move this to a top level issue and it worked. [~anoop.hbase]
> Check DBEs where fields are being read from Bytebuffers but unused.
> ---
>
> Key: HBASE-14117
> URL: https://issues.apache.org/jira/browse/HBASE-14117
> Project: HBase
> Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: Jingcheng Du
>
> {code}
> public Cell getFirstKeyCellInBlock(ByteBuff block) {
> block.mark();
> block.position(Bytes.SIZEOF_INT);
> int keyLength = ByteBuff.readCompressedInt(block);
> // TODO : See if we can avoid these reads as the read values are not
> getting used
> ByteBuff.readCompressedInt(block);
> {code}
> In DBEs many a places we read the integers just to skip them. This JIRA is to
> see if we can avoid this and rather go position based, as per a review
> comment in HBASE-12213.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13082) Coarsen StoreScanner locks to RegionScanner
[ https://issues.apache.org/jira/browse/HBASE-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946193#comment-14946193 ] ramkrishna.s.vasudevan commented on HBASE-13082: bq.So, bulk load won't show mid-scan... you have to get to the end? That would be fine. New files added by bulk load during an on going scan won't be included in the current scan. The new files will be included only when a new scan is started. Currently during a course of a scan the bulk loaded files can be included too. bq.On the patch, can we get more of Lars comments in on what is going on Could we get rid of some of these getReaderLocks too... in hstorefile, in hstore, etc would be good to not let this stuff out if we can. Yes we can. I have a working version of a patch. Some issues with Merge regions and splits. Working on them. All the locks will be removed that are currently in StoreScanner and also the notifyReaders() will also go away totally. Just because we are updating the readers during a course of a scan we needed all those locks. > Coarsen StoreScanner locks to RegionScanner > --- > > Key: HBASE-13082 > URL: https://issues.apache.org/jira/browse/HBASE-13082 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: ramkrishna.s.vasudevan > Attachments: 13082-test.txt, 13082-v2.txt, 13082-v3.txt, > 13082-v4.txt, 13082.txt, 13082.txt, gc.png, gc.png, gc.png, hits.png, > next.png, next.png > > > Continuing where HBASE-10015 left of. > We can avoid locking (and memory fencing) inside StoreScanner by deferring to > the lock already held by the RegionScanner. > In tests this shows quite a scan improvement and reduced CPU (the fences make > the cores wait for memory fetches). > There are some drawbacks too: > * All calls to RegionScanner need to be remain synchronized > * Implementors of coprocessors need to be diligent in following the locking > contract. For example Phoenix does not lock RegionScanner.nextRaw() and > required in the documentation (not picking on Phoenix, this one is my fault > as I told them it's OK) > * possible starving of flushes and compaction with heavy read load. > RegionScanner operations would keep getting the locks and the > flushes/compactions would not be able finalize the set of files. > I'll have a patch soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12291) Create Read only buffers where ever possible
[ https://issues.apache.org/jira/browse/HBASE-12291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-12291: --- Issue Type: Bug (was: Sub-task) Parent: (was: HBASE-11425) > Create Read only buffers where ever possible > > > Key: HBASE-12291 > URL: https://issues.apache.org/jira/browse/HBASE-12291 > Project: HBase > Issue Type: Bug > Components: regionserver, Scanners >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > > This issue is to see if we can really create a Read only buffer in the read > path. Later can see if this needs to be BR or our own BB impl. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12291) Create Read only buffers where ever possible
[ https://issues.apache.org/jira/browse/HBASE-12291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946189#comment-14946189 ] ramkrishna.s.vasudevan commented on HBASE-12291: [~saint@gmail.com] Converted this to an issue rather than sub task as suggested. Your suggestion makes sense. > Create Read only buffers where ever possible > > > Key: HBASE-12291 > URL: https://issues.apache.org/jira/browse/HBASE-12291 > Project: HBase > Issue Type: Bug > Components: regionserver, Scanners >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > > This issue is to see if we can really create a Read only buffer in the read > path. Later can see if this needs to be BR or our own BB impl. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14479) Apply the Leader/Followers pattern to RpcServer's Reader
[
https://issues.apache.org/jira/browse/HBASE-14479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946178#comment-14946178
]
Hadoop QA commented on HBASE-14479:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12765295/HBASE-14479-V2.patch
against master branch at commit 0ea1f8122709302ee19279aaa438b37dac30c25b.
ATTACHMENT ID: 12765295
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15893//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15893//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15893//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15893//console
This message is automatically generated.
> Apply the Leader/Followers pattern to RpcServer's Reader
>
>
> Key: HBASE-14479
> URL: https://issues.apache.org/jira/browse/HBASE-14479
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC, Performance
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Attachments: HBASE-14479-V2.patch, HBASE-14479-V2.patch,
> HBASE-14479.patch
>
>
> {{RpcServer}} uses multiple selectors to read data for load distribution, but
> the distribution is just done by round-robin. It is uncertain, especially for
> long run, whether load is equally divided and resources are used without
> being wasted.
> Moreover, multiple selectors may cause excessive context switches which give
> priority to low latency (while we just add the requests to queues), and it is
> possible to reduce throughput of the whole server.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14117) Check DBEs where fields are being read from Bytebuffers but unused.
[
https://issues.apache.org/jira/browse/HBASE-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946174#comment-14946174
]
Anoop Sam John commented on HBASE-14117:
Not able to move it out as subtask and make a top level issue. May be will
close it and open another improvement issue?
> Check DBEs where fields are being read from Bytebuffers but unused.
> ---
>
> Key: HBASE-14117
> URL: https://issues.apache.org/jira/browse/HBASE-14117
> Project: HBase
> Issue Type: Sub-task
>Reporter: ramkrishna.s.vasudevan
>Assignee: Jingcheng Du
>
> {code}
> public Cell getFirstKeyCellInBlock(ByteBuff block) {
> block.mark();
> block.position(Bytes.SIZEOF_INT);
> int keyLength = ByteBuff.readCompressedInt(block);
> // TODO : See if we can avoid these reads as the read values are not
> getting used
> ByteBuff.readCompressedInt(block);
> {code}
> In DBEs many a places we read the integers just to skip them. This JIRA is to
> see if we can avoid this and rather go position based, as per a review
> comment in HBASE-12213.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-9260) Timestamp Compactions
[
https://issues.apache.org/jira/browse/HBASE-9260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946173#comment-14946173
]
Anoop Sam John commented on HBASE-9260:
---
Is this similar to idea in HBASE-14477?
> Timestamp Compactions
> -
>
> Key: HBASE-9260
> URL: https://issues.apache.org/jira/browse/HBASE-9260
> Project: HBase
> Issue Type: New Feature
> Components: Compaction
>Affects Versions: 0.94.10
>Reporter: Adrien Mogenet
>Priority: Minor
> Labels: features, performance
>
> h1.TSCompactions
> h2.The issue
> One of the biggest issue I currently deal with is compacting big
> stores, i.e. when HBase cluster is 80% full on 4 TB nodes (let say
> with a single big table), compactions might take several hours (from
> 15 to 20 in my case).
> In 'time series' workloads, we could avoid compacting everything
> everytime. Think about OpenTSDB-like systems, or write-heavy,
> TTL based workloads where you want to free space everyday, deleting
> oldest data, and you're not concerned about read latency (i.e. read
> into a single bigger StoreFile).
> > Note: in this draft, I currently consider that we get free space from
> > the TTL behavior only, not really from the Delete operations.
> h2.Proposal and benefits
> For such cases, StoreFiles could be organized and managed in a way
> that would compact:
> * recent StoreFiles with recent data
> * oldest StoreFiles that are concerned by TTL eviction
> By the way, it would help when scanning with a timestamp criterion.
> h2.Configuration
> * {{hbase.hstore.compaction.sortByTS}} (boolean, default=false)
> This indicates if new behavior is enabled or not. Set it to
> {{false}} and compactions will remain the same than current ones.
> * {{hbase.hstore.compaction.ts.bucketSize}} (integer)
> If `sortByTS` is enabled, tells to HBase the target size of
> buckets. The lower, the more StoreFiles you'll get, but you should
> save more IO's. Higher values will generate less StoreFiles, but
> theses will be bigger and thus compactions could generate more
> IO's.
> h2.Examples
> Here is how a common store could look like after some flushes and
> perhaps some minor compactions:
> {noformat}
>,---, ,---, ,---,
>| | | | ,---, | |
>| | | | | | | |
>`---' `---' `---' `---'
> SF1 SF2 SF3 SF4
>\__ __/
> V
>for all of these Storefiles,
>let say minimum TS is 01/01/2013
>and maximum TS is 31/03/2013
> {noformat}
> Set the bucket size to 1 month, and that's what we have after
> compaction:
> {noformat}
> ,---, ,---,
> | | | |
> ,---, | | | |
> | | | | | |
> `---' `---' `---'
>SF1 SF2 SF3
>,-,
>| minimum TS | maximum TS |
> ,---'
> | SF1 | 03/03/2013 | 31/03/2013 | most recent, growing
> | SF2 | 31/01/2013 | 02/03/2013 | old data, "sealed"
> | SF3 | 01/01/2013 | 30/01/2013 | oldest data, "sealed"
> '---'
> {noformat}
> h2.StoreFile selection
> * for minor compactions, current algorithm should already do the
> right job. Pick up `n` eldest files that are small enough, and
> write a bigger file. Remember, TSCompaction are designed for time
> series, so this 'minor selection' should leave "sealed" big old
> files as they are.
> * for major compactions, when all the StoreFiles have been selected,
> apply the TTL first. StoreFiles that are entirely out of time just
> don't need to be rewritten. They'll be deleted in one time,
> avoiding lots of IO's.
> h2.New issues and trade-offs
> 1. In that case ({{bucketSize=1 month}}), after 1+ year, we'll have lots
> of StoreFiles (and more generally after `n * bucketSize` seconds) if
> there is no TTL eviction. In any case, a clever threshold should be
> implemented to limit the maximum number of StoreFiles.
> 2. If we later add old data that matches timerange of a StoreFile
> which has already been compacted, this could generate lots of IO's
> to reconstruct a single StoreFile for this time bucket, perhaps just
> to merge a few lines.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14268) Improve KeyLocker
[
https://issues.apache.org/jira/browse/HBASE-14268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946172#comment-14946172
]
Hadoop QA commented on HBASE-14268:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12765287/HBASE-14268-V7.patch
against master branch at commit 0ea1f8122709302ee19279aaa438b37dac30c25b.
ATTACHMENT ID: 12765287
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at org.apache.phoenix.pherf.ResultTest.testResult(ResultTest.java:119)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15891//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15891//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15891//artifact/patchprocess/checkstyle-aggregate.html
Javadoc warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15891//artifact/patchprocess/patchJavadocWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15891//console
This message is automatically generated.
> Improve KeyLocker
> -
>
> Key: HBASE-14268
> URL: https://issues.apache.org/jira/browse/HBASE-14268
> Project: HBase
> Issue Type: Improvement
> Components: util
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Fix For: 2.0.0, 1.3.0
>
> Attachments: 14268-V5.patch, HBASE-14268-V2.patch,
> HBASE-14268-V3.patch, HBASE-14268-V4.patch, HBASE-14268-V5.patch,
> HBASE-14268-V5.patch, HBASE-14268-V6.patch, HBASE-14268-V7.patch,
> HBASE-14268-V7.patch, HBASE-14268-V7.patch, HBASE-14268-V7.patch,
> HBASE-14268-V7.patch, HBASE-14268.patch, KeyLockerIncrKeysPerformance.java,
> KeyLockerPerformance.java, ReferenceTestApp.java
>
>
> 1. In the implementation of {{KeyLocker}} it uses atomic variables inside a
> synchronized block, which doesn't make sense. Moreover, logic inside the
> synchronized block is not trivial so that it makes less performance in heavy
> multi-threaded environment.
> 2. {{KeyLocker}} gives an instance of {{RentrantLock}} which is already
> locked, but it doesn't follow the contract of {{ReentrantLock}} because you
> are not allowed to freely invoke lock/unlock methods under that contract.
> That introduces a potential risk; Whenever you see a variable of the type
> {{RentrantLock}}, you should pay attention to what the included instance is
> coming from.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14451) Move on to htrace-4.0.1 (from htrace-3.2.0)
[ https://issues.apache.org/jira/browse/HBASE-14451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14451: -- Attachment: 14451.v10.txt > Move on to htrace-4.0.1 (from htrace-3.2.0) > --- > > Key: HBASE-14451 > URL: https://issues.apache.org/jira/browse/HBASE-14451 > Project: HBase > Issue Type: Task >Reporter: stack >Assignee: stack > Attachments: 14451.txt, 14451.v10.txt, 14451.v10.txt, 14451v2.txt, > 14451v3.txt, 14451v4.txt, 14451v5.txt, 14451v6.txt, 14451v7.txt, 14451v8.txt, > 14451v9.txt > > > htrace-4.0.0 was just release with a new API. Get up on it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12593) Tags and Tag dictionary to work with BB
[ https://issues.apache.org/jira/browse/HBASE-12593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946170#comment-14946170 ] Anoop Sam John commented on HBASE-12593: This need some change and some what a bigger patch. Our Tag can work only with byte[]. So following the same line of BB backed cell in server side, we have to make BB backed Tags as well.. I have a WIP patch. Missed this some how till now.. Let me rebase that patch and put here. > Tags and Tag dictionary to work with BB > --- > > Key: HBASE-12593 > URL: https://issues.apache.org/jira/browse/HBASE-12593 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Reporter: ramkrishna.s.vasudevan >Assignee: Anoop Sam John > > Adding the subtask so that we don't forget it. Came up while reviewing the > items required for this parent task. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14451) Move on to htrace-4.0.1 (from htrace-3.2.0)
[
https://issues.apache.org/jira/browse/HBASE-14451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946171#comment-14946171
]
stack commented on HBASE-14451:
---
Killed...
Running org.apache.hadoop.hbase.ipc.TestSimpleRpcScheduler
Killed
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/test-framework/dev-support/test-patch.sh:
line 838: 14645 Killed $MVN clean test
-Dsurefire.rerunFailingTestsCount=2 -P runAllTests -D${PROJECT_NAME}PatchProcess
We're ok: there is no zombie test
> Move on to htrace-4.0.1 (from htrace-3.2.0)
> ---
>
> Key: HBASE-14451
> URL: https://issues.apache.org/jira/browse/HBASE-14451
> Project: HBase
> Issue Type: Task
>Reporter: stack
>Assignee: stack
> Attachments: 14451.txt, 14451.v10.txt, 14451v2.txt, 14451v3.txt,
> 14451v4.txt, 14451v5.txt, 14451v6.txt, 14451v7.txt, 14451v8.txt, 14451v9.txt
>
>
> htrace-4.0.0 was just release with a new API. Get up on it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14436) HTableDescriptor#addCoprocessor will always make RegionCoprocessorHost create new Configuration
[
https://issues.apache.org/jira/browse/HBASE-14436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946166#comment-14946166
]
Hudson commented on HBASE-14436:
FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #1097 (See
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1097/])
HBASE-14436 HTableDescriptor#addCoprocessor will always make (stack: rev
6c0f501ece9b7c31f6ce64c85289e55b4f7cc875)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java
> HTableDescriptor#addCoprocessor will always make RegionCoprocessorHost create
> new Configuration
> ---
>
> Key: HBASE-14436
> URL: https://issues.apache.org/jira/browse/HBASE-14436
> Project: HBase
> Issue Type: Improvement
> Components: Coprocessors
>Affects Versions: 1.2.1
>Reporter: Jianwei Cui
>Assignee: stack
>Priority: Minor
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: HBASE-14436-trunk-v1.patch, HBASE-14436-trunk-v2.patch
>
>
> HTableDescriptor#addCoprocessor will set the coprocessor value as following
> format:
> {code}
> public HTableDescriptor addCoprocessor(String className, Path jarFilePath,
> int priority, final Map kvs)
> throws IOException {
> ...
> String value = ((jarFilePath == null)? "" : jarFilePath.toString()) +
> "|" + className + "|" + Integer.toString(priority) + "|" +
> kvString.toString();
> ...
> }
> {code}
> If the 'jarFilePath' is null, the 'value' will always has the format
> '|className|priority|' even if 'kvs' is null, which means no extra arguments
> for the coprocessor. Then, in the server side,
> RegionCoprocessorHost#getTableCoprocessorAttrsFromSchema will load the table
> coprocessors as:
> {code}
> static List
> getTableCoprocessorAttrsFromSchema(Configuration conf,
> HTableDescriptor htd) {
> ...
> try {
> cfgSpec = matcher.group(4); // => cfgSpec will be '|' for the
> format '|className|priority|'
> } catch (IndexOutOfBoundsException ex) {
> // ignore
> }
> Configuration ourConf;
> if (cfgSpec != null) { // => cfgSpec will be '|' for the format
> '|className|priority|'
> ourConf = new Configuration(false);
> HBaseConfiguration.merge(ourConf, conf);
> }
> ...
> }
> {code}
> The 'cfgSpec' will be '|' for the coprocessor formatted as
> '|className|priority|', so that always create a new Configuration.
> In our production, there are a lot of tables having table-level coprocessors,
> so that the region server will create new Configurations for each region of
> the table, this will consume a certain number of memory when we have many
> such regions.
> To fix the problem, we can make the HTableDescriptor not append the '|' if no
> extra arguments for the coprocessor, or check the 'cfgSpec' more strictly in
> server side which could avoid creating new Configurations for existed such
> regions after the regions reopened. Discussions and suggestions are welcomed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14436) HTableDescriptor#addCoprocessor will always make RegionCoprocessorHost create new Configuration
[
https://issues.apache.org/jira/browse/HBASE-14436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946154#comment-14946154
]
Hudson commented on HBASE-14436:
SUCCESS: Integrated in HBase-0.98 #1145 (See
[https://builds.apache.org/job/HBase-0.98/1145/])
HBASE-14436 HTableDescriptor#addCoprocessor will always make (stack: rev
6c0f501ece9b7c31f6ce64c85289e55b4f7cc875)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java
> HTableDescriptor#addCoprocessor will always make RegionCoprocessorHost create
> new Configuration
> ---
>
> Key: HBASE-14436
> URL: https://issues.apache.org/jira/browse/HBASE-14436
> Project: HBase
> Issue Type: Improvement
> Components: Coprocessors
>Affects Versions: 1.2.1
>Reporter: Jianwei Cui
>Assignee: stack
>Priority: Minor
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: HBASE-14436-trunk-v1.patch, HBASE-14436-trunk-v2.patch
>
>
> HTableDescriptor#addCoprocessor will set the coprocessor value as following
> format:
> {code}
> public HTableDescriptor addCoprocessor(String className, Path jarFilePath,
> int priority, final Map kvs)
> throws IOException {
> ...
> String value = ((jarFilePath == null)? "" : jarFilePath.toString()) +
> "|" + className + "|" + Integer.toString(priority) + "|" +
> kvString.toString();
> ...
> }
> {code}
> If the 'jarFilePath' is null, the 'value' will always has the format
> '|className|priority|' even if 'kvs' is null, which means no extra arguments
> for the coprocessor. Then, in the server side,
> RegionCoprocessorHost#getTableCoprocessorAttrsFromSchema will load the table
> coprocessors as:
> {code}
> static List
> getTableCoprocessorAttrsFromSchema(Configuration conf,
> HTableDescriptor htd) {
> ...
> try {
> cfgSpec = matcher.group(4); // => cfgSpec will be '|' for the
> format '|className|priority|'
> } catch (IndexOutOfBoundsException ex) {
> // ignore
> }
> Configuration ourConf;
> if (cfgSpec != null) { // => cfgSpec will be '|' for the format
> '|className|priority|'
> ourConf = new Configuration(false);
> HBaseConfiguration.merge(ourConf, conf);
> }
> ...
> }
> {code}
> The 'cfgSpec' will be '|' for the coprocessor formatted as
> '|className|priority|', so that always create a new Configuration.
> In our production, there are a lot of tables having table-level coprocessors,
> so that the region server will create new Configurations for each region of
> the table, this will consume a certain number of memory when we have many
> such regions.
> To fix the problem, we can make the HTableDescriptor not append the '|' if no
> extra arguments for the coprocessor, or check the 'cfgSpec' more strictly in
> server side which could avoid creating new Configurations for existed such
> regions after the regions reopened. Discussions and suggestions are welcomed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14497) Reverse Scan threw StackOverflow caused by readPt checking
[ https://issues.apache.org/jira/browse/HBASE-14497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946153#comment-14946153 ] Hudson commented on HBASE-14497: FAILURE: Integrated in HBase-1.3-IT #215 (See [https://builds.apache.org/job/HBase-1.3-IT/215/]) HBASE-14497 Reverse Scan threw StackOverflow caused by readPt checking (tedyu: rev 5e2db42d680f35c1f0a345344a6555ea319d870b) * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultMemStore.java > Reverse Scan threw StackOverflow caused by readPt checking > -- > > Key: HBASE-14497 > URL: https://issues.apache.org/jira/browse/HBASE-14497 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 0.98.14, 1.3.0 >Reporter: Yerui Sun >Assignee: Yerui Sun > Fix For: 2.0.0, 1.3.0 > > Attachments: 14497-branch-1-v6.patch, 14497-master-v6.patch, > HBASE-14497-0.98.patch, HBASE-14497-branch-1-v2.patch, > HBASE-14497-branch-1-v3.patch, HBASE-14497-branch-1-v6.patch, > HBASE-14497-branch-1.patch, HBASE-14497-master-v2.patch, > HBASE-14497-master-v3.patch, HBASE-14497-master-v3.patch, > HBASE-14497-master-v4.patch, HBASE-14497-master-v5.patch, > HBASE-14497-master.patch > > > I met stack overflow error in StoreFileScanner.seekToPreviousRow using > reversed scan. I searched and founded HBASE-14155, but it seems to be a > different reason. > The seekToPreviousRow will fetch the row which closest before, and compare > mvcc to the readPt, which acquired when scanner created. If the row's mvcc is > bigger than readPt, an recursive call of seekToPreviousRow will invoked, to > find the next closest before row. > Considering we created a scanner for reversed scan, and some data with > smaller rows was written and flushed, before calling scanner next. When > seekToPreviousRow was invoked, it would call itself recursively, until all > rows which written after scanner created were iterated. The depth of > recursive calling stack depends on the count of rows, the stack overflow > error will be threw if the count of rows is large, like 1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14436) HTableDescriptor#addCoprocessor will always make RegionCoprocessorHost create new Configuration
[
https://issues.apache.org/jira/browse/HBASE-14436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946152#comment-14946152
]
Hudson commented on HBASE-14436:
FAILURE: Integrated in HBase-1.3-IT #215 (See
[https://builds.apache.org/job/HBase-1.3-IT/215/])
HBASE-14436 HTableDescriptor#addCoprocessor will always make (stack: rev
ab33a65a1adae33025a1034568397d6d67bf476a)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java
> HTableDescriptor#addCoprocessor will always make RegionCoprocessorHost create
> new Configuration
> ---
>
> Key: HBASE-14436
> URL: https://issues.apache.org/jira/browse/HBASE-14436
> Project: HBase
> Issue Type: Improvement
> Components: Coprocessors
>Affects Versions: 1.2.1
>Reporter: Jianwei Cui
>Assignee: stack
>Priority: Minor
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: HBASE-14436-trunk-v1.patch, HBASE-14436-trunk-v2.patch
>
>
> HTableDescriptor#addCoprocessor will set the coprocessor value as following
> format:
> {code}
> public HTableDescriptor addCoprocessor(String className, Path jarFilePath,
> int priority, final Map kvs)
> throws IOException {
> ...
> String value = ((jarFilePath == null)? "" : jarFilePath.toString()) +
> "|" + className + "|" + Integer.toString(priority) + "|" +
> kvString.toString();
> ...
> }
> {code}
> If the 'jarFilePath' is null, the 'value' will always has the format
> '|className|priority|' even if 'kvs' is null, which means no extra arguments
> for the coprocessor. Then, in the server side,
> RegionCoprocessorHost#getTableCoprocessorAttrsFromSchema will load the table
> coprocessors as:
> {code}
> static List
> getTableCoprocessorAttrsFromSchema(Configuration conf,
> HTableDescriptor htd) {
> ...
> try {
> cfgSpec = matcher.group(4); // => cfgSpec will be '|' for the
> format '|className|priority|'
> } catch (IndexOutOfBoundsException ex) {
> // ignore
> }
> Configuration ourConf;
> if (cfgSpec != null) { // => cfgSpec will be '|' for the format
> '|className|priority|'
> ourConf = new Configuration(false);
> HBaseConfiguration.merge(ourConf, conf);
> }
> ...
> }
> {code}
> The 'cfgSpec' will be '|' for the coprocessor formatted as
> '|className|priority|', so that always create a new Configuration.
> In our production, there are a lot of tables having table-level coprocessors,
> so that the region server will create new Configurations for each region of
> the table, this will consume a certain number of memory when we have many
> such regions.
> To fix the problem, we can make the HTableDescriptor not append the '|' if no
> extra arguments for the coprocessor, or check the 'cfgSpec' more strictly in
> server side which could avoid creating new Configurations for existed such
> regions after the regions reopened. Discussions and suggestions are welcomed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14565) Make ZK connection timeout configurable in MiniZooKeeperCluster
[
https://issues.apache.org/jira/browse/HBASE-14565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946151#comment-14946151
]
Hadoop QA commented on HBASE-14565:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12765275/14565-v1.txt
against master branch at commit 0ea1f8122709302ee19279aaa438b37dac30c25b.
ATTACHMENT ID: 12765275
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.util.TestProcessBasedCluster
org.apache.hadoop.hbase.mapreduce.TestImportExport
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15890//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15890//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15890//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15890//console
This message is automatically generated.
> Make ZK connection timeout configurable in MiniZooKeeperCluster
> ---
>
> Key: HBASE-14565
> URL: https://issues.apache.org/jira/browse/HBASE-14565
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 14565-v1.txt
>
>
> This request was made by [~swagle] who works on Ambari Metrics System.
> Currently a hardcoded timeout of 30s is used by MiniZooKeeperCluster
> This affects operation of Ambari Metrics System in standalone mode.
> This JIRA is to make the connection timeout configurable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14451) Move on to htrace-4.0.1 (from htrace-3.2.0)
[
https://issues.apache.org/jira/browse/HBASE-14451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946150#comment-14946150
]
Hadoop QA commented on HBASE-14451:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12765293/14451.v10.txt
against master branch at commit 0ea1f8122709302ee19279aaa438b37dac30c25b.
ATTACHMENT ID: 12765293
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 29 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15892//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15892//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15892//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15892//console
This message is automatically generated.
> Move on to htrace-4.0.1 (from htrace-3.2.0)
> ---
>
> Key: HBASE-14451
> URL: https://issues.apache.org/jira/browse/HBASE-14451
> Project: HBase
> Issue Type: Task
>Reporter: stack
>Assignee: stack
> Attachments: 14451.txt, 14451.v10.txt, 14451v2.txt, 14451v3.txt,
> 14451v4.txt, 14451v5.txt, 14451v6.txt, 14451v7.txt, 14451v8.txt, 14451v9.txt
>
>
> htrace-4.0.0 was just release with a new API. Get up on it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14398) Create the fake keys required in the scan path to avoid copy to byte[]
[ https://issues.apache.org/jira/browse/HBASE-14398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946149#comment-14946149 ] Anoop Sam John commented on HBASE-14398: bq.Why does ByteBufferedCell have to have getFamilyPositionInByteBuffer at all? Why can't I just call getFamilyOffset on the ByteBufferedCell implementation and it returns me an offset that makes sense on the ByteBuffer returned out of getFamilyByteBuffer? When the cell is backed by DBB or byte[], we have to support the getFamilyArray() API. So when it is DBB, we have to do copy to a temp byte[]. So when getFamilyOffset() is used along with getFamilyArray(), we have to return 0 as the offset. Whereas the offset to family in get BB, will be a non 0 value. We have to support getFamilyArray API on BBCell also. Else we will have to have the hasArray() API. We had these discussion long back. > Create the fake keys required in the scan path to avoid copy to byte[] > -- > > Key: HBASE-14398 > URL: https://issues.apache.org/jira/browse/HBASE-14398 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-14398.patch, HBASE-14398_1.patch > > > Already we have created some fake keys for the ByteBufferedCells so that we > can avoid the copy requried to create fake keys. This JIRA aims to fill up > all such places so that the Offheap BBs are not copied to onheap byte[]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12911) Client-side metrics
[
https://issues.apache.org/jira/browse/HBASE-12911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946147#comment-14946147
]
Hadoop QA commented on HBASE-12911:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12765277/12911.yammer.v03.branch-1.patch
against branch-1 branch at commit 0ea1f8122709302ee19279aaa438b37dac30c25b.
ATTACHMENT ID: 12765277
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 31 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
3779 checkstyle errors (more than the master's current 3762 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15889//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15889//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15889//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15889//console
This message is automatically generated.
> Client-side metrics
> ---
>
> Key: HBASE-12911
> URL: https://issues.apache.org/jira/browse/HBASE-12911
> Project: HBase
> Issue Type: New Feature
> Components: Client, Operability, Performance
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
> Fix For: 2.0.0, 1.3.0
>
> Attachments: 0001-HBASE-12911-Client-side-metrics.patch,
> 0001-HBASE-12911-Client-side-metrics.patch,
> 0001-HBASE-12911-Client-side-metrics.patch,
> 0001-HBASE-12911-Client-side-metrics.patch,
> 0001-HBASE-12911-Client-side-metrics.patch,
> 0001-HBASE-12911-Client-side-metrics.patch,
> 0001-HBASE-12911-Client-side-metrics.patch,
> 0001-HBASE-12911-Client-side-metrics.patch,
> 0001-HBASE-12911-Client-side-metrics.patch,
> 0001-HBASE-12911-Client-side-metrics.patch,
> 0001-HBASE-12911-Client-side-metrics.patch, 12911-0.98.00.patch,
> 12911-branch-1.00.patch, 12911.yammer.jpg, 12911.yammer.v00.patch,
> 12911.yammer.v01.patch, 12911.yammer.v02.patch, 12911.yammer.v02.patch,
> 12911.yammer.v03.branch-1.patch, 12911.yammer.v03.patch,
> 12911.yammer.v03.patch, am.jpg, client metrics RS-Master.jpg, client metrics
> client.jpg, conn_agg.jpg, connection attributes.jpg, ltt.jpg, standalone.jpg
>
>
> There's very little visibility into the hbase client. Folks who care to add
> some kind of metrics collection end up wrapping Table method invocations with
> {{System.currentTimeMillis()}}. For a crude example of this, have a look at
> what I did in {{PerformanceEvaluation}} for exposing requests latencies up to
> {{IntegrationTestRegionReplicaPerf}}. The client is quite complex, there's a
> lot going on under the hood that is impossible to see right now without a
> profiler. Being a crucial part of the performance of this distributed system,
> we should have deeper visibility into the client's function.
> I'm not sure that wiring into the hadoop metrics system is the right choice
> because the client is often embedded as a library in a user's application. We
> should have integration with our metrics tools so that, i.e., a client
> embedded in a coprocessor can report metrics through the usual RS channels,
> or a client used in a MR job can do the same.
> I would propose an interface-based system with pluggable implementations. Out
> of the box we'd include a hadoop-metrics implementation and one other,
> possibly [dropwizard/metrics|https://github.com/dropwizard/metrics].
> Thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14158) Add documentation for Initial Release for HBase-Spark Module integration
[ https://issues.apache.org/jira/browse/HBASE-14158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946145#comment-14946145 ] Misty Stanley-Jones commented on HBASE-14158: - +1 there is one very long line here. Also if you can make your patch with git format-patch instead of git diff, we can give you credit for the commit easier. > Add documentation for Initial Release for HBase-Spark Module integration > - > > Key: HBASE-14158 > URL: https://issues.apache.org/jira/browse/HBASE-14158 > Project: HBase > Issue Type: Improvement > Components: documentation, spark >Reporter: Ted Malaska >Assignee: Ted Malaska > Fix For: 2.0.0 > > Attachments: HBASE-14158.1.patch, HBASE-14158.2.patch, > HBASE-14158.5.patch, HBASE-14158.5.patch, HBASE-14158.6.patch > > > Add documentation for Initial Release for HBase-Spark Module integration -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14271) Improve Nexus staging instructions
[ https://issues.apache.org/jira/browse/HBASE-14271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-14271: Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) +1 LGTM, committed to master. > Improve Nexus staging instructions > -- > > Key: HBASE-14271 > URL: https://issues.apache.org/jira/browse/HBASE-14271 > Project: HBase > Issue Type: Task > Components: build, documentation >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-14271.patch > > > Refine the Nexus staging instructions a bit. (A promise I made a long time > ago.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14346) Typo in FamilyFilter
[ https://issues.apache.org/jira/browse/HBASE-14346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-14346: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.0.0 Status: Resolved (was: Patch Available) Thanks [~lars_francke] > Typo in FamilyFilter > > > Key: HBASE-14346 > URL: https://issues.apache.org/jira/browse/HBASE-14346 > Project: HBase > Issue Type: Bug > Components: documentation >Reporter: Joshua Batson >Assignee: Lars Francke >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-14346.patch > > Original Estimate: 5m > Remaining Estimate: 5m > > I think there's a typo. "qualifier name" should read "column family name" > Family Filter > This filter takes a compare operator and a comparator. It compares each > qualifier name with the comparator using the compare operator and if the > comparison returns true, it returns all the key-values in that column. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14502) Purge use of jmock and remove as dependency
[ https://issues.apache.org/jira/browse/HBASE-14502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946142#comment-14946142 ] stack commented on HBASE-14502: --- Excellent [~gliptak] Thanks. Lets see how it does against patch build. Thats sweet undoing a dependency altogether. > Purge use of jmock and remove as dependency > --- > > Key: HBASE-14502 > URL: https://issues.apache.org/jira/browse/HBASE-14502 > Project: HBase > Issue Type: Task > Components: test >Reporter: stack >Assignee: Gabor Liptak > Labels: beginner > Attachments: HBASE-14502.1.patch > > > jmock is a dependency used by one test only, TestBulkLoad. It looks like you > can do anything in mockito that can be done in jmock. Lets purge it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14502) Purge use of jmock and remove as dependency
[ https://issues.apache.org/jira/browse/HBASE-14502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Liptak updated HBASE-14502: - Assignee: Gabor Liptak Release Note: HBASE-14502 Purge use of jmock and remove as dependency Status: Patch Available (was: Open) > Purge use of jmock and remove as dependency > --- > > Key: HBASE-14502 > URL: https://issues.apache.org/jira/browse/HBASE-14502 > Project: HBase > Issue Type: Task > Components: test >Reporter: stack >Assignee: Gabor Liptak > Labels: beginner > Attachments: HBASE-14502.1.patch > > > jmock is a dependency used by one test only, TestBulkLoad. It looks like you > can do anything in mockito that can be done in jmock. Lets purge it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14346) Typo in FamilyFilter
[ https://issues.apache.org/jira/browse/HBASE-14346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946139#comment-14946139 ] Misty Stanley-Jones commented on HBASE-14346: - +1 I will commit it. > Typo in FamilyFilter > > > Key: HBASE-14346 > URL: https://issues.apache.org/jira/browse/HBASE-14346 > Project: HBase > Issue Type: Bug > Components: documentation >Reporter: Joshua Batson >Assignee: Lars Francke >Priority: Trivial > Attachments: HBASE-14346.patch > > Original Estimate: 5m > Remaining Estimate: 5m > > I think there's a typo. "qualifier name" should read "column family name" > Family Filter > This filter takes a compare operator and a comparator. It compares each > qualifier name with the comparator using the compare operator and if the > comparison returns true, it returns all the key-values in that column. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14502) Purge use of jmock and remove as dependency
[ https://issues.apache.org/jira/browse/HBASE-14502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Liptak updated HBASE-14502: - Attachment: HBASE-14502.1.patch > Purge use of jmock and remove as dependency > --- > > Key: HBASE-14502 > URL: https://issues.apache.org/jira/browse/HBASE-14502 > Project: HBase > Issue Type: Task > Components: test >Reporter: stack > Labels: beginner > Attachments: HBASE-14502.1.patch > > > jmock is a dependency used by one test only, TestBulkLoad. It looks like you > can do anything in mockito that can be done in jmock. Lets purge it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14436) HTableDescriptor#addCoprocessor will always make RegionCoprocessorHost create new Configuration
[
https://issues.apache.org/jira/browse/HBASE-14436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946136#comment-14946136
]
Hudson commented on HBASE-14436:
FAILURE: Integrated in HBase-1.2 #231 (See
[https://builds.apache.org/job/HBase-1.2/231/])
HBASE-14436 HTableDescriptor#addCoprocessor will always make (stack: rev
f4ea1862ac98c85fddf8e018beca7fb5b85c1f95)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java
> HTableDescriptor#addCoprocessor will always make RegionCoprocessorHost create
> new Configuration
> ---
>
> Key: HBASE-14436
> URL: https://issues.apache.org/jira/browse/HBASE-14436
> Project: HBase
> Issue Type: Improvement
> Components: Coprocessors
>Affects Versions: 1.2.1
>Reporter: Jianwei Cui
>Assignee: stack
>Priority: Minor
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: HBASE-14436-trunk-v1.patch, HBASE-14436-trunk-v2.patch
>
>
> HTableDescriptor#addCoprocessor will set the coprocessor value as following
> format:
> {code}
> public HTableDescriptor addCoprocessor(String className, Path jarFilePath,
> int priority, final Map kvs)
> throws IOException {
> ...
> String value = ((jarFilePath == null)? "" : jarFilePath.toString()) +
> "|" + className + "|" + Integer.toString(priority) + "|" +
> kvString.toString();
> ...
> }
> {code}
> If the 'jarFilePath' is null, the 'value' will always has the format
> '|className|priority|' even if 'kvs' is null, which means no extra arguments
> for the coprocessor. Then, in the server side,
> RegionCoprocessorHost#getTableCoprocessorAttrsFromSchema will load the table
> coprocessors as:
> {code}
> static List
> getTableCoprocessorAttrsFromSchema(Configuration conf,
> HTableDescriptor htd) {
> ...
> try {
> cfgSpec = matcher.group(4); // => cfgSpec will be '|' for the
> format '|className|priority|'
> } catch (IndexOutOfBoundsException ex) {
> // ignore
> }
> Configuration ourConf;
> if (cfgSpec != null) { // => cfgSpec will be '|' for the format
> '|className|priority|'
> ourConf = new Configuration(false);
> HBaseConfiguration.merge(ourConf, conf);
> }
> ...
> }
> {code}
> The 'cfgSpec' will be '|' for the coprocessor formatted as
> '|className|priority|', so that always create a new Configuration.
> In our production, there are a lot of tables having table-level coprocessors,
> so that the region server will create new Configurations for each region of
> the table, this will consume a certain number of memory when we have many
> such regions.
> To fix the problem, we can make the HTableDescriptor not append the '|' if no
> extra arguments for the coprocessor, or check the 'cfgSpec' more strictly in
> server side which could avoid creating new Configurations for existed such
> regions after the regions reopened. Discussions and suggestions are welcomed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14522) Document and/or disable hsperfdata GC pauses
[
https://issues.apache.org/jira/browse/HBASE-14522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946138#comment-14946138
]
Misty Stanley-Jones commented on HBASE-14522:
-
This says that it is ready for review, but the last comment seems to indicate
that there is still work to do. Update?
> Document and/or disable hsperfdata GC pauses
>
>
> Key: HBASE-14522
> URL: https://issues.apache.org/jira/browse/HBASE-14522
> Project: HBase
> Issue Type: Task
> Components: documentation, Performance
>Reporter: Nick Dimiduk
>Assignee: Lars Francke
> Attachments: HBASE-14522.patch
>
>
> {quote}
> The JVM by default exports statistics by mmap-ing a file in /tmp
> (hsperfdata). On Linux, modifying a memory mapped file can block until disk
> I/O completes, which can be hundreds of milliseconds. Since the JVM modifies
> these statistics during garbage collection and safepoints, this causes pauses
> that are hundreds of milliseconds long.
> {quote}
> Via [JVM mmap pause|http://www.evanjones.ca/jvm-mmap-pause.html].
> We should add {{-XX:+PerfDisableSharedMem}} to our default options as was
> apparently done in CASSANDRA-9242 and/or document the presence of this bug so
> operators know to use tmpfs.
> Hat-tip [~vrodionov]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14420) Zombie Stomping Session
[ https://issues.apache.org/jira/browse/HBASE-14420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946135#comment-14946135 ] stack commented on HBASE-14420: --- Looks like we are getting somewhere with my no-change patch. Let me rerun. > Zombie Stomping Session > --- > > Key: HBASE-14420 > URL: https://issues.apache.org/jira/browse/HBASE-14420 > Project: HBase > Issue Type: Umbrella > Components: test >Reporter: stack >Assignee: stack >Priority: Critical > Attachments: hangers.txt, none_fix.txt, none_fix.txt > > > Patch build are now failing most of the time because we are dropping zombies. > I confirm we are doing this on non-apache build boxes too. > Left-over zombies consume resources on build boxes (OOME cannot create native > threads). Having to do multiple test runs in the hope that we can get a > non-zombie-making build or making (arbitrary) rulings that the zombies are > 'not related' is a productivity sink. And so on... > This is an umbrella issue for a zombie stomping session that started earlier > this week. Will hang sub-issues of this one. Am running builds back-to-back > on little cluster to turn out the monsters. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14420) Zombie Stomping Session
[ https://issues.apache.org/jira/browse/HBASE-14420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14420: -- Attachment: none_fix.txt > Zombie Stomping Session > --- > > Key: HBASE-14420 > URL: https://issues.apache.org/jira/browse/HBASE-14420 > Project: HBase > Issue Type: Umbrella > Components: test >Reporter: stack >Assignee: stack >Priority: Critical > Attachments: hangers.txt, none_fix.txt, none_fix.txt > > > Patch build are now failing most of the time because we are dropping zombies. > I confirm we are doing this on non-apache build boxes too. > Left-over zombies consume resources on build boxes (OOME cannot create native > threads). Having to do multiple test runs in the hope that we can get a > non-zombie-making build or making (arbitrary) rulings that the zombies are > 'not related' is a productivity sink. And so on... > This is an umbrella issue for a zombie stomping session that started earlier > this week. Will hang sub-issues of this one. Am running builds back-to-back > on little cluster to turn out the monsters. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14424) Document that DisabledRegionSplitPolicy blocks manual splits
[ https://issues.apache.org/jira/browse/HBASE-14424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-14424: Resolution: Fixed Status: Resolved (was: Patch Available) Made the requested change, pushed to master. > Document that DisabledRegionSplitPolicy blocks manual splits > > > Key: HBASE-14424 > URL: https://issues.apache.org/jira/browse/HBASE-14424 > Project: HBase > Issue Type: Task > Components: documentation >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-14424.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14424) Document that DisabledRegionSplitPolicy blocks manual splits
[ https://issues.apache.org/jira/browse/HBASE-14424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-14424: Hadoop Flags: Reviewed Fix Version/s: 2.0.0 > Document that DisabledRegionSplitPolicy blocks manual splits > > > Key: HBASE-14424 > URL: https://issues.apache.org/jira/browse/HBASE-14424 > Project: HBase > Issue Type: Task > Components: documentation >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-14424.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12983) HBase book mentions hadoop.ssl.enabled when it should be hbase.ssl.enabled
[
https://issues.apache.org/jira/browse/HBASE-12983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Misty Stanley-Jones updated HBASE-12983:
Resolution: Fixed
Fix Version/s: 2.0.0
Status: Resolved (was: Patch Available)
> HBase book mentions hadoop.ssl.enabled when it should be hbase.ssl.enabled
> --
>
> Key: HBASE-12983
> URL: https://issues.apache.org/jira/browse/HBASE-12983
> Project: HBase
> Issue Type: Bug
> Components: documentation
>Reporter: Esteban Gutierrez
>Assignee: Misty Stanley-Jones
> Fix For: 2.0.0
>
> Attachments: HBASE-12983.patch
>
>
> In the HBase book we say the following:
> {quote}
> A default HBase install uses insecure HTTP connections for web UIs for the
> master and region servers. To enable secure HTTP (HTTPS) connections instead,
> set *hadoop.ssl.enabled* to true in hbase-site.xml. This does not change the
> port used by the Web UI. To change the port for the web UI for a given HBase
> component, configure that port’s setting in hbase-site.xml. These settings
> are:
> {quote}
> The property should be *hbase.ssl.enabled* instead.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12983) HBase book mentions hadoop.ssl.enabled when it should be hbase.ssl.enabled
[
https://issues.apache.org/jira/browse/HBASE-12983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Misty Stanley-Jones updated HBASE-12983:
Hadoop Flags: Reviewed
Committed to master.
> HBase book mentions hadoop.ssl.enabled when it should be hbase.ssl.enabled
> --
>
> Key: HBASE-12983
> URL: https://issues.apache.org/jira/browse/HBASE-12983
> Project: HBase
> Issue Type: Bug
> Components: documentation
>Reporter: Esteban Gutierrez
>Assignee: Misty Stanley-Jones
> Fix For: 2.0.0
>
> Attachments: HBASE-12983.patch
>
>
> In the HBase book we say the following:
> {quote}
> A default HBase install uses insecure HTTP connections for web UIs for the
> master and region servers. To enable secure HTTP (HTTPS) connections instead,
> set *hadoop.ssl.enabled* to true in hbase-site.xml. This does not change the
> port used by the Web UI. To change the port for the web UI for a given HBase
> component, configure that port’s setting in hbase-site.xml. These settings
> are:
> {quote}
> The property should be *hbase.ssl.enabled* instead.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12615) Document GC conserving guidelines for contributors
[ https://issues.apache.org/jira/browse/HBASE-12615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-12615: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.0.0 Status: Resolved (was: Patch Available) Verified by Jon. Pushed to master. > Document GC conserving guidelines for contributors > -- > > Key: HBASE-12615 > URL: https://issues.apache.org/jira/browse/HBASE-12615 > Project: HBase > Issue Type: Bug > Components: documentation >Reporter: Andrew Purtell >Assignee: Misty Stanley-Jones > Fix For: 2.0.0 > > Attachments: HBASE-12615.patch > > > LinkedIn put up a blog post with a nice concise list of GC conserving > techniques we should document for contributors. Additionally, when we're at a > point our build supports custom error-prone plugins, we can develop warnings > for some of them. > Source: > http://engineering.linkedin.com/performance/linkedin-feed-faster-less-jvm-garbage > - Be careful with Iterators > - Estimate the size of a collection when initializing > - Defer expression evaluation > - Compile the regex patterns in advance > - Cache it if you can > - String Interns are useful but dangerous > All good advice and practice that I know we aim for. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14436) HTableDescriptor#addCoprocessor will always make RegionCoprocessorHost create new Configuration
[
https://issues.apache.org/jira/browse/HBASE-14436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946124#comment-14946124
]
Hudson commented on HBASE-14436:
SUCCESS: Integrated in HBase-1.2-IT #193 (See
[https://builds.apache.org/job/HBase-1.2-IT/193/])
HBASE-14436 HTableDescriptor#addCoprocessor will always make (stack: rev
f4ea1862ac98c85fddf8e018beca7fb5b85c1f95)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java
> HTableDescriptor#addCoprocessor will always make RegionCoprocessorHost create
> new Configuration
> ---
>
> Key: HBASE-14436
> URL: https://issues.apache.org/jira/browse/HBASE-14436
> Project: HBase
> Issue Type: Improvement
> Components: Coprocessors
>Affects Versions: 1.2.1
>Reporter: Jianwei Cui
>Assignee: stack
>Priority: Minor
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: HBASE-14436-trunk-v1.patch, HBASE-14436-trunk-v2.patch
>
>
> HTableDescriptor#addCoprocessor will set the coprocessor value as following
> format:
> {code}
> public HTableDescriptor addCoprocessor(String className, Path jarFilePath,
> int priority, final Map kvs)
> throws IOException {
> ...
> String value = ((jarFilePath == null)? "" : jarFilePath.toString()) +
> "|" + className + "|" + Integer.toString(priority) + "|" +
> kvString.toString();
> ...
> }
> {code}
> If the 'jarFilePath' is null, the 'value' will always has the format
> '|className|priority|' even if 'kvs' is null, which means no extra arguments
> for the coprocessor. Then, in the server side,
> RegionCoprocessorHost#getTableCoprocessorAttrsFromSchema will load the table
> coprocessors as:
> {code}
> static List
> getTableCoprocessorAttrsFromSchema(Configuration conf,
> HTableDescriptor htd) {
> ...
> try {
> cfgSpec = matcher.group(4); // => cfgSpec will be '|' for the
> format '|className|priority|'
> } catch (IndexOutOfBoundsException ex) {
> // ignore
> }
> Configuration ourConf;
> if (cfgSpec != null) { // => cfgSpec will be '|' for the format
> '|className|priority|'
> ourConf = new Configuration(false);
> HBaseConfiguration.merge(ourConf, conf);
> }
> ...
> }
> {code}
> The 'cfgSpec' will be '|' for the coprocessor formatted as
> '|className|priority|', so that always create a new Configuration.
> In our production, there are a lot of tables having table-level coprocessors,
> so that the region server will create new Configurations for each region of
> the table, this will consume a certain number of memory when we have many
> such regions.
> To fix the problem, we can make the HTableDescriptor not append the '|' if no
> extra arguments for the coprocessor, or check the 'cfgSpec' more strictly in
> server side which could avoid creating new Configurations for existed such
> regions after the regions reopened. Discussions and suggestions are welcomed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14563) Disable zombie TestHFileOutputFormat2
[ https://issues.apache.org/jira/browse/HBASE-14563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946123#comment-14946123 ] Hudson commented on HBASE-14563: SUCCESS: Integrated in HBase-1.2-IT #193 (See [https://builds.apache.org/job/HBase-1.2-IT/193/]) HBASE-14563 Disable zombie TestHFileOutputFormat2 (stack: rev 22c87d9644c600788a0df5456333464cba969c49) * hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat2.java > Disable zombie TestHFileOutputFormat2 > - > > Key: HBASE-14563 > URL: https://issues.apache.org/jira/browse/HBASE-14563 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack >Assignee: stack > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: 14563.txt > > > Disabling until someone has a chance to look at it. > I watched it in jvisualvm a while. Its starting and stopping clusters > multiple times and then running mr jobs. Needs a rewrite at least and some > shrinking of scope on what is tested. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14420) Zombie Stomping Session
[
https://issues.apache.org/jira/browse/HBASE-14420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946114#comment-14946114
]
Hadoop QA commented on HBASE-14420:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12765252/none_fix.txt
against master branch at commit 0ea1f8122709302ee19279aaa438b37dac30c25b.
ATTACHMENT ID: 12765252
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation, build,
or dev-support patch that doesn't require tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15886//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15886//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15886//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15886//console
This message is automatically generated.
> Zombie Stomping Session
> ---
>
> Key: HBASE-14420
> URL: https://issues.apache.org/jira/browse/HBASE-14420
> Project: HBase
> Issue Type: Umbrella
> Components: test
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: hangers.txt, none_fix.txt
>
>
> Patch build are now failing most of the time because we are dropping zombies.
> I confirm we are doing this on non-apache build boxes too.
> Left-over zombies consume resources on build boxes (OOME cannot create native
> threads). Having to do multiple test runs in the hope that we can get a
> non-zombie-making build or making (arbitrary) rulings that the zombies are
> 'not related' is a productivity sink. And so on...
> This is an umbrella issue for a zombie stomping session that started earlier
> this week. Will hang sub-issues of this one. Am running builds back-to-back
> on little cluster to turn out the monsters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-14566) RegionServer status UI shows maxes out on number of blocks in a blockcache level at 100K
Ashu Pachauri created HBASE-14566: - Summary: RegionServer status UI shows maxes out on number of blocks in a blockcache level at 100K Key: HBASE-14566 URL: https://issues.apache.org/jira/browse/HBASE-14566 Project: HBase Issue Type: Bug Components: UI Affects Versions: 1.1.2 Reporter: Ashu Pachauri Priority: Minor Numbers greater than 100K are shown as 100K while those lesser are correctly displayed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14566) RegionServer status UI maxes out on number of blocks in a blockcache level at 100K
[ https://issues.apache.org/jira/browse/HBASE-14566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashu Pachauri updated HBASE-14566: -- Summary: RegionServer status UI maxes out on number of blocks in a blockcache level at 100K (was: RegionServer status UI shows maxes out on number of blocks in a blockcache level at 100K) > RegionServer status UI maxes out on number of blocks in a blockcache level at > 100K > -- > > Key: HBASE-14566 > URL: https://issues.apache.org/jira/browse/HBASE-14566 > Project: HBase > Issue Type: Bug > Components: UI >Affects Versions: 1.1.2 >Reporter: Ashu Pachauri >Priority: Minor > > Numbers greater than 100K are shown as 100K while those lesser are correctly > displayed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13819) Make RPC layer CellBlock buffer a DirectByteBuffer
[
https://issues.apache.org/jira/browse/HBASE-13819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946078#comment-14946078
]
Hadoop QA commented on HBASE-13819:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12765268/HBASE-13819_branch-1.patch
against branch-1 branch at commit 0ea1f8122709302ee19279aaa438b37dac30c25b.
ATTACHMENT ID: 12765268
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hbase.security.access.TestCellACLs.testCoveringCheck(TestCellACLs.java:402)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15887//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15887//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15887//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15887//console
This message is automatically generated.
> Make RPC layer CellBlock buffer a DirectByteBuffer
> --
>
> Key: HBASE-13819
> URL: https://issues.apache.org/jira/browse/HBASE-13819
> Project: HBase
> Issue Type: Sub-task
> Components: Scanners
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-13819.patch, HBASE-13819_branch-1.patch,
> HBASE-13819_branch-1.patch, HBASE-13819_branch-1.patch
>
>
> In RPC layer, when we make a cellBlock to put as RPC payload, we will make an
> on heap byte buffer (via BoundedByteBufferPool). The pool will keep upto
> certain number of buffers. This jira aims at testing possibility for making
> this buffers off heap ones. (DBB) The advantages
> 1. Unsafe based writes to off heap is faster than that to on heap. Now we are
> not using unsafe based writes at all. Even if we add, DBB will be better
> 2. When Cells are backed by off heap (HBASE-11425) off heap to off heap
> writes will be better
> 3. When checked the code in SocketChannel impl, if we pass a HeapByteBuffer
> to the socket channel, it will create a temp DBB and copy data to there and
> only DBBs will be moved to Sockets. If we make DBB 1st hand itself, we can
> avoid this one more level of copying.
> Will do different perf testing with changed and report back.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14436) HTableDescriptor#addCoprocessor will always make RegionCoprocessorHost create new Configuration
[
https://issues.apache.org/jira/browse/HBASE-14436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946068#comment-14946068
]
Hudson commented on HBASE-14436:
FAILURE: Integrated in HBase-1.3 #239 (See
[https://builds.apache.org/job/HBase-1.3/239/])
HBASE-14436 HTableDescriptor#addCoprocessor will always make (stack: rev
ab33a65a1adae33025a1034568397d6d67bf476a)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java
> HTableDescriptor#addCoprocessor will always make RegionCoprocessorHost create
> new Configuration
> ---
>
> Key: HBASE-14436
> URL: https://issues.apache.org/jira/browse/HBASE-14436
> Project: HBase
> Issue Type: Improvement
> Components: Coprocessors
>Affects Versions: 1.2.1
>Reporter: Jianwei Cui
>Assignee: stack
>Priority: Minor
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: HBASE-14436-trunk-v1.patch, HBASE-14436-trunk-v2.patch
>
>
> HTableDescriptor#addCoprocessor will set the coprocessor value as following
> format:
> {code}
> public HTableDescriptor addCoprocessor(String className, Path jarFilePath,
> int priority, final Map kvs)
> throws IOException {
> ...
> String value = ((jarFilePath == null)? "" : jarFilePath.toString()) +
> "|" + className + "|" + Integer.toString(priority) + "|" +
> kvString.toString();
> ...
> }
> {code}
> If the 'jarFilePath' is null, the 'value' will always has the format
> '|className|priority|' even if 'kvs' is null, which means no extra arguments
> for the coprocessor. Then, in the server side,
> RegionCoprocessorHost#getTableCoprocessorAttrsFromSchema will load the table
> coprocessors as:
> {code}
> static List
> getTableCoprocessorAttrsFromSchema(Configuration conf,
> HTableDescriptor htd) {
> ...
> try {
> cfgSpec = matcher.group(4); // => cfgSpec will be '|' for the
> format '|className|priority|'
> } catch (IndexOutOfBoundsException ex) {
> // ignore
> }
> Configuration ourConf;
> if (cfgSpec != null) { // => cfgSpec will be '|' for the format
> '|className|priority|'
> ourConf = new Configuration(false);
> HBaseConfiguration.merge(ourConf, conf);
> }
> ...
> }
> {code}
> The 'cfgSpec' will be '|' for the coprocessor formatted as
> '|className|priority|', so that always create a new Configuration.
> In our production, there are a lot of tables having table-level coprocessors,
> so that the region server will create new Configurations for each region of
> the table, this will consume a certain number of memory when we have many
> such regions.
> To fix the problem, we can make the HTableDescriptor not append the '|' if no
> extra arguments for the coprocessor, or check the 'cfgSpec' more strictly in
> server side which could avoid creating new Configurations for existed such
> regions after the regions reopened. Discussions and suggestions are welcomed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14497) Reverse Scan threw StackOverflow caused by readPt checking
[ https://issues.apache.org/jira/browse/HBASE-14497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946069#comment-14946069 ] Hudson commented on HBASE-14497: FAILURE: Integrated in HBase-1.3 #239 (See [https://builds.apache.org/job/HBase-1.3/239/]) HBASE-14497 Reverse Scan threw StackOverflow caused by readPt checking (tedyu: rev 5e2db42d680f35c1f0a345344a6555ea319d870b) * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultMemStore.java > Reverse Scan threw StackOverflow caused by readPt checking > -- > > Key: HBASE-14497 > URL: https://issues.apache.org/jira/browse/HBASE-14497 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 0.98.14, 1.3.0 >Reporter: Yerui Sun >Assignee: Yerui Sun > Fix For: 2.0.0, 1.3.0 > > Attachments: 14497-branch-1-v6.patch, 14497-master-v6.patch, > HBASE-14497-0.98.patch, HBASE-14497-branch-1-v2.patch, > HBASE-14497-branch-1-v3.patch, HBASE-14497-branch-1-v6.patch, > HBASE-14497-branch-1.patch, HBASE-14497-master-v2.patch, > HBASE-14497-master-v3.patch, HBASE-14497-master-v3.patch, > HBASE-14497-master-v4.patch, HBASE-14497-master-v5.patch, > HBASE-14497-master.patch > > > I met stack overflow error in StoreFileScanner.seekToPreviousRow using > reversed scan. I searched and founded HBASE-14155, but it seems to be a > different reason. > The seekToPreviousRow will fetch the row which closest before, and compare > mvcc to the readPt, which acquired when scanner created. If the row's mvcc is > bigger than readPt, an recursive call of seekToPreviousRow will invoked, to > find the next closest before row. > Considering we created a scanner for reversed scan, and some data with > smaller rows was written and flushed, before calling scanner next. When > seekToPreviousRow was invoked, it would call itself recursively, until all > rows which written after scanner created were iterated. The depth of > recursive calling stack depends on the count of rows, the stack overflow > error will be threw if the count of rows is large, like 1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14563) Disable zombie TestHFileOutputFormat2
[ https://issues.apache.org/jira/browse/HBASE-14563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946067#comment-14946067 ] Hudson commented on HBASE-14563: FAILURE: Integrated in HBase-1.3 #239 (See [https://builds.apache.org/job/HBase-1.3/239/]) HBASE-14563 Disable zombie TestHFileOutputFormat2 (stack: rev aeb3a624590be8bd276e58bba9d4debfb3e7759f) * hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat2.java > Disable zombie TestHFileOutputFormat2 > - > > Key: HBASE-14563 > URL: https://issues.apache.org/jira/browse/HBASE-14563 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack >Assignee: stack > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: 14563.txt > > > Disabling until someone has a chance to look at it. > I watched it in jvisualvm a while. Its starting and stopping clusters > multiple times and then running mr jobs. Needs a rewrite at least and some > shrinking of scope on what is tested. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14436) HTableDescriptor#addCoprocessor will always make RegionCoprocessorHost create new Configuration
[
https://issues.apache.org/jira/browse/HBASE-14436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946065#comment-14946065
]
Hudson commented on HBASE-14436:
FAILURE: Integrated in HBase-TRUNK #6879 (See
[https://builds.apache.org/job/HBase-TRUNK/6879/])
HBASE-14436 HTableDescriptor#addCoprocessor will always make (stack: rev
0ea1f8122709302ee19279aaa438b37dac30c25b)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java
> HTableDescriptor#addCoprocessor will always make RegionCoprocessorHost create
> new Configuration
> ---
>
> Key: HBASE-14436
> URL: https://issues.apache.org/jira/browse/HBASE-14436
> Project: HBase
> Issue Type: Improvement
> Components: Coprocessors
>Affects Versions: 1.2.1
>Reporter: Jianwei Cui
>Assignee: stack
>Priority: Minor
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: HBASE-14436-trunk-v1.patch, HBASE-14436-trunk-v2.patch
>
>
> HTableDescriptor#addCoprocessor will set the coprocessor value as following
> format:
> {code}
> public HTableDescriptor addCoprocessor(String className, Path jarFilePath,
> int priority, final Map kvs)
> throws IOException {
> ...
> String value = ((jarFilePath == null)? "" : jarFilePath.toString()) +
> "|" + className + "|" + Integer.toString(priority) + "|" +
> kvString.toString();
> ...
> }
> {code}
> If the 'jarFilePath' is null, the 'value' will always has the format
> '|className|priority|' even if 'kvs' is null, which means no extra arguments
> for the coprocessor. Then, in the server side,
> RegionCoprocessorHost#getTableCoprocessorAttrsFromSchema will load the table
> coprocessors as:
> {code}
> static List
> getTableCoprocessorAttrsFromSchema(Configuration conf,
> HTableDescriptor htd) {
> ...
> try {
> cfgSpec = matcher.group(4); // => cfgSpec will be '|' for the
> format '|className|priority|'
> } catch (IndexOutOfBoundsException ex) {
> // ignore
> }
> Configuration ourConf;
> if (cfgSpec != null) { // => cfgSpec will be '|' for the format
> '|className|priority|'
> ourConf = new Configuration(false);
> HBaseConfiguration.merge(ourConf, conf);
> }
> ...
> }
> {code}
> The 'cfgSpec' will be '|' for the coprocessor formatted as
> '|className|priority|', so that always create a new Configuration.
> In our production, there are a lot of tables having table-level coprocessors,
> so that the region server will create new Configurations for each region of
> the table, this will consume a certain number of memory when we have many
> such regions.
> To fix the problem, we can make the HTableDescriptor not append the '|' if no
> extra arguments for the coprocessor, or check the 'cfgSpec' more strictly in
> server side which could avoid creating new Configurations for existed such
> regions after the regions reopened. Discussions and suggestions are welcomed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14563) Disable zombie TestHFileOutputFormat2
[ https://issues.apache.org/jira/browse/HBASE-14563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946064#comment-14946064 ] Hudson commented on HBASE-14563: FAILURE: Integrated in HBase-TRUNK #6879 (See [https://builds.apache.org/job/HBase-TRUNK/6879/]) HBASE-14563 Disable zombie TestHFileOutputFormat2 (stack: rev 8fcc8155042766121cb4e99433f23affe2d9ae2d) * hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat2.java > Disable zombie TestHFileOutputFormat2 > - > > Key: HBASE-14563 > URL: https://issues.apache.org/jira/browse/HBASE-14563 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack >Assignee: stack > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: 14563.txt > > > Disabling until someone has a chance to look at it. > I watched it in jvisualvm a while. Its starting and stopping clusters > multiple times and then running mr jobs. Needs a rewrite at least and some > shrinking of scope on what is tested. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12911) Client-side metrics
[
https://issues.apache.org/jira/browse/HBASE-12911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946058#comment-14946058
]
Nick Dimiduk commented on HBASE-12911:
--
bq. there is no unpacking that I saw...
Unpacking in that I'm reading into the PB Method and switching on the index of
the entry; it's based on the generated code so I assume it's an implementation
detail that could change in the future. See {{MetricsConnection#updateRpc}}.
bq. How does an operator use this stuff?
Let me add a release note. Right now they have to look at the JMX of the
machine running the client. After HBASE-14381 we'll be exposing the metrics
programatically. Do we want another follow-on to allow changing the reporter?
This version of yammer also ships with a {{ConsoleReporter}} that allows
reporting to System.out. What about disabling client-side metrics collection
entirely?
> Client-side metrics
> ---
>
> Key: HBASE-12911
> URL: https://issues.apache.org/jira/browse/HBASE-12911
> Project: HBase
> Issue Type: New Feature
> Components: Client, Operability, Performance
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
> Fix For: 2.0.0, 1.3.0
>
> Attachments: 0001-HBASE-12911-Client-side-metrics.patch,
> 0001-HBASE-12911-Client-side-metrics.patch,
> 0001-HBASE-12911-Client-side-metrics.patch,
> 0001-HBASE-12911-Client-side-metrics.patch,
> 0001-HBASE-12911-Client-side-metrics.patch,
> 0001-HBASE-12911-Client-side-metrics.patch,
> 0001-HBASE-12911-Client-side-metrics.patch,
> 0001-HBASE-12911-Client-side-metrics.patch,
> 0001-HBASE-12911-Client-side-metrics.patch,
> 0001-HBASE-12911-Client-side-metrics.patch,
> 0001-HBASE-12911-Client-side-metrics.patch, 12911-0.98.00.patch,
> 12911-branch-1.00.patch, 12911.yammer.jpg, 12911.yammer.v00.patch,
> 12911.yammer.v01.patch, 12911.yammer.v02.patch, 12911.yammer.v02.patch,
> 12911.yammer.v03.branch-1.patch, 12911.yammer.v03.patch,
> 12911.yammer.v03.patch, am.jpg, client metrics RS-Master.jpg, client metrics
> client.jpg, conn_agg.jpg, connection attributes.jpg, ltt.jpg, standalone.jpg
>
>
> There's very little visibility into the hbase client. Folks who care to add
> some kind of metrics collection end up wrapping Table method invocations with
> {{System.currentTimeMillis()}}. For a crude example of this, have a look at
> what I did in {{PerformanceEvaluation}} for exposing requests latencies up to
> {{IntegrationTestRegionReplicaPerf}}. The client is quite complex, there's a
> lot going on under the hood that is impossible to see right now without a
> profiler. Being a crucial part of the performance of this distributed system,
> we should have deeper visibility into the client's function.
> I'm not sure that wiring into the hadoop metrics system is the right choice
> because the client is often embedded as a library in a user's application. We
> should have integration with our metrics tools so that, i.e., a client
> embedded in a coprocessor can report metrics through the usual RS channels,
> or a client used in a MR job can do the same.
> I would propose an interface-based system with pluggable implementations. Out
> of the box we'd include a hadoop-metrics implementation and one other,
> possibly [dropwizard/metrics|https://github.com/dropwizard/metrics].
> Thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14458) AsyncRpcClient#createRpcChannel() should check and remove dead channel before creating new one to same server
[
https://issues.apache.org/jira/browse/HBASE-14458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-14458:
--
Attachment: HBASE-14458 (1).patch
Retry
> AsyncRpcClient#createRpcChannel() should check and remove dead channel before
> creating new one to same server
> -
>
> Key: HBASE-14458
> URL: https://issues.apache.org/jira/browse/HBASE-14458
> Project: HBase
> Issue Type: Bug
> Components: IPC/RPC
>Affects Versions: 2.0.0, 1.2.0, 1.3.0, 1.1.3
>Reporter: Samir Ahmic
>Assignee: Samir Ahmic
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-14458 (1).patch, HBASE-14458.patch,
> HBASE-14458.patch
>
>
> I have notice this issue while testing master branch in distributed mode.
> Reproduction steps:
> 1. Write some data with hbase ltt
> 2. While ltt is writing execute $graceful_stop.sh --restart --reload [rs]
> 3. Wait until script start to reload regions to restarted server. In that
> moment ltt will stop writing and eventually fail.
> After some digging i have notice that while ltt is working correctly there is
> single connection per regionserver (lsof for single connection, 27109 is ltt
> PID )
> {code}
> java 27109 hbase 143u210579579 0t0TCP
> hnode1:40423->hnode5:16020 (ESTABLISHED)
> {code}
> and when in this example hnode5 server is restarted and script starts to
> reload regions on this server ltt start creating thousands of new tcp
> connections to this server:
> {code}
> java 27109 hbase *623u 210674415 0t0TCP
> hnode1:52948->hnode5:16020 (ESTABLISHED)
> java 27109 hbase *624u 210674416 0t0TCP
> hnode1:52949->hnode5:16020 (ESTABLISHED)
> java 27109 hbase *625u 210674417 0t0TCP
> hnode1:52950->hnode5:16020 (ESTABLISHED)
> java 27109 hbase *627u 210674419 0t0TCP
> hnode1:52952->hnode5:16020 (ESTABLISHED)
> java 27109 hbase *628u 210674420 0t0TCP
> hnode1:52953->hnode5:16020 (ESTABLISHED)
> java 27109 hbase *633u 210674425 0t0TCP
> hnode1:52958->hnode5:16020 (ESTABLISHED)
> ...
> {code}
> So here is what happened based on some additional logging and debugging:
> - AsyncRpcClient never detected that regionserver is restarted because
> regions were moved and there was no write/read requests to this server and
> there is no some sort of heart-bit mechanism implemented
> - because of above dead {code}AsyncRpcChannel{code} stayed in
> {code}PoolMap connections{code}
> - when ltt detected that regions are moved back to hnode5 it tried to
> reconnect to hnode5 leading this issue
> I was able to resolve this issue by adding following to
> AsyncRpcClient#createRpcChannel():
> {code}
> synchronized (connections) {
> if (closed) {
> throw new StoppedRpcClientException();
> }
> rpcChannel = connections.get(hashCode);
> +if (rpcChannel != null && !rpcChannel.isAlive()) {
> +LOG.debug(Removing dead channel from "+
> rpcChannel.address.toString());
> +connections.remove(hashCode);
> + }
> if (rpcChannel == null || !rpcChannel.isAlive()) {
> rpcChannel = new AsyncRpcChannel(this.bootstrap, this, ticket,
> serviceName, location);
> connections.put(hashCode, rpcChannel);
> {code}
> I will attach patch after some more testing.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14479) Apply the Leader/Followers pattern to RpcServer's Reader
[
https://issues.apache.org/jira/browse/HBASE-14479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946052#comment-14946052
]
stack commented on HBASE-14479:
---
Here's link http://www.kircher-schwanninger.de/michael/publications/lf.pdf I
like the explanation here too:
http://stackoverflow.com/questions/3058272/explain-leader-follower-pattern
Patch seems good. You tried it [~ikeda] (if you'd messed up, unit tests would
be failing...) Anyway we could figure if a benefit? I can try running on a
cluster and see Thanks [~ikeda]
> Apply the Leader/Followers pattern to RpcServer's Reader
>
>
> Key: HBASE-14479
> URL: https://issues.apache.org/jira/browse/HBASE-14479
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC, Performance
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Attachments: HBASE-14479-V2.patch, HBASE-14479-V2.patch,
> HBASE-14479.patch
>
>
> {{RpcServer}} uses multiple selectors to read data for load distribution, but
> the distribution is just done by round-robin. It is uncertain, especially for
> long run, whether load is equally divided and resources are used without
> being wasted.
> Moreover, multiple selectors may cause excessive context switches which give
> priority to low latency (while we just add the requests to queues), and it is
> possible to reduce throughput of the whole server.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14479) Apply the Leader/Followers pattern to RpcServer's Reader
[
https://issues.apache.org/jira/browse/HBASE-14479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-14479:
--
Attachment: HBASE-14479-V2.patch
Retry.
> Apply the Leader/Followers pattern to RpcServer's Reader
>
>
> Key: HBASE-14479
> URL: https://issues.apache.org/jira/browse/HBASE-14479
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC, Performance
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Attachments: HBASE-14479-V2.patch, HBASE-14479-V2.patch,
> HBASE-14479.patch
>
>
> {{RpcServer}} uses multiple selectors to read data for load distribution, but
> the distribution is just done by round-robin. It is uncertain, especially for
> long run, whether load is equally divided and resources are used without
> being wasted.
> Moreover, multiple selectors may cause excessive context switches which give
> priority to low latency (while we just add the requests to queues), and it is
> possible to reduce throughput of the whole server.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14451) Move on to htrace-4.0.1 (from htrace-3.2.0)
[ https://issues.apache.org/jira/browse/HBASE-14451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14451: -- Attachment: 14451.v10.txt Retry. Rebase. > Move on to htrace-4.0.1 (from htrace-3.2.0) > --- > > Key: HBASE-14451 > URL: https://issues.apache.org/jira/browse/HBASE-14451 > Project: HBase > Issue Type: Task >Reporter: stack >Assignee: stack > Attachments: 14451.txt, 14451.v10.txt, 14451v2.txt, 14451v3.txt, > 14451v4.txt, 14451v5.txt, 14451v6.txt, 14451v7.txt, 14451v8.txt, 14451v9.txt > > > htrace-4.0.0 was just release with a new API. Get up on it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12911) Client-side metrics
[
https://issues.apache.org/jira/browse/HBASE-12911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946029#comment-14946029
]
stack commented on HBASE-12911:
---
I'm good w/ use of pb (there is no unpacking that I saw...) +1'd the patch.
How does an operator use this stuff? They'd have to look for client jmx
footprint on a machine? Needs a bit of doc in the release note. Nice addition.
> Client-side metrics
> ---
>
> Key: HBASE-12911
> URL: https://issues.apache.org/jira/browse/HBASE-12911
> Project: HBase
> Issue Type: New Feature
> Components: Client, Operability, Performance
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
> Fix For: 2.0.0, 1.3.0
>
> Attachments: 0001-HBASE-12911-Client-side-metrics.patch,
> 0001-HBASE-12911-Client-side-metrics.patch,
> 0001-HBASE-12911-Client-side-metrics.patch,
> 0001-HBASE-12911-Client-side-metrics.patch,
> 0001-HBASE-12911-Client-side-metrics.patch,
> 0001-HBASE-12911-Client-side-metrics.patch,
> 0001-HBASE-12911-Client-side-metrics.patch,
> 0001-HBASE-12911-Client-side-metrics.patch,
> 0001-HBASE-12911-Client-side-metrics.patch,
> 0001-HBASE-12911-Client-side-metrics.patch,
> 0001-HBASE-12911-Client-side-metrics.patch, 12911-0.98.00.patch,
> 12911-branch-1.00.patch, 12911.yammer.jpg, 12911.yammer.v00.patch,
> 12911.yammer.v01.patch, 12911.yammer.v02.patch, 12911.yammer.v02.patch,
> 12911.yammer.v03.branch-1.patch, 12911.yammer.v03.patch,
> 12911.yammer.v03.patch, am.jpg, client metrics RS-Master.jpg, client metrics
> client.jpg, conn_agg.jpg, connection attributes.jpg, ltt.jpg, standalone.jpg
>
>
> There's very little visibility into the hbase client. Folks who care to add
> some kind of metrics collection end up wrapping Table method invocations with
> {{System.currentTimeMillis()}}. For a crude example of this, have a look at
> what I did in {{PerformanceEvaluation}} for exposing requests latencies up to
> {{IntegrationTestRegionReplicaPerf}}. The client is quite complex, there's a
> lot going on under the hood that is impossible to see right now without a
> profiler. Being a crucial part of the performance of this distributed system,
> we should have deeper visibility into the client's function.
> I'm not sure that wiring into the hadoop metrics system is the right choice
> because the client is often embedded as a library in a user's application. We
> should have integration with our metrics tools so that, i.e., a client
> embedded in a coprocessor can report metrics through the usual RS channels,
> or a client used in a MR job can do the same.
> I would propose an interface-based system with pluggable implementations. Out
> of the box we'd include a hadoop-metrics implementation and one other,
> possibly [dropwizard/metrics|https://github.com/dropwizard/metrics].
> Thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14436) HTableDescriptor#addCoprocessor will always make RegionCoprocessorHost create new Configuration
[
https://issues.apache.org/jira/browse/HBASE-14436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946028#comment-14946028
]
Hudson commented on HBASE-14436:
FAILURE: Integrated in HBase-1.1 #697 (See
[https://builds.apache.org/job/HBase-1.1/697/])
HBASE-14436 HTableDescriptor#addCoprocessor will always make (stack: rev
2c662898037b6ad9e17399f0c7914bc785622202)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java
> HTableDescriptor#addCoprocessor will always make RegionCoprocessorHost create
> new Configuration
> ---
>
> Key: HBASE-14436
> URL: https://issues.apache.org/jira/browse/HBASE-14436
> Project: HBase
> Issue Type: Improvement
> Components: Coprocessors
>Affects Versions: 1.2.1
>Reporter: Jianwei Cui
>Assignee: stack
>Priority: Minor
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: HBASE-14436-trunk-v1.patch, HBASE-14436-trunk-v2.patch
>
>
> HTableDescriptor#addCoprocessor will set the coprocessor value as following
> format:
> {code}
> public HTableDescriptor addCoprocessor(String className, Path jarFilePath,
> int priority, final Map kvs)
> throws IOException {
> ...
> String value = ((jarFilePath == null)? "" : jarFilePath.toString()) +
> "|" + className + "|" + Integer.toString(priority) + "|" +
> kvString.toString();
> ...
> }
> {code}
> If the 'jarFilePath' is null, the 'value' will always has the format
> '|className|priority|' even if 'kvs' is null, which means no extra arguments
> for the coprocessor. Then, in the server side,
> RegionCoprocessorHost#getTableCoprocessorAttrsFromSchema will load the table
> coprocessors as:
> {code}
> static List
> getTableCoprocessorAttrsFromSchema(Configuration conf,
> HTableDescriptor htd) {
> ...
> try {
> cfgSpec = matcher.group(4); // => cfgSpec will be '|' for the
> format '|className|priority|'
> } catch (IndexOutOfBoundsException ex) {
> // ignore
> }
> Configuration ourConf;
> if (cfgSpec != null) { // => cfgSpec will be '|' for the format
> '|className|priority|'
> ourConf = new Configuration(false);
> HBaseConfiguration.merge(ourConf, conf);
> }
> ...
> }
> {code}
> The 'cfgSpec' will be '|' for the coprocessor formatted as
> '|className|priority|', so that always create a new Configuration.
> In our production, there are a lot of tables having table-level coprocessors,
> so that the region server will create new Configurations for each region of
> the table, this will consume a certain number of memory when we have many
> such regions.
> To fix the problem, we can make the HTableDescriptor not append the '|' if no
> extra arguments for the coprocessor, or check the 'cfgSpec' more strictly in
> server side which could avoid creating new Configurations for existed such
> regions after the regions reopened. Discussions and suggestions are welcomed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14268) Improve KeyLocker
[
https://issues.apache.org/jira/browse/HBASE-14268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-14268:
--
Attachment: HBASE-14268-V7.patch
Reattach.
[~ikeda] That is interesting. Weak references will be collected by GC if in new
gen but not if it makes it up into old gen (You should do a blog post on your
findings here).
> Improve KeyLocker
> -
>
> Key: HBASE-14268
> URL: https://issues.apache.org/jira/browse/HBASE-14268
> Project: HBase
> Issue Type: Improvement
> Components: util
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Fix For: 2.0.0, 1.3.0
>
> Attachments: 14268-V5.patch, HBASE-14268-V2.patch,
> HBASE-14268-V3.patch, HBASE-14268-V4.patch, HBASE-14268-V5.patch,
> HBASE-14268-V5.patch, HBASE-14268-V6.patch, HBASE-14268-V7.patch,
> HBASE-14268-V7.patch, HBASE-14268-V7.patch, HBASE-14268-V7.patch,
> HBASE-14268-V7.patch, HBASE-14268.patch, KeyLockerIncrKeysPerformance.java,
> KeyLockerPerformance.java, ReferenceTestApp.java
>
>
> 1. In the implementation of {{KeyLocker}} it uses atomic variables inside a
> synchronized block, which doesn't make sense. Moreover, logic inside the
> synchronized block is not trivial so that it makes less performance in heavy
> multi-threaded environment.
> 2. {{KeyLocker}} gives an instance of {{RentrantLock}} which is already
> locked, but it doesn't follow the contract of {{ReentrantLock}} because you
> are not allowed to freely invoke lock/unlock methods under that contract.
> That introduces a potential risk; Whenever you see a variable of the type
> {{RentrantLock}}, you should pay attention to what the included instance is
> coming from.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14563) Disable zombie TestHFileOutputFormat2
[ https://issues.apache.org/jira/browse/HBASE-14563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946013#comment-14946013 ] Hudson commented on HBASE-14563: FAILURE: Integrated in HBase-1.2 #230 (See [https://builds.apache.org/job/HBase-1.2/230/]) HBASE-14563 Disable zombie TestHFileOutputFormat2 (stack: rev 22c87d9644c600788a0df5456333464cba969c49) * hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat2.java > Disable zombie TestHFileOutputFormat2 > - > > Key: HBASE-14563 > URL: https://issues.apache.org/jira/browse/HBASE-14563 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack >Assignee: stack > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: 14563.txt > > > Disabling until someone has a chance to look at it. > I watched it in jvisualvm a while. Its starting and stopping clusters > multiple times and then running mr jobs. Needs a rewrite at least and some > shrinking of scope on what is tested. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14493) Upgrade the jamon-runtime dependency to the newer version MPL 2.0
[ https://issues.apache.org/jira/browse/HBASE-14493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946001#comment-14946001 ] stack commented on HBASE-14493: --- +1 from me [~apurtell] Yeah, you good w/ this [~busbey]? > Upgrade the jamon-runtime dependency to the newer version MPL 2.0 > - > > Key: HBASE-14493 > URL: https://issues.apache.org/jira/browse/HBASE-14493 > Project: HBase > Issue Type: Task >Affects Versions: 1.1.1 >Reporter: Newton Alex >Assignee: Andrew Purtell >Priority: Minor > Fix For: 2.0.0, 1.2.0, 1.3.0, 0.98.16 > > Attachments: HBASE-14493-0.98.patch, HBASE-14493-branch-1.patch, > HBASE-14493.patch, HBASE-14493.patch > > > Current version of HBase uses MPL 1.1 which has legal restrictions. Newer > versions of jamon-runtime appear to be MPL 2.0. HBase should upgrade to a > safer licensed version of jamon. > 2.4.0 is MPL 1.1 : > http://grepcode.com/snapshot/repo1.maven.org/maven2/org.jamon/jamon-runtime/2.4.0 > 2.4.1 is MPL 2.0 : > http://grepcode.com/snapshot/repo1.maven.org/maven2/org.jamon/jamon-runtime/2.4.1 > Here’s a comparison of the equivalent sections of the respective licenses > dealing w/ Termination: > MPL 1.1 - Section 8 (Termination) Subsection 2: > 8.2. If You initiate litigation by asserting a patent infringement claim > (excluding declatory judgment actions) against Initial Developer or a > Contributor (the Initial Developer or Contributor against whom You file such > action is referred to as "Participant") alleging that: > such Participant's Contributor Version directly or indirectly infringes any > patent, then any and all rights granted by such Participant to You under > Sections 2.1 and/or 2.2 of this License shall, upon 60 days notice from > Participant terminate prospectively, unless if within 60 days after receipt > of notice You either: (i) agree in writing to pay Participant a mutually > agreeable reasonable royalty for Your past and future use of Modifications > made by such Participant, or (ii) withdraw Your litigation claim with respect > to the Contributor Version against such Participant. If within 60 days of > notice, a reasonable royalty and payment arrangement are not mutually agreed > upon in writing by the parties or the litigation claim is not withdrawn, the > rights granted by Participant to You under Sections 2.1 and/or 2.2 > automatically terminate at the expiration of the 60 day notice period > specified above. > any software, hardware, or device, other than such Participant's Contributor > Version, directly or indirectly infringes any patent, then any rights granted > to You by such Participant under Sections 2.1(b) and 2.2(b) are revoked > effective as of the date You first made, used, sold, distributed, or had > made, Modifications made by that Participant. > MPL 2.0 - Section 5 (Termination) Subsection 2: > 5.2. If You initiate litigation against any entity by asserting a patent > infringement claim (excluding declaratory judgment actions, counter-claims, > and cross-claims) alleging that a Contributor Version directly or indirectly > infringes any patent, then the rights granted to You by any and all > Contributors for the Covered Software under Section 2.1 of this License shall > terminate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14517) Show regionserver's version in master status page
[ https://issues.apache.org/jira/browse/HBASE-14517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946000#comment-14946000 ] stack commented on HBASE-14517: --- This looks really nice [~liushaohui] Operators will like it. Why you move the VersionInfo from RPC to HBase protos? Will that break anyone (I don't think so.. since you do not change the pb data structure) > Show regionserver's version in master status page > - > > Key: HBASE-14517 > URL: https://issues.apache.org/jira/browse/HBASE-14517 > Project: HBase > Issue Type: Improvement > Components: monitoring >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-14517-v1.diff > > > In production env, regionservers may be removed from the cluster for hardware > problems and rejoined the cluster after the repair. There is a potential risk > that the version of rejoined regionserver may diff from others because the > cluster has been upgraded through many versions. > To solve this, we can show the all regionservers' version in the server list > of master's status page, and highlight the regionserver when its version is > different from the master's version, similar to HDFS-3245 > Suggestions are welcome~ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14436) HTableDescriptor#addCoprocessor will always make RegionCoprocessorHost create new Configuration
[
https://issues.apache.org/jira/browse/HBASE-14436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14945999#comment-14945999
]
Hudson commented on HBASE-14436:
FAILURE: Integrated in HBase-1.0 #1073 (See
[https://builds.apache.org/job/HBase-1.0/1073/])
HBASE-14436 HTableDescriptor#addCoprocessor will always make (stack: rev
c1890b5b15a3cb3ed9c00f4326e4eb6b583c55a6)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java
> HTableDescriptor#addCoprocessor will always make RegionCoprocessorHost create
> new Configuration
> ---
>
> Key: HBASE-14436
> URL: https://issues.apache.org/jira/browse/HBASE-14436
> Project: HBase
> Issue Type: Improvement
> Components: Coprocessors
>Affects Versions: 1.2.1
>Reporter: Jianwei Cui
>Assignee: stack
>Priority: Minor
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: HBASE-14436-trunk-v1.patch, HBASE-14436-trunk-v2.patch
>
>
> HTableDescriptor#addCoprocessor will set the coprocessor value as following
> format:
> {code}
> public HTableDescriptor addCoprocessor(String className, Path jarFilePath,
> int priority, final Map kvs)
> throws IOException {
> ...
> String value = ((jarFilePath == null)? "" : jarFilePath.toString()) +
> "|" + className + "|" + Integer.toString(priority) + "|" +
> kvString.toString();
> ...
> }
> {code}
> If the 'jarFilePath' is null, the 'value' will always has the format
> '|className|priority|' even if 'kvs' is null, which means no extra arguments
> for the coprocessor. Then, in the server side,
> RegionCoprocessorHost#getTableCoprocessorAttrsFromSchema will load the table
> coprocessors as:
> {code}
> static List
> getTableCoprocessorAttrsFromSchema(Configuration conf,
> HTableDescriptor htd) {
> ...
> try {
> cfgSpec = matcher.group(4); // => cfgSpec will be '|' for the
> format '|className|priority|'
> } catch (IndexOutOfBoundsException ex) {
> // ignore
> }
> Configuration ourConf;
> if (cfgSpec != null) { // => cfgSpec will be '|' for the format
> '|className|priority|'
> ourConf = new Configuration(false);
> HBaseConfiguration.merge(ourConf, conf);
> }
> ...
> }
> {code}
> The 'cfgSpec' will be '|' for the coprocessor formatted as
> '|className|priority|', so that always create a new Configuration.
> In our production, there are a lot of tables having table-level coprocessors,
> so that the region server will create new Configurations for each region of
> the table, this will consume a certain number of memory when we have many
> such regions.
> To fix the problem, we can make the HTableDescriptor not append the '|' if no
> extra arguments for the coprocessor, or check the 'cfgSpec' more strictly in
> server side which could avoid creating new Configurations for existed such
> regions after the regions reopened. Discussions and suggestions are welcomed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14497) Reverse Scan threw StackOverflow caused by readPt checking
[ https://issues.apache.org/jira/browse/HBASE-14497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14497: --- Resolution: Fixed Status: Resolved (was: Patch Available) > Reverse Scan threw StackOverflow caused by readPt checking > -- > > Key: HBASE-14497 > URL: https://issues.apache.org/jira/browse/HBASE-14497 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 0.98.14, 1.3.0 >Reporter: Yerui Sun >Assignee: Yerui Sun > Fix For: 2.0.0, 1.3.0 > > Attachments: 14497-branch-1-v6.patch, 14497-master-v6.patch, > HBASE-14497-0.98.patch, HBASE-14497-branch-1-v2.patch, > HBASE-14497-branch-1-v3.patch, HBASE-14497-branch-1-v6.patch, > HBASE-14497-branch-1.patch, HBASE-14497-master-v2.patch, > HBASE-14497-master-v3.patch, HBASE-14497-master-v3.patch, > HBASE-14497-master-v4.patch, HBASE-14497-master-v5.patch, > HBASE-14497-master.patch > > > I met stack overflow error in StoreFileScanner.seekToPreviousRow using > reversed scan. I searched and founded HBASE-14155, but it seems to be a > different reason. > The seekToPreviousRow will fetch the row which closest before, and compare > mvcc to the readPt, which acquired when scanner created. If the row's mvcc is > bigger than readPt, an recursive call of seekToPreviousRow will invoked, to > find the next closest before row. > Considering we created a scanner for reversed scan, and some data with > smaller rows was written and flushed, before calling scanner next. When > seekToPreviousRow was invoked, it would call itself recursively, until all > rows which written after scanner created were iterated. The depth of > recursive calling stack depends on the count of rows, the stack overflow > error will be threw if the count of rows is large, like 1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
