[jira] [Commented] (HBASE-12072) We are doing 35 x 35 retries for master operations
[
https://issues.apache.org/jira/browse/HBASE-12072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14191446#comment-14191446
]
Hadoop QA commented on HBASE-12072:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12678413/hbase-12072_v2.patch
against trunk revision .
ATTACHMENT ID: 12678413
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 18 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
3775 checkstyle errors (more than the trunk's current 3774 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/11535//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11535//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11535//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11535//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11535//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11535//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11535//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11535//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11535//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11535//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11535//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11535//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/11535//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/11535//console
This message is automatically generated.
We are doing 35 x 35 retries for master operations
--
Key: HBASE-12072
URL: https://issues.apache.org/jira/browse/HBASE-12072
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.6
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 2.0.0, 0.99.2
Attachments: 12072-v1.txt, 12072-v2.txt, hbase-12072_v1.patch,
hbase-12072_v2.patch
For master requests, there are two retry mechanisms in effect. The first one
is from HBaseAdmin.executeCallable()
{code}
private V V executeCallable(MasterCallableV callable) throws
IOException {
RpcRetryingCallerV caller = rpcCallerFactory.newCaller();
try {
return caller.callWithRetries(callable);
} finally {
callable.close();
}
}
{code}
And inside, the other one is from StubMaker.makeStub():
{code}
/**
* Create a stub against the master. Retry if necessary.
* @return A stub to do codeintf/code against the master
* @throws MasterNotRunningException
*/
@edu.umd.cs.findbugs.annotations.SuppressWarnings
(value=SWL_SLEEP_WITH_LOCK_HELD)
Object makeStub() throws MasterNotRunningException {
{code}
The tests will just hang for 10 min * 35 ~= 6hours.
{code}
2014-09-23 16:19:05,151 INFO [main]
[jira] [Updated] (HBASE-12331) Shorten the mob snapshot unit tests
[
https://issues.apache.org/jira/browse/HBASE-12331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Li Jiajia updated HBASE-12331:
--
Attachment: HBASE-12331-V1.diff
change some UTs to be integration tests and shorten some UTs.
Shorten the mob snapshot unit tests
---
Key: HBASE-12331
URL: https://issues.apache.org/jira/browse/HBASE-12331
Project: HBase
Issue Type: Sub-task
Components: mob
Affects Versions: hbase-11339
Reporter: Jonathan Hsieh
Fix For: hbase-11339
Attachments: HBASE-12331-V1.diff
The mob snapshot patch introduced a whole log of tests that take a long time
to run and would be better as integration tests.
{code}
---
T E S T S
---
Running
org.apache.hadoop.hbase.client.TestMobRestoreSnapshotFromClientWithRegionReplicas
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 394.803 sec -
in
org.apache.hadoop.hbase.client.TestMobRestoreSnapshotFromClientWithRegionReplicas
Running org.apache.hadoop.hbase.client.TestMobRestoreSnapshotFromClient
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 212.377 sec -
in org.apache.hadoop.hbase.client.TestMobRestoreSnapshotFromClient
Running
org.apache.hadoop.hbase.client.TestMobSnapshotFromClientWithRegionReplicas
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 49.463 sec -
in org.apache.hadoop.hbase.client.TestMobSnapshotFromClientWithRegionReplicas
Running org.apache.hadoop.hbase.client.TestMobSnapshotFromClient
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 46.724 sec -
in org.apache.hadoop.hbase.client.TestMobSnapshotFromClient
Running org.apache.hadoop.hbase.client.TestMobCloneSnapshotFromClient
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 204.03 sec -
in org.apache.hadoop.hbase.client.TestMobCloneSnapshotFromClient
Running
org.apache.hadoop.hbase.client.TestMobCloneSnapshotFromClientWithRegionReplicas
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 214.052 sec -
in
org.apache.hadoop.hbase.client.TestMobCloneSnapshotFromClientWithRegionReplicas
Running org.apache.hadoop.hbase.client.TestMobSnapshotCloneIndependence
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 105.139 sec -
in org.apache.hadoop.hbase.client.TestMobSnapshotCloneIndependence
Running org.apache.hadoop.hbase.regionserver.TestMobStoreScanner
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.42 sec -
in org.apache.hadoop.hbase.regionserver.TestMobStoreScanner
Running org.apache.hadoop.hbase.regionserver.TestDeleteMobTable
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.136 sec -
in org.apache.hadoop.hbase.regionserver.TestDeleteMobTable
Running org.apache.hadoop.hbase.regionserver.TestHMobStore
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.09 sec - in
org.apache.hadoop.hbase.regionserver.TestHMobStore
Running org.apache.hadoop.hbase.regionserver.TestMobCompaction
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.629 sec -
in org.apache.hadoop.hbase.regionserver.TestMobCompaction
Running org.apache.hadoop.hbase.mob.TestCachedMobFile
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.301 sec -
in org.apache.hadoop.hbase.mob.TestCachedMobFile
Running org.apache.hadoop.hbase.mob.mapreduce.TestMobSweepJob
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.752 sec -
in org.apache.hadoop.hbase.mob.mapreduce.TestMobSweepJob
Running org.apache.hadoop.hbase.mob.mapreduce.TestMobSweepReducer
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.276 sec -
in org.apache.hadoop.hbase.mob.mapreduce.TestMobSweepReducer
Running org.apache.hadoop.hbase.mob.mapreduce.TestMobSweepMapper
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.46 sec -
in org.apache.hadoop.hbase.mob.mapreduce.TestMobSweepMapper
Running org.apache.hadoop.hbase.mob.mapreduce.TestMobSweeper
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 173.05 sec -
in org.apache.hadoop.hbase.mob.mapreduce.TestMobSweeper
Running org.apache.hadoop.hbase.mob.TestMobDataBlockEncoding
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.86 sec -
in org.apache.hadoop.hbase.mob.TestMobDataBlockEncoding
Running org.apache.hadoop.hbase.mob.TestExpiredMobFileCleaner
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.029 sec -
in org.apache.hadoop.hbase.mob.TestExpiredMobFileCleaner
Running org.apache.hadoop.hbase.mob.TestMobFile
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.562 sec -
in org.apache.hadoop.hbase.mob.TestMobFile
Running
[jira] [Created] (HBASE-12391) Correct a typo in the mob metrics
Jingcheng Du created HBASE-12391: Summary: Correct a typo in the mob metrics Key: HBASE-12391 URL: https://issues.apache.org/jira/browse/HBASE-12391 Project: HBase Issue Type: Sub-task Reporter: Jingcheng Du Assignee: Jingcheng Du Priority: Minor There's a typo in the temp variable in the region server metrics for mob. It's now testMobCompactedFromMobCellsSize, and should be changed to tempMobCompactedFromMobCellsSize -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12391) Correct a typo in the mob metrics
[ https://issues.apache.org/jira/browse/HBASE-12391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingcheng Du updated HBASE-12391: - Attachment: HBASE-12391.diff Update the patch to fix the typo. Correct a typo in the mob metrics - Key: HBASE-12391 URL: https://issues.apache.org/jira/browse/HBASE-12391 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Jingcheng Du Assignee: Jingcheng Du Priority: Minor Fix For: hbase-11339 Attachments: HBASE-12391.diff There's a typo in the temp variable in the region server metrics for mob. It's now testMobCompactedFromMobCellsSize, and should be changed to tempMobCompactedFromMobCellsSize -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12391) Correct a typo in the mob metrics
[ https://issues.apache.org/jira/browse/HBASE-12391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingcheng Du updated HBASE-12391: - Status: Patch Available (was: Open) Correct a typo in the mob metrics - Key: HBASE-12391 URL: https://issues.apache.org/jira/browse/HBASE-12391 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Jingcheng Du Assignee: Jingcheng Du Priority: Minor Fix For: hbase-11339 Attachments: HBASE-12391.diff There's a typo in the temp variable in the region server metrics for mob. It's now testMobCompactedFromMobCellsSize, and should be changed to tempMobCompactedFromMobCellsSize -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12391) Correct a typo in the mob metrics
[ https://issues.apache.org/jira/browse/HBASE-12391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14191509#comment-14191509 ] Jingcheng Du commented on HBASE-12391: -- Hi [~jmhsieh], [~anoopsamjohn] and [~ram_krish], would you please look at it and commit it? Thanks a lot! Correct a typo in the mob metrics - Key: HBASE-12391 URL: https://issues.apache.org/jira/browse/HBASE-12391 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Jingcheng Du Assignee: Jingcheng Du Priority: Minor Fix For: hbase-11339 Attachments: HBASE-12391.diff There's a typo in the temp variable in the region server metrics for mob. It's now testMobCompactedFromMobCellsSize, and should be changed to tempMobCompactedFromMobCellsSize -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12391) Correct a typo in the mob metrics
[
https://issues.apache.org/jira/browse/HBASE-12391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14191513#comment-14191513
]
Hadoop QA commented on HBASE-12391:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12678430/HBASE-12391.diff
against trunk revision .
ATTACHMENT ID: 12678430
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/11537//console
This message is automatically generated.
Correct a typo in the mob metrics
-
Key: HBASE-12391
URL: https://issues.apache.org/jira/browse/HBASE-12391
Project: HBase
Issue Type: Sub-task
Components: regionserver, Scanners
Reporter: Jingcheng Du
Assignee: Jingcheng Du
Priority: Minor
Fix For: hbase-11339
Attachments: HBASE-12391.diff
There's a typo in the temp variable in the region server metrics for mob.
It's now testMobCompactedFromMobCellsSize, and should be changed to
tempMobCompactedFromMobCellsSize
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-12392) Incorrect implementation of CompactionRequest.isRetainDeleteMarkers
Jingcheng Du created HBASE-12392:
Summary: Incorrect implementation of
CompactionRequest.isRetainDeleteMarkers
Key: HBASE-12392
URL: https://issues.apache.org/jira/browse/HBASE-12392
Project: HBase
Issue Type: Sub-task
Reporter: Jingcheng Du
Assignee: Jingcheng Du
Now in the implementation of the isRetainDeleteMarkers method, the code look
like,
{code}
return (this.retainDeleteMarkers != null) ?
this.retainDeleteMarkers.booleanValue()
: isAllFiles();
{code}
It means for a major compaction in a normal store, this method returns true.
Consequently the delete marks could not be deleted in the major compaction,
which leads the unit test TestKeepDeletes fails.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12392) Incorrect implementation of CompactionRequest.isRetainDeleteMarkers
[
https://issues.apache.org/jira/browse/HBASE-12392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingcheng Du updated HBASE-12392:
-
Affects Version/s: hbase-11339
Incorrect implementation of CompactionRequest.isRetainDeleteMarkers
---
Key: HBASE-12392
URL: https://issues.apache.org/jira/browse/HBASE-12392
Project: HBase
Issue Type: Sub-task
Components: regionserver, Scanners
Affects Versions: hbase-11339
Reporter: Jingcheng Du
Assignee: Jingcheng Du
Fix For: hbase-11339
Now in the implementation of the isRetainDeleteMarkers method, the code look
like,
{code}
return (this.retainDeleteMarkers != null) ?
this.retainDeleteMarkers.booleanValue()
: isAllFiles();
{code}
It means for a major compaction in a normal store, this method returns true.
Consequently the delete marks could not be deleted in the major compaction,
which leads the unit test TestKeepDeletes fails.
The correct implementation should be,
{code}
return (this.retainDeleteMarkers != null) ?
this.retainDeleteMarkers.booleanValue()
: !isAllFiles();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12392) Incorrect implementation of CompactionRequest.isRetainDeleteMarkers
[
https://issues.apache.org/jira/browse/HBASE-12392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingcheng Du updated HBASE-12392:
-
Description:
Now in the implementation of the isRetainDeleteMarkers method, the code look
like,
{code}
return (this.retainDeleteMarkers != null) ?
this.retainDeleteMarkers.booleanValue()
: isAllFiles();
{code}
It means for a major compaction in a normal store, this method returns true.
Consequently the delete marks could not be deleted in the major compaction,
which leads the unit test TestKeepDeletes fails.
The correct implementation should be,
{code}
return (this.retainDeleteMarkers != null) ?
this.retainDeleteMarkers.booleanValue()
: !isAllFiles();
{code}
was:
Now in the implementation of the isRetainDeleteMarkers method, the code look
like,
{code}
return (this.retainDeleteMarkers != null) ?
this.retainDeleteMarkers.booleanValue()
: isAllFiles();
{code}
It means for a major compaction in a normal store, this method returns true.
Consequently the delete marks could not be deleted in the major compaction,
which leads the unit test TestKeepDeletes fails.
Incorrect implementation of CompactionRequest.isRetainDeleteMarkers
---
Key: HBASE-12392
URL: https://issues.apache.org/jira/browse/HBASE-12392
Project: HBase
Issue Type: Sub-task
Components: regionserver, Scanners
Affects Versions: hbase-11339
Reporter: Jingcheng Du
Assignee: Jingcheng Du
Fix For: hbase-11339
Now in the implementation of the isRetainDeleteMarkers method, the code look
like,
{code}
return (this.retainDeleteMarkers != null) ?
this.retainDeleteMarkers.booleanValue()
: isAllFiles();
{code}
It means for a major compaction in a normal store, this method returns true.
Consequently the delete marks could not be deleted in the major compaction,
which leads the unit test TestKeepDeletes fails.
The correct implementation should be,
{code}
return (this.retainDeleteMarkers != null) ?
this.retainDeleteMarkers.booleanValue()
: !isAllFiles();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12274) Race between RegionScannerImpl#nextInternal() and RegionScannerImpl#close() may produce null pointer exception
[
https://issues.apache.org/jira/browse/HBASE-12274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14191527#comment-14191527
]
Hudson commented on HBASE-12274:
SUCCESS: Integrated in HBase-0.98 #643 (See
[https://builds.apache.org/job/HBase-0.98/643/])
HBASE-12274 addendum removes synchronized keyword for nextRaw() (tedyu: rev
f1cabd378512f1b18003afed9782d88d022d6850)
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
Race between RegionScannerImpl#nextInternal() and RegionScannerImpl#close()
may produce null pointer exception
--
Key: HBASE-12274
URL: https://issues.apache.org/jira/browse/HBASE-12274
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.6.1
Reporter: Ted Yu
Assignee: Ted Yu
Fix For: 2.0.0, 0.98.8, 0.99.2
Attachments: 12274-0.98.addendum, 12274-region-server.log,
12274-v2.txt, 12274-v2.txt, 12274-v3.txt
I saw the following in region server log:
{code}
2014-10-15 03:28:36,976 ERROR
[B.DefaultRpcServer.handler=0,queue=0,port=60020] ipc.RpcServer: Unexpected
throwable object
java.lang.NullPointerException
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:5023)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:4932)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:4923)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3245)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29994)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2078)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
at java.lang.Thread.run(Thread.java:745)
{code}
This is where the NPE happened:
{code}
// Let's see what we have in the storeHeap.
KeyValue current = this.storeHeap.peek();
{code}
The cause was race between nextInternal(called through nextRaw) and close
methods.
nextRaw() is not synchronized.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12392) Incorrect implementation of CompactionRequest.isRetainDeleteMarkers
[
https://issues.apache.org/jira/browse/HBASE-12392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingcheng Du updated HBASE-12392:
-
Attachment: HBASE-12392.diff
Upload the patch to fix this issue.
Hi [~anoopsamjohn], [~jmhsieh], could you please help look at it and commit?
Thanks.
Incorrect implementation of CompactionRequest.isRetainDeleteMarkers
---
Key: HBASE-12392
URL: https://issues.apache.org/jira/browse/HBASE-12392
Project: HBase
Issue Type: Sub-task
Components: regionserver, Scanners
Affects Versions: hbase-11339
Reporter: Jingcheng Du
Assignee: Jingcheng Du
Fix For: hbase-11339
Attachments: HBASE-12392.diff
Now in the implementation of the isRetainDeleteMarkers method, the code look
like,
{code}
return (this.retainDeleteMarkers != null) ?
this.retainDeleteMarkers.booleanValue()
: isAllFiles();
{code}
It means for a major compaction in a normal store, this method returns true.
Consequently the delete marks could not be deleted in the major compaction,
which leads the unit test TestKeepDeletes fails.
The correct implementation should be,
{code}
return (this.retainDeleteMarkers != null) ?
this.retainDeleteMarkers.booleanValue()
: !isAllFiles();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12392) Incorrect implementation of CompactionRequest.isRetainDeleteMarkers
[
https://issues.apache.org/jira/browse/HBASE-12392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingcheng Du updated HBASE-12392:
-
Status: Patch Available (was: Open)
Incorrect implementation of CompactionRequest.isRetainDeleteMarkers
---
Key: HBASE-12392
URL: https://issues.apache.org/jira/browse/HBASE-12392
Project: HBase
Issue Type: Sub-task
Components: regionserver, Scanners
Affects Versions: hbase-11339
Reporter: Jingcheng Du
Assignee: Jingcheng Du
Fix For: hbase-11339
Attachments: HBASE-12392.diff
Now in the implementation of the isRetainDeleteMarkers method, the code look
like,
{code}
return (this.retainDeleteMarkers != null) ?
this.retainDeleteMarkers.booleanValue()
: isAllFiles();
{code}
It means for a major compaction in a normal store, this method returns true.
Consequently the delete marks could not be deleted in the major compaction,
which leads the unit test TestKeepDeletes fails.
The correct implementation should be,
{code}
return (this.retainDeleteMarkers != null) ?
this.retainDeleteMarkers.booleanValue()
: !isAllFiles();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12392) Incorrect implementation of CompactionRequest.isRetainDeleteMarkers
[
https://issues.apache.org/jira/browse/HBASE-12392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14191533#comment-14191533
]
Hadoop QA commented on HBASE-12392:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12678433/HBASE-12392.diff
against trunk revision .
ATTACHMENT ID: 12678433
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/11538//console
This message is automatically generated.
Incorrect implementation of CompactionRequest.isRetainDeleteMarkers
---
Key: HBASE-12392
URL: https://issues.apache.org/jira/browse/HBASE-12392
Project: HBase
Issue Type: Sub-task
Components: regionserver, Scanners
Affects Versions: hbase-11339
Reporter: Jingcheng Du
Assignee: Jingcheng Du
Fix For: hbase-11339
Attachments: HBASE-12392.diff
Now in the implementation of the isRetainDeleteMarkers method, the code look
like,
{code}
return (this.retainDeleteMarkers != null) ?
this.retainDeleteMarkers.booleanValue()
: isAllFiles();
{code}
It means for a major compaction in a normal store, this method returns true.
Consequently the delete marks could not be deleted in the major compaction,
which leads the unit test TestKeepDeletes fails.
The correct implementation should be,
{code}
return (this.retainDeleteMarkers != null) ?
this.retainDeleteMarkers.booleanValue()
: !isAllFiles();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12377) HBaseAdmin#deleteTable fails when META region is moved around the same timeframe
[
https://issues.apache.org/jira/browse/HBASE-12377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14191534#comment-14191534
]
Hadoop QA commented on HBASE-12377:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12678417/HBASE-12377.v3-2.0.patch
against trunk revision .
ATTACHMENT ID: 12678417
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.util.TestProcessBasedCluster
org.apache.hadoop.hbase.mapreduce.TestImportExport
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/11536//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11536//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11536//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11536//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11536//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11536//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11536//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11536//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11536//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11536//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11536//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11536//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/11536//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/11536//console
This message is automatically generated.
HBaseAdmin#deleteTable fails when META region is moved around the same
timeframe
Key: HBASE-12377
URL: https://issues.apache.org/jira/browse/HBASE-12377
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 0.98.4
Reporter: Stephen Yuan Jiang
Assignee: Stephen Yuan Jiang
Fix For: 2.0.0, 0.98.8, 0.99.2
Attachments: HBASE-12377.v1-2.0.patch, HBASE-12377.v2-2.0.patch,
HBASE-12377.v3-2.0.patch
This is the same issue that HBASE-10809 tried to address. The fix of
HBASE-10809 refetch the latest meta location in retry-loop. However, there
are 2 problems: (1). inside the retry loop, there is another try-catch block
that would throw the exception before retry can kick in; (2). It looks like
that HBaseAdmin::getFirstMetaServerForTable() always tries to get meta data
from meta cache, which means if the meta cache is stale and out of date,
retries would not solve the problem by fetching from the
[jira] [Commented] (HBASE-12274) Race between RegionScannerImpl#nextInternal() and RegionScannerImpl#close() may produce null pointer exception
[
https://issues.apache.org/jira/browse/HBASE-12274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14191539#comment-14191539
]
Hudson commented on HBASE-12274:
FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #612 (See
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/612/])
HBASE-12274 addendum removes synchronized keyword for nextRaw() (tedyu: rev
f1cabd378512f1b18003afed9782d88d022d6850)
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
Race between RegionScannerImpl#nextInternal() and RegionScannerImpl#close()
may produce null pointer exception
--
Key: HBASE-12274
URL: https://issues.apache.org/jira/browse/HBASE-12274
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.6.1
Reporter: Ted Yu
Assignee: Ted Yu
Fix For: 2.0.0, 0.98.8, 0.99.2
Attachments: 12274-0.98.addendum, 12274-region-server.log,
12274-v2.txt, 12274-v2.txt, 12274-v3.txt
I saw the following in region server log:
{code}
2014-10-15 03:28:36,976 ERROR
[B.DefaultRpcServer.handler=0,queue=0,port=60020] ipc.RpcServer: Unexpected
throwable object
java.lang.NullPointerException
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:5023)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:4932)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:4923)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3245)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29994)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2078)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
at java.lang.Thread.run(Thread.java:745)
{code}
This is where the NPE happened:
{code}
// Let's see what we have in the storeHeap.
KeyValue current = this.storeHeap.peek();
{code}
The cause was race between nextInternal(called through nextRaw) and close
methods.
nextRaw() is not synchronized.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12392) Incorrect implementation of CompactionRequest.isRetainDeleteMarkers
[
https://issues.apache.org/jira/browse/HBASE-12392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John updated HBASE-12392:
---
Priority: Critical (was: Major)
Incorrect implementation of CompactionRequest.isRetainDeleteMarkers
---
Key: HBASE-12392
URL: https://issues.apache.org/jira/browse/HBASE-12392
Project: HBase
Issue Type: Sub-task
Components: regionserver, Scanners
Affects Versions: hbase-11339
Reporter: Jingcheng Du
Assignee: Jingcheng Du
Priority: Critical
Fix For: hbase-11339
Attachments: HBASE-12392.diff
Now in the implementation of the isRetainDeleteMarkers method, the code look
like,
{code}
return (this.retainDeleteMarkers != null) ?
this.retainDeleteMarkers.booleanValue()
: isAllFiles();
{code}
It means for a major compaction in a normal store, this method returns true.
Consequently the delete marks could not be deleted in the major compaction,
which leads the unit test TestKeepDeletes fails.
The correct implementation should be,
{code}
return (this.retainDeleteMarkers != null) ?
this.retainDeleteMarkers.booleanValue()
: !isAllFiles();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12392) Incorrect implementation of CompactionRequest.isRetainDeleteMarkers
[
https://issues.apache.org/jira/browse/HBASE-12392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14191553#comment-14191553
]
Anoop Sam John commented on HBASE-12392:
Oh that is bad :(
+1. Thanks for the find Jingcheng
Incorrect implementation of CompactionRequest.isRetainDeleteMarkers
---
Key: HBASE-12392
URL: https://issues.apache.org/jira/browse/HBASE-12392
Project: HBase
Issue Type: Sub-task
Components: regionserver, Scanners
Affects Versions: hbase-11339
Reporter: Jingcheng Du
Assignee: Jingcheng Du
Fix For: hbase-11339
Attachments: HBASE-12392.diff
Now in the implementation of the isRetainDeleteMarkers method, the code look
like,
{code}
return (this.retainDeleteMarkers != null) ?
this.retainDeleteMarkers.booleanValue()
: isAllFiles();
{code}
It means for a major compaction in a normal store, this method returns true.
Consequently the delete marks could not be deleted in the major compaction,
which leads the unit test TestKeepDeletes fails.
The correct implementation should be,
{code}
return (this.retainDeleteMarkers != null) ?
this.retainDeleteMarkers.booleanValue()
: !isAllFiles();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12387) committer guidelines should include patch signoff
[
https://issues.apache.org/jira/browse/HBASE-12387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14191567#comment-14191567
]
Anoop Sam John commented on HBASE-12387:
So we should always go with making author to contributor and signed off? Even
now also many patch commits are like old way (adding contributor name in commit
message). What do you say Stack?
committer guidelines should include patch signoff
-
Key: HBASE-12387
URL: https://issues.apache.org/jira/browse/HBASE-12387
Project: HBase
Issue Type: Task
Components: documentation
Reporter: Sean Busbey
Right now our guide for committers apply patches has them use {{git am}}
without a signoff flag. This works okay, but it misses adding the
signed-off-by blurb in the commit message.
Those messages make it easier to see at a glance with e.g. {{git log}} which
committer applied the patch.
this section:
{quote}
The directive to use git format-patch rather than git diff, and not to use
--no-prefix, is a new one. See the second example for how to apply a patch
created with git diff, and educate the person who created the patch.
{code}
$ git checkout -b HBASE-
$ git am ~/Downloads/HBASE--v2.patch
$ git checkout master
$ git pull --rebase
$ git cherry-pick sha-from-commit
# Resolve conflicts if necessary or ask the submitter to do it
$ git pull --rebase # Better safe than sorry
$ git push origin master
$ git checkout branch-1
$ git pull --rebase
$ git cherry-pick sha-from-commit
# Resolve conflicts if necessary
$ git pull --rebase # Better safe than sorry
$ git push origin branch-1
$ git branch -D HBASE-
{code}
{quote}
Should be
{quote}
The directive to use git format-patch rather than git diff, and not to use
--no-prefix, is a new one. See the second example for how to apply a patch
created with git diff, and educate the person who created the patch.
Note that the {{--signoff}} flag to {{git am}} will insert a line in the
commit message that the patch was checked by your author string. This
addition to your inclusion as the commit's committer makes your participation
more prominent to users browsing {{git log}}.
{code}
$ git checkout -b HBASE-
$ git am --signoff ~/Downloads/HBASE--v2.patch
$ git checkout master
$ git pull --rebase
$ git cherry-pick sha-from-commit
# Resolve conflicts if necessary or ask the submitter to do it
$ git pull --rebase # Better safe than sorry
$ git push origin master
$ git checkout branch-1
$ git pull --rebase
$ git cherry-pick sha-from-commit
# Resolve conflicts if necessary
$ git pull --rebase # Better safe than sorry
$ git push origin branch-1
$ git branch -D HBASE-
{code}
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12219) Cache more efficiently getAll() and get() in FSTableDescriptors
[ https://issues.apache.org/jira/browse/HBASE-12219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esteban Gutierrez updated HBASE-12219: -- Attachment: HBASE-12219-0.99.patch Attaching patch for 0.99. Is that good for you [~enis]? Cache more efficiently getAll() and get() in FSTableDescriptors --- Key: HBASE-12219 URL: https://issues.apache.org/jira/browse/HBASE-12219 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.94.24, 0.99.1, 0.98.6.1 Reporter: Esteban Gutierrez Assignee: Esteban Gutierrez Labels: scalability Attachments: HBASE-12219-0.99.patch, HBASE-12219-v1.patch, HBASE-12219-v1.patch, HBASE-12219.v0.txt, HBASE-12219.v2.patch, HBASE-12219.v3.patch, list.png Currently table descriptors and tables are cached once they are accessed for the first time. Next calls to the master only require a trip to HDFS to lookup the modified time in order to reload the table descriptors if modified. However in clusters with a large number of tables or concurrent clients and this can be too aggressive to HDFS and the master causing contention to process other requests. A simple solution is to have a TTL based cached for FSTableDescriptors#getAll() and FSTableDescriptors#TableDescriptorAndModtime() that can allow the master to process those calls faster without causing contention without having to perform a trip to HDFS for every call. to listtables() or getTableDescriptor() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12219) Cache more efficiently getAll() and get() in FSTableDescriptors
[
https://issues.apache.org/jira/browse/HBASE-12219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14191614#comment-14191614
]
Hadoop QA commented on HBASE-12219:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12678443/HBASE-12219-0.99.patch
against trunk revision .
ATTACHMENT ID: 12678443
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 8 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/11539//console
This message is automatically generated.
Cache more efficiently getAll() and get() in FSTableDescriptors
---
Key: HBASE-12219
URL: https://issues.apache.org/jira/browse/HBASE-12219
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.94.24, 0.99.1, 0.98.6.1
Reporter: Esteban Gutierrez
Assignee: Esteban Gutierrez
Labels: scalability
Attachments: HBASE-12219-0.99.patch, HBASE-12219-v1.patch,
HBASE-12219-v1.patch, HBASE-12219.v0.txt, HBASE-12219.v2.patch,
HBASE-12219.v3.patch, list.png
Currently table descriptors and tables are cached once they are accessed for
the first time. Next calls to the master only require a trip to HDFS to
lookup the modified time in order to reload the table descriptors if
modified. However in clusters with a large number of tables or concurrent
clients and this can be too aggressive to HDFS and the master causing
contention to process other requests. A simple solution is to have a TTL
based cached for FSTableDescriptors#getAll() and
FSTableDescriptors#TableDescriptorAndModtime() that can allow the master to
process those calls faster without causing contention without having to
perform a trip to HDFS for every call. to listtables() or getTableDescriptor()
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12391) Correct a typo in the mob metrics
[ https://issues.apache.org/jira/browse/HBASE-12391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14191626#comment-14191626 ] ramkrishna.s.vasudevan commented on HBASE-12391: +1 Correct a typo in the mob metrics - Key: HBASE-12391 URL: https://issues.apache.org/jira/browse/HBASE-12391 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Jingcheng Du Assignee: Jingcheng Du Priority: Minor Fix For: hbase-11339 Attachments: HBASE-12391.diff There's a typo in the temp variable in the region server metrics for mob. It's now testMobCompactedFromMobCellsSize, and should be changed to tempMobCompactedFromMobCellsSize -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12392) Incorrect implementation of CompactionRequest.isRetainDeleteMarkers
[
https://issues.apache.org/jira/browse/HBASE-12392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14191627#comment-14191627
]
Jingcheng Du commented on HBASE-12392:
--
Hi Ram [~ram_krish], do you want to look at this patch? Thanks.
Incorrect implementation of CompactionRequest.isRetainDeleteMarkers
---
Key: HBASE-12392
URL: https://issues.apache.org/jira/browse/HBASE-12392
Project: HBase
Issue Type: Sub-task
Components: regionserver, Scanners
Affects Versions: hbase-11339
Reporter: Jingcheng Du
Assignee: Jingcheng Du
Priority: Critical
Fix For: hbase-11339
Attachments: HBASE-12392.diff
Now in the implementation of the isRetainDeleteMarkers method, the code look
like,
{code}
return (this.retainDeleteMarkers != null) ?
this.retainDeleteMarkers.booleanValue()
: isAllFiles();
{code}
It means for a major compaction in a normal store, this method returns true.
Consequently the delete marks could not be deleted in the major compaction,
which leads the unit test TestKeepDeletes fails.
The correct implementation should be,
{code}
return (this.retainDeleteMarkers != null) ?
this.retainDeleteMarkers.booleanValue()
: !isAllFiles();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12392) Incorrect implementation of CompactionRequest.isRetainDeleteMarkers
[
https://issues.apache.org/jira/browse/HBASE-12392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14191632#comment-14191632
]
ramkrishna.s.vasudevan commented on HBASE-12392:
+1 on patch. Good catch.
Incorrect implementation of CompactionRequest.isRetainDeleteMarkers
---
Key: HBASE-12392
URL: https://issues.apache.org/jira/browse/HBASE-12392
Project: HBase
Issue Type: Sub-task
Components: regionserver, Scanners
Affects Versions: hbase-11339
Reporter: Jingcheng Du
Assignee: Jingcheng Du
Priority: Critical
Fix For: hbase-11339
Attachments: HBASE-12392.diff
Now in the implementation of the isRetainDeleteMarkers method, the code look
like,
{code}
return (this.retainDeleteMarkers != null) ?
this.retainDeleteMarkers.booleanValue()
: isAllFiles();
{code}
It means for a major compaction in a normal store, this method returns true.
Consequently the delete marks could not be deleted in the major compaction,
which leads the unit test TestKeepDeletes fails.
The correct implementation should be,
{code}
return (this.retainDeleteMarkers != null) ?
this.retainDeleteMarkers.booleanValue()
: !isAllFiles();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12391) Correct a typo in the mob metrics
[ https://issues.apache.org/jira/browse/HBASE-12391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14191640#comment-14191640 ] ramkrishna.s.vasudevan commented on HBASE-12391: Pushed to HBASE-11339 branch. Thanks for the patch Jingcheng. Correct a typo in the mob metrics - Key: HBASE-12391 URL: https://issues.apache.org/jira/browse/HBASE-12391 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Jingcheng Du Assignee: Jingcheng Du Priority: Minor Fix For: hbase-11339 Attachments: HBASE-12391.diff There's a typo in the temp variable in the region server metrics for mob. It's now testMobCompactedFromMobCellsSize, and should be changed to tempMobCompactedFromMobCellsSize -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12393) The regionserver web UI will throw exception when we set block cache to zero
ChiaPing Tsai created HBASE-12393:
-
Summary: The regionserver web UI will throw exception when we set
block cache to zero
Key: HBASE-12393
URL: https://issues.apache.org/jira/browse/HBASE-12393
Project: HBase
Issue Type: Bug
Components: BlockCache
Environment: ubuntu 12.04 64bits, hadoop-2.2.0, hbase-0.98.7-hadoop2
Reporter: ChiaPing Tsai
Priority: Minor
The CacheConfig.getBlockCache() will return the null point when we set
hfile.block.cache.size to zero.
It caused the BlockCacheTmplImpl.java:123 to throw null exception.
{code}
org.jamon.escaping.Escaping.HTML.write(org.jamon.emit.StandardEmitter.valueOf(StringUtils.humanReadableInt(cacheConfig.getBlockCache().size())),
jamonWriter);
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12393) The regionserver web UI will throw exception when we set block cache to zero
[
https://issues.apache.org/jira/browse/HBASE-12393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ChiaPing Tsai updated HBASE-12393:
--
Component/s: (was: BlockCache)
regionserver
The regionserver web UI will throw exception when we set block cache to zero
Key: HBASE-12393
URL: https://issues.apache.org/jira/browse/HBASE-12393
Project: HBase
Issue Type: Bug
Components: regionserver
Environment: ubuntu 12.04 64bits, hadoop-2.2.0, hbase-0.98.7-hadoop2
Reporter: ChiaPing Tsai
Priority: Minor
The CacheConfig.getBlockCache() will return the null point when we set
hfile.block.cache.size to zero.
It caused the BlockCacheTmplImpl.java:123 to throw null exception.
{code}
org.jamon.escaping.Escaping.HTML.write(org.jamon.emit.StandardEmitter.valueOf(StringUtils.humanReadableInt(cacheConfig.getBlockCache().size())),
jamonWriter);
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12391) Correct a typo in the mob metrics
[ https://issues.apache.org/jira/browse/HBASE-12391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14191641#comment-14191641 ] ramkrishna.s.vasudevan commented on HBASE-12391: Should we wait for [~jmhsieh] +1? Or can we commit this. Change is fine anyway. Correct a typo in the mob metrics - Key: HBASE-12391 URL: https://issues.apache.org/jira/browse/HBASE-12391 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Jingcheng Du Assignee: Jingcheng Du Priority: Minor Fix For: hbase-11339 Attachments: HBASE-12391.diff There's a typo in the temp variable in the region server metrics for mob. It's now testMobCompactedFromMobCellsSize, and should be changed to tempMobCompactedFromMobCellsSize -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12393) The regionserver web UI will throw exception when we set block cache to zero
[
https://issues.apache.org/jira/browse/HBASE-12393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ChiaPing Tsai updated HBASE-12393:
--
Affects Version/s: 0.98.7
The regionserver web UI will throw exception when we set block cache to zero
Key: HBASE-12393
URL: https://issues.apache.org/jira/browse/HBASE-12393
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.98.7
Environment: ubuntu 12.04 64bits, hadoop-2.2.0, hbase-0.98.7-hadoop2
Reporter: ChiaPing Tsai
Priority: Minor
The CacheConfig.getBlockCache() will return the null point when we set
hfile.block.cache.size to zero.
It caused the BlockCacheTmplImpl.java:123 to throw null exception.
{code}
org.jamon.escaping.Escaping.HTML.write(org.jamon.emit.StandardEmitter.valueOf(StringUtils.humanReadableInt(cacheConfig.getBlockCache().size())),
jamonWriter);
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-12394) Support multiple regions as input to each mapper in map/reduce jobs
Weichen Ye created HBASE-12394: -- Summary: Support multiple regions as input to each mapper in map/reduce jobs Key: HBASE-12394 URL: https://issues.apache.org/jira/browse/HBASE-12394 Project: HBase Issue Type: Improvement Components: mapreduce Affects Versions: 0.98.6.1 Reporter: Weichen Ye -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12290) Column trackers and delete trackers should deal with BBs
[ https://issues.apache.org/jira/browse/HBASE-12290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-12290: --- Attachment: HBASE-12290.patch Adding some methods that deals with BB. Currently they wil be unused. Modified some existing APIs to work witih BB by wrapping the exisiting byte[] with BB. The new code here will be unused but once all the subtasks are completed would be useful. This would help in easier review. Also DeleteTracker cannot be changed now because it already deals with Cell. So all the impl of delete tracker should change based on it. Column trackers and delete trackers should deal with BBs Key: HBASE-12290 URL: https://issues.apache.org/jira/browse/HBASE-12290 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Affects Versions: 0.99.1 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Attachments: HBASE-12290.patch All the trackers should deal with BBs if we need E2E BB usage in the read path. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12290) Column trackers and delete trackers should deal with BBs
[ https://issues.apache.org/jira/browse/HBASE-12290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-12290: --- Status: Patch Available (was: Open) Column trackers and delete trackers should deal with BBs Key: HBASE-12290 URL: https://issues.apache.org/jira/browse/HBASE-12290 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Affects Versions: 0.99.1 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Attachments: HBASE-12290.patch All the trackers should deal with BBs if we need E2E BB usage in the read path. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12219) Cache more efficiently getAll() and get() in FSTableDescriptors
[ https://issues.apache.org/jira/browse/HBASE-12219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esteban Gutierrez updated HBASE-12219: -- Attachment: (was: HBASE-12219-0.99.patch) Cache more efficiently getAll() and get() in FSTableDescriptors --- Key: HBASE-12219 URL: https://issues.apache.org/jira/browse/HBASE-12219 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.94.24, 0.99.1, 0.98.6.1 Reporter: Esteban Gutierrez Assignee: Esteban Gutierrez Labels: scalability Attachments: HBASE-12219-v1.patch, HBASE-12219-v1.patch, HBASE-12219.v0.txt, HBASE-12219.v2.patch, HBASE-12219.v3.patch, list.png Currently table descriptors and tables are cached once they are accessed for the first time. Next calls to the master only require a trip to HDFS to lookup the modified time in order to reload the table descriptors if modified. However in clusters with a large number of tables or concurrent clients and this can be too aggressive to HDFS and the master causing contention to process other requests. A simple solution is to have a TTL based cached for FSTableDescriptors#getAll() and FSTableDescriptors#TableDescriptorAndModtime() that can allow the master to process those calls faster without causing contention without having to perform a trip to HDFS for every call. to listtables() or getTableDescriptor() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12392) Incorrect implementation of CompactionRequest.isRetainDeleteMarkers
[
https://issues.apache.org/jira/browse/HBASE-12392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14191647#comment-14191647
]
ramkrishna.s.vasudevan commented on HBASE-12392:
Should we wait for [~jmhsieh] +1? Or can we commit this. Change is fine anyway.
Incorrect implementation of CompactionRequest.isRetainDeleteMarkers
---
Key: HBASE-12392
URL: https://issues.apache.org/jira/browse/HBASE-12392
Project: HBase
Issue Type: Sub-task
Components: regionserver, Scanners
Affects Versions: hbase-11339
Reporter: Jingcheng Du
Assignee: Jingcheng Du
Priority: Critical
Fix For: hbase-11339
Attachments: HBASE-12392.diff
Now in the implementation of the isRetainDeleteMarkers method, the code look
like,
{code}
return (this.retainDeleteMarkers != null) ?
this.retainDeleteMarkers.booleanValue()
: isAllFiles();
{code}
It means for a major compaction in a normal store, this method returns true.
Consequently the delete marks could not be deleted in the major compaction,
which leads the unit test TestKeepDeletes fails.
The correct implementation should be,
{code}
return (this.retainDeleteMarkers != null) ?
this.retainDeleteMarkers.booleanValue()
: !isAllFiles();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12391) Correct a typo in the mob metrics
[ https://issues.apache.org/jira/browse/HBASE-12391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14191646#comment-14191646 ] ramkrishna.s.vasudevan commented on HBASE-12391: Sorry the previous comment was applicable for HBASE-12392. Correct a typo in the mob metrics - Key: HBASE-12391 URL: https://issues.apache.org/jira/browse/HBASE-12391 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Jingcheng Du Assignee: Jingcheng Du Priority: Minor Fix For: hbase-11339 Attachments: HBASE-12391.diff There's a typo in the temp variable in the region server metrics for mob. It's now testMobCompactedFromMobCellsSize, and should be changed to tempMobCompactedFromMobCellsSize -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12391) Correct a typo in the mob metrics
[ https://issues.apache.org/jira/browse/HBASE-12391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-12391: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Correct a typo in the mob metrics - Key: HBASE-12391 URL: https://issues.apache.org/jira/browse/HBASE-12391 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Jingcheng Du Assignee: Jingcheng Du Priority: Minor Fix For: hbase-11339 Attachments: HBASE-12391.diff There's a typo in the temp variable in the region server metrics for mob. It's now testMobCompactedFromMobCellsSize, and should be changed to tempMobCompactedFromMobCellsSize -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12219) Cache more efficiently getAll() and get() in FSTableDescriptors
[ https://issues.apache.org/jira/browse/HBASE-12219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14191648#comment-14191648 ] Esteban Gutierrez commented on HBASE-12219: --- Cancelled 0.99 patch for now, it was consistent but had some formatting issues. Cache more efficiently getAll() and get() in FSTableDescriptors --- Key: HBASE-12219 URL: https://issues.apache.org/jira/browse/HBASE-12219 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.94.24, 0.99.1, 0.98.6.1 Reporter: Esteban Gutierrez Assignee: Esteban Gutierrez Labels: scalability Attachments: HBASE-12219-v1.patch, HBASE-12219-v1.patch, HBASE-12219.v0.txt, HBASE-12219.v2.patch, HBASE-12219.v3.patch, list.png Currently table descriptors and tables are cached once they are accessed for the first time. Next calls to the master only require a trip to HDFS to lookup the modified time in order to reload the table descriptors if modified. However in clusters with a large number of tables or concurrent clients and this can be too aggressive to HDFS and the master causing contention to process other requests. A simple solution is to have a TTL based cached for FSTableDescriptors#getAll() and FSTableDescriptors#TableDescriptorAndModtime() that can allow the master to process those calls faster without causing contention without having to perform a trip to HDFS for every call. to listtables() or getTableDescriptor() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12393) The regionserver web UI will throw exception when we disable block cache
[
https://issues.apache.org/jira/browse/HBASE-12393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ChiaPing Tsai updated HBASE-12393:
--
Summary: The regionserver web UI will throw exception when we disable block
cache (was: The regionserver web UI will throw exception when we set block
cache to zero)
The regionserver web UI will throw exception when we disable block cache
Key: HBASE-12393
URL: https://issues.apache.org/jira/browse/HBASE-12393
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.98.7
Environment: ubuntu 12.04 64bits, hadoop-2.2.0, hbase-0.98.7-hadoop2
Reporter: ChiaPing Tsai
Priority: Minor
The CacheConfig.getBlockCache() will return the null point when we set
hfile.block.cache.size to zero.
It caused the BlockCacheTmplImpl.java:123 to throw null exception.
{code}
org.jamon.escaping.Escaping.HTML.write(org.jamon.emit.StandardEmitter.valueOf(StringUtils.humanReadableInt(cacheConfig.getBlockCache().size())),
jamonWriter);
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12393) The regionserver web will throw exception when we disable block cache
[
https://issues.apache.org/jira/browse/HBASE-12393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ChiaPing Tsai updated HBASE-12393:
--
Summary: The regionserver web will throw exception when we disable block
cache (was: The regionserver web UI will throw exception when we disable block
cache)
The regionserver web will throw exception when we disable block cache
-
Key: HBASE-12393
URL: https://issues.apache.org/jira/browse/HBASE-12393
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.98.7
Environment: ubuntu 12.04 64bits, hadoop-2.2.0, hbase-0.98.7-hadoop2
Reporter: ChiaPing Tsai
Priority: Minor
The CacheConfig.getBlockCache() will return the null point when we set
hfile.block.cache.size to zero.
It caused the BlockCacheTmplImpl.java:123 to throw null exception.
{code}
org.jamon.escaping.Escaping.HTML.write(org.jamon.emit.StandardEmitter.valueOf(StringUtils.humanReadableInt(cacheConfig.getBlockCache().size())),
jamonWriter);
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12394) Support multiple regions as input to each mapper in map/reduce jobs
[ https://issues.apache.org/jira/browse/HBASE-12394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Ye updated HBASE-12394: --- Description: For Hadoop cluster, a job with large HBase table as input always consumes a large amount of computing resources. For example, we need to create a job with 1000 mappers to scan a table with 1000 regions. This patch is to support one mapper using multiple regions as input. The following new files are including in this patch: TableMultiRegionInputFormat.java TableMultiRegionInputFormatBase.java TableMultiRegionMapReduceUtil.java *TestTableMultiRegionInputFormatScan1.java *TestTableMultiRegionInputFormatScan2.java *TestTableMultiRegionInputFormatScanBase.java *TestTableMultiRegionMapReduceUtil.java The files start with * are tests. In order to support multiple regions for one mapper, users can add a property in configuration--hbase.mapreduce.scan.regionspermapper This is an example,This means each mapper can have 3 regions as input. property namehbase.mapreduce.scan.regionspermapper/name value3/value /property This is an example for Java code: TableMultiRegionMapReduceUtil.initTableMapperJob(tablename, scan, Map.class, Text.class, Text.class, job); Support multiple regions as input to each mapper in map/reduce jobs --- Key: HBASE-12394 URL: https://issues.apache.org/jira/browse/HBASE-12394 Project: HBase Issue Type: Improvement Components: mapreduce Affects Versions: 0.98.6.1 Reporter: Weichen Ye For Hadoop cluster, a job with large HBase table as input always consumes a large amount of computing resources. For example, we need to create a job with 1000 mappers to scan a table with 1000 regions. This patch is to support one mapper using multiple regions as input. The following new files are including in this patch: TableMultiRegionInputFormat.java TableMultiRegionInputFormatBase.java TableMultiRegionMapReduceUtil.java *TestTableMultiRegionInputFormatScan1.java *TestTableMultiRegionInputFormatScan2.java *TestTableMultiRegionInputFormatScanBase.java *TestTableMultiRegionMapReduceUtil.java The files start with * are tests. In order to support multiple regions for one mapper, users can add a property in configuration--hbase.mapreduce.scan.regionspermapper This is an example,This means each mapper can have 3 regions as input. property namehbase.mapreduce.scan.regionspermapper/name value3/value /property This is an example for Java code: TableMultiRegionMapReduceUtil.initTableMapperJob(tablename, scan, Map.class, Text.class, Text.class, job); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12394) Support multiple regions as input to each mapper in map/reduce jobs
[ https://issues.apache.org/jira/browse/HBASE-12394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Ye updated HBASE-12394: --- Attachment: HBASE-12394.patch Support multiple regions as input to each mapper in map/reduce jobs --- Key: HBASE-12394 URL: https://issues.apache.org/jira/browse/HBASE-12394 Project: HBase Issue Type: Improvement Components: mapreduce Affects Versions: 0.98.6.1 Reporter: Weichen Ye Attachments: HBASE-12394.patch For Hadoop cluster, a job with large HBase table as input always consumes a large amount of computing resources. For example, we need to create a job with 1000 mappers to scan a table with 1000 regions. This patch is to support one mapper using multiple regions as input. The following new files are including in this patch: TableMultiRegionInputFormat.java TableMultiRegionInputFormatBase.java TableMultiRegionMapReduceUtil.java *TestTableMultiRegionInputFormatScan1.java *TestTableMultiRegionInputFormatScan2.java *TestTableMultiRegionInputFormatScanBase.java *TestTableMultiRegionMapReduceUtil.java The files start with * are tests. In order to support multiple regions for one mapper, users can add a property in configuration--hbase.mapreduce.scan.regionspermapper This is an example,This means each mapper can have 3 regions as input. property namehbase.mapreduce.scan.regionspermapper/name value3/value /property This is an example for Java code: TableMultiRegionMapReduceUtil.initTableMapperJob(tablename, scan, Map.class, Text.class, Text.class, job); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12394) Support multiple regions as input to each mapper in map/reduce jobs
[ https://issues.apache.org/jira/browse/HBASE-12394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Ye updated HBASE-12394: --- Description: For Hadoop cluster, a job with large HBase table as input always consumes a large amount of computing resources. For example, we need to create a job with 1000 mappers to scan a table with 1000 regions. This patch is to support one mapper using multiple regions as input. The following new files are including in this patch: TableMultiRegionInputFormat.java TableMultiRegionInputFormatBase.java TableMultiRegionMapReduceUtil.java *TestTableMultiRegionInputFormatScan1.java *TestTableMultiRegionInputFormatScan2.java *TestTableMultiRegionInputFormatScanBase.java *TestTableMultiRegionMapReduceUtil.java The files start with * are tests. In order to support multiple regions for one mapper, we need a new property in configuration--hbase.mapreduce.scan.regionspermapper This is an example,which means each mapper can have 3 regions as input. property namehbase.mapreduce.scan.regionspermapper/name value3/value /property This is an example for Java code: TableMultiRegionMapReduceUtil.initTableMapperJob(tablename, scan, Map.class, Text.class, Text.class, job); was: For Hadoop cluster, a job with large HBase table as input always consumes a large amount of computing resources. For example, we need to create a job with 1000 mappers to scan a table with 1000 regions. This patch is to support one mapper using multiple regions as input. The following new files are including in this patch: TableMultiRegionInputFormat.java TableMultiRegionInputFormatBase.java TableMultiRegionMapReduceUtil.java *TestTableMultiRegionInputFormatScan1.java *TestTableMultiRegionInputFormatScan2.java *TestTableMultiRegionInputFormatScanBase.java *TestTableMultiRegionMapReduceUtil.java The files start with * are tests. In order to support multiple regions for one mapper, users can add a property in configuration--hbase.mapreduce.scan.regionspermapper This is an example,This means each mapper can have 3 regions as input. property namehbase.mapreduce.scan.regionspermapper/name value3/value /property This is an example for Java code: TableMultiRegionMapReduceUtil.initTableMapperJob(tablename, scan, Map.class, Text.class, Text.class, job); Support multiple regions as input to each mapper in map/reduce jobs --- Key: HBASE-12394 URL: https://issues.apache.org/jira/browse/HBASE-12394 Project: HBase Issue Type: Improvement Components: mapreduce Affects Versions: 0.98.6.1 Reporter: Weichen Ye Attachments: HBASE-12394.patch For Hadoop cluster, a job with large HBase table as input always consumes a large amount of computing resources. For example, we need to create a job with 1000 mappers to scan a table with 1000 regions. This patch is to support one mapper using multiple regions as input. The following new files are including in this patch: TableMultiRegionInputFormat.java TableMultiRegionInputFormatBase.java TableMultiRegionMapReduceUtil.java *TestTableMultiRegionInputFormatScan1.java *TestTableMultiRegionInputFormatScan2.java *TestTableMultiRegionInputFormatScanBase.java *TestTableMultiRegionMapReduceUtil.java The files start with * are tests. In order to support multiple regions for one mapper, we need a new property in configuration--hbase.mapreduce.scan.regionspermapper This is an example,which means each mapper can have 3 regions as input. property namehbase.mapreduce.scan.regionspermapper/name value3/value /property This is an example for Java code: TableMultiRegionMapReduceUtil.initTableMapperJob(tablename, scan, Map.class, Text.class, Text.class, job); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12394) Support multiple regions as input to each mapper in map/reduce jobs
[ https://issues.apache.org/jira/browse/HBASE-12394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Ye updated HBASE-12394: --- Description: For Hadoop cluster, a job with large HBase table as input always consumes a large amount of computing resources. For example, we need to create a job with 1000 mappers to scan a table with 1000 regions. This patch is to support one mapper using multiple regions as input. The following new files are including in this patch: TableMultiRegionInputFormat.java TableMultiRegionInputFormatBase.java TableMultiRegionMapReduceUtil.java *TestTableMultiRegionInputFormatScan1.java *TestTableMultiRegionInputFormatScan2.java *TestTableMultiRegionInputFormatScanBase.java *TestTableMultiRegionMapReduceUtil.java The files start with * are tests. In order to support multiple regions for one mapper, we need a new property in configuration--hbase.mapreduce.scan.regionspermapper This is an example,which means each mapper has 3 regions as input. property namehbase.mapreduce.scan.regionspermapper/name value3/value /property This is an example for Java code: TableMultiRegionMapReduceUtil.initTableMapperJob(tablename, scan, Map.class, Text.class, Text.class, job); was: For Hadoop cluster, a job with large HBase table as input always consumes a large amount of computing resources. For example, we need to create a job with 1000 mappers to scan a table with 1000 regions. This patch is to support one mapper using multiple regions as input. The following new files are including in this patch: TableMultiRegionInputFormat.java TableMultiRegionInputFormatBase.java TableMultiRegionMapReduceUtil.java *TestTableMultiRegionInputFormatScan1.java *TestTableMultiRegionInputFormatScan2.java *TestTableMultiRegionInputFormatScanBase.java *TestTableMultiRegionMapReduceUtil.java The files start with * are tests. In order to support multiple regions for one mapper, we need a new property in configuration--hbase.mapreduce.scan.regionspermapper This is an example,which means each mapper can have 3 regions as input. property namehbase.mapreduce.scan.regionspermapper/name value3/value /property This is an example for Java code: TableMultiRegionMapReduceUtil.initTableMapperJob(tablename, scan, Map.class, Text.class, Text.class, job); Support multiple regions as input to each mapper in map/reduce jobs --- Key: HBASE-12394 URL: https://issues.apache.org/jira/browse/HBASE-12394 Project: HBase Issue Type: Improvement Components: mapreduce Affects Versions: 0.98.6.1 Reporter: Weichen Ye Attachments: HBASE-12394.patch For Hadoop cluster, a job with large HBase table as input always consumes a large amount of computing resources. For example, we need to create a job with 1000 mappers to scan a table with 1000 regions. This patch is to support one mapper using multiple regions as input. The following new files are including in this patch: TableMultiRegionInputFormat.java TableMultiRegionInputFormatBase.java TableMultiRegionMapReduceUtil.java *TestTableMultiRegionInputFormatScan1.java *TestTableMultiRegionInputFormatScan2.java *TestTableMultiRegionInputFormatScanBase.java *TestTableMultiRegionMapReduceUtil.java The files start with * are tests. In order to support multiple regions for one mapper, we need a new property in configuration--hbase.mapreduce.scan.regionspermapper This is an example,which means each mapper has 3 regions as input. property namehbase.mapreduce.scan.regionspermapper/name value3/value /property This is an example for Java code: TableMultiRegionMapReduceUtil.initTableMapperJob(tablename, scan, Map.class, Text.class, Text.class, job); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12392) Incorrect implementation of CompactionRequest.isRetainDeleteMarkers
[
https://issues.apache.org/jira/browse/HBASE-12392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-12392:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed to HBASE-11339. Thanks for the patch Jingcheng.
Incorrect implementation of CompactionRequest.isRetainDeleteMarkers
---
Key: HBASE-12392
URL: https://issues.apache.org/jira/browse/HBASE-12392
Project: HBase
Issue Type: Sub-task
Components: regionserver, Scanners
Affects Versions: hbase-11339
Reporter: Jingcheng Du
Assignee: Jingcheng Du
Priority: Critical
Fix For: hbase-11339
Attachments: HBASE-12392.diff
Now in the implementation of the isRetainDeleteMarkers method, the code look
like,
{code}
return (this.retainDeleteMarkers != null) ?
this.retainDeleteMarkers.booleanValue()
: isAllFiles();
{code}
It means for a major compaction in a normal store, this method returns true.
Consequently the delete marks could not be deleted in the major compaction,
which leads the unit test TestKeepDeletes fails.
The correct implementation should be,
{code}
return (this.retainDeleteMarkers != null) ?
this.retainDeleteMarkers.booleanValue()
: !isAllFiles();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12394) Support multiple regions as input to each mapper in map/reduce jobs
[ https://issues.apache.org/jira/browse/HBASE-12394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Ye updated HBASE-12394: --- Description: For Hadoop cluster, a job with large HBase table as input always consumes a large amount of computing resources. For example, we need to create a job with 1000 mappers to scan a table with 1000 regions. This patch is to support one mapper using multiple regions as input. The following new files are included in this patch: TableMultiRegionInputFormat.java TableMultiRegionInputFormatBase.java TableMultiRegionMapReduceUtil.java *TestTableMultiRegionInputFormatScan1.java *TestTableMultiRegionInputFormatScan2.java *TestTableMultiRegionInputFormatScanBase.java *TestTableMultiRegionMapReduceUtil.java The files start with * are tests. In order to support multiple regions for one mapper, we need a new property in configuration--hbase.mapreduce.scan.regionspermapper This is an example,which means each mapper has 3 regions as input. property namehbase.mapreduce.scan.regionspermapper/name value3/value /property This is an example for Java code: TableMultiRegionMapReduceUtil.initTableMapperJob(tablename, scan, Map.class, Text.class, Text.class, job); was: For Hadoop cluster, a job with large HBase table as input always consumes a large amount of computing resources. For example, we need to create a job with 1000 mappers to scan a table with 1000 regions. This patch is to support one mapper using multiple regions as input. The following new files are including in this patch: TableMultiRegionInputFormat.java TableMultiRegionInputFormatBase.java TableMultiRegionMapReduceUtil.java *TestTableMultiRegionInputFormatScan1.java *TestTableMultiRegionInputFormatScan2.java *TestTableMultiRegionInputFormatScanBase.java *TestTableMultiRegionMapReduceUtil.java The files start with * are tests. In order to support multiple regions for one mapper, we need a new property in configuration--hbase.mapreduce.scan.regionspermapper This is an example,which means each mapper has 3 regions as input. property namehbase.mapreduce.scan.regionspermapper/name value3/value /property This is an example for Java code: TableMultiRegionMapReduceUtil.initTableMapperJob(tablename, scan, Map.class, Text.class, Text.class, job); Support multiple regions as input to each mapper in map/reduce jobs --- Key: HBASE-12394 URL: https://issues.apache.org/jira/browse/HBASE-12394 Project: HBase Issue Type: Improvement Components: mapreduce Affects Versions: 0.98.6.1 Reporter: Weichen Ye Attachments: HBASE-12394.patch For Hadoop cluster, a job with large HBase table as input always consumes a large amount of computing resources. For example, we need to create a job with 1000 mappers to scan a table with 1000 regions. This patch is to support one mapper using multiple regions as input. The following new files are included in this patch: TableMultiRegionInputFormat.java TableMultiRegionInputFormatBase.java TableMultiRegionMapReduceUtil.java *TestTableMultiRegionInputFormatScan1.java *TestTableMultiRegionInputFormatScan2.java *TestTableMultiRegionInputFormatScanBase.java *TestTableMultiRegionMapReduceUtil.java The files start with * are tests. In order to support multiple regions for one mapper, we need a new property in configuration--hbase.mapreduce.scan.regionspermapper This is an example,which means each mapper has 3 regions as input. property namehbase.mapreduce.scan.regionspermapper/name value3/value /property This is an example for Java code: TableMultiRegionMapReduceUtil.initTableMapperJob(tablename, scan, Map.class, Text.class, Text.class, job); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12394) Support multiple regions as input to each mapper in map/reduce jobs
[ https://issues.apache.org/jira/browse/HBASE-12394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Ye updated HBASE-12394: --- Affects Version/s: 2.0.0 Status: Patch Available (was: Open) Support multiple regions as input to each mapper in map/reduce jobs --- Key: HBASE-12394 URL: https://issues.apache.org/jira/browse/HBASE-12394 Project: HBase Issue Type: Improvement Components: mapreduce Affects Versions: 0.98.6.1, 2.0.0 Reporter: Weichen Ye Attachments: HBASE-12394.patch For Hadoop cluster, a job with large HBase table as input always consumes a large amount of computing resources. For example, we need to create a job with 1000 mappers to scan a table with 1000 regions. This patch is to support one mapper using multiple regions as input. The following new files are included in this patch: TableMultiRegionInputFormat.java TableMultiRegionInputFormatBase.java TableMultiRegionMapReduceUtil.java *TestTableMultiRegionInputFormatScan1.java *TestTableMultiRegionInputFormatScan2.java *TestTableMultiRegionInputFormatScanBase.java *TestTableMultiRegionMapReduceUtil.java The files start with * are tests. In order to support multiple regions for one mapper, we need a new property in configuration--hbase.mapreduce.scan.regionspermapper This is an example,which means each mapper has 3 regions as input. property namehbase.mapreduce.scan.regionspermapper/name value3/value /property This is an example for Java code: TableMultiRegionMapReduceUtil.initTableMapperJob(tablename, scan, Map.class, Text.class, Text.class, job); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12358) Create ByteBuffer backed Cell
[ https://issues.apache.org/jira/browse/HBASE-12358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14191673#comment-14191673 ] ramkrishna.s.vasudevan commented on HBASE-12358: Some APIs needs some small changes. Will update the patch once done and adding test cases. Create ByteBuffer backed Cell - Key: HBASE-12358 URL: https://issues.apache.org/jira/browse/HBASE-12358 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Attachments: HBASE-12358.patch, HBASE-12358_1.patch As part of HBASE-12224 and HBASE-12282 we wanted a Cell that is backed by BB. Changing the core Cell impl would not be needed as it is used in server only. So we will create a BB backed Cell and use it in the Server side read path. This JIRA just creates an interface that extends Cell and adds the needed API. The getTimeStamp and getTypebyte() can still refer to the original Cell API only. The getXXxOffset() and getXXXLength() can also refer to the original Cell only. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12393) The regionserver web will throw exception if we disable block cache
[
https://issues.apache.org/jira/browse/HBASE-12393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ChiaPing Tsai updated HBASE-12393:
--
Summary: The regionserver web will throw exception if we disable block
cache (was: The regionserver web will throw exception when we disable block
cache)
The regionserver web will throw exception if we disable block cache
---
Key: HBASE-12393
URL: https://issues.apache.org/jira/browse/HBASE-12393
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.98.7
Environment: ubuntu 12.04 64bits, hadoop-2.2.0, hbase-0.98.7-hadoop2
Reporter: ChiaPing Tsai
Priority: Minor
The CacheConfig.getBlockCache() will return the null point when we set
hfile.block.cache.size to zero.
It caused the BlockCacheTmplImpl.java:123 to throw null exception.
{code}
org.jamon.escaping.Escaping.HTML.write(org.jamon.emit.StandardEmitter.valueOf(StringUtils.humanReadableInt(cacheConfig.getBlockCache().size())),
jamonWriter);
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12363) KEEP_DELETED_CELLS considered harmful?
[ https://issues.apache.org/jira/browse/HBASE-12363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14191690#comment-14191690 ] Ted Yu commented on HBASE-12363: +1 on getting test in. Nit: fix typo on commit: compactin KEEP_DELETED_CELLS considered harmful? -- Key: HBASE-12363 URL: https://issues.apache.org/jira/browse/HBASE-12363 Project: HBase Issue Type: Sub-task Components: regionserver Reporter: Lars Hofhansl Labels: Phoenix Attachments: 12363-test.txt Brainstorming... This morning in the train (of all places) I realized a fundamental issue in how KEEP_DELETED_CELLS is implemented. The problem is around knowing when it is safe to remove a delete marker (we cannot remove it unless all cells affected by it are remove otherwise). This was particularly hard for family marker, since they sort before all cells of a row, and hence scanning forward through an HFile you cannot know whether the family markers are still needed until at least the entire row is scanned. My solution was to keep the TS of the oldest put in any given HFile, and only remove delete markers older than that TS. That sounds good on the face of it... But now imagine you wrote a version of ROW 1 and then never update it again. Then later you write a billion other rows and delete them all. Since the TS of the cells in ROW 1 is older than all the delete markers for the other billion rows, these will never be collected... At least for the region that hosts ROW 1 after a major compaction. Note, in a sense that is what HBase is supposed to do when keeping deleted cells: Keep them until they would be removed by some other means (for example TTL, or MAX_VERSION when new versions are inserted). The specific problem here is that even as all KVs affected by a delete marker are expired this way the marker would not be removed if there just one older KV in the HStore. I don't see a good way out of this. In parent I outlined these four solutions: So there are three options I think: # Only allow the new flag set on CFs with TTL set. MIN_VERSIONS would not apply to deleted rows or delete marker rows (wouldn't know how long to keep family deletes in that case). (MAX)VERSIONS would still be enforced on all rows types except for family delete markers. # Translate family delete markers to column delete marker at (major) compaction time. # Change HFileWriterV* to keep track of the earliest put TS in a store and write it to the file metadata. Use that use expire delete marker that are older and hence can't affect any puts in the file. # Have Store.java keep track of the earliest put in internalFlushCache and compactStore and then append it to the file metadata. That way HFileWriterV* would not need to know about KVs. And I implemented #4. I'd love to get input on ideas. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12290) Column trackers and delete trackers should deal with BBs
[
https://issues.apache.org/jira/browse/HBASE-12290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14191706#comment-14191706
]
Hadoop QA commented on HBASE-12290:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12678449/HBASE-12290.patch
against trunk revision .
ATTACHMENT ID: 12678449
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 2
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/11540//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11540//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11540//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11540//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11540//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11540//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11540//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11540//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11540//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11540//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11540//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11540//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/11540//artifact/patchprocess/checkstyle-aggregate.html
Javadoc warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11540//artifact/patchprocess/patchJavadocWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/11540//console
This message is automatically generated.
Column trackers and delete trackers should deal with BBs
Key: HBASE-12290
URL: https://issues.apache.org/jira/browse/HBASE-12290
Project: HBase
Issue Type: Sub-task
Components: regionserver, Scanners
Affects Versions: 0.99.1
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Attachments: HBASE-12290.patch
All the trackers should deal with BBs if we need E2E BB usage in the read
path.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12394) Support multiple regions as input to each mapper in map/reduce jobs
[
https://issues.apache.org/jira/browse/HBASE-12394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14191741#comment-14191741
]
Hadoop QA commented on HBASE-12394:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12678452/HBASE-12394.patch
against trunk revision .
ATTACHMENT ID: 12678452
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 8 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+ * See {@link
org.apache.hadoop.hbase.mapreduce.TableMultiRegionMapReduceUtil#convertScanToString(org.apache.hadoop.hbase.client.Scan)}
for more details.
+ * Overrides previous calls to {@link
org.apache.hadoop.hbase.client.Scan#addColumn(byte[], byte[])}for any families
in the
+ String regionPerMapper
=context.getConfiguration().get(hbase.mapreduce.scan.regionspermapper,1);
+ LOG.error(ERROR when parseInt: hbase.mapreduce.scan.regionspermapper
must be an integer );
+int
stopRegion=(i*regionPerMapperInt+regionPerMapperInt-1keys.getFirst().length)?(i*regionPerMapperInt+regionPerMapperInt-1):(keys.getFirst().length-1);
+InetSocketAddress isa = new
InetSocketAddress(location.getHostname(), location.getPort());
+ * This optimization is effective when there is a specific reasoning to
exclude an entire region from the M-R job,
+ * Useful when we need to remember the last-processed top record and revisit
the [last, current) interval for M-R processing,
+ * continuously. In addition to reducing InputSplits, reduces the load on
the region server as well, due to the ordering of the keys.
+ * Override this method, if you want to bulk exclude regions altogether from
M-R. By default, no region is excluded( i.e. all regions are included).
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/11541//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11541//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11541//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11541//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11541//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11541//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11541//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11541//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11541//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11541//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11541//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11541//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/11541//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/11541//console
This message is automatically generated.
Support multiple regions as input to each mapper in map/reduce jobs
---
Key:
[jira] [Commented] (HBASE-12391) Correct a typo in the mob metrics
[ https://issues.apache.org/jira/browse/HBASE-12391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14191748#comment-14191748 ] Jonathan Hsieh commented on HBASE-12391: +1, lgtm. Correct a typo in the mob metrics - Key: HBASE-12391 URL: https://issues.apache.org/jira/browse/HBASE-12391 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Jingcheng Du Assignee: Jingcheng Du Priority: Minor Fix For: hbase-11339 Attachments: HBASE-12391.diff There's a typo in the temp variable in the region server metrics for mob. It's now testMobCompactedFromMobCellsSize, and should be changed to tempMobCompactedFromMobCellsSize -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12392) Incorrect implementation of CompactionRequest.isRetainDeleteMarkers
[
https://issues.apache.org/jira/browse/HBASE-12392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14191749#comment-14191749
]
Jonathan Hsieh commented on HBASE-12392:
+1, lgtm.
Incorrect implementation of CompactionRequest.isRetainDeleteMarkers
---
Key: HBASE-12392
URL: https://issues.apache.org/jira/browse/HBASE-12392
Project: HBase
Issue Type: Sub-task
Components: regionserver, Scanners
Affects Versions: hbase-11339
Reporter: Jingcheng Du
Assignee: Jingcheng Du
Priority: Critical
Fix For: hbase-11339
Attachments: HBASE-12392.diff
Now in the implementation of the isRetainDeleteMarkers method, the code look
like,
{code}
return (this.retainDeleteMarkers != null) ?
this.retainDeleteMarkers.booleanValue()
: isAllFiles();
{code}
It means for a major compaction in a normal store, this method returns true.
Consequently the delete marks could not be deleted in the major compaction,
which leads the unit test TestKeepDeletes fails.
The correct implementation should be,
{code}
return (this.retainDeleteMarkers != null) ?
this.retainDeleteMarkers.booleanValue()
: !isAllFiles();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12358) Create ByteBuffer backed Cell
[ https://issues.apache.org/jira/browse/HBASE-12358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-12358: --- Status: Patch Available (was: Open) Create ByteBuffer backed Cell - Key: HBASE-12358 URL: https://issues.apache.org/jira/browse/HBASE-12358 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Attachments: HBASE-12358.patch, HBASE-12358_1.patch, HBASE-12358_2.patch As part of HBASE-12224 and HBASE-12282 we wanted a Cell that is backed by BB. Changing the core Cell impl would not be needed as it is used in server only. So we will create a BB backed Cell and use it in the Server side read path. This JIRA just creates an interface that extends Cell and adds the needed API. The getTimeStamp and getTypebyte() can still refer to the original Cell API only. The getXXxOffset() and getXXXLength() can also refer to the original Cell only. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12358) Create ByteBuffer backed Cell
[ https://issues.apache.org/jira/browse/HBASE-12358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-12358: --- Attachment: HBASE-12358_2.patch Updated patch with test cases. Also adds ByteBufferBackedKeyOnlyKeyValue. Create ByteBuffer backed Cell - Key: HBASE-12358 URL: https://issues.apache.org/jira/browse/HBASE-12358 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Attachments: HBASE-12358.patch, HBASE-12358_1.patch, HBASE-12358_2.patch As part of HBASE-12224 and HBASE-12282 we wanted a Cell that is backed by BB. Changing the core Cell impl would not be needed as it is used in server only. So we will create a BB backed Cell and use it in the Server side read path. This JIRA just creates an interface that extends Cell and adds the needed API. The getTimeStamp and getTypebyte() can still refer to the original Cell API only. The getXXxOffset() and getXXXLength() can also refer to the original Cell only. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12329) Table create with duplicate column family names quietly succeeds
[
https://issues.apache.org/jira/browse/HBASE-12329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingcheng Du updated HBASE-12329:
-
Attachment: HBASE-12329-V2.diff
The latest patch is uploaded.
bq. either update the API example in the ref guide (currently Example 10.2) to
show update an existing CF or file a follow on ticket that such docs are
needed
I think it's better to be done by Misty for the ref guide?
Hi, [~misty], do you want to update the ref guide(currently Example 10.2 to
show update an existing CF with the new API modifyFamily in master) after
this JIRA is committed? Thanks a lot!
Hi all, please be noted, this patch is targeted to master. If it's backported
to lower versions, should have the last one wins warning in addFamily method
instead of throwing a runtime exception.
Thanks.
Table create with duplicate column family names quietly succeeds
Key: HBASE-12329
URL: https://issues.apache.org/jira/browse/HBASE-12329
Project: HBase
Issue Type: Bug
Components: Client, shell
Reporter: Sean Busbey
Assignee: Jingcheng Du
Priority: Minor
Attachments: HBASE-12329-V2.diff, HBASE-12329.diff
From the mailing list
{quote}
I was expecting that it is forbidden, **but** this call does not throw any
exception
{code}
String[] families = {cf, cf};
HTableDescriptor desc = new HTableDescriptor(name);
for (String cf : families) {
HColumnDescriptor coldef = new HColumnDescriptor(cf);
desc.addFamily(coldef);
}
try {
admin.createTable(desc);
} catch (TableExistsException e) {
throw new IOException(table \' + name + \' already exists);
}
{code}
{quote}
And Ted's follow up replicates in the shell
{code}
hbase(main):001:0 create 't2', {NAME = 'f1'}, {NAME = 'f1'}
The table got created - with 1 column family:
hbase(main):002:0 describe 't2'
DESCRIPTION
ENABLED
't2', {NAME = 'f1', DATA_BLOCK_ENCODING = 'NONE', BLOOMFILTER = 'ROW',
REPLICATION_SCOPE = '0 true
', VERSIONS = '1', COMPRESSION = 'NONE', MIN_VERSIONS = '0', TTL =
'2147483647', KEEP_DELETED
_CELLS = 'false', BLOCKSIZE = '65536', IN_MEMORY = 'false', BLOCKCACHE
= 'true'}
1 row(s) in 0.1000 seconds
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12332) [mob] use filelink instad of retry when resolving an hfilelink.
[ https://issues.apache.org/jira/browse/HBASE-12332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14191806#comment-14191806 ] Jingcheng Du commented on HBASE-12332: -- Hi [~jmhsieh], any ideas on the comments? Thanks! [mob] use filelink instad of retry when resolving an hfilelink. --- Key: HBASE-12332 URL: https://issues.apache.org/jira/browse/HBASE-12332 Project: HBase Issue Type: Sub-task Components: mob Affects Versions: hbase-11339 Reporter: Jonathan Hsieh Fix For: hbase-11339 in the snapshot code, hmobstore was modified to traverse an hfile link to a mob. Ideally this should use the transparent filelink code to read the data. Also there will likely be some issues with the mob file cache with these links. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12393) The regionserver web will throw exception if we disable block cache
[
https://issues.apache.org/jira/browse/HBASE-12393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14191818#comment-14191818
]
Jean-Marc Spaggiari commented on HBASE-12393:
-
Should we not simply throw exception when someone set the blockcache to 0 and
not start the RS?
The regionserver web will throw exception if we disable block cache
---
Key: HBASE-12393
URL: https://issues.apache.org/jira/browse/HBASE-12393
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.98.7
Environment: ubuntu 12.04 64bits, hadoop-2.2.0, hbase-0.98.7-hadoop2
Reporter: ChiaPing Tsai
Priority: Minor
The CacheConfig.getBlockCache() will return the null point when we set
hfile.block.cache.size to zero.
It caused the BlockCacheTmplImpl.java:123 to throw null exception.
{code}
org.jamon.escaping.Escaping.HTML.write(org.jamon.emit.StandardEmitter.valueOf(StringUtils.humanReadableInt(cacheConfig.getBlockCache().size())),
jamonWriter);
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HBASE-12387) committer guidelines should include patch signoff
[
https://issues.apache.org/jira/browse/HBASE-12387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey reassigned HBASE-12387:
---
Assignee: Sean Busbey
committer guidelines should include patch signoff
-
Key: HBASE-12387
URL: https://issues.apache.org/jira/browse/HBASE-12387
Project: HBase
Issue Type: Task
Components: documentation
Reporter: Sean Busbey
Assignee: Sean Busbey
Right now our guide for committers apply patches has them use {{git am}}
without a signoff flag. This works okay, but it misses adding the
signed-off-by blurb in the commit message.
Those messages make it easier to see at a glance with e.g. {{git log}} which
committer applied the patch.
this section:
{quote}
The directive to use git format-patch rather than git diff, and not to use
--no-prefix, is a new one. See the second example for how to apply a patch
created with git diff, and educate the person who created the patch.
{code}
$ git checkout -b HBASE-
$ git am ~/Downloads/HBASE--v2.patch
$ git checkout master
$ git pull --rebase
$ git cherry-pick sha-from-commit
# Resolve conflicts if necessary or ask the submitter to do it
$ git pull --rebase # Better safe than sorry
$ git push origin master
$ git checkout branch-1
$ git pull --rebase
$ git cherry-pick sha-from-commit
# Resolve conflicts if necessary
$ git pull --rebase # Better safe than sorry
$ git push origin branch-1
$ git branch -D HBASE-
{code}
{quote}
Should be
{quote}
The directive to use git format-patch rather than git diff, and not to use
--no-prefix, is a new one. See the second example for how to apply a patch
created with git diff, and educate the person who created the patch.
Note that the {{--signoff}} flag to {{git am}} will insert a line in the
commit message that the patch was checked by your author string. This
addition to your inclusion as the commit's committer makes your participation
more prominent to users browsing {{git log}}.
{code}
$ git checkout -b HBASE-
$ git am --signoff ~/Downloads/HBASE--v2.patch
$ git checkout master
$ git pull --rebase
$ git cherry-pick sha-from-commit
# Resolve conflicts if necessary or ask the submitter to do it
$ git pull --rebase # Better safe than sorry
$ git push origin master
$ git checkout branch-1
$ git pull --rebase
$ git cherry-pick sha-from-commit
# Resolve conflicts if necessary
$ git pull --rebase # Better safe than sorry
$ git push origin branch-1
$ git branch -D HBASE-
{code}
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12387) committer guidelines should include patch signoff
[
https://issues.apache.org/jira/browse/HBASE-12387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14191827#comment-14191827
]
Sean Busbey commented on HBASE-12387:
-
If we want to do that we'll need to expand the committer guidelines to cover
setting things up.
Something like
{quote}
This example shows how to commit a patch that was created using git diff
without --no-prefix. If the patch was created with --no-prefix, add -p0 to the
git apply command. *Before following these steps, you must ensure your working
directory is clean.* Otherwise, the autocommit step (commit -a) will coalesce
your local changes with those of the contributor.
Note that unlike the case of a patch made with git format-patch, the patch
itself doesn't include information on the contributor. Normally, you should be
able to use the name and email address from the user's ASF Jira account to fill
in the author details, below we use the example Prathia Hall
prathia.h...@example.com.
{code}
$ git apply ~/Downloads/HBASE--v2.patch
$ git commit -m HBASE- Really Good Code --author Prathia Hall
prathia.h...@example.com \
--signoff -a # This extra step is needed for patches created with 'git diff'
$ git checkout master
$ git pull --rebase
$ git cherry-pick sha-from-commit
# Resolve conflicts if necessary or ask the submitter to do it
$ git pull --rebase # Better safe than sorry
$ git push origin master
$ git checkout branch-1
$ git pull --rebase
$ git cherry-pick sha-from-commit
# Resolve conflicts if necessary or ask the submitter to do it
$ git pull --rebase # Better safe than sorry
$ git push origin branch-1
$ git branch -D HBASE-
{code}
{quote}
Actually, that's not that much more work. But it sounds like enough of a change
to warrant a DISCUSS thread?
committer guidelines should include patch signoff
-
Key: HBASE-12387
URL: https://issues.apache.org/jira/browse/HBASE-12387
Project: HBase
Issue Type: Task
Components: documentation
Reporter: Sean Busbey
Right now our guide for committers apply patches has them use {{git am}}
without a signoff flag. This works okay, but it misses adding the
signed-off-by blurb in the commit message.
Those messages make it easier to see at a glance with e.g. {{git log}} which
committer applied the patch.
this section:
{quote}
The directive to use git format-patch rather than git diff, and not to use
--no-prefix, is a new one. See the second example for how to apply a patch
created with git diff, and educate the person who created the patch.
{code}
$ git checkout -b HBASE-
$ git am ~/Downloads/HBASE--v2.patch
$ git checkout master
$ git pull --rebase
$ git cherry-pick sha-from-commit
# Resolve conflicts if necessary or ask the submitter to do it
$ git pull --rebase # Better safe than sorry
$ git push origin master
$ git checkout branch-1
$ git pull --rebase
$ git cherry-pick sha-from-commit
# Resolve conflicts if necessary
$ git pull --rebase # Better safe than sorry
$ git push origin branch-1
$ git branch -D HBASE-
{code}
{quote}
Should be
{quote}
The directive to use git format-patch rather than git diff, and not to use
--no-prefix, is a new one. See the second example for how to apply a patch
created with git diff, and educate the person who created the patch.
Note that the {{--signoff}} flag to {{git am}} will insert a line in the
commit message that the patch was checked by your author string. This
addition to your inclusion as the commit's committer makes your participation
more prominent to users browsing {{git log}}.
{code}
$ git checkout -b HBASE-
$ git am --signoff ~/Downloads/HBASE--v2.patch
$ git checkout master
$ git pull --rebase
$ git cherry-pick sha-from-commit
# Resolve conflicts if necessary or ask the submitter to do it
$ git pull --rebase # Better safe than sorry
$ git push origin master
$ git checkout branch-1
$ git pull --rebase
$ git cherry-pick sha-from-commit
# Resolve conflicts if necessary
$ git pull --rebase # Better safe than sorry
$ git push origin branch-1
$ git branch -D HBASE-
{code}
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12329) Table create with duplicate column family names quietly succeeds
[
https://issues.apache.org/jira/browse/HBASE-12329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14191840#comment-14191840
]
Sean Busbey commented on HBASE-12329:
-
{quote}
I think it's better to be done by Misty for the ref guide?
Hi, Misty Stanley-Jones, do you want to update the ref guide(currently Example
10.2 to show update an existing CF with the new API modifyFamily in master)
after this JIRA is committed? Thanks a lot!
{quote}
Please file as a follow on ticket. If Misty wants to take it up, then she can.
Otherwise someone else can do it.
{code}
+ @Test
+ public void testModifyFamily() {
+HTableDescriptor htd = new HTableDescriptor(TableName.valueOf(table));
+byte[] familyName = Bytes.toBytes(cf);
+HColumnDescriptor hcd = new HColumnDescriptor(familyName);
+hcd.setBlocksize(1000);
+htd.addFamily(hcd);
+assertEquals(1000, htd.getFamily(familyName).getBlocksize());
+hcd.setBlocksize(2000);
+htd.modifyFamily(hcd);
+assertEquals(2000, htd.getFamily(familyName).getBlocksize());
+ }
{code}
Make a second HColumnDescriptor for the update to make sure we're not just
mutating state on an old version in the htd.
{code}
+ @Test
+ public void testModifyInexistentFamily() {
+HTableDescriptor htd = new HTableDescriptor(TableName.valueOf(table));
+byte[] familyName = Bytes.toBytes(cf);
+HColumnDescriptor hcd = new HColumnDescriptor(familyName);
+boolean hasException = false;
+try {
+ htd.modifyFamily(hcd);
+} catch (Exception e) {
+ hasException = true;
+}
+assertTrue(hasException);
+ }
+
+ @Test
+ public void testAddDuplicateFamilies() {
+HTableDescriptor htd = new HTableDescriptor(TableName.valueOf(table));
+byte[] familyName = Bytes.toBytes(cf);
+HColumnDescriptor hcd = new HColumnDescriptor(familyName);
+hcd.setBlocksize(1000);
+htd.addFamily(hcd);
+assertEquals(1000, htd.getFamily(familyName).getBlocksize());
+hcd.setBlocksize(2000);
+boolean hasException = false;
+try {
+ htd.addFamily(hcd);
+} catch (Exception e) {
+ hasException = true;
+}
+assertTrue(hasException);
+ }
{code}
Use the {{@Test(expected=IllegalArgumentException.class)}} form for both of
these instead.
{code}
+if (htd.hasFamily(hcd.getName())) {
+ htd.modifyFamily(hcd);
+} else {
+ htd.addFamily(hcd);
+}
{code}
I'm just thinking out loud here, but do we think this is going to be a common
idiom? Should we add a third method setFamily that behaves like the old add?
Table create with duplicate column family names quietly succeeds
Key: HBASE-12329
URL: https://issues.apache.org/jira/browse/HBASE-12329
Project: HBase
Issue Type: Bug
Components: Client, shell
Reporter: Sean Busbey
Assignee: Jingcheng Du
Priority: Minor
Attachments: HBASE-12329-V2.diff, HBASE-12329.diff
From the mailing list
{quote}
I was expecting that it is forbidden, **but** this call does not throw any
exception
{code}
String[] families = {cf, cf};
HTableDescriptor desc = new HTableDescriptor(name);
for (String cf : families) {
HColumnDescriptor coldef = new HColumnDescriptor(cf);
desc.addFamily(coldef);
}
try {
admin.createTable(desc);
} catch (TableExistsException e) {
throw new IOException(table \' + name + \' already exists);
}
{code}
{quote}
And Ted's follow up replicates in the shell
{code}
hbase(main):001:0 create 't2', {NAME = 'f1'}, {NAME = 'f1'}
The table got created - with 1 column family:
hbase(main):002:0 describe 't2'
DESCRIPTION
ENABLED
't2', {NAME = 'f1', DATA_BLOCK_ENCODING = 'NONE', BLOOMFILTER = 'ROW',
REPLICATION_SCOPE = '0 true
', VERSIONS = '1', COMPRESSION = 'NONE', MIN_VERSIONS = '0', TTL =
'2147483647', KEEP_DELETED
_CELLS = 'false', BLOCKSIZE = '65536', IN_MEMORY = 'false', BLOCKCACHE
= 'true'}
1 row(s) in 0.1000 seconds
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12358) Create ByteBuffer backed Cell
[
https://issues.apache.org/jira/browse/HBASE-12358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14191878#comment-14191878
]
Hadoop QA commented on HBASE-12358:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12678464/HBASE-12358_2.patch
against trunk revision .
ATTACHMENT ID: 12678464
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 11 new
or modified tests.
{color:red}-1 javac{color}. The applied patch generated 115 javac compiler
warnings (more than the trunk's current 102 warnings).
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 3
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
3792 checkstyle errors (more than the trunk's current 3774 errors).
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+ public static final int ROW_OFFSET = Bytes.SIZEOF_INT /* keylength */+
Bytes.SIZEOF_INT /* valuelength */;
+ public static long getKeyValueDataStructureSize(int rlength, int flength,
int qlength, int vlength) {
+ public static ByteBufferBackedCell createFirstOnRow(final ByteBuffer row,
int roffset, short rlength) {
+ public static ByteBufferBackedCell createLastOnRow(final ByteBuffer row,
final int roffset, final int rlength,
+return new ByteBufferBackedKeyValue(row, roffset, rlength, family,
foffset, flength, qualifier, qoffset,
+ public static boolean equals(final ByteBuffer left, int leftOffset, int
leftLen, final ByteBuffer right,
+ public static boolean equals(final ByteBuffer left, int leftOffset, int
leftLen, final byte[] right,
+ public static boolean equals(final byte[] left, int leftOffset, int leftLen,
final ByteBuffer right,
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/11542//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11542//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11542//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11542//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11542//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11542//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11542//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11542//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11542//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11542//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11542//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11542//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/11542//artifact/patchprocess/checkstyle-aggregate.html
Javadoc warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11542//artifact/patchprocess/patchJavadocWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/11542//console
This message is automatically generated.
Create ByteBuffer backed Cell
-
Key: HBASE-12358
URL: https://issues.apache.org/jira/browse/HBASE-12358
Project: HBase
Issue Type: Sub-task
Components: regionserver, Scanners
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Attachments:
[jira] [Commented] (HBASE-12393) The regionserver web will throw exception if we disable block cache
[
https://issues.apache.org/jira/browse/HBASE-12393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14191900#comment-14191900
]
ChiaPing Tsai commented on HBASE-12393:
---
It is only a little bug which we don't handle null point(cache block).
Maybe we can take null point as zero.
For example:
{code}
org.jamon.escaping.Escaping.HTML.write(org.jamon.emit.StandardEmitter.valueOf(StringUtils.humanReadableInt(checkSize(cacheConfig.getBlockCache(,
jamonWriter);
static long checkSize(BlockCache blockCache)
{
return blockCache == null ? 0 : blockCache.size();
}
{code}
The regionserver web will throw exception if we disable block cache
---
Key: HBASE-12393
URL: https://issues.apache.org/jira/browse/HBASE-12393
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.98.7
Environment: ubuntu 12.04 64bits, hadoop-2.2.0, hbase-0.98.7-hadoop2
Reporter: ChiaPing Tsai
Priority: Minor
The CacheConfig.getBlockCache() will return the null point when we set
hfile.block.cache.size to zero.
It caused the BlockCacheTmplImpl.java:123 to throw null exception.
{code}
org.jamon.escaping.Escaping.HTML.write(org.jamon.emit.StandardEmitter.valueOf(StringUtils.humanReadableInt(cacheConfig.getBlockCache().size())),
jamonWriter);
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HBASE-10870) Deprecate and replace HCD methods that have a 'should' prefix with a 'get' instead
[ https://issues.apache.org/jira/browse/HBASE-10870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi reassigned HBASE-10870: - Assignee: Ashish Singhi Deprecate and replace HCD methods that have a 'should' prefix with a 'get' instead -- Key: HBASE-10870 URL: https://issues.apache.org/jira/browse/HBASE-10870 Project: HBase Issue Type: Task Reporter: stack Assignee: Ashish Singhi HColumnDescriptor has a bunch of methods that have 'should' for a prefix. Deprecate and give them a javabean 'get' or 'is' instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-10870) Deprecate and replace HCD methods that have a 'should' prefix with a 'get' instead
[ https://issues.apache.org/jira/browse/HBASE-10870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-10870: -- Attachment: HBASE-10870.patch Patch for master branch. Replaced and deprecated all apis in HCD having prefix 'should' with 'is'. Some one please review. Deprecate and replace HCD methods that have a 'should' prefix with a 'get' instead -- Key: HBASE-10870 URL: https://issues.apache.org/jira/browse/HBASE-10870 Project: HBase Issue Type: Task Reporter: stack Assignee: Ashish Singhi Attachments: HBASE-10870.patch HColumnDescriptor has a bunch of methods that have 'should' for a prefix. Deprecate and give them a javabean 'get' or 'is' instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-10870) Deprecate and replace HCD methods that have a 'should' prefix with a 'get' instead
[ https://issues.apache.org/jira/browse/HBASE-10870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-10870: -- Affects Version/s: 2.0.0 Status: Patch Available (was: Open) Deprecate and replace HCD methods that have a 'should' prefix with a 'get' instead -- Key: HBASE-10870 URL: https://issues.apache.org/jira/browse/HBASE-10870 Project: HBase Issue Type: Task Affects Versions: 2.0.0 Reporter: stack Assignee: Ashish Singhi Attachments: HBASE-10870.patch HColumnDescriptor has a bunch of methods that have 'should' for a prefix. Deprecate and give them a javabean 'get' or 'is' instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-10870) Deprecate and replace HCD methods that have a 'should' prefix with a 'get' instead
[ https://issues.apache.org/jira/browse/HBASE-10870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14191941#comment-14191941 ] stack commented on HBASE-10870: --- Nice. +1. Deprecate and replace HCD methods that have a 'should' prefix with a 'get' instead -- Key: HBASE-10870 URL: https://issues.apache.org/jira/browse/HBASE-10870 Project: HBase Issue Type: Task Affects Versions: 2.0.0 Reporter: stack Assignee: Ashish Singhi Attachments: HBASE-10870.patch HColumnDescriptor has a bunch of methods that have 'should' for a prefix. Deprecate and give them a javabean 'get' or 'is' instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-10870) Deprecate and replace HCD methods that have a 'should' prefix with a 'get' instead
[ https://issues.apache.org/jira/browse/HBASE-10870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-10870: -- Resolution: Fixed Fix Version/s: 0.99.2 2.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to branch-1+. Thanks for patch [~ashish singhi] Deprecate and replace HCD methods that have a 'should' prefix with a 'get' instead -- Key: HBASE-10870 URL: https://issues.apache.org/jira/browse/HBASE-10870 Project: HBase Issue Type: Task Affects Versions: 2.0.0 Reporter: stack Assignee: Ashish Singhi Fix For: 2.0.0, 0.99.2 Attachments: HBASE-10870.patch HColumnDescriptor has a bunch of methods that have 'should' for a prefix. Deprecate and give them a javabean 'get' or 'is' instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12393) The regionserver web will throw exception if we disable block cache
[
https://issues.apache.org/jira/browse/HBASE-12393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14191953#comment-14191953
]
stack commented on HBASE-12393:
---
Why set it to zero? When would it make sense? i.e. loading index blocks every
time rather than keeping them around in mem. But yeah, shouldn't NPE. If you
have a patch that fixes the NPE and it works for you, attach and we'll commit
(but will add WARNINGs that 0 block cache is bad idea as per [~jmspaggi])
The regionserver web will throw exception if we disable block cache
---
Key: HBASE-12393
URL: https://issues.apache.org/jira/browse/HBASE-12393
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.98.7
Environment: ubuntu 12.04 64bits, hadoop-2.2.0, hbase-0.98.7-hadoop2
Reporter: ChiaPing Tsai
Priority: Minor
The CacheConfig.getBlockCache() will return the null point when we set
hfile.block.cache.size to zero.
It caused the BlockCacheTmplImpl.java:123 to throw null exception.
{code}
org.jamon.escaping.Escaping.HTML.write(org.jamon.emit.StandardEmitter.valueOf(StringUtils.humanReadableInt(cacheConfig.getBlockCache().size())),
jamonWriter);
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12387) committer guidelines should include patch signoff
[
https://issues.apache.org/jira/browse/HBASE-12387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14191956#comment-14191956
]
stack commented on HBASE-12387:
---
bq. So we should always go with making author to contributor and signed off?
I like [~busbey]'s prescription [~anoopsamjohn]. He has the tooling doing the
work for us. Encouraging folks to produce patches that we can just do 'git am
--signoff' makes for less work for committers; any savings are appreciated when
we are running multiple branches as we are doing currently. What you think
[~anoopsamjohn]?
committer guidelines should include patch signoff
-
Key: HBASE-12387
URL: https://issues.apache.org/jira/browse/HBASE-12387
Project: HBase
Issue Type: Task
Components: documentation
Reporter: Sean Busbey
Assignee: Sean Busbey
Right now our guide for committers apply patches has them use {{git am}}
without a signoff flag. This works okay, but it misses adding the
signed-off-by blurb in the commit message.
Those messages make it easier to see at a glance with e.g. {{git log}} which
committer applied the patch.
this section:
{quote}
The directive to use git format-patch rather than git diff, and not to use
--no-prefix, is a new one. See the second example for how to apply a patch
created with git diff, and educate the person who created the patch.
{code}
$ git checkout -b HBASE-
$ git am ~/Downloads/HBASE--v2.patch
$ git checkout master
$ git pull --rebase
$ git cherry-pick sha-from-commit
# Resolve conflicts if necessary or ask the submitter to do it
$ git pull --rebase # Better safe than sorry
$ git push origin master
$ git checkout branch-1
$ git pull --rebase
$ git cherry-pick sha-from-commit
# Resolve conflicts if necessary
$ git pull --rebase # Better safe than sorry
$ git push origin branch-1
$ git branch -D HBASE-
{code}
{quote}
Should be
{quote}
The directive to use git format-patch rather than git diff, and not to use
--no-prefix, is a new one. See the second example for how to apply a patch
created with git diff, and educate the person who created the patch.
Note that the {{--signoff}} flag to {{git am}} will insert a line in the
commit message that the patch was checked by your author string. This
addition to your inclusion as the commit's committer makes your participation
more prominent to users browsing {{git log}}.
{code}
$ git checkout -b HBASE-
$ git am --signoff ~/Downloads/HBASE--v2.patch
$ git checkout master
$ git pull --rebase
$ git cherry-pick sha-from-commit
# Resolve conflicts if necessary or ask the submitter to do it
$ git pull --rebase # Better safe than sorry
$ git push origin master
$ git checkout branch-1
$ git pull --rebase
$ git cherry-pick sha-from-commit
# Resolve conflicts if necessary
$ git pull --rebase # Better safe than sorry
$ git push origin branch-1
$ git branch -D HBASE-
{code}
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12072) We are doing 35 x 35 retries for master operations
[
https://issues.apache.org/jira/browse/HBASE-12072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-12072:
--
Attachment: hbase-12072_v2.patch
Retry
We are doing 35 x 35 retries for master operations
--
Key: HBASE-12072
URL: https://issues.apache.org/jira/browse/HBASE-12072
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.6
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 2.0.0, 0.99.2
Attachments: 12072-v1.txt, 12072-v2.txt, hbase-12072_v1.patch,
hbase-12072_v2.patch, hbase-12072_v2.patch
For master requests, there are two retry mechanisms in effect. The first one
is from HBaseAdmin.executeCallable()
{code}
private V V executeCallable(MasterCallableV callable) throws
IOException {
RpcRetryingCallerV caller = rpcCallerFactory.newCaller();
try {
return caller.callWithRetries(callable);
} finally {
callable.close();
}
}
{code}
And inside, the other one is from StubMaker.makeStub():
{code}
/**
* Create a stub against the master. Retry if necessary.
* @return A stub to do codeintf/code against the master
* @throws MasterNotRunningException
*/
@edu.umd.cs.findbugs.annotations.SuppressWarnings
(value=SWL_SLEEP_WITH_LOCK_HELD)
Object makeStub() throws MasterNotRunningException {
{code}
The tests will just hang for 10 min * 35 ~= 6hours.
{code}
2014-09-23 16:19:05,151 INFO [main]
client.ConnectionManager$HConnectionImplementation: getMaster attempt 1 of 35
failed; retrying after sleep of 100, exception=java.io.IOException: Can't get
master address from ZooKeeper; znode data == null
2014-09-23 16:19:05,253 INFO [main]
client.ConnectionManager$HConnectionImplementation: getMaster attempt 2 of 35
failed; retrying after sleep of 200, exception=java.io.IOException: Can't get
master address from ZooKeeper; znode data == null
2014-09-23 16:19:05,456 INFO [main]
client.ConnectionManager$HConnectionImplementation: getMaster attempt 3 of 35
failed; retrying after sleep of 300, exception=java.io.IOException: Can't get
master address from ZooKeeper; znode data == null
2014-09-23 16:19:05,759 INFO [main]
client.ConnectionManager$HConnectionImplementation: getMaster attempt 4 of 35
failed; retrying after sleep of 500, exception=java.io.IOException: Can't get
master address from ZooKeeper; znode data == null
2014-09-23 16:19:06,262 INFO [main]
client.ConnectionManager$HConnectionImplementation: getMaster attempt 5 of 35
failed; retrying after sleep of 1008, exception=java.io.IOException: Can't
get master address from ZooKeeper; znode data == null
2014-09-23 16:19:07,273 INFO [main]
client.ConnectionManager$HConnectionImplementation: getMaster attempt 6 of 35
failed; retrying after sleep of 2011, exception=java.io.IOException: Can't
get master address from ZooKeeper; znode data == null
2014-09-23 16:19:09,286 INFO [main]
client.ConnectionManager$HConnectionImplementation: getMaster attempt 7 of 35
failed; retrying after sleep of 4012, exception=java.io.IOException: Can't
get master address from ZooKeeper; znode data == null
2014-09-23 16:19:13,303 INFO [main]
client.ConnectionManager$HConnectionImplementation: getMaster attempt 8 of 35
failed; retrying after sleep of 10033, exception=java.io.IOException: Can't
get master address from ZooKeeper; znode data == null
2014-09-23 16:19:23,343 INFO [main]
client.ConnectionManager$HConnectionImplementation: getMaster attempt 9 of 35
failed; retrying after sleep of 10089, exception=java.io.IOException: Can't
get master address from ZooKeeper; znode data == null
2014-09-23 16:19:33,439 INFO [main]
client.ConnectionManager$HConnectionImplementation: getMaster attempt 10 of
35 failed; retrying after sleep of 10027, exception=java.io.IOException:
Can't get master address from ZooKeeper; znode data == null
2014-09-23 16:19:43,473 INFO [main]
client.ConnectionManager$HConnectionImplementation: getMaster attempt 11 of
35 failed; retrying after sleep of 10004, exception=java.io.IOException:
Can't get master address from ZooKeeper; znode data == null
2014-09-23 16:19:53,485 INFO [main]
client.ConnectionManager$HConnectionImplementation: getMaster attempt 12 of
35 failed; retrying after sleep of 20160, exception=java.io.IOException:
Can't get master address from ZooKeeper; znode data == null
2014-09-23 16:20:13,656 INFO [main]
client.ConnectionManager$HConnectionImplementation: getMaster attempt 13 of
35 failed; retrying after sleep of 20006, exception=java.io.IOException:
Can't get master address from ZooKeeper; znode data == null
2014-09-23 16:20:33,675 INFO [main]
client.ConnectionManager$HConnectionImplementation: getMaster attempt 14
[jira] [Created] (HBASE-12395) Some internal classes are logging to DEBUG what should be logged to TRACE
Dima Spivak created HBASE-12395: --- Summary: Some internal classes are logging to DEBUG what should be logged to TRACE Key: HBASE-12395 URL: https://issues.apache.org/jira/browse/HBASE-12395 Project: HBase Issue Type: Improvement Reporter: Dima Spivak Assignee: Dima Spivak e.g. RpcExecutor is doing this a lot. This leads to 1) huge log files that waste disk space and IO and 2) difficulty debugging tests themselves since you need to wade through thousands of lines to get to what your test is doing (see TestDistributedLogSplitting-output for 25 MB of thread information). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12393) The regionserver web will throw exception if we disable block cache
[
https://issues.apache.org/jira/browse/HBASE-12393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14191969#comment-14191969
]
ChiaPing Tsai commented on HBASE-12393:
---
hi stack
Thanks for your helpful suggestion. We test the lower-bound performance by
disabling the block cache and others, and then we found this bug.
We just thought the 0 block cache shouldn't affect the regionserver web
The regionserver web will throw exception if we disable block cache
---
Key: HBASE-12393
URL: https://issues.apache.org/jira/browse/HBASE-12393
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.98.7
Environment: ubuntu 12.04 64bits, hadoop-2.2.0, hbase-0.98.7-hadoop2
Reporter: ChiaPing Tsai
Priority: Minor
The CacheConfig.getBlockCache() will return the null point when we set
hfile.block.cache.size to zero.
It caused the BlockCacheTmplImpl.java:123 to throw null exception.
{code}
org.jamon.escaping.Escaping.HTML.write(org.jamon.emit.StandardEmitter.valueOf(StringUtils.humanReadableInt(cacheConfig.getBlockCache().size())),
jamonWriter);
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12393) The regionserver web will throw exception if we disable block cache
[
https://issues.apache.org/jira/browse/HBASE-12393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14191995#comment-14191995
]
stack commented on HBASE-12393:
---
[~chia7712] You are right. UI should keep going regardless. Attach a patch w/
your fix above and we'll get it in.
The regionserver web will throw exception if we disable block cache
---
Key: HBASE-12393
URL: https://issues.apache.org/jira/browse/HBASE-12393
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.98.7
Environment: ubuntu 12.04 64bits, hadoop-2.2.0, hbase-0.98.7-hadoop2
Reporter: ChiaPing Tsai
Priority: Minor
The CacheConfig.getBlockCache() will return the null point when we set
hfile.block.cache.size to zero.
It caused the BlockCacheTmplImpl.java:123 to throw null exception.
{code}
org.jamon.escaping.Escaping.HTML.write(org.jamon.emit.StandardEmitter.valueOf(StringUtils.humanReadableInt(cacheConfig.getBlockCache().size())),
jamonWriter);
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-12396) Document suggested use of log levels in dev guide
Sean Busbey created HBASE-12396: --- Summary: Document suggested use of log levels in dev guide Key: HBASE-12396 URL: https://issues.apache.org/jira/browse/HBASE-12396 Project: HBase Issue Type: Improvement Components: documentation Reporter: Sean Busbey Priority: Minor Right now we don't provide any guidance on appropriate use of log levels, which leads to inconsistent use and in some cases exacerbating problems for tests and troubleshooting (see HBASE-12395). We should add a section on suggested use of log levels to the guide. Some related reading * [a good ops-focused blog post on levels|http://watchitlater.com/blog/2009/12/logging-guidelines/] * [another, focused on devs|http://www.codeproject.com/Articles/42354/The-Art-of-Logging] * [guidelines from a user on stackoverflow (that I like)|http://stackoverflow.com/a/2031209] also has some good discussion. * [extended logging discussion with some level use guidelines|http://www.javacodegeeks.com/2011/01/10-tips-proper-application-logging.html] * [guidelines for Atlassian devs|https://developer.atlassian.com/display/CONFDEV/Logging+Guidelines] * [guidelines for OpenStack devs|https://wiki.openstack.org/wiki/LoggingStandards] * [the Kafka dev guide|http://kafka.apache.org/coding-guide.html] has a good section on their use titled Logging -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12396) Document suggested use of log levels in dev guide
[ https://issues.apache.org/jira/browse/HBASE-12396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14192008#comment-14192008 ] Sean Busbey commented on HBASE-12396: - Personally, I like the focus on INFO and more severe is for operator use. * ERROR if at all possible should include a plain description of the problem and an action the operator can take to either correct or troubleshoot * Stack traces should be rare above DEBUG. We allow changing the level of a particular logger at run time, if an operator needs that level of detail they can alter the level (we should improve the docs on doing this). * I think a good heuristic for DEBUG v TRACE is if you need multiple messages that show the flow of control within part of the codebase, you ought to be logging at TRACE. Document suggested use of log levels in dev guide - Key: HBASE-12396 URL: https://issues.apache.org/jira/browse/HBASE-12396 Project: HBase Issue Type: Improvement Components: documentation Reporter: Sean Busbey Priority: Minor Right now we don't provide any guidance on appropriate use of log levels, which leads to inconsistent use and in some cases exacerbating problems for tests and troubleshooting (see HBASE-12395). We should add a section on suggested use of log levels to the guide. Some related reading * [a good ops-focused blog post on levels|http://watchitlater.com/blog/2009/12/logging-guidelines/] * [another, focused on devs|http://www.codeproject.com/Articles/42354/The-Art-of-Logging] * [guidelines from a user on stackoverflow (that I like)|http://stackoverflow.com/a/2031209] also has some good discussion. * [extended logging discussion with some level use guidelines|http://www.javacodegeeks.com/2011/01/10-tips-proper-application-logging.html] * [guidelines for Atlassian devs|https://developer.atlassian.com/display/CONFDEV/Logging+Guidelines] * [guidelines for OpenStack devs|https://wiki.openstack.org/wiki/LoggingStandards] * [the Kafka dev guide|http://kafka.apache.org/coding-guide.html] has a good section on their use titled Logging -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-10870) Deprecate and replace HCD methods that have a 'should' prefix with a 'get' instead
[ https://issues.apache.org/jira/browse/HBASE-10870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14192066#comment-14192066 ] Hudson commented on HBASE-10870: SUCCESS: Integrated in HBase-1.0 #399 (See [https://builds.apache.org/job/HBase-1.0/399/]) HBASE-10870 Deprecate and replace HCD methods that have a 'should' prefix with a 'is' instead (stack: rev ae8462b3a2de710ba2b0c49fbddf46b2316a61e3) * hbase-client/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java Deprecate and replace HCD methods that have a 'should' prefix with a 'get' instead -- Key: HBASE-10870 URL: https://issues.apache.org/jira/browse/HBASE-10870 Project: HBase Issue Type: Task Affects Versions: 2.0.0 Reporter: stack Assignee: Ashish Singhi Fix For: 2.0.0, 0.99.2 Attachments: HBASE-10870.patch HColumnDescriptor has a bunch of methods that have 'should' for a prefix. Deprecate and give them a javabean 'get' or 'is' instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12219) Cache more efficiently getAll() and get() in FSTableDescriptors
[ https://issues.apache.org/jira/browse/HBASE-12219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esteban Gutierrez updated HBASE-12219: -- Attachment: HBASE-12219-0.99.patch Cache more efficiently getAll() and get() in FSTableDescriptors --- Key: HBASE-12219 URL: https://issues.apache.org/jira/browse/HBASE-12219 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.94.24, 0.99.1, 0.98.6.1 Reporter: Esteban Gutierrez Assignee: Esteban Gutierrez Labels: scalability Attachments: HBASE-12219-0.99.patch, HBASE-12219-v1.patch, HBASE-12219-v1.patch, HBASE-12219.v0.txt, HBASE-12219.v2.patch, HBASE-12219.v3.patch, list.png Currently table descriptors and tables are cached once they are accessed for the first time. Next calls to the master only require a trip to HDFS to lookup the modified time in order to reload the table descriptors if modified. However in clusters with a large number of tables or concurrent clients and this can be too aggressive to HDFS and the master causing contention to process other requests. A simple solution is to have a TTL based cached for FSTableDescriptors#getAll() and FSTableDescriptors#TableDescriptorAndModtime() that can allow the master to process those calls faster without causing contention without having to perform a trip to HDFS for every call. to listtables() or getTableDescriptor() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-10870) Deprecate and replace HCD methods that have a 'should' prefix with a 'get' instead
[
https://issues.apache.org/jira/browse/HBASE-10870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14192081#comment-14192081
]
Hadoop QA commented on HBASE-10870:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12678484/HBASE-10870.patch
against trunk revision .
ATTACHMENT ID: 12678484
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.mapreduce.TestSecureLoadIncrementalHFilesSplitRecovery
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hbase.regionserver.TestHRegion.testWritesWhileScanning(TestHRegion.java:3615)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/11543//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11543//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11543//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11543//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11543//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11543//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11543//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11543//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11543//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11543//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11543//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11543//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/11543//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/11543//console
This message is automatically generated.
Deprecate and replace HCD methods that have a 'should' prefix with a 'get'
instead
--
Key: HBASE-10870
URL: https://issues.apache.org/jira/browse/HBASE-10870
Project: HBase
Issue Type: Task
Affects Versions: 2.0.0
Reporter: stack
Assignee: Ashish Singhi
Fix For: 2.0.0, 0.99.2
Attachments: HBASE-10870.patch
HColumnDescriptor has a bunch of methods that have 'should' for a prefix.
Deprecate and give them a javabean 'get' or 'is' instead.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12329) Table create with duplicate column family names quietly succeeds
[
https://issues.apache.org/jira/browse/HBASE-12329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14192082#comment-14192082
]
Jingcheng Du commented on HBASE-12329:
--
Thanks [~busbey] for the review.
bq. Please file as a follow on ticket. If Misty wants to take it up, then she
can. Otherwise someone else can do it.
Sure, will create a follow on jira.
bq. Should we add a third method setFamily that behaves like the old add?
If it's added, does it imply such an operation (add or update in one method) is
allowed, and users should be aware of last one wins at that time? If so why
do we re-implement addFamily and add modifyFamily? The existing addFamily is
enough. Is it?
Table create with duplicate column family names quietly succeeds
Key: HBASE-12329
URL: https://issues.apache.org/jira/browse/HBASE-12329
Project: HBase
Issue Type: Bug
Components: Client, shell
Reporter: Sean Busbey
Assignee: Jingcheng Du
Priority: Minor
Attachments: HBASE-12329-V2.diff, HBASE-12329.diff
From the mailing list
{quote}
I was expecting that it is forbidden, **but** this call does not throw any
exception
{code}
String[] families = {cf, cf};
HTableDescriptor desc = new HTableDescriptor(name);
for (String cf : families) {
HColumnDescriptor coldef = new HColumnDescriptor(cf);
desc.addFamily(coldef);
}
try {
admin.createTable(desc);
} catch (TableExistsException e) {
throw new IOException(table \' + name + \' already exists);
}
{code}
{quote}
And Ted's follow up replicates in the shell
{code}
hbase(main):001:0 create 't2', {NAME = 'f1'}, {NAME = 'f1'}
The table got created - with 1 column family:
hbase(main):002:0 describe 't2'
DESCRIPTION
ENABLED
't2', {NAME = 'f1', DATA_BLOCK_ENCODING = 'NONE', BLOOMFILTER = 'ROW',
REPLICATION_SCOPE = '0 true
', VERSIONS = '1', COMPRESSION = 'NONE', MIN_VERSIONS = '0', TTL =
'2147483647', KEEP_DELETED
_CELLS = 'false', BLOCKSIZE = '65536', IN_MEMORY = 'false', BLOCKCACHE
= 'true'}
1 row(s) in 0.1000 seconds
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12219) Cache more efficiently getAll() and get() in FSTableDescriptors
[
https://issues.apache.org/jira/browse/HBASE-12219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14192091#comment-14192091
]
Hadoop QA commented on HBASE-12219:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12678498/HBASE-12219-0.99.patch
against trunk revision .
ATTACHMENT ID: 12678498
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 8 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/11545//console
This message is automatically generated.
Cache more efficiently getAll() and get() in FSTableDescriptors
---
Key: HBASE-12219
URL: https://issues.apache.org/jira/browse/HBASE-12219
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.94.24, 0.99.1, 0.98.6.1
Reporter: Esteban Gutierrez
Assignee: Esteban Gutierrez
Labels: scalability
Attachments: HBASE-12219-0.99.patch, HBASE-12219-v1.patch,
HBASE-12219-v1.patch, HBASE-12219.v0.txt, HBASE-12219.v2.patch,
HBASE-12219.v3.patch, list.png
Currently table descriptors and tables are cached once they are accessed for
the first time. Next calls to the master only require a trip to HDFS to
lookup the modified time in order to reload the table descriptors if
modified. However in clusters with a large number of tables or concurrent
clients and this can be too aggressive to HDFS and the master causing
contention to process other requests. A simple solution is to have a TTL
based cached for FSTableDescriptors#getAll() and
FSTableDescriptors#TableDescriptorAndModtime() that can allow the master to
process those calls faster without causing contention without having to
perform a trip to HDFS for every call. to listtables() or getTableDescriptor()
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12329) Table create with duplicate column family names quietly succeeds
[
https://issues.apache.org/jira/browse/HBASE-12329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingcheng Du updated HBASE-12329:
-
Attachment: HBASE-12329-V3.diff
Table create with duplicate column family names quietly succeeds
Key: HBASE-12329
URL: https://issues.apache.org/jira/browse/HBASE-12329
Project: HBase
Issue Type: Bug
Components: Client, shell
Reporter: Sean Busbey
Assignee: Jingcheng Du
Priority: Minor
Attachments: HBASE-12329-V2.diff, HBASE-12329-V3.diff,
HBASE-12329.diff
From the mailing list
{quote}
I was expecting that it is forbidden, **but** this call does not throw any
exception
{code}
String[] families = {cf, cf};
HTableDescriptor desc = new HTableDescriptor(name);
for (String cf : families) {
HColumnDescriptor coldef = new HColumnDescriptor(cf);
desc.addFamily(coldef);
}
try {
admin.createTable(desc);
} catch (TableExistsException e) {
throw new IOException(table \' + name + \' already exists);
}
{code}
{quote}
And Ted's follow up replicates in the shell
{code}
hbase(main):001:0 create 't2', {NAME = 'f1'}, {NAME = 'f1'}
The table got created - with 1 column family:
hbase(main):002:0 describe 't2'
DESCRIPTION
ENABLED
't2', {NAME = 'f1', DATA_BLOCK_ENCODING = 'NONE', BLOOMFILTER = 'ROW',
REPLICATION_SCOPE = '0 true
', VERSIONS = '1', COMPRESSION = 'NONE', MIN_VERSIONS = '0', TTL =
'2147483647', KEEP_DELETED
_CELLS = 'false', BLOCKSIZE = '65536', IN_MEMORY = 'false', BLOCKCACHE
= 'true'}
1 row(s) in 0.1000 seconds
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12329) Table create with duplicate column family names quietly succeeds
[
https://issues.apache.org/jira/browse/HBASE-12329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14192097#comment-14192097
]
Sean Busbey commented on HBASE-12329:
-
{quote}
bq. Should we add a third method setFamily that behaves like the old add?
If it's added, does it imply such an operation (add or update in one method) is
allowed, and users should be aware of last one wins at that time? If so why
do we re-implement addFamily and add modifyFamily? The existing addFamily is
enough. Is it?
{quote}
It's API semantics. addXXX means to put something new in, modifyXXX means
to update something already present. setXXX is used in other places to mean
add or update, knowledge of last write wins is already baked into the
naming.
Table create with duplicate column family names quietly succeeds
Key: HBASE-12329
URL: https://issues.apache.org/jira/browse/HBASE-12329
Project: HBase
Issue Type: Bug
Components: Client, shell
Reporter: Sean Busbey
Assignee: Jingcheng Du
Priority: Minor
Attachments: HBASE-12329-V2.diff, HBASE-12329-V3.diff,
HBASE-12329.diff
From the mailing list
{quote}
I was expecting that it is forbidden, **but** this call does not throw any
exception
{code}
String[] families = {cf, cf};
HTableDescriptor desc = new HTableDescriptor(name);
for (String cf : families) {
HColumnDescriptor coldef = new HColumnDescriptor(cf);
desc.addFamily(coldef);
}
try {
admin.createTable(desc);
} catch (TableExistsException e) {
throw new IOException(table \' + name + \' already exists);
}
{code}
{quote}
And Ted's follow up replicates in the shell
{code}
hbase(main):001:0 create 't2', {NAME = 'f1'}, {NAME = 'f1'}
The table got created - with 1 column family:
hbase(main):002:0 describe 't2'
DESCRIPTION
ENABLED
't2', {NAME = 'f1', DATA_BLOCK_ENCODING = 'NONE', BLOOMFILTER = 'ROW',
REPLICATION_SCOPE = '0 true
', VERSIONS = '1', COMPRESSION = 'NONE', MIN_VERSIONS = '0', TTL =
'2147483647', KEEP_DELETED
_CELLS = 'false', BLOCKSIZE = '65536', IN_MEMORY = 'false', BLOCKCACHE
= 'true'}
1 row(s) in 0.1000 seconds
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12329) Table create with duplicate column family names quietly succeeds
[
https://issues.apache.org/jira/browse/HBASE-12329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14192100#comment-14192100
]
Sean Busbey commented on HBASE-12329:
-
Thinking through things more, let's skip the third method. We can always add it
later if the idiom shows up often downstream. Once it's there removing it is
hard.
Table create with duplicate column family names quietly succeeds
Key: HBASE-12329
URL: https://issues.apache.org/jira/browse/HBASE-12329
Project: HBase
Issue Type: Bug
Components: Client, shell
Reporter: Sean Busbey
Assignee: Jingcheng Du
Priority: Minor
Attachments: HBASE-12329-V2.diff, HBASE-12329-V3.diff,
HBASE-12329.diff
From the mailing list
{quote}
I was expecting that it is forbidden, **but** this call does not throw any
exception
{code}
String[] families = {cf, cf};
HTableDescriptor desc = new HTableDescriptor(name);
for (String cf : families) {
HColumnDescriptor coldef = new HColumnDescriptor(cf);
desc.addFamily(coldef);
}
try {
admin.createTable(desc);
} catch (TableExistsException e) {
throw new IOException(table \' + name + \' already exists);
}
{code}
{quote}
And Ted's follow up replicates in the shell
{code}
hbase(main):001:0 create 't2', {NAME = 'f1'}, {NAME = 'f1'}
The table got created - with 1 column family:
hbase(main):002:0 describe 't2'
DESCRIPTION
ENABLED
't2', {NAME = 'f1', DATA_BLOCK_ENCODING = 'NONE', BLOOMFILTER = 'ROW',
REPLICATION_SCOPE = '0 true
', VERSIONS = '1', COMPRESSION = 'NONE', MIN_VERSIONS = '0', TTL =
'2147483647', KEEP_DELETED
_CELLS = 'false', BLOCKSIZE = '65536', IN_MEMORY = 'false', BLOCKCACHE
= 'true'}
1 row(s) in 0.1000 seconds
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-10870) Deprecate and replace HCD methods that have a 'should' prefix with a 'get' instead
[ https://issues.apache.org/jira/browse/HBASE-10870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14192114#comment-14192114 ] Ashish Singhi commented on HBASE-10870: --- Thanks for your time Stack. Test failures should not be related to patch. Deprecate and replace HCD methods that have a 'should' prefix with a 'get' instead -- Key: HBASE-10870 URL: https://issues.apache.org/jira/browse/HBASE-10870 Project: HBase Issue Type: Task Affects Versions: 2.0.0 Reporter: stack Assignee: Ashish Singhi Fix For: 2.0.0, 0.99.2 Attachments: HBASE-10870.patch HColumnDescriptor has a bunch of methods that have 'should' for a prefix. Deprecate and give them a javabean 'get' or 'is' instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-10870) Deprecate and replace HCD methods that have a 'should' prefix with a 'get' instead
[ https://issues.apache.org/jira/browse/HBASE-10870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14192115#comment-14192115 ] Hudson commented on HBASE-10870: SUCCESS: Integrated in HBase-TRUNK #5730 (See [https://builds.apache.org/job/HBase-TRUNK/5730/]) HBASE-10870 Deprecate and replace HCD methods that have a 'should' prefix with a 'is' instead (stack: rev cacdb89e0345d4d507ae0ae04628d871d636fbca) * hbase-client/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java Deprecate and replace HCD methods that have a 'should' prefix with a 'get' instead -- Key: HBASE-10870 URL: https://issues.apache.org/jira/browse/HBASE-10870 Project: HBase Issue Type: Task Affects Versions: 2.0.0 Reporter: stack Assignee: Ashish Singhi Fix For: 2.0.0, 0.99.2 Attachments: HBASE-10870.patch HColumnDescriptor has a bunch of methods that have 'should' for a prefix. Deprecate and give them a javabean 'get' or 'is' instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12397) CopyTable fails to compile with the Hadoop 1 profile
Andrew Purtell created HBASE-12397:
--
Summary: CopyTable fails to compile with the Hadoop 1 profile
Key: HBASE-12397
URL: https://issues.apache.org/jira/browse/HBASE-12397
Project: HBase
Issue Type: Bug
Reporter: Andrew Purtell
Fix For: 0.98.8
[ERROR]
/usr/src/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/CopyTable.java:[88,17]
error: cannot find symbol
This was introduced in f31edd80
{noformat}
commit f31edd8004226c795ee46fbe9e93d10671ab895a
Author: Ted Yu te...@apache.org
Date: Thu Oct 9 15:52:18 2014 +
HBASE-11997 CopyTable with bulkload (Yi Deng)
{noformat}
[~tedyu], [~daviddengcn], please have a look at this or I will revert it on
0.98.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12072) We are doing 35 x 35 retries for master operations
[
https://issues.apache.org/jira/browse/HBASE-12072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14192129#comment-14192129
]
Hadoop QA commented on HBASE-12072:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12678488/hbase-12072_v2.patch
against trunk revision .
ATTACHMENT ID: 12678488
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 18 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
3775 checkstyle errors (more than the trunk's current 3774 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/11544//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11544//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11544//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11544//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11544//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11544//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11544//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11544//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11544//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11544//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11544//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11544//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/11544//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/11544//console
This message is automatically generated.
We are doing 35 x 35 retries for master operations
--
Key: HBASE-12072
URL: https://issues.apache.org/jira/browse/HBASE-12072
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.6
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 2.0.0, 0.99.2
Attachments: 12072-v1.txt, 12072-v2.txt, hbase-12072_v1.patch,
hbase-12072_v2.patch, hbase-12072_v2.patch
For master requests, there are two retry mechanisms in effect. The first one
is from HBaseAdmin.executeCallable()
{code}
private V V executeCallable(MasterCallableV callable) throws
IOException {
RpcRetryingCallerV caller = rpcCallerFactory.newCaller();
try {
return caller.callWithRetries(callable);
} finally {
callable.close();
}
}
{code}
And inside, the other one is from StubMaker.makeStub():
{code}
/**
* Create a stub against the master. Retry if necessary.
* @return A stub to do codeintf/code against the master
* @throws MasterNotRunningException
*/
@edu.umd.cs.findbugs.annotations.SuppressWarnings
(value=SWL_SLEEP_WITH_LOCK_HELD)
Object makeStub() throws MasterNotRunningException {
{code}
The tests will just hang for 10 min * 35 ~= 6hours.
{code}
2014-09-23 16:19:05,151 INFO
[jira] [Commented] (HBASE-12396) Document suggested use of log levels in dev guide
[ https://issues.apache.org/jira/browse/HBASE-12396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14192133#comment-14192133 ] stack commented on HBASE-12396: --- Nice one Sean. Kept meaning to do this. Here's another few items to throw on the list. + Logs should be info-dense (don't just write 'starting', but rather 'starting ...' and a listing of the config the service was started with). + Don't have two log lines where one would do. + Logs should be absent repetition and no need of verbiage that is obvious from context. ++ For example, don't write 'opening region xyz' when the context is the RegionOpenHandler + Logs should use same vocabulary (to be published) everywhere and the same format when referring to entities throughout; it makes the logs greppable. ++ We should not log an edits sequenceid with the label sequenceid in one location, seqid in another, and id somewhere else again. ++ We should not log sequenceid=XYZ in one log message and sequenceid: XYZ in another. ++ For example, tracing the history of a region, always refer to it the same when when making mention in the logs: if we use its encoded name everywhere, then a grep on this will turn up all mentions Logs should be digestible, actionable. They should be so regularized, 'standardized' (as it is termed in one of the articles noted above by Sean) so that monitoring of logs can be done by tools and building a tool like http://www.ymc.ch/en/hbase-split-visualisation-introducing-hannibal is easy-to-do and continues to work across versions (you shouldn't need to install hannibal to get a historic, cluster-wide view on compactions/flushes -- but that is another issue). I like the idea of standard context to dump on DEBUG (one of the articles talks of log4j MDC). We had a hack of this where we'd dump context in JSON when stuff was slow. Document suggested use of log levels in dev guide - Key: HBASE-12396 URL: https://issues.apache.org/jira/browse/HBASE-12396 Project: HBase Issue Type: Improvement Components: documentation Reporter: Sean Busbey Priority: Minor Right now we don't provide any guidance on appropriate use of log levels, which leads to inconsistent use and in some cases exacerbating problems for tests and troubleshooting (see HBASE-12395). We should add a section on suggested use of log levels to the guide. Some related reading * [a good ops-focused blog post on levels|http://watchitlater.com/blog/2009/12/logging-guidelines/] * [another, focused on devs|http://www.codeproject.com/Articles/42354/The-Art-of-Logging] * [guidelines from a user on stackoverflow (that I like)|http://stackoverflow.com/a/2031209] also has some good discussion. * [extended logging discussion with some level use guidelines|http://www.javacodegeeks.com/2011/01/10-tips-proper-application-logging.html] * [guidelines for Atlassian devs|https://developer.atlassian.com/display/CONFDEV/Logging+Guidelines] * [guidelines for OpenStack devs|https://wiki.openstack.org/wiki/LoggingStandards] * [the Kafka dev guide|http://kafka.apache.org/coding-guide.html] has a good section on their use titled Logging -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12397) CopyTable fails to compile with the Hadoop 1 profile
[
https://issues.apache.org/jira/browse/HBASE-12397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14192135#comment-14192135
]
Andrew Purtell commented on HBASE-12397:
This will fail with '-Dhadoop.profile=1.0' but not '-Dhadoop.profile=1.1' so
one option is dropping support for compiling against 1.0 in the POMs. That
would be one less source of surprise for devs working with 0.98. Thoughts?
CopyTable fails to compile with the Hadoop 1 profile
Key: HBASE-12397
URL: https://issues.apache.org/jira/browse/HBASE-12397
Project: HBase
Issue Type: Bug
Reporter: Andrew Purtell
Fix For: 0.98.8
[ERROR]
/usr/src/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/CopyTable.java:[88,17]
error: cannot find symbol
This was introduced in f31edd80
{noformat}
commit f31edd8004226c795ee46fbe9e93d10671ab895a
Author: Ted Yu te...@apache.org
Date: Thu Oct 9 15:52:18 2014 +
HBASE-11997 CopyTable with bulkload (Yi Deng)
{noformat}
[~tedyu], [~daviddengcn], please have a look at this or I will revert it on
0.98.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12397) CopyTable fails to compile with the Hadoop 1 profile
[
https://issues.apache.org/jira/browse/HBASE-12397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14192140#comment-14192140
]
Ted Yu commented on HBASE-12397:
That would be nice.
CopyTable fails to compile with the Hadoop 1 profile
Key: HBASE-12397
URL: https://issues.apache.org/jira/browse/HBASE-12397
Project: HBase
Issue Type: Bug
Reporter: Andrew Purtell
Fix For: 0.98.8
[ERROR]
/usr/src/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/CopyTable.java:[88,17]
error: cannot find symbol
This was introduced in f31edd80
{noformat}
commit f31edd8004226c795ee46fbe9e93d10671ab895a
Author: Ted Yu te...@apache.org
Date: Thu Oct 9 15:52:18 2014 +
HBASE-11997 CopyTable with bulkload (Yi Deng)
{noformat}
[~tedyu], [~daviddengcn], please have a look at this or I will revert it on
0.98.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12396) Document suggested use of log levels in dev guide
[ https://issues.apache.org/jira/browse/HBASE-12396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14192142#comment-14192142 ] Sean Busbey commented on HBASE-12396: - We should also document what user-provided data might be included in log messages (like keys in region ids) so downstream people can reason about the protection level needed for them. Document suggested use of log levels in dev guide - Key: HBASE-12396 URL: https://issues.apache.org/jira/browse/HBASE-12396 Project: HBase Issue Type: Improvement Components: documentation Reporter: Sean Busbey Priority: Minor Right now we don't provide any guidance on appropriate use of log levels, which leads to inconsistent use and in some cases exacerbating problems for tests and troubleshooting (see HBASE-12395). We should add a section on suggested use of log levels to the guide. Some related reading * [a good ops-focused blog post on levels|http://watchitlater.com/blog/2009/12/logging-guidelines/] * [another, focused on devs|http://www.codeproject.com/Articles/42354/The-Art-of-Logging] * [guidelines from a user on stackoverflow (that I like)|http://stackoverflow.com/a/2031209] also has some good discussion. * [extended logging discussion with some level use guidelines|http://www.javacodegeeks.com/2011/01/10-tips-proper-application-logging.html] * [guidelines for Atlassian devs|https://developer.atlassian.com/display/CONFDEV/Logging+Guidelines] * [guidelines for OpenStack devs|https://wiki.openstack.org/wiki/LoggingStandards] * [the Kafka dev guide|http://kafka.apache.org/coding-guide.html] has a good section on their use titled Logging -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12329) Table create with duplicate column family names quietly succeeds
[
https://issues.apache.org/jira/browse/HBASE-12329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-12329:
Status: Patch Available (was: Open)
+1 pending QA.
applies cleanly, ran through the tests altered by the patch.
switching to Patch Available to get a QA run.
Table create with duplicate column family names quietly succeeds
Key: HBASE-12329
URL: https://issues.apache.org/jira/browse/HBASE-12329
Project: HBase
Issue Type: Bug
Components: Client, shell
Reporter: Sean Busbey
Assignee: Jingcheng Du
Priority: Minor
Attachments: HBASE-12329-V2.diff, HBASE-12329-V3.diff,
HBASE-12329.diff
From the mailing list
{quote}
I was expecting that it is forbidden, **but** this call does not throw any
exception
{code}
String[] families = {cf, cf};
HTableDescriptor desc = new HTableDescriptor(name);
for (String cf : families) {
HColumnDescriptor coldef = new HColumnDescriptor(cf);
desc.addFamily(coldef);
}
try {
admin.createTable(desc);
} catch (TableExistsException e) {
throw new IOException(table \' + name + \' already exists);
}
{code}
{quote}
And Ted's follow up replicates in the shell
{code}
hbase(main):001:0 create 't2', {NAME = 'f1'}, {NAME = 'f1'}
The table got created - with 1 column family:
hbase(main):002:0 describe 't2'
DESCRIPTION
ENABLED
't2', {NAME = 'f1', DATA_BLOCK_ENCODING = 'NONE', BLOOMFILTER = 'ROW',
REPLICATION_SCOPE = '0 true
', VERSIONS = '1', COMPRESSION = 'NONE', MIN_VERSIONS = '0', TTL =
'2147483647', KEEP_DELETED
_CELLS = 'false', BLOCKSIZE = '65536', IN_MEMORY = 'false', BLOCKCACHE
= 'true'}
1 row(s) in 0.1000 seconds
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12219) Cache more efficiently getAll() and get() in FSTableDescriptors
[ https://issues.apache.org/jira/browse/HBASE-12219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-12219: -- Resolution: Fixed Fix Version/s: 0.99.2 2.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed the branch-1 patch (Pushed the master patch yesterday). Thanks [~esteban] Cache more efficiently getAll() and get() in FSTableDescriptors --- Key: HBASE-12219 URL: https://issues.apache.org/jira/browse/HBASE-12219 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.94.24, 0.99.1, 0.98.6.1 Reporter: Esteban Gutierrez Assignee: Esteban Gutierrez Labels: scalability Fix For: 2.0.0, 0.99.2 Attachments: HBASE-12219-0.99.patch, HBASE-12219-v1.patch, HBASE-12219-v1.patch, HBASE-12219.v0.txt, HBASE-12219.v2.patch, HBASE-12219.v3.patch, list.png Currently table descriptors and tables are cached once they are accessed for the first time. Next calls to the master only require a trip to HDFS to lookup the modified time in order to reload the table descriptors if modified. However in clusters with a large number of tables or concurrent clients and this can be too aggressive to HDFS and the master causing contention to process other requests. A simple solution is to have a TTL based cached for FSTableDescriptors#getAll() and FSTableDescriptors#TableDescriptorAndModtime() that can allow the master to process those calls faster without causing contention without having to perform a trip to HDFS for every call. to listtables() or getTableDescriptor() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12397) CopyTable fails to compile with the Hadoop 1 profile
[
https://issues.apache.org/jira/browse/HBASE-12397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14192163#comment-14192163
]
Andrew Purtell commented on HBASE-12397:
I sent a mail to dev@.
CopyTable fails to compile with the Hadoop 1 profile
Key: HBASE-12397
URL: https://issues.apache.org/jira/browse/HBASE-12397
Project: HBase
Issue Type: Bug
Reporter: Andrew Purtell
Fix For: 0.98.8
[ERROR]
/usr/src/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/CopyTable.java:[88,17]
error: cannot find symbol
This was introduced in f31edd80
{noformat}
commit f31edd8004226c795ee46fbe9e93d10671ab895a
Author: Ted Yu te...@apache.org
Date: Thu Oct 9 15:52:18 2014 +
HBASE-11997 CopyTable with bulkload (Yi Deng)
{noformat}
[~tedyu], [~daviddengcn], please have a look at this or I will revert it on
0.98.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12358) Create ByteBuffer backed Cell
[
https://issues.apache.org/jira/browse/HBASE-12358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14192171#comment-14192171
]
stack commented on HBASE-12358:
---
As a POC, v2 looks cleaner.
What is ByteBufferBackedKeyValue then? An old-school KeyValue only backed by
BB? Thats a good POC exercise I'd say.
You don't need HeapSize and Settable... because ByteBufferBackedCell implements
them already:
+public class ByteBufferBackedKeyValue implements ByteBufferBackedCell,
HeapSize, Cloneable,
+SettableSequenceId {
Don't repeat these defines in new class I'd say... just refer to them from
KeyValue?
+ /** Size of the key length field in bytes */
+ public static final int KEY_LENGTH_SIZE = Bytes.SIZEOF_INT;
Or if you do bring them over, make them private so this madness (smile) doesn't
leak about.
On below, you can actually instantiate an empty one?
+ public ByteBufferBackedKeyValue() {
+
+ }
We can't have a util per type as in ByteBufferBackedKeyValueUtil
Could we have single util and it uses factory to get type particular methods?
Can do later.
What you fellas think of adding this?
+ public boolean hasArray() {
Does it belong in Cell? I mean, if it returns false, what are you to do? Go
find BB version? Where you find that? A factory can be used to figure the
configured type inside server/client but as to whether or not we should use BBs
rather than arrays... This is one way adding it to Cell but if DBBs, what you
to do?
On other hand, we are not going to have that many implementations of Cell... so
if Cell answer false, then you presume it has implemented BBBCell... and use
the BB methods. SOmething like that.
Create ByteBuffer backed Cell
-
Key: HBASE-12358
URL: https://issues.apache.org/jira/browse/HBASE-12358
Project: HBase
Issue Type: Sub-task
Components: regionserver, Scanners
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Attachments: HBASE-12358.patch, HBASE-12358_1.patch,
HBASE-12358_2.patch
As part of HBASE-12224 and HBASE-12282 we wanted a Cell that is backed by BB.
Changing the core Cell impl would not be needed as it is used in server
only. So we will create a BB backed Cell and use it in the Server side read
path. This JIRA just creates an interface that extends Cell and adds the
needed API.
The getTimeStamp and getTypebyte() can still refer to the original Cell API
only. The getXXxOffset() and getXXXLength() can also refer to the original
Cell only.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12390) Change revision style from svn to git
[
https://issues.apache.org/jira/browse/HBASE-12390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-12390:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Pushed this to branch-1+. Thanks Stack for review.
Change revision style from svn to git
-
Key: HBASE-12390
URL: https://issues.apache.org/jira/browse/HBASE-12390
Project: HBase
Issue Type: Improvement
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Priority: Minor
Fix For: 2.0.0, 0.99.2
Attachments: hbase-12390_v1.patch
This was bothering me. We should change the {{-r revision_id}} style that
is an svn thing.
We can do:
{code}
2.0.0-SNAPSHOT, revision=64b6109ce917a47e4fa4b88cdb800bcc7a228484
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12397) CopyTable fails to compile with the Hadoop 1 profile
[
https://issues.apache.org/jira/browse/HBASE-12397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14192185#comment-14192185
]
Yi Deng commented on HBASE-12397:
-
I'm not familar with -Dhadoop.profile=1.0 but what could be changed between
different profiles?
CopyTable fails to compile with the Hadoop 1 profile
Key: HBASE-12397
URL: https://issues.apache.org/jira/browse/HBASE-12397
Project: HBase
Issue Type: Bug
Reporter: Andrew Purtell
Fix For: 0.98.8
[ERROR]
/usr/src/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/CopyTable.java:[88,17]
error: cannot find symbol
This was introduced in f31edd80
{noformat}
commit f31edd8004226c795ee46fbe9e93d10671ab895a
Author: Ted Yu te...@apache.org
Date: Thu Oct 9 15:52:18 2014 +
HBASE-11997 CopyTable with bulkload (Yi Deng)
{noformat}
[~tedyu], [~daviddengcn], please have a look at this or I will revert it on
0.98.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12072) We are doing 35 x 35 retries for master operations
[
https://issues.apache.org/jira/browse/HBASE-12072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14192187#comment-14192187
]
Enis Soztutar commented on HBASE-12072:
---
There seems to be a timeout in one of these tests. let me look into it.
We are doing 35 x 35 retries for master operations
--
Key: HBASE-12072
URL: https://issues.apache.org/jira/browse/HBASE-12072
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.6
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 2.0.0, 0.99.2
Attachments: 12072-v1.txt, 12072-v2.txt, hbase-12072_v1.patch,
hbase-12072_v2.patch, hbase-12072_v2.patch
For master requests, there are two retry mechanisms in effect. The first one
is from HBaseAdmin.executeCallable()
{code}
private V V executeCallable(MasterCallableV callable) throws
IOException {
RpcRetryingCallerV caller = rpcCallerFactory.newCaller();
try {
return caller.callWithRetries(callable);
} finally {
callable.close();
}
}
{code}
And inside, the other one is from StubMaker.makeStub():
{code}
/**
* Create a stub against the master. Retry if necessary.
* @return A stub to do codeintf/code against the master
* @throws MasterNotRunningException
*/
@edu.umd.cs.findbugs.annotations.SuppressWarnings
(value=SWL_SLEEP_WITH_LOCK_HELD)
Object makeStub() throws MasterNotRunningException {
{code}
The tests will just hang for 10 min * 35 ~= 6hours.
{code}
2014-09-23 16:19:05,151 INFO [main]
client.ConnectionManager$HConnectionImplementation: getMaster attempt 1 of 35
failed; retrying after sleep of 100, exception=java.io.IOException: Can't get
master address from ZooKeeper; znode data == null
2014-09-23 16:19:05,253 INFO [main]
client.ConnectionManager$HConnectionImplementation: getMaster attempt 2 of 35
failed; retrying after sleep of 200, exception=java.io.IOException: Can't get
master address from ZooKeeper; znode data == null
2014-09-23 16:19:05,456 INFO [main]
client.ConnectionManager$HConnectionImplementation: getMaster attempt 3 of 35
failed; retrying after sleep of 300, exception=java.io.IOException: Can't get
master address from ZooKeeper; znode data == null
2014-09-23 16:19:05,759 INFO [main]
client.ConnectionManager$HConnectionImplementation: getMaster attempt 4 of 35
failed; retrying after sleep of 500, exception=java.io.IOException: Can't get
master address from ZooKeeper; znode data == null
2014-09-23 16:19:06,262 INFO [main]
client.ConnectionManager$HConnectionImplementation: getMaster attempt 5 of 35
failed; retrying after sleep of 1008, exception=java.io.IOException: Can't
get master address from ZooKeeper; znode data == null
2014-09-23 16:19:07,273 INFO [main]
client.ConnectionManager$HConnectionImplementation: getMaster attempt 6 of 35
failed; retrying after sleep of 2011, exception=java.io.IOException: Can't
get master address from ZooKeeper; znode data == null
2014-09-23 16:19:09,286 INFO [main]
client.ConnectionManager$HConnectionImplementation: getMaster attempt 7 of 35
failed; retrying after sleep of 4012, exception=java.io.IOException: Can't
get master address from ZooKeeper; znode data == null
2014-09-23 16:19:13,303 INFO [main]
client.ConnectionManager$HConnectionImplementation: getMaster attempt 8 of 35
failed; retrying after sleep of 10033, exception=java.io.IOException: Can't
get master address from ZooKeeper; znode data == null
2014-09-23 16:19:23,343 INFO [main]
client.ConnectionManager$HConnectionImplementation: getMaster attempt 9 of 35
failed; retrying after sleep of 10089, exception=java.io.IOException: Can't
get master address from ZooKeeper; znode data == null
2014-09-23 16:19:33,439 INFO [main]
client.ConnectionManager$HConnectionImplementation: getMaster attempt 10 of
35 failed; retrying after sleep of 10027, exception=java.io.IOException:
Can't get master address from ZooKeeper; znode data == null
2014-09-23 16:19:43,473 INFO [main]
client.ConnectionManager$HConnectionImplementation: getMaster attempt 11 of
35 failed; retrying after sleep of 10004, exception=java.io.IOException:
Can't get master address from ZooKeeper; znode data == null
2014-09-23 16:19:53,485 INFO [main]
client.ConnectionManager$HConnectionImplementation: getMaster attempt 12 of
35 failed; retrying after sleep of 20160, exception=java.io.IOException:
Can't get master address from ZooKeeper; znode data == null
2014-09-23 16:20:13,656 INFO [main]
client.ConnectionManager$HConnectionImplementation: getMaster attempt 13 of
35 failed; retrying after sleep of 20006, exception=java.io.IOException:
Can't get master address from ZooKeeper; znode data == null
2014-09-23
[jira] [Updated] (HBASE-12219) Cache more efficiently getAll() and get() in FSTableDescriptors
[ https://issues.apache.org/jira/browse/HBASE-12219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esteban Gutierrez updated HBASE-12219: -- Attachment: HBASE-12219-0.99.addendum.patch Missing change. Cache more efficiently getAll() and get() in FSTableDescriptors --- Key: HBASE-12219 URL: https://issues.apache.org/jira/browse/HBASE-12219 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.94.24, 0.99.1, 0.98.6.1 Reporter: Esteban Gutierrez Assignee: Esteban Gutierrez Labels: scalability Fix For: 2.0.0, 0.99.2 Attachments: HBASE-12219-0.99.addendum.patch, HBASE-12219-0.99.patch, HBASE-12219-v1.patch, HBASE-12219-v1.patch, HBASE-12219.v0.txt, HBASE-12219.v2.patch, HBASE-12219.v3.patch, list.png Currently table descriptors and tables are cached once they are accessed for the first time. Next calls to the master only require a trip to HDFS to lookup the modified time in order to reload the table descriptors if modified. However in clusters with a large number of tables or concurrent clients and this can be too aggressive to HDFS and the master causing contention to process other requests. A simple solution is to have a TTL based cached for FSTableDescriptors#getAll() and FSTableDescriptors#TableDescriptorAndModtime() that can allow the master to process those calls faster without causing contention without having to perform a trip to HDFS for every call. to listtables() or getTableDescriptor() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
