[jira] [Commented] (HBASE-12016) Reduce number of versions in Meta table. Make it configurable
[ https://issues.apache.org/jira/browse/HBASE-12016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14622260#comment-14622260 ] Andrey Stepachev commented on HBASE-12016: -- [~anoop.hbase] Thank for pointing that. Updated. Reduce number of versions in Meta table. Make it configurable - Key: HBASE-12016 URL: https://issues.apache.org/jira/browse/HBASE-12016 Project: HBase Issue Type: Improvement Affects Versions: 2.0.0 Reporter: Andrey Stepachev Assignee: Andrey Stepachev Priority: Minor Fix For: 1.0.0, 2.0.0 Attachments: HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch Currently meta keeps up to 10 versions of each KV. For big metas it leads to substantial memory overhead and scan slowdowns. (see https://issues.apache.org/jira/browse/HBASE-11165 ) Need to keep reasonable number of versions (suggested value is 3). Number of versions configurable via parameter: hbase.meta.versions -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12016) Reduce number of versions in Meta table. Make it configurable
[ https://issues.apache.org/jira/browse/HBASE-12016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14620258#comment-14620258 ] Anoop Sam John commented on HBASE-12016: Mind adding how we can change the #versions for META table as well as its block size on a new cluster. We are exposing some configs for those. Reduce number of versions in Meta table. Make it configurable - Key: HBASE-12016 URL: https://issues.apache.org/jira/browse/HBASE-12016 Project: HBase Issue Type: Improvement Affects Versions: 2.0.0 Reporter: Andrey Stepachev Assignee: Andrey Stepachev Priority: Minor Fix For: 1.0.0, 2.0.0 Attachments: HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch Currently meta keeps up to 10 versions of each KV. For big metas it leads to substantial memory overhead and scan slowdowns. (see https://issues.apache.org/jira/browse/HBASE-11165 ) Need to keep reasonable number of versions (suggested value is 3). Number of versions configurable via parameter: hbase.meta.versions -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12016) Reduce number of versions in Meta table. Make it configurable
[ https://issues.apache.org/jira/browse/HBASE-12016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14613064#comment-14613064 ] Anoop Sam John commented on HBASE-12016: It would have been better to add a Release Note for this issue. [~octo47] Reduce number of versions in Meta table. Make it configurable - Key: HBASE-12016 URL: https://issues.apache.org/jira/browse/HBASE-12016 Project: HBase Issue Type: Improvement Affects Versions: 2.0.0 Reporter: Andrey Stepachev Assignee: Andrey Stepachev Priority: Minor Fix For: 1.0.0, 2.0.0 Attachments: HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch Currently meta keeps up to 10 versions of each KV. For big metas it leads to substantial memory overhead and scan slowdowns. (see https://issues.apache.org/jira/browse/HBASE-11165 ) Need to keep reasonable number of versions (suggested value is 3). Number of versions configurable via parameter: hbase.meta.versions -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12016) Reduce number of versions in Meta table. Make it configurable
[ https://issues.apache.org/jira/browse/HBASE-12016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234691#comment-14234691 ] Hudson commented on HBASE-12016: SUCCESS: Integrated in HBase-1.0 #540 (See [https://builds.apache.org/job/HBase-1.0/540/]) HBASE-12204 Backport HBASE-12016 'Reduce number of versions in Meta table. Make it configurable' to branch-1 (tedyu: rev 2e47c37b7102aa88ac99bacb8950105178ec0e29) * hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestGetClosestAtOrBefore.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionManager.java * hbase-client/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * hbase-server/src/test/java/org/apache/hadoop/hbase/migration/TestNamespaceUpgrade.java * hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotManifest.java * hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java * hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java * hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestMergeTool.java * hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java * hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestFSTableDescriptors.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/MetaUtils.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java * hbase-server/src/test/java/org/apache/hadoop/hbase/TestFSTableDescriptorForceCreation.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAdmin2.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java * hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java Reduce number of versions in Meta table. Make it configurable - Key: HBASE-12016 URL: https://issues.apache.org/jira/browse/HBASE-12016 Project: HBase Issue Type: Improvement Affects Versions: 2.0.0 Reporter: Andrey Stepachev Assignee: Andrey Stepachev Priority: Minor Fix For: 2.0.0 Attachments: HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch Currently meta keeps up to 10 versions of each KV. For big metas it leads to substantial memory overhead and scan slowdowns. (see https://issues.apache.org/jira/browse/HBASE-11165 ) Need to keep reasonable number of versions (suggested value is 3). Number of versions configurable via parameter: hbase.meta.versions -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12016) Reduce number of versions in Meta table. Make it configurable
[ https://issues.apache.org/jira/browse/HBASE-12016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14187961#comment-14187961 ] Hudson commented on HBASE-12016: FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #605 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/605/]) HBASE-12205 Backport HBASE-12016 'Reduce number of versions in Meta table. Make it configurable' to 0.98 (apurtell: rev 81f78fce17f0c67f5b21ca86825bba87ebb534a0) * hbase-server/src/test/java/org/apache/hadoop/hbase/migration/TestNamespaceUpgrade.java * hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java * hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * hbase-client/src/main/java/org/apache/hadoop/hbase/catalog/MetaReader.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRollingNoCluster.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * hbase-client/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestGetClosestAtOrBefore.java * hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java * hbase-server/src/main/java/org/apache/hadoop/hbase/migration/NamespaceUpgrade.java * hbase-server/src/test/java/org/apache/hadoop/hbase/TestFSTableDescriptorForceCreation.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/MetaUtils.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java * hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java * hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotManifest.java * hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestMergeTool.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java * hbase-server/src/test/java/org/apache/hadoop/hbase/TestHColumnDescriptor.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java * hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java * hbase-server/src/test/java/org/apache/hadoop/hbase/TestHTableDescriptor.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java * hbase-server/src/test/java/org/apache/hadoop/hbase/util/hbck/OfflineMetaRebuildTestCore.java * hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestFSTableDescriptors.java * hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java Reduce number of versions in Meta table. Make it configurable - Key: HBASE-12016 URL: https://issues.apache.org/jira/browse/HBASE-12016 Project: HBase Issue Type: Improvement Affects Versions: 2.0.0 Reporter: Andrey Stepachev Assignee: Andrey Stepachev Priority: Minor Fix For: 2.0.0 Attachments: HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch Currently meta keeps up to 10 versions of each KV. For big metas it leads to substantial memory overhead and scan slowdowns. (see https://issues.apache.org/jira/browse/HBASE-11165 ) Need to keep reasonable number of versions (suggested value is 3). Number of versions configurable via parameter: hbase.meta.versions -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12016) Reduce number of versions in Meta table. Make it configurable
[ https://issues.apache.org/jira/browse/HBASE-12016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14187974#comment-14187974 ] Hudson commented on HBASE-12016: FAILURE: Integrated in HBase-0.98 #635 (See [https://builds.apache.org/job/HBase-0.98/635/]) HBASE-12205 Backport HBASE-12016 'Reduce number of versions in Meta table. Make it configurable' to 0.98 (apurtell: rev 81f78fce17f0c67f5b21ca86825bba87ebb534a0) * hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java * hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestFSTableDescriptors.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java * hbase-client/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java * hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotManifest.java * hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRollingNoCluster.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * hbase-server/src/test/java/org/apache/hadoop/hbase/TestFSTableDescriptorForceCreation.java * hbase-server/src/test/java/org/apache/hadoop/hbase/migration/TestNamespaceUpgrade.java * hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestGetClosestAtOrBefore.java * hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * hbase-server/src/main/java/org/apache/hadoop/hbase/migration/NamespaceUpgrade.java * hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * hbase-server/src/test/java/org/apache/hadoop/hbase/TestHTableDescriptor.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java * hbase-server/src/test/java/org/apache/hadoop/hbase/util/hbck/OfflineMetaRebuildTestCore.java * hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestMergeTool.java * hbase-server/src/test/java/org/apache/hadoop/hbase/TestHColumnDescriptor.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java * hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java * hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/MetaUtils.java * hbase-client/src/main/java/org/apache/hadoop/hbase/catalog/MetaReader.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java Reduce number of versions in Meta table. Make it configurable - Key: HBASE-12016 URL: https://issues.apache.org/jira/browse/HBASE-12016 Project: HBase Issue Type: Improvement Affects Versions: 2.0.0 Reporter: Andrey Stepachev Assignee: Andrey Stepachev Priority: Minor Fix For: 2.0.0 Attachments: HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch Currently meta keeps up to 10 versions of each KV. For big metas it leads to substantial memory overhead and scan slowdowns. (see https://issues.apache.org/jira/browse/HBASE-11165 ) Need to keep reasonable number of versions (suggested value is 3). Number of versions configurable via parameter: hbase.meta.versions -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12016) Reduce number of versions in Meta table. Make it configurable
[ https://issues.apache.org/jira/browse/HBASE-12016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14163575#comment-14163575 ] Andrey Stepachev commented on HBASE-12016: -- [~tedyu] file not exists due of missing HBASE-7767. Backporting will take some time. Or HBASE-12203 can be made instead for adoption of HBASE-7767 to branch1 and this patch will apply. Reduce number of versions in Meta table. Make it configurable - Key: HBASE-12016 URL: https://issues.apache.org/jira/browse/HBASE-12016 Project: HBase Issue Type: Improvement Affects Versions: 2.0.0 Reporter: Andrey Stepachev Assignee: Andrey Stepachev Priority: Minor Fix For: 2.0.0, 0.98.8, 0.99.1 Attachments: HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch Currently meta keeps up to 10 versions of each KV. For big metas it leads to substantial memory overhead and scan slowdowns. (see https://issues.apache.org/jira/browse/HBASE-11165 ) Need to keep reasonable number of versions (suggested value is 3). Number of versions configurable via parameter: hbase.meta.versions -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12016) Reduce number of versions in Meta table. Make it configurable
[ https://issues.apache.org/jira/browse/HBASE-12016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14161541#comment-14161541 ] Andrew Purtell commented on HBASE-12016: Unable to apply latest patch to trunk Reduce number of versions in Meta table. Make it configurable - Key: HBASE-12016 URL: https://issues.apache.org/jira/browse/HBASE-12016 Project: HBase Issue Type: Improvement Affects Versions: 2.0.0 Reporter: Andrey Stepachev Assignee: Andrey Stepachev Priority: Minor Fix For: 2.0.0, 0.98.8, 0.99.1 Attachments: HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch Currently meta keeps up to 10 versions of each KV. For big metas it leads to substantial memory overhead and scan slowdowns. (see https://issues.apache.org/jira/browse/HBASE-11165 ) Need to keep reasonable number of versions (suggested value is 3). Number of versions configurable via parameter: hbase.meta.versions -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12016) Reduce number of versions in Meta table. Make it configurable
[
https://issues.apache.org/jira/browse/HBASE-12016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14161720#comment-14161720
]
Hadoop QA commented on HBASE-12016:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12673314/HBASE-12016.patch
against trunk revision .
ATTACHMENT ID: 12673314
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 48 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.util.TestFSTableDescriptors
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/11235//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11235//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11235//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11235//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11235//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11235//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11235//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11235//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11235//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11235//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11235//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/11235//console
This message is automatically generated.
Reduce number of versions in Meta table. Make it configurable
-
Key: HBASE-12016
URL: https://issues.apache.org/jira/browse/HBASE-12016
Project: HBase
Issue Type: Improvement
Affects Versions: 2.0.0
Reporter: Andrey Stepachev
Assignee: Andrey Stepachev
Priority: Minor
Fix For: 2.0.0, 0.98.8, 0.99.1
Attachments: HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch,
HBASE-12016.patch
Currently meta keeps up to 10 versions of each KV.
For big metas it leads to substantial memory overhead and scan slowdowns.
(see https://issues.apache.org/jira/browse/HBASE-11165 )
Need to keep reasonable number of versions (suggested value is 3).
Number of versions configurable via parameter: hbase.meta.versions
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12016) Reduce number of versions in Meta table. Make it configurable
[
https://issues.apache.org/jira/browse/HBASE-12016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14161879#comment-14161879
]
Hadoop QA commented on HBASE-12016:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12673344/HBASE-12016.patch
against trunk revision .
ATTACHMENT ID: 12673344
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 48 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestShell
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/11236//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11236//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11236//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11236//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11236//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11236//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11236//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11236//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11236//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11236//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11236//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/11236//console
This message is automatically generated.
Reduce number of versions in Meta table. Make it configurable
-
Key: HBASE-12016
URL: https://issues.apache.org/jira/browse/HBASE-12016
Project: HBase
Issue Type: Improvement
Affects Versions: 2.0.0
Reporter: Andrey Stepachev
Assignee: Andrey Stepachev
Priority: Minor
Fix For: 2.0.0, 0.98.8, 0.99.1
Attachments: HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch,
HBASE-12016.patch, HBASE-12016.patch
Currently meta keeps up to 10 versions of each KV.
For big metas it leads to substantial memory overhead and scan slowdowns.
(see https://issues.apache.org/jira/browse/HBASE-11165 )
Need to keep reasonable number of versions (suggested value is 3).
Number of versions configurable via parameter: hbase.meta.versions
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12016) Reduce number of versions in Meta table. Make it configurable
[ https://issues.apache.org/jira/browse/HBASE-12016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14161953#comment-14161953 ] Matteo Bertozzi commented on HBASE-12016: - why are you adding back archive to the blacklist? with namespace archive is a valid name Reduce number of versions in Meta table. Make it configurable - Key: HBASE-12016 URL: https://issues.apache.org/jira/browse/HBASE-12016 Project: HBase Issue Type: Improvement Affects Versions: 2.0.0 Reporter: Andrey Stepachev Assignee: Andrey Stepachev Priority: Minor Fix For: 2.0.0, 0.98.8, 0.99.1 Attachments: HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch Currently meta keeps up to 10 versions of each KV. For big metas it leads to substantial memory overhead and scan slowdowns. (see https://issues.apache.org/jira/browse/HBASE-11165 ) Need to keep reasonable number of versions (suggested value is 3). Number of versions configurable via parameter: hbase.meta.versions -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12016) Reduce number of versions in Meta table. Make it configurable
[ https://issues.apache.org/jira/browse/HBASE-12016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14161962#comment-14161962 ] Andrey Stepachev commented on HBASE-12016: -- Thank you [~mbertozzi], you are right. With roodir/data directory it is not a forbidden name. Reduce number of versions in Meta table. Make it configurable - Key: HBASE-12016 URL: https://issues.apache.org/jira/browse/HBASE-12016 Project: HBase Issue Type: Improvement Affects Versions: 2.0.0 Reporter: Andrey Stepachev Assignee: Andrey Stepachev Priority: Minor Fix For: 2.0.0, 0.98.8, 0.99.1 Attachments: HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch Currently meta keeps up to 10 versions of each KV. For big metas it leads to substantial memory overhead and scan slowdowns. (see https://issues.apache.org/jira/browse/HBASE-11165 ) Need to keep reasonable number of versions (suggested value is 3). Number of versions configurable via parameter: hbase.meta.versions -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12016) Reduce number of versions in Meta table. Make it configurable
[
https://issues.apache.org/jira/browse/HBASE-12016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14162059#comment-14162059
]
Ted Yu commented on HBASE-12016:
Looks good overall.
{code}
+ public static int DEFAULT_HBASE_META_BLOCK_SIZE = 8 * 1024;
{code}
How was the block size determined ?
{code}
+ public static void setInfoFamilyCachingForMeta(HTableDescriptor
metaDescriptor, final boolean b) {
{code}
Since table descriptor is an argument, method name can be changed to
setInfoFamilyCaching.
Reduce number of versions in Meta table. Make it configurable
-
Key: HBASE-12016
URL: https://issues.apache.org/jira/browse/HBASE-12016
Project: HBase
Issue Type: Improvement
Affects Versions: 2.0.0
Reporter: Andrey Stepachev
Assignee: Andrey Stepachev
Priority: Minor
Fix For: 2.0.0, 0.98.8, 0.99.1
Attachments: HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch,
HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch
Currently meta keeps up to 10 versions of each KV.
For big metas it leads to substantial memory overhead and scan slowdowns.
(see https://issues.apache.org/jira/browse/HBASE-11165 )
Need to keep reasonable number of versions (suggested value is 3).
Number of versions configurable via parameter: hbase.meta.versions
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12016) Reduce number of versions in Meta table. Make it configurable
[
https://issues.apache.org/jira/browse/HBASE-12016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14162086#comment-14162086
]
Hadoop QA commented on HBASE-12016:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12673364/HBASE-12016.patch
against trunk revision .
ATTACHMENT ID: 12673364
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 48 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestShell
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/11238//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11238//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11238//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11238//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11238//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11238//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11238//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11238//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11238//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11238//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11238//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/11238//console
This message is automatically generated.
Reduce number of versions in Meta table. Make it configurable
-
Key: HBASE-12016
URL: https://issues.apache.org/jira/browse/HBASE-12016
Project: HBase
Issue Type: Improvement
Affects Versions: 2.0.0
Reporter: Andrey Stepachev
Assignee: Andrey Stepachev
Priority: Minor
Fix For: 2.0.0, 0.98.8, 0.99.1
Attachments: HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch,
HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch
Currently meta keeps up to 10 versions of each KV.
For big metas it leads to substantial memory overhead and scan slowdowns.
(see https://issues.apache.org/jira/browse/HBASE-11165 )
Need to keep reasonable number of versions (suggested value is 3).
Number of versions configurable via parameter: hbase.meta.versions
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12016) Reduce number of versions in Meta table. Make it configurable
[ https://issues.apache.org/jira/browse/HBASE-12016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14162097#comment-14162097 ] Ted Yu commented on HBASE-12016: TestShell failure is related to the patch. Reduce number of versions in Meta table. Make it configurable - Key: HBASE-12016 URL: https://issues.apache.org/jira/browse/HBASE-12016 Project: HBase Issue Type: Improvement Affects Versions: 2.0.0 Reporter: Andrey Stepachev Assignee: Andrey Stepachev Priority: Minor Fix For: 2.0.0, 0.98.8, 0.99.1 Attachments: HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch Currently meta keeps up to 10 versions of each KV. For big metas it leads to substantial memory overhead and scan slowdowns. (see https://issues.apache.org/jira/browse/HBASE-11165 ) Need to keep reasonable number of versions (suggested value is 3). Number of versions configurable via parameter: hbase.meta.versions -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12016) Reduce number of versions in Meta table. Make it configurable
[ https://issues.apache.org/jira/browse/HBASE-12016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14162260#comment-14162260 ] Andrey Stepachev commented on HBASE-12016: -- [~apurtell], can you look at code org/apache/hadoop/hbase/security/access/AccessController.java:2177, I think we need to remove checks for isTableModifiable here. Only validate that table exists. [~ted_yu] thank you for reviewing. block size was got from previous code just by extracting constant, so I didn't thought much about why that value was selected. Regarding setInfoFamilyCaching, this method is static and applied to Meta only. So it is good to be explicit here, so I'd prefer to state in method name that this method is for meta only. Reduce number of versions in Meta table. Make it configurable - Key: HBASE-12016 URL: https://issues.apache.org/jira/browse/HBASE-12016 Project: HBase Issue Type: Improvement Affects Versions: 2.0.0 Reporter: Andrey Stepachev Assignee: Andrey Stepachev Priority: Minor Fix For: 2.0.0, 0.98.8, 0.99.1 Attachments: HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch Currently meta keeps up to 10 versions of each KV. For big metas it leads to substantial memory overhead and scan slowdowns. (see https://issues.apache.org/jira/browse/HBASE-11165 ) Need to keep reasonable number of versions (suggested value is 3). Number of versions configurable via parameter: hbase.meta.versions -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12016) Reduce number of versions in Meta table. Make it configurable
[ https://issues.apache.org/jira/browse/HBASE-12016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14162281#comment-14162281 ] stack commented on HBASE-12016: --- bq. + public static int DEFAULT_HBASE_META_BLOCK_SIZE = 8 * 1024; Yeah, this is an oldie. Usual reasons for doing a smaller block apply (small cells, random access). Suggest holding with this default unless evidence to do otherwise. Reduce number of versions in Meta table. Make it configurable - Key: HBASE-12016 URL: https://issues.apache.org/jira/browse/HBASE-12016 Project: HBase Issue Type: Improvement Affects Versions: 2.0.0 Reporter: Andrey Stepachev Assignee: Andrey Stepachev Priority: Minor Fix For: 2.0.0, 0.98.8, 0.99.1 Attachments: HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch Currently meta keeps up to 10 versions of each KV. For big metas it leads to substantial memory overhead and scan slowdowns. (see https://issues.apache.org/jira/browse/HBASE-11165 ) Need to keep reasonable number of versions (suggested value is 3). Number of versions configurable via parameter: hbase.meta.versions -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12016) Reduce number of versions in Meta table. Make it configurable
[ https://issues.apache.org/jira/browse/HBASE-12016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14162341#comment-14162341 ] Andrew Purtell commented on HBASE-12016: bq. I think we need to remove checks for isTableModifiable here. Only validate that table exists. Sure, the table doesn't need to be modifiable, the call to checkTableModifiable was a convenient shortcut to existence testing (if IIRC). Reduce number of versions in Meta table. Make it configurable - Key: HBASE-12016 URL: https://issues.apache.org/jira/browse/HBASE-12016 Project: HBase Issue Type: Improvement Affects Versions: 2.0.0 Reporter: Andrey Stepachev Assignee: Andrey Stepachev Priority: Minor Fix For: 2.0.0, 0.98.8, 0.99.1 Attachments: HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch Currently meta keeps up to 10 versions of each KV. For big metas it leads to substantial memory overhead and scan slowdowns. (see https://issues.apache.org/jira/browse/HBASE-11165 ) Need to keep reasonable number of versions (suggested value is 3). Number of versions configurable via parameter: hbase.meta.versions -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12016) Reduce number of versions in Meta table. Make it configurable
[ https://issues.apache.org/jira/browse/HBASE-12016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14162345#comment-14162345 ] Andrey Stepachev commented on HBASE-12016: -- [~apurtell] great, I've replaced it with table existence check. Reduce number of versions in Meta table. Make it configurable - Key: HBASE-12016 URL: https://issues.apache.org/jira/browse/HBASE-12016 Project: HBase Issue Type: Improvement Affects Versions: 2.0.0 Reporter: Andrey Stepachev Assignee: Andrey Stepachev Priority: Minor Fix For: 2.0.0, 0.98.8, 0.99.1 Attachments: HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch Currently meta keeps up to 10 versions of each KV. For big metas it leads to substantial memory overhead and scan slowdowns. (see https://issues.apache.org/jira/browse/HBASE-11165 ) Need to keep reasonable number of versions (suggested value is 3). Number of versions configurable via parameter: hbase.meta.versions -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12016) Reduce number of versions in Meta table. Make it configurable
[
https://issues.apache.org/jira/browse/HBASE-12016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14162458#comment-14162458
]
Hadoop QA commented on HBASE-12016:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12673405/HBASE-12016.patch
against trunk revision .
ATTACHMENT ID: 12673405
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 48 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/11244//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11244//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11244//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11244//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11244//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11244//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11244//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11244//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11244//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11244//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11244//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/11244//console
This message is automatically generated.
Reduce number of versions in Meta table. Make it configurable
-
Key: HBASE-12016
URL: https://issues.apache.org/jira/browse/HBASE-12016
Project: HBase
Issue Type: Improvement
Affects Versions: 2.0.0
Reporter: Andrey Stepachev
Assignee: Andrey Stepachev
Priority: Minor
Fix For: 2.0.0, 0.98.8, 0.99.1
Attachments: HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch,
HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch
Currently meta keeps up to 10 versions of each KV.
For big metas it leads to substantial memory overhead and scan slowdowns.
(see https://issues.apache.org/jira/browse/HBASE-11165 )
Need to keep reasonable number of versions (suggested value is 3).
Number of versions configurable via parameter: hbase.meta.versions
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12016) Reduce number of versions in Meta table. Make it configurable
[ https://issues.apache.org/jira/browse/HBASE-12016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14162534#comment-14162534 ] Ted Yu commented on HBASE-12016: Integrated to master branch. Please attach patches for 0.98 and branch-1 e.g. the following file doesn't exist in branch-1: hbase-server/src/main/java/org/apache/hadoop/hbase/TableDescriptor.java Thanks Andrey Reduce number of versions in Meta table. Make it configurable - Key: HBASE-12016 URL: https://issues.apache.org/jira/browse/HBASE-12016 Project: HBase Issue Type: Improvement Affects Versions: 2.0.0 Reporter: Andrey Stepachev Assignee: Andrey Stepachev Priority: Minor Fix For: 2.0.0, 0.98.8, 0.99.1 Attachments: HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch Currently meta keeps up to 10 versions of each KV. For big metas it leads to substantial memory overhead and scan slowdowns. (see https://issues.apache.org/jira/browse/HBASE-11165 ) Need to keep reasonable number of versions (suggested value is 3). Number of versions configurable via parameter: hbase.meta.versions -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12016) Reduce number of versions in Meta table. Make it configurable
[ https://issues.apache.org/jira/browse/HBASE-12016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14162691#comment-14162691 ] Hudson commented on HBASE-12016: SUCCESS: Integrated in HBase-TRUNK #5632 (See [https://builds.apache.org/job/HBase-TRUNK/5632/]) HBASE-12016 Reduce number of versions in Meta table. Make it configurable (Andrey Stepachev) (tedyu: rev a8be606145b979be89d85e6191a3e3df8ab05545) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * hbase-server/src/test/java/org/apache/hadoop/hbase/TestTableDescriptor.java * hbase-server/src/main/java/org/apache/hadoop/hbase/migration/NamespaceUpgrade.java * hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java * hbase-client/src/test/java/org/apache/hadoop/hbase/TestHColumnDescriptor.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestGetClosestAtOrBefore.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationWALEntryFilters.java * hbase-server/src/test/java/org/apache/hadoop/hbase/TestFSTableDescriptorForceCreation.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java * hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java * hbase-client/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java * hbase-server/src/main/java/org/apache/hadoop/hbase/TableDescriptor.java * hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestFSTableDescriptors.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionManager.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/MetaUtils.java * hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java * hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java * hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java * hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotManifest.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java * hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java * hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestMergeTool.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRollingNoCluster.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * hbase-client/src/test/java/org/apache/hadoop/hbase/TestHTableDescriptor.java Reduce number of versions in Meta table. Make it configurable - Key: HBASE-12016 URL: https://issues.apache.org/jira/browse/HBASE-12016 Project: HBase Issue Type: Improvement Affects Versions: 2.0.0 Reporter: Andrey Stepachev Assignee: Andrey Stepachev Priority: Minor Fix For: 2.0.0, 0.98.8, 0.99.1 Attachments: HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch, HBASE-12016.patch Currently meta keeps up to 10 versions of each KV. For big metas it leads to substantial memory overhead and scan slowdowns. (see https://issues.apache.org/jira/browse/HBASE-11165 ) Need to keep reasonable number of versions (suggested value is 3). Number of versions configurable via parameter: hbase.meta.versions -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12016) Reduce number of versions in Meta table. Make it configurable
[ https://issues.apache.org/jira/browse/HBASE-12016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14138805#comment-14138805 ] Andrey Stepachev commented on HBASE-12016: -- This patch is a bit more, it remove static defined META_TABLEDESC and make sure, that all code query for descriptor or lookup it through FSTableDescriptors. So now it is one place to get them and not static dependency in client code (meta descriptor will be queried as usual via master RPC call) Reduce number of versions in Meta table. Make it configurable - Key: HBASE-12016 URL: https://issues.apache.org/jira/browse/HBASE-12016 Project: HBase Issue Type: Improvement Affects Versions: 1.0.0, 2.0.0 Reporter: Andrey Stepachev Assignee: Andrey Stepachev Priority: Minor Attachments: HBASE-12016.patch Currently meta keeps up to 10 versions of each KV. For big metas it leads to substantial memory overhead and scan slowdowns. (see https://issues.apache.org/jira/browse/HBASE-11165 ) Need to keep reasonable number of versions (suggested value is 3). Number of versions configurable via parameter: hbase.meta.versions -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12016) Reduce number of versions in Meta table. Make it configurable
[
https://issues.apache.org/jira/browse/HBASE-12016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14138881#comment-14138881
]
Hadoop QA commented on HBASE-12016:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12669705/HBASE-12016.patch
against trunk revision .
ATTACHMENT ID: 12669705
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 51 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.wal.TestHLog
org.apache.hadoop.hbase.regionserver.TestDefaultMemStore
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/10966//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10966//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10966//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10966//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10966//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10966//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10966//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10966//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10966//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10966//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/10966//console
This message is automatically generated.
Reduce number of versions in Meta table. Make it configurable
-
Key: HBASE-12016
URL: https://issues.apache.org/jira/browse/HBASE-12016
Project: HBase
Issue Type: Improvement
Affects Versions: 2.0.0
Reporter: Andrey Stepachev
Assignee: Andrey Stepachev
Priority: Minor
Attachments: HBASE-12016.patch
Currently meta keeps up to 10 versions of each KV.
For big metas it leads to substantial memory overhead and scan slowdowns.
(see https://issues.apache.org/jira/browse/HBASE-11165 )
Need to keep reasonable number of versions (suggested value is 3).
Number of versions configurable via parameter: hbase.meta.versions
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12016) Reduce number of versions in Meta table. Make it configurable
[
https://issues.apache.org/jira/browse/HBASE-12016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14139201#comment-14139201
]
Hadoop QA commented on HBASE-12016:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12669738/HBASE-12016.patch
against trunk revision .
ATTACHMENT ID: 12669738
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 51 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction
org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster
org.apache.hadoop.hbase.client.TestReplicaWithCluster
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/10968//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10968//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10968//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10968//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10968//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10968//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10968//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10968//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10968//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10968//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/10968//console
This message is automatically generated.
Reduce number of versions in Meta table. Make it configurable
-
Key: HBASE-12016
URL: https://issues.apache.org/jira/browse/HBASE-12016
Project: HBase
Issue Type: Improvement
Affects Versions: 2.0.0
Reporter: Andrey Stepachev
Assignee: Andrey Stepachev
Priority: Minor
Attachments: HBASE-12016.patch, HBASE-12016.patch
Currently meta keeps up to 10 versions of each KV.
For big metas it leads to substantial memory overhead and scan slowdowns.
(see https://issues.apache.org/jira/browse/HBASE-11165 )
Need to keep reasonable number of versions (suggested value is 3).
Number of versions configurable via parameter: hbase.meta.versions
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12016) Reduce number of versions in Meta table. Make it configurable
[ https://issues.apache.org/jira/browse/HBASE-12016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14139217#comment-14139217 ] Virag Kothari commented on HBASE-12016: --- [~octo47] Can we have this for 0.98.7 also? Just skimmed. The config change looks good and a nit is why to put the meta table descriptor instantiation in public constructor? Dont we want to make sure that its only initialized once like the earlier public static final way? For 0.98, we should make sure that we dont remove the META_TABLEDESC from HTableDescriptor as that would be client incompatible. Reduce number of versions in Meta table. Make it configurable - Key: HBASE-12016 URL: https://issues.apache.org/jira/browse/HBASE-12016 Project: HBase Issue Type: Improvement Affects Versions: 2.0.0 Reporter: Andrey Stepachev Assignee: Andrey Stepachev Priority: Minor Attachments: HBASE-12016.patch, HBASE-12016.patch Currently meta keeps up to 10 versions of each KV. For big metas it leads to substantial memory overhead and scan slowdowns. (see https://issues.apache.org/jira/browse/HBASE-11165 ) Need to keep reasonable number of versions (suggested value is 3). Number of versions configurable via parameter: hbase.meta.versions -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12016) Reduce number of versions in Meta table. Make it configurable
[ https://issues.apache.org/jira/browse/HBASE-12016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14139241#comment-14139241 ] Andrey Stepachev commented on HBASE-12016: -- [~virag] Thank you for looking at this patch. bq. Just skimmed. The config change looks good and a nit is why to put the meta table descriptor instantiation in public constructor? Dont we want to make sure that its only initialized once like the earlier public static final way? bq. For 0.98, we should make sure that we dont remove the META_TABLEDESC from HTableDescriptor as that would be client incompatible. I can create static HTableDescriptor META_TABLEDESC back, just for compatibility and with deprecation warning. HBase codebase will not use this instance. Does it sounds good? I think that code should not use static variables, and should query for meta descriptor as any other descriptors. That ensures identical descriptor across cluster. As for initialisation, do you have a suggestion, where to place initialised instance. In general I don't like an idea to have a static instance, which will be initialised by some unknown config. I think it is better to strictly correlate config and instance of HTableDescriptor. In proposed patch it is strictly correlated with FSTableDescriptor instance, which was instantiated with some config (config now mandatory parameter). Does it sounds good, or we need to guarantee single instance of HTabelDescriptor across vm? Reduce number of versions in Meta table. Make it configurable - Key: HBASE-12016 URL: https://issues.apache.org/jira/browse/HBASE-12016 Project: HBase Issue Type: Improvement Affects Versions: 2.0.0 Reporter: Andrey Stepachev Assignee: Andrey Stepachev Priority: Minor Fix For: 2.0.0, 0.98.7, 0.99.1 Attachments: HBASE-12016.patch, HBASE-12016.patch Currently meta keeps up to 10 versions of each KV. For big metas it leads to substantial memory overhead and scan slowdowns. (see https://issues.apache.org/jira/browse/HBASE-11165 ) Need to keep reasonable number of versions (suggested value is 3). Number of versions configurable via parameter: hbase.meta.versions -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12016) Reduce number of versions in Meta table. Make it configurable
[ https://issues.apache.org/jira/browse/HBASE-12016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14139243#comment-14139243 ] Andrew Purtell commented on HBASE-12016: Updated fix versions Reduce number of versions in Meta table. Make it configurable - Key: HBASE-12016 URL: https://issues.apache.org/jira/browse/HBASE-12016 Project: HBase Issue Type: Improvement Affects Versions: 2.0.0 Reporter: Andrey Stepachev Assignee: Andrey Stepachev Priority: Minor Fix For: 2.0.0, 0.98.7, 0.99.1 Attachments: HBASE-12016.patch, HBASE-12016.patch Currently meta keeps up to 10 versions of each KV. For big metas it leads to substantial memory overhead and scan slowdowns. (see https://issues.apache.org/jira/browse/HBASE-11165 ) Need to keep reasonable number of versions (suggested value is 3). Number of versions configurable via parameter: hbase.meta.versions -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12016) Reduce number of versions in Meta table. Make it configurable
[ https://issues.apache.org/jira/browse/HBASE-12016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14139787#comment-14139787 ] Virag Kothari commented on HBASE-12016: --- bq. I can create static HTableDescriptor META_TABLEDESC back, just for compatibility and with deprecation warning. HBase codebase will not use this instance. Does it sounds good? Good for me. One minor issue is if they query the HTableDescriptor.META_TABLEDESC for versions, will they get 3 (the default)? Existing 0.98 deployments will have meta with 10 versions. So, should we have 10 as default for 0.98? bq. Does it sounds good, or we need to guarantee single instance of HTabelDescriptor across vm? I see what you are saying. Your approach looks good to me. Thanks for the explanation. Reduce number of versions in Meta table. Make it configurable - Key: HBASE-12016 URL: https://issues.apache.org/jira/browse/HBASE-12016 Project: HBase Issue Type: Improvement Affects Versions: 2.0.0 Reporter: Andrey Stepachev Assignee: Andrey Stepachev Priority: Minor Fix For: 2.0.0, 0.98.7, 0.99.1 Attachments: HBASE-12016.patch, HBASE-12016.patch Currently meta keeps up to 10 versions of each KV. For big metas it leads to substantial memory overhead and scan slowdowns. (see https://issues.apache.org/jira/browse/HBASE-11165 ) Need to keep reasonable number of versions (suggested value is 3). Number of versions configurable via parameter: hbase.meta.versions -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12016) Reduce number of versions in Meta table. Make it configurable
[ https://issues.apache.org/jira/browse/HBASE-12016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14139799#comment-14139799 ] stack commented on HBASE-12016: --- Undoing the hardcoding of meta is the right way to go. I see this patch as fine for now but imagine we could be in for some interesting times when meta actually changes an attribute Lets see how it goes.. Patch looks good to me except for the nit that meta is defined in FSTableDescriptor rather than say in TableDescriptor. If we have another registry implementation, it would have to copy the FSTD definition for meta? Reduce number of versions in Meta table. Make it configurable - Key: HBASE-12016 URL: https://issues.apache.org/jira/browse/HBASE-12016 Project: HBase Issue Type: Improvement Affects Versions: 2.0.0 Reporter: Andrey Stepachev Assignee: Andrey Stepachev Priority: Minor Fix For: 2.0.0, 0.98.7, 0.99.1 Attachments: HBASE-12016.patch, HBASE-12016.patch Currently meta keeps up to 10 versions of each KV. For big metas it leads to substantial memory overhead and scan slowdowns. (see https://issues.apache.org/jira/browse/HBASE-11165 ) Need to keep reasonable number of versions (suggested value is 3). Number of versions configurable via parameter: hbase.meta.versions -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12016) Reduce number of versions in Meta table. Make it configurable
[ https://issues.apache.org/jira/browse/HBASE-12016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14140029#comment-14140029 ] Andrey Stepachev commented on HBASE-12016: -- [~virag] bq. Good for me. One minor issue is if they query the HTableDescriptor.META_TABLEDESC for versions, will they get 3 (the default)? Existing 0.98 deployments will have meta with 10 versions. So, should we have 10 as default for 0.98? I think that should be done in back porting phase. I'm sure it is a good idea to keep same value for old versions. [~stack] Thank you for looking at the patch. bq. Undoing the hardcoding of meta is the right way to go. I see this patch as fine for now but imagine we could be in for some interesting times when meta actually changes an attribute Lets see how it goes.. That was why I've implemented different access stories. Client side will always go through getDescriptor(TableName.META_TABLE), so meta will have the same method of descriptor resolution as regular tables. That gives identical descriptors across the cluster. For server side it is a bit awkward, because it depends on config and looks more like static thing. I think we have to introduce, say, TableManager, and put all tables in it (real or virtual). I'm working on proposal, but that is out of the scope of this patch. bq. Patch looks good to me except for the nit that meta is defined in FSTableDescriptor rather than say in TableDescriptor. If we have another registry implementation, it would have to copy the FSTD definition for meta? Thank you for good suggestion! Moved. Reduce number of versions in Meta table. Make it configurable - Key: HBASE-12016 URL: https://issues.apache.org/jira/browse/HBASE-12016 Project: HBase Issue Type: Improvement Affects Versions: 2.0.0 Reporter: Andrey Stepachev Assignee: Andrey Stepachev Priority: Minor Fix For: 2.0.0, 0.98.7, 0.99.1 Attachments: HBASE-12016.patch, HBASE-12016.patch Currently meta keeps up to 10 versions of each KV. For big metas it leads to substantial memory overhead and scan slowdowns. (see https://issues.apache.org/jira/browse/HBASE-11165 ) Need to keep reasonable number of versions (suggested value is 3). Number of versions configurable via parameter: hbase.meta.versions -- This message was sent by Atlassian JIRA (v6.3.4#6332)
