[jira] [Commented] (HBASE-14727) Add block cache stats for regions
[
https://issues.apache.org/jira/browse/HBASE-14727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983858#comment-14983858
]
Hadoop QA commented on HBASE-14727:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12769909/HBASE-14727-v1.patch
against master branch at commit 683f84e6a217dfd872e5f1be82c7aa4cdf232ec1.
ATTACHMENT ID: 12769909
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 22 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1748 checkstyle errors (more than the master's current 1730 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+result = result && (Float.floatToIntBits(getCacheHitRatio())==
Float.floatToIntBits(other.getCacheHitRatio()));
+ result = result && (hasCacheHitRatioLatestNPeriods() ==
other.hasCacheHitRatioLatestNPeriods());
+ new java.lang.String[] { "RegionSpecifier", "Stores",
"Storefiles", "StoreUncompressedSizeMB", "StorefileSizeMB", "MemstoreSizeMB",
"StorefileIndexSizeMB", "ReadRequestsCount", "WriteRequestsCount",
"TotalCompactingKVs", "CurrentCompactedKVs", "RootIndexSizeKB",
"TotalStaticIndexSizeKB", "TotalStaticBloomSizeKB", "CompleteSequenceId",
"DataLocality", "LastMajorCompactionTs", "StoreCompleteSequenceId",
"CacheHitCount", "CacheMissCount", "CacheEvictedBlockCount", "CacheSize",
"CacheBlockCount", "CacheHitRatio", "CacheHitRatioLatestNPeriods", });
+ BlockCacheKey cacheKey = new
BlockCacheKey(BlockCacheKey.parseRegion(path), name, metaBlockOffset,
+BlockCacheKey cacheKey = new
BlockCacheKey(BlockCacheKey.parseRegion(path), name, dataBlockOffset,
+stats.evicted(block.getCacheKey().getRegion(), block.getCachedTime(),
block.getCacheKey().isPrimary());
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16328//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16328//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16328//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16328//console
This message is automatically generated.
> Add block cache stats for regions
> -
>
> Key: HBASE-14727
> URL: https://issues.apache.org/jira/browse/HBASE-14727
> Project: HBase
> Issue Type: New Feature
>Reporter: Eungsop Yoo
>Priority: Minor
> Attachments: HBASE-14727-v1.patch, HBASE-14727.patch, rs-web-ui.png
>
>
> The stats of block cache are calculated at region server level only. So at
> region level, we could not know about the numbers of blocks cached, the sizes
> of cached blocks, the numbers of cache hit, etc. I suggest to add the stats
> of block cache at region level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14703) update the per-region stats twice for the call on return
[
https://issues.apache.org/jira/browse/HBASE-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-14703:
--
Attachment: HBASE-14703-async.patch
I upload a sample patch, suggestions? [~jesse_yates]
> update the per-region stats twice for the call on return
>
>
> Key: HBASE-14703
> URL: https://issues.apache.org/jira/browse/HBASE-14703
> Project: HBase
> Issue Type: Bug
>Reporter: Heng Chen
>Assignee: Heng Chen
> Attachments: HBASE-14703-async.patch, HBASE-14703-start.patch,
> HBASE-14703.patch
>
>
> In {{AsyncProcess.SingleServerRequestRunnable}}, it seems we update
> serverStatistics twice.
> The first one is that we wrapper {{RetryingCallable}} by
> {{StatsTrackingRpcRetryingCaller}}, and do serverStatistics update when we
> call {{callWithRetries}} and {{callWithoutRetries}}. Relates code like below:
> {code}
> @Override
> public T callWithRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> @Override
> public T callWithoutRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> {code}
> The secondary one is after we get response, in {{receiveMultiAction}}, we do
> update again.
> {code}
> // update the stats about the region, if its a user table. We don't want to
> slow down
> // updates to meta tables, especially from internal updates (master, etc).
> if (AsyncProcess.this.connection.getStatisticsTracker() != null) {
> result = ResultStatsUtil.updateStats(result,
> AsyncProcess.this.connection.getStatisticsTracker(), server, regionName);
> }
> {code}
> It seems that {{StatsTrackingRpcRetryingCaller}} is NOT necessary, remove it?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14733) Minor typo in alter_namespace.rb
[
https://issues.apache.org/jira/browse/HBASE-14733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983848#comment-14983848
]
Hudson commented on HBASE-14733:
SUCCESS: Integrated in HBase-1.2-IT #256 (See
[https://builds.apache.org/job/HBase-1.2-IT/256/])
HBASE-14733 Minor typo in alter_namespace.rb (enis: rev
68eea04745786e829bb1a8a6b4e0a481fa6a7656)
* hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
* hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
> Minor typo in alter_namespace.rb
>
>
> Key: HBASE-14733
> URL: https://issues.apache.org/jira/browse/HBASE-14733
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Trivial
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: hbase-14733.patch
>
>
> {code}
> diff --git hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
> hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
> index 760bbf7..a16e10d 100644
> --- hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
> +++ hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
> @@ -26,11 +26,11 @@ Alter namespace properties.
>
> To add/modify a property:
>
> - hbase> alter_namespace 'ns1', {METHOD => 'set', 'PROERTY_NAME' =>
> 'PROPERTY_VALUE'}
> + hbase> alter_namespace 'ns1', {METHOD => 'set', 'PROPERTY_NAME' =>
> 'PROPERTY_VALUE'}
>
> To delete a property:
>
> - hbase> alter_namespace 'ns1', {METHOD => 'unset', NAME=>'PROERTY_NAME'}
> + hbase> alter_namespace 'ns1', {METHOD => 'unset', NAME=>'PROPERTY_NAME'}
> EOF
>end
>
> diff --git hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
> hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
> index 3259eb6..adb6897 100644
> --- hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
> +++ hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
> @@ -27,7 +27,7 @@ and optionally a dictionary of namespace configuration.
> Examples:
>
>hbase> create_namespace 'ns1'
> - hbase> create_namespace 'ns1', {'PROERTY_NAME'=>'PROPERTY_VALUE'}
> + hbase> create_namespace 'ns1', {'PROPERTY_NAME'=>'PROPERTY_VALUE'}
> EOF
>end
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14700) Support a "permissive" mode for secure clusters to allow "simple" auth clients
[ https://issues.apache.org/jira/browse/HBASE-14700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983847#comment-14983847 ] Hudson commented on HBASE-14700: SUCCESS: Integrated in HBase-1.2-IT #256 (See [https://builds.apache.org/job/HBase-1.2-IT/256/]) HBASE-14700 Support a permissive mode for secure clusters to allow (garyh: rev f273d147b7ecdc30484e94a8b2fed622e53d7fbd) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * hbase-server/src/test/java/org/apache/hadoop/hbase/security/TestSecureRPC.java * hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSource.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServer.java * hbase-common/src/main/resources/hbase-default.xml * hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSourceImpl.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java > Support a "permissive" mode for secure clusters to allow "simple" auth clients > -- > > Key: HBASE-14700 > URL: https://issues.apache.org/jira/browse/HBASE-14700 > Project: HBase > Issue Type: Improvement > Components: security >Reporter: Gary Helmling >Assignee: Gary Helmling > Fix For: 2.0.0, 1.2.0, 0.98.16 > > Attachments: HBASE-14700-v2.patch, HBASE-14700-v3.patch, > HBASE-14700.patch > > > When implementing HBase security for an existing cluster, it can be useful to > support mixed secure and insecure clients while all client configurations are > migrated over to secure authentication. > We currently have an option to allow secure clients to fallback to simple > auth against insecure clusters. By providing an analogous setting for > servers, we would allow a phased rollout of security: > # First, security can be enabled on the cluster servers, with the > "permissive" mode enabled > # Clients can be converting to using secure authentication incrementally > # The server audit logs allow identification of clients still using simple > auth to connect > # Finally, when sufficient clients have been converted to secure operation, > the server-side "permissive" mode can be removed, allowing completely secure > operation. > Obviously with this enabled, there is no effective access control, but this > would still be a useful tool to enable a smooth operational rollout of > security. Permissive mode would of course be disabled by default. Enabling > it should provide a big scary warning in the logs on startup, and possibly be > flagged on relevant UIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14725) Vet categorization of tests so they for sure go into the right small/medium/large buckets
[ https://issues.apache.org/jira/browse/HBASE-14725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14725: -- Attachment: 14725v4.txt Retry to ensure not a fluke > Vet categorization of tests so they for sure go into the right > small/medium/large buckets > - > > Key: HBASE-14725 > URL: https://issues.apache.org/jira/browse/HBASE-14725 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > Attachments: 14725v2 (1).txt, 14725v2 (1).txt, 14725v2.txt, > 14725v2.txt, 14725v4.txt, 14725v4.txt, categorization.patch > > > I tried doing runSmallTests, runMediumTests, etc., and I noticed that some > tests are larger than our categorization. At least for small tests it means > they area all running in the one JVM. I also noticed that the categorization > only takes effect in hbase-server. > This patch makes it so runSmallTests runs all the small tests only in each > category. Also moves tests that were larger than small out to medium so they > don't run in the one JVM anymore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14722) Update surefire to 2.19
[ https://issues.apache.org/jira/browse/HBASE-14722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983831#comment-14983831 ] Hudson commented on HBASE-14722: FAILURE: Integrated in HBase-Trunk_matrix #415 (See [https://builds.apache.org/job/HBase-Trunk_matrix/415/]) Revert "HBASE-14722 Update surefire to 2.19" (stack: rev c6d3dcdea33efa4fa745b323f6ff46b8b0711309) * pom.xml > Update surefire to 2.19 > --- > > Key: HBASE-14722 > URL: https://issues.apache.org/jira/browse/HBASE-14722 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack >Assignee: stack > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: surefire.patch > > > Bug fixes: > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12317927&version=12331679 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14733) Minor typo in alter_namespace.rb
[
https://issues.apache.org/jira/browse/HBASE-14733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983832#comment-14983832
]
Hudson commented on HBASE-14733:
FAILURE: Integrated in HBase-Trunk_matrix #415 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/415/])
HBASE-14733 Minor typo in alter_namespace.rb (enis: rev
4534f8ed0c60974bbd1d3343248735b9f60e0082)
* hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
* hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
> Minor typo in alter_namespace.rb
>
>
> Key: HBASE-14733
> URL: https://issues.apache.org/jira/browse/HBASE-14733
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Trivial
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: hbase-14733.patch
>
>
> {code}
> diff --git hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
> hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
> index 760bbf7..a16e10d 100644
> --- hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
> +++ hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
> @@ -26,11 +26,11 @@ Alter namespace properties.
>
> To add/modify a property:
>
> - hbase> alter_namespace 'ns1', {METHOD => 'set', 'PROERTY_NAME' =>
> 'PROPERTY_VALUE'}
> + hbase> alter_namespace 'ns1', {METHOD => 'set', 'PROPERTY_NAME' =>
> 'PROPERTY_VALUE'}
>
> To delete a property:
>
> - hbase> alter_namespace 'ns1', {METHOD => 'unset', NAME=>'PROERTY_NAME'}
> + hbase> alter_namespace 'ns1', {METHOD => 'unset', NAME=>'PROPERTY_NAME'}
> EOF
>end
>
> diff --git hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
> hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
> index 3259eb6..adb6897 100644
> --- hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
> +++ hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
> @@ -27,7 +27,7 @@ and optionally a dictionary of namespace configuration.
> Examples:
>
>hbase> create_namespace 'ns1'
> - hbase> create_namespace 'ns1', {'PROERTY_NAME'=>'PROPERTY_VALUE'}
> + hbase> create_namespace 'ns1', {'PROPERTY_NAME'=>'PROPERTY_VALUE'}
> EOF
>end
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14703) update the per-region stats twice for the call on return
[
https://issues.apache.org/jira/browse/HBASE-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983830#comment-14983830
]
Heng Chen commented on HBASE-14703:
---
IMO the point confused us is there are two ways we make communications with
server.
One sync way we do RPC call directly and client will block.
The other seems to be an async way using asyncProcess.
So we have to collect stats in two places.
But {{HTable.batch}} method seems to be sync although it use asyncProcess
{code}
public void batch(final List actions, final Object[] results)
throws InterruptedException, IOException {
AsyncRequestFuture ars = multiAp.submitAll(pool, tableName, actions, null,
results);
ars.waitUntilDone();
if (ars.hasError()) {
throw ars.getErrors();
}
}
{code}
So can we change all method to use AsyncProcess to do communications with
server ?
As you mentioned, {{HTable.mutateRow}} should return List instead of
one Result. Then we parse the information in {{StatsTrackingRpcRetryingCaller}}.
But If we use asyncProcess to do that, it seems to be more simple.
Any concerns?
> update the per-region stats twice for the call on return
>
>
> Key: HBASE-14703
> URL: https://issues.apache.org/jira/browse/HBASE-14703
> Project: HBase
> Issue Type: Bug
>Reporter: Heng Chen
>Assignee: Heng Chen
> Attachments: HBASE-14703-start.patch, HBASE-14703.patch
>
>
> In {{AsyncProcess.SingleServerRequestRunnable}}, it seems we update
> serverStatistics twice.
> The first one is that we wrapper {{RetryingCallable}} by
> {{StatsTrackingRpcRetryingCaller}}, and do serverStatistics update when we
> call {{callWithRetries}} and {{callWithoutRetries}}. Relates code like below:
> {code}
> @Override
> public T callWithRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> @Override
> public T callWithoutRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> {code}
> The secondary one is after we get response, in {{receiveMultiAction}}, we do
> update again.
> {code}
> // update the stats about the region, if its a user table. We don't want to
> slow down
> // updates to meta tables, especially from internal updates (master, etc).
> if (AsyncProcess.this.connection.getStatisticsTracker() != null) {
> result = ResultStatsUtil.updateStats(result,
> AsyncProcess.this.connection.getStatisticsTracker(), server, regionName);
> }
> {code}
> It seems that {{StatsTrackingRpcRetryingCaller}} is NOT necessary, remove it?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14725) Vet categorization of tests so they for sure go into the right small/medium/large buckets
[
https://issues.apache.org/jira/browse/HBASE-14725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983823#comment-14983823
]
Hadoop QA commented on HBASE-14725:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12769907/14725v4.txt
against master branch at commit 683f84e6a217dfd872e5f1be82c7aa4cdf232ec1.
ATTACHMENT ID: 12769907
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 27 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16329//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16329//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16329//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16329//console
This message is automatically generated.
> Vet categorization of tests so they for sure go into the right
> small/medium/large buckets
> -
>
> Key: HBASE-14725
> URL: https://issues.apache.org/jira/browse/HBASE-14725
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: stack
> Attachments: 14725v2 (1).txt, 14725v2 (1).txt, 14725v2.txt,
> 14725v2.txt, 14725v4.txt, categorization.patch
>
>
> I tried doing runSmallTests, runMediumTests, etc., and I noticed that some
> tests are larger than our categorization. At least for small tests it means
> they area all running in the one JVM. I also noticed that the categorization
> only takes effect in hbase-server.
> This patch makes it so runSmallTests runs all the small tests only in each
> category. Also moves tests that were larger than small out to medium so they
> don't run in the one JVM anymore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11386) Replication#table,CF config will be wrong if the table name includes namespace
[
https://issues.apache.org/jira/browse/HBASE-11386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983821#comment-14983821
]
Qianxi Zhang commented on HBASE-11386:
--
Thanks [~ashish singhi]
I will be appreciate for your work!
> Replication#table,CF config will be wrong if the table name includes namespace
> --
>
> Key: HBASE-11386
> URL: https://issues.apache.org/jira/browse/HBASE-11386
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Qianxi Zhang
>Assignee: Qianxi Zhang
>Priority: Critical
> Attachments: HBASE_11386_trunk_v1.patch, HBASE_11386_trunk_v2.patch
>
>
> Now we can config the table and CF in Replication, but I think the parse will
> be wrong if the table name includes namespace
> ReplicationPeer#parseTableCFsFromConfig(line 125)
> {code}
> Map> tableCFsMap = null;
> // parse out (table, cf-list) pairs from tableCFsConfig
> // format: "table1:cf1,cf2;table2:cfA,cfB"
> String[] tables = tableCFsConfig.split(";");
> for (String tab : tables) {
> // 1 ignore empty table config
> tab = tab.trim();
> if (tab.length() == 0) {
> continue;
> }
> // 2 split to "table" and "cf1,cf2"
> // for each table: "table:cf1,cf2" or "table"
> String[] pair = tab.split(":");
> String tabName = pair[0].trim();
> if (pair.length > 2 || tabName.length() == 0) {
> LOG.error("ignore invalid tableCFs setting: " + tab);
> continue;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14733) Minor typo in alter_namespace.rb
[
https://issues.apache.org/jira/browse/HBASE-14733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983816#comment-14983816
]
Hudson commented on HBASE-14733:
SUCCESS: Integrated in HBase-1.3-IT #285 (See
[https://builds.apache.org/job/HBase-1.3-IT/285/])
HBASE-14733 Minor typo in alter_namespace.rb (enis: rev
b8a2caf159f74421c4cb786d3e68c2fe98d0918a)
* hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
* hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
> Minor typo in alter_namespace.rb
>
>
> Key: HBASE-14733
> URL: https://issues.apache.org/jira/browse/HBASE-14733
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Trivial
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: hbase-14733.patch
>
>
> {code}
> diff --git hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
> hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
> index 760bbf7..a16e10d 100644
> --- hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
> +++ hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
> @@ -26,11 +26,11 @@ Alter namespace properties.
>
> To add/modify a property:
>
> - hbase> alter_namespace 'ns1', {METHOD => 'set', 'PROERTY_NAME' =>
> 'PROPERTY_VALUE'}
> + hbase> alter_namespace 'ns1', {METHOD => 'set', 'PROPERTY_NAME' =>
> 'PROPERTY_VALUE'}
>
> To delete a property:
>
> - hbase> alter_namespace 'ns1', {METHOD => 'unset', NAME=>'PROERTY_NAME'}
> + hbase> alter_namespace 'ns1', {METHOD => 'unset', NAME=>'PROPERTY_NAME'}
> EOF
>end
>
> diff --git hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
> hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
> index 3259eb6..adb6897 100644
> --- hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
> +++ hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
> @@ -27,7 +27,7 @@ and optionally a dictionary of namespace configuration.
> Examples:
>
>hbase> create_namespace 'ns1'
> - hbase> create_namespace 'ns1', {'PROERTY_NAME'=>'PROPERTY_VALUE'}
> + hbase> create_namespace 'ns1', {'PROPERTY_NAME'=>'PROPERTY_VALUE'}
> EOF
>end
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14687) Un-synchronize BufferedMutator
[ https://issues.apache.org/jira/browse/HBASE-14687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983807#comment-14983807 ] Hudson commented on HBASE-14687: FAILURE: Integrated in HBase-1.3 #329 (See [https://builds.apache.org/job/HBase-1.3/329/]) HBASE-14687 Un-synchronize BufferedMutator (eclark: rev 3bdfa782f34666423254735591c9fd017cc8f3cd) * hbase-client/src/main/java/org/apache/hadoop/hbase/client/BufferedMutatorImpl.java * hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestAsyncProcess.java > Un-synchronize BufferedMutator > -- > > Key: HBASE-14687 > URL: https://issues.apache.org/jira/browse/HBASE-14687 > Project: HBase > Issue Type: Improvement > Components: Client, Performance >Affects Versions: 2.0.0, 1.2.0 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Critical > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14687-v1.patch, HBASE-14687-v2.patch, > HBASE-14687-v3.patch, HBASE-14687-v4.patch, HBASE-14687-v5.patch, > HBASE-14687.patch > > > It should totally be possible to make BufferedMutatorImpl not use much > locking. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14722) Update surefire to 2.19
[ https://issues.apache.org/jira/browse/HBASE-14722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983806#comment-14983806 ] Hudson commented on HBASE-14722: FAILURE: Integrated in HBase-1.3 #329 (See [https://builds.apache.org/job/HBase-1.3/329/]) Revert "HBASE-14722 Update surefire to 2.19" (stack: rev c48d16ae2b4e8884a5af39b08080f27a2c03159b) * pom.xml > Update surefire to 2.19 > --- > > Key: HBASE-14722 > URL: https://issues.apache.org/jira/browse/HBASE-14722 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack >Assignee: stack > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: surefire.patch > > > Bug fixes: > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12317927&version=12331679 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14722) Update surefire to 2.19
[ https://issues.apache.org/jira/browse/HBASE-14722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983801#comment-14983801 ] Hudson commented on HBASE-14722: FAILURE: Integrated in HBase-1.2 #329 (See [https://builds.apache.org/job/HBase-1.2/329/]) Revert "HBASE-14722 Update surefire to 2.19" (stack: rev 3aca67cc5da750115c05b2cebc14226a0f9bf624) * pom.xml > Update surefire to 2.19 > --- > > Key: HBASE-14722 > URL: https://issues.apache.org/jira/browse/HBASE-14722 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack >Assignee: stack > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: surefire.patch > > > Bug fixes: > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12317927&version=12331679 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-6721) RegionServer Group based Assignment
[
https://issues.apache.org/jira/browse/HBASE-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983799#comment-14983799
]
Hadoop QA commented on HBASE-6721:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12769912/hbase-6721-v16.patch
against master branch at commit 683f84e6a217dfd872e5f1be82c7aa4cdf232ec1.
ATTACHMENT ID: 12769912
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 33 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 4
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1755 checkstyle errors (more than the master's current 1730 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+ * rpc GetGroupInfo(.hbase.pb.GetGroupInfoRequest) returns
(.hbase.pb.GetGroupInfoResponse);
+ * rpc GetGroupInfoOfTable(.hbase.pb.GetGroupInfoOfTableRequest)
returns (.hbase.pb.GetGroupInfoOfTableResponse);
+ * rpc GetGroupInfoOfServer(.hbase.pb.GetGroupInfoOfServerRequest)
returns (.hbase.pb.GetGroupInfoOfServerResponse);
+ * rpc MoveServers(.hbase.pb.MoveServersRequest) returns
(.hbase.pb.MoveServersResponse);
+ * rpc MoveTables(.hbase.pb.MoveTablesRequest) returns
(.hbase.pb.MoveTablesResponse);
+ * rpc RemoveGroup(.hbase.pb.RemoveGroupRequest) returns
(.hbase.pb.RemoveGroupResponse);
+ * rpc BalanceGroup(.hbase.pb.BalanceGroupRequest) returns
(.hbase.pb.BalanceGroupResponse);
+ * rpc ListGroupInfos(.hbase.pb.ListGroupInfosRequest) returns
(.hbase.pb.ListGroupInfosResponse);
+ * rpc GetGroupInfo(.hbase.pb.GetGroupInfoRequest) returns
(.hbase.pb.GetGroupInfoResponse);
+ * rpc GetGroupInfoOfTable(.hbase.pb.GetGroupInfoOfTableRequest)
returns (.hbase.pb.GetGroupInfoOfTableResponse);
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestAsyncProcess
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16327//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16327//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16327//artifact/patchprocess/checkstyle-aggregate.html
Javadoc warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16327//artifact/patchprocess/patchJavadocWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16327//console
This message is automatically generated.
> RegionServer Group based Assignment
> ---
>
> Key: HBASE-6721
> URL: https://issues.apache.org/jira/browse/HBASE-6721
> Project: HBase
> Issue Type: New Feature
>Reporter: Francis Liu
>Assignee: Francis Liu
> Labels: hbase-6721
> Attachments: 6721-master-webUI.patch, HBASE-6721
> GroupBasedLoadBalancer Sequence Diagram.xml, HBASE-6721-DesigDoc.pdf,
> HBASE-6721-DesigDoc.pdf, HBASE-6721-DesigDoc.pdf, HBASE-6721-DesigDoc.pdf,
> HBASE-6721_0.98_2.patch, HBASE-6721_10.patch, HBASE-6721_11.patch,
> HBASE-6721_12.patch, HBASE-6721_13.patch, HBASE-6721_14.patch,
> HBASE-6721_15.patch, HBASE-6721_8.patch, HBASE-6721_9.patch,
> HBASE-6721_9.patch, HBASE-6721_94.patch, HBASE-6721_94.patch,
> HBASE-6721_94_2.patch, HBASE-6721_94_3.patch, HBASE-6721_94_3.patch,
> HBASE-6721_94_4.patch, HBASE-6721_94_5.patch, HBASE-6721_94_6.patch,
> HBASE-6721_94_7.patch, HBASE-6721_98_1.patch, HBASE-6721_98_2.patch,
> HBASE-6721_hbase-6721_addendum.patch, HBASE-6721_trunk.patch,
> HBASE-6721_trunk.patch, HBASE-6721_trunk.patch, HBASE-6721_trunk1.patch,
> HBASE-6721_trunk2.patch, balanceCluster Sequence Diagram.svg,
> hbase-6721-v15-branch-1.1.patch, hbase-6721-v16.patch, immediateAssignments
> Sequence Diagram.svg, randomAssignment Sequence D
[jira] [Commented] (HBASE-14700) Support a "permissive" mode for secure clusters to allow "simple" auth clients
[ https://issues.apache.org/jira/browse/HBASE-14700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983779#comment-14983779 ] Gary Helmling commented on HBASE-14700: --- Committed to master, branch-1 and branch-1.2. Still working on applying to 0.98, it requires a bit more change there. > Support a "permissive" mode for secure clusters to allow "simple" auth clients > -- > > Key: HBASE-14700 > URL: https://issues.apache.org/jira/browse/HBASE-14700 > Project: HBase > Issue Type: Improvement > Components: security >Reporter: Gary Helmling >Assignee: Gary Helmling > Fix For: 2.0.0, 1.2.0, 0.98.16 > > Attachments: HBASE-14700-v2.patch, HBASE-14700-v3.patch, > HBASE-14700.patch > > > When implementing HBase security for an existing cluster, it can be useful to > support mixed secure and insecure clients while all client configurations are > migrated over to secure authentication. > We currently have an option to allow secure clients to fallback to simple > auth against insecure clusters. By providing an analogous setting for > servers, we would allow a phased rollout of security: > # First, security can be enabled on the cluster servers, with the > "permissive" mode enabled > # Clients can be converting to using secure authentication incrementally > # The server audit logs allow identification of clients still using simple > auth to connect > # Finally, when sufficient clients have been converted to secure operation, > the server-side "permissive" mode can be removed, allowing completely secure > operation. > Obviously with this enabled, there is no effective access control, but this > would still be a useful tool to enable a smooth operational rollout of > security. Permissive mode would of course be disabled by default. Enabling > it should provide a big scary warning in the logs on startup, and possibly be > flagged on relevant UIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14727) Add block cache stats for regions
[ https://issues.apache.org/jira/browse/HBASE-14727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-14727: Status: Open (was: Patch Available) > Add block cache stats for regions > - > > Key: HBASE-14727 > URL: https://issues.apache.org/jira/browse/HBASE-14727 > Project: HBase > Issue Type: New Feature >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-14727-v1.patch, HBASE-14727.patch, rs-web-ui.png > > > The stats of block cache are calculated at region server level only. So at > region level, we could not know about the numbers of blocks cached, the sizes > of cached blocks, the numbers of cache hit, etc. I suggest to add the stats > of block cache at region level. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14727) Add block cache stats for regions
[ https://issues.apache.org/jira/browse/HBASE-14727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-14727: Status: Patch Available (was: Open) > Add block cache stats for regions > - > > Key: HBASE-14727 > URL: https://issues.apache.org/jira/browse/HBASE-14727 > Project: HBase > Issue Type: New Feature >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-14727-v1.patch, HBASE-14727.patch, rs-web-ui.png > > > The stats of block cache are calculated at region server level only. So at > region level, we could not know about the numbers of blocks cached, the sizes > of cached blocks, the numbers of cache hit, etc. I suggest to add the stats > of block cache at region level. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14687) Un-synchronize BufferedMutator
[ https://issues.apache.org/jira/browse/HBASE-14687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983748#comment-14983748 ] Hudson commented on HBASE-14687: SUCCESS: Integrated in HBase-1.2-IT #255 (See [https://builds.apache.org/job/HBase-1.2-IT/255/]) HBASE-14687 Un-synchronize BufferedMutator (eclark: rev 6308401b6a26551629dfec772fa13408a39936f4) * hbase-client/src/main/java/org/apache/hadoop/hbase/client/BufferedMutatorImpl.java * hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestAsyncProcess.java > Un-synchronize BufferedMutator > -- > > Key: HBASE-14687 > URL: https://issues.apache.org/jira/browse/HBASE-14687 > Project: HBase > Issue Type: Improvement > Components: Client, Performance >Affects Versions: 2.0.0, 1.2.0 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Critical > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14687-v1.patch, HBASE-14687-v2.patch, > HBASE-14687-v3.patch, HBASE-14687-v4.patch, HBASE-14687-v5.patch, > HBASE-14687.patch > > > It should totally be possible to make BufferedMutatorImpl not use much > locking. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14722) Update surefire to 2.19
[ https://issues.apache.org/jira/browse/HBASE-14722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983747#comment-14983747 ] Hudson commented on HBASE-14722: SUCCESS: Integrated in HBase-1.2-IT #255 (See [https://builds.apache.org/job/HBase-1.2-IT/255/]) Revert "HBASE-14722 Update surefire to 2.19" (stack: rev 3aca67cc5da750115c05b2cebc14226a0f9bf624) * pom.xml > Update surefire to 2.19 > --- > > Key: HBASE-14722 > URL: https://issues.apache.org/jira/browse/HBASE-14722 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack >Assignee: stack > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: surefire.patch > > > Bug fixes: > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12317927&version=12331679 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-6721) RegionServer Group based Assignment
[ https://issues.apache.org/jira/browse/HBASE-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-6721: - Attachment: hbase-6721-v16.patch Here is a master patch that is: - rebased to latest master. - Addresses (1), (10), (11), and (12) as above. > RegionServer Group based Assignment > --- > > Key: HBASE-6721 > URL: https://issues.apache.org/jira/browse/HBASE-6721 > Project: HBase > Issue Type: New Feature >Reporter: Francis Liu >Assignee: Francis Liu > Labels: hbase-6721 > Attachments: 6721-master-webUI.patch, HBASE-6721 > GroupBasedLoadBalancer Sequence Diagram.xml, HBASE-6721-DesigDoc.pdf, > HBASE-6721-DesigDoc.pdf, HBASE-6721-DesigDoc.pdf, HBASE-6721-DesigDoc.pdf, > HBASE-6721_0.98_2.patch, HBASE-6721_10.patch, HBASE-6721_11.patch, > HBASE-6721_12.patch, HBASE-6721_13.patch, HBASE-6721_14.patch, > HBASE-6721_15.patch, HBASE-6721_8.patch, HBASE-6721_9.patch, > HBASE-6721_9.patch, HBASE-6721_94.patch, HBASE-6721_94.patch, > HBASE-6721_94_2.patch, HBASE-6721_94_3.patch, HBASE-6721_94_3.patch, > HBASE-6721_94_4.patch, HBASE-6721_94_5.patch, HBASE-6721_94_6.patch, > HBASE-6721_94_7.patch, HBASE-6721_98_1.patch, HBASE-6721_98_2.patch, > HBASE-6721_hbase-6721_addendum.patch, HBASE-6721_trunk.patch, > HBASE-6721_trunk.patch, HBASE-6721_trunk.patch, HBASE-6721_trunk1.patch, > HBASE-6721_trunk2.patch, balanceCluster Sequence Diagram.svg, > hbase-6721-v15-branch-1.1.patch, hbase-6721-v16.patch, immediateAssignments > Sequence Diagram.svg, randomAssignment Sequence Diagram.svg, retainAssignment > Sequence Diagram.svg, roundRobinAssignment Sequence Diagram.svg > > > In multi-tenant deployments of HBase, it is likely that a RegionServer will > be serving out regions from a number of different tables owned by various > client applications. Being able to group a subset of running RegionServers > and assign specific tables to it, provides a client application a level of > isolation and resource allocation. > The proposal essentially is to have an AssignmentManager which is aware of > RegionServer groups and assigns tables to region servers based on groupings. > Load balancing will occur on a per group basis as well. > This is essentially a simplification of the approach taken in HBASE-4120. See > attached document. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-6721) RegionServer Group based Assignment
[ https://issues.apache.org/jira/browse/HBASE-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983743#comment-14983743 ] Enis Soztutar commented on HBASE-6721: -- We may have a problem with IntegrationTestGroup.testMoveServers. 12. Also the test leaves the balancer turned off, which should be be the case. > RegionServer Group based Assignment > --- > > Key: HBASE-6721 > URL: https://issues.apache.org/jira/browse/HBASE-6721 > Project: HBase > Issue Type: New Feature >Reporter: Francis Liu >Assignee: Francis Liu > Labels: hbase-6721 > Attachments: 6721-master-webUI.patch, HBASE-6721 > GroupBasedLoadBalancer Sequence Diagram.xml, HBASE-6721-DesigDoc.pdf, > HBASE-6721-DesigDoc.pdf, HBASE-6721-DesigDoc.pdf, HBASE-6721-DesigDoc.pdf, > HBASE-6721_0.98_2.patch, HBASE-6721_10.patch, HBASE-6721_11.patch, > HBASE-6721_12.patch, HBASE-6721_13.patch, HBASE-6721_14.patch, > HBASE-6721_15.patch, HBASE-6721_8.patch, HBASE-6721_9.patch, > HBASE-6721_9.patch, HBASE-6721_94.patch, HBASE-6721_94.patch, > HBASE-6721_94_2.patch, HBASE-6721_94_3.patch, HBASE-6721_94_3.patch, > HBASE-6721_94_4.patch, HBASE-6721_94_5.patch, HBASE-6721_94_6.patch, > HBASE-6721_94_7.patch, HBASE-6721_98_1.patch, HBASE-6721_98_2.patch, > HBASE-6721_hbase-6721_addendum.patch, HBASE-6721_trunk.patch, > HBASE-6721_trunk.patch, HBASE-6721_trunk.patch, HBASE-6721_trunk1.patch, > HBASE-6721_trunk2.patch, balanceCluster Sequence Diagram.svg, > hbase-6721-v15-branch-1.1.patch, immediateAssignments Sequence Diagram.svg, > randomAssignment Sequence Diagram.svg, retainAssignment Sequence Diagram.svg, > roundRobinAssignment Sequence Diagram.svg > > > In multi-tenant deployments of HBase, it is likely that a RegionServer will > be serving out regions from a number of different tables owned by various > client applications. Being able to group a subset of running RegionServers > and assign specific tables to it, provides a client application a level of > isolation and resource allocation. > The proposal essentially is to have an AssignmentManager which is aware of > RegionServer groups and assigns tables to region servers based on groupings. > Load balancing will occur on a per group basis as well. > This is essentially a simplification of the approach taken in HBASE-4120. See > attached document. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-6721) RegionServer Group based Assignment
[ https://issues.apache.org/jira/browse/HBASE-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983740#comment-14983740 ] Enis Soztutar commented on HBASE-6721: -- Even more testing: Test 4: Create namespace with hbase.rsgroup.name pointing to a group, then create a table in that namespace. Works as expected, the new table created in correct group Run the integration test on the 7 node cluster: hbase org.apache.hadoop.hbase.IntegrationTestsDriver -regex IntegrationTestGroup Failed. Inspecting. 8. I was thinking of testing to remove the default group, but did not do it yet. Reading of the code seems to allow it. We should fix that. 9. We can add a description to the rsgroup table (minor). The master UI displays the description. 10. In IntegrationTestGroup.java, the wait condition should not wait for exactly NUM_SLAVES_BASE, there can be more servers that NUM_SLAVES_BASE in an actual cluster. 11. Once balancer is turned off, there is no way to turn it on since GroupAdminEndpoint.preBalanceSwitch() always returns false. > RegionServer Group based Assignment > --- > > Key: HBASE-6721 > URL: https://issues.apache.org/jira/browse/HBASE-6721 > Project: HBase > Issue Type: New Feature >Reporter: Francis Liu >Assignee: Francis Liu > Labels: hbase-6721 > Attachments: 6721-master-webUI.patch, HBASE-6721 > GroupBasedLoadBalancer Sequence Diagram.xml, HBASE-6721-DesigDoc.pdf, > HBASE-6721-DesigDoc.pdf, HBASE-6721-DesigDoc.pdf, HBASE-6721-DesigDoc.pdf, > HBASE-6721_0.98_2.patch, HBASE-6721_10.patch, HBASE-6721_11.patch, > HBASE-6721_12.patch, HBASE-6721_13.patch, HBASE-6721_14.patch, > HBASE-6721_15.patch, HBASE-6721_8.patch, HBASE-6721_9.patch, > HBASE-6721_9.patch, HBASE-6721_94.patch, HBASE-6721_94.patch, > HBASE-6721_94_2.patch, HBASE-6721_94_3.patch, HBASE-6721_94_3.patch, > HBASE-6721_94_4.patch, HBASE-6721_94_5.patch, HBASE-6721_94_6.patch, > HBASE-6721_94_7.patch, HBASE-6721_98_1.patch, HBASE-6721_98_2.patch, > HBASE-6721_hbase-6721_addendum.patch, HBASE-6721_trunk.patch, > HBASE-6721_trunk.patch, HBASE-6721_trunk.patch, HBASE-6721_trunk1.patch, > HBASE-6721_trunk2.patch, balanceCluster Sequence Diagram.svg, > hbase-6721-v15-branch-1.1.patch, immediateAssignments Sequence Diagram.svg, > randomAssignment Sequence Diagram.svg, retainAssignment Sequence Diagram.svg, > roundRobinAssignment Sequence Diagram.svg > > > In multi-tenant deployments of HBase, it is likely that a RegionServer will > be serving out regions from a number of different tables owned by various > client applications. Being able to group a subset of running RegionServers > and assign specific tables to it, provides a client application a level of > isolation and resource allocation. > The proposal essentially is to have an AssignmentManager which is aware of > RegionServer groups and assigns tables to region servers based on groupings. > Load balancing will occur on a per group basis as well. > This is essentially a simplification of the approach taken in HBASE-4120. See > attached document. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14687) Un-synchronize BufferedMutator
[ https://issues.apache.org/jira/browse/HBASE-14687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983737#comment-14983737 ] Hudson commented on HBASE-14687: SUCCESS: Integrated in HBase-1.3-IT #284 (See [https://builds.apache.org/job/HBase-1.3-IT/284/]) HBASE-14687 Un-synchronize BufferedMutator (eclark: rev 3bdfa782f34666423254735591c9fd017cc8f3cd) * hbase-client/src/main/java/org/apache/hadoop/hbase/client/BufferedMutatorImpl.java * hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestAsyncProcess.java > Un-synchronize BufferedMutator > -- > > Key: HBASE-14687 > URL: https://issues.apache.org/jira/browse/HBASE-14687 > Project: HBase > Issue Type: Improvement > Components: Client, Performance >Affects Versions: 2.0.0, 1.2.0 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Critical > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14687-v1.patch, HBASE-14687-v2.patch, > HBASE-14687-v3.patch, HBASE-14687-v4.patch, HBASE-14687-v5.patch, > HBASE-14687.patch > > > It should totally be possible to make BufferedMutatorImpl not use much > locking. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14722) Update surefire to 2.19
[ https://issues.apache.org/jira/browse/HBASE-14722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983736#comment-14983736 ] Hudson commented on HBASE-14722: SUCCESS: Integrated in HBase-1.3-IT #284 (See [https://builds.apache.org/job/HBase-1.3-IT/284/]) Revert "HBASE-14722 Update surefire to 2.19" (stack: rev c48d16ae2b4e8884a5af39b08080f27a2c03159b) * pom.xml > Update surefire to 2.19 > --- > > Key: HBASE-14722 > URL: https://issues.apache.org/jira/browse/HBASE-14722 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack >Assignee: stack > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: surefire.patch > > > Bug fixes: > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12317927&version=12331679 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14727) Add block cache stats for regions
[ https://issues.apache.org/jira/browse/HBASE-14727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-14727: Attachment: HBASE-14727-v1.patch fix compilation error > Add block cache stats for regions > - > > Key: HBASE-14727 > URL: https://issues.apache.org/jira/browse/HBASE-14727 > Project: HBase > Issue Type: New Feature >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-14727-v1.patch, HBASE-14727.patch, rs-web-ui.png > > > The stats of block cache are calculated at region server level only. So at > region level, we could not know about the numbers of blocks cached, the sizes > of cached blocks, the numbers of cache hit, etc. I suggest to add the stats > of block cache at region level. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14733) Minor typo in alter_namespace.rb
[
https://issues.apache.org/jira/browse/HBASE-14733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-14733:
--
Fix Version/s: 0.98.16
1.1.3
1.0.3
1.3.0
1.2.0
2.0.0
> Minor typo in alter_namespace.rb
>
>
> Key: HBASE-14733
> URL: https://issues.apache.org/jira/browse/HBASE-14733
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Trivial
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: hbase-14733.patch
>
>
> {code}
> diff --git hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
> hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
> index 760bbf7..a16e10d 100644
> --- hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
> +++ hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
> @@ -26,11 +26,11 @@ Alter namespace properties.
>
> To add/modify a property:
>
> - hbase> alter_namespace 'ns1', {METHOD => 'set', 'PROERTY_NAME' =>
> 'PROPERTY_VALUE'}
> + hbase> alter_namespace 'ns1', {METHOD => 'set', 'PROPERTY_NAME' =>
> 'PROPERTY_VALUE'}
>
> To delete a property:
>
> - hbase> alter_namespace 'ns1', {METHOD => 'unset', NAME=>'PROERTY_NAME'}
> + hbase> alter_namespace 'ns1', {METHOD => 'unset', NAME=>'PROPERTY_NAME'}
> EOF
>end
>
> diff --git hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
> hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
> index 3259eb6..adb6897 100644
> --- hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
> +++ hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
> @@ -27,7 +27,7 @@ and optionally a dictionary of namespace configuration.
> Examples:
>
>hbase> create_namespace 'ns1'
> - hbase> create_namespace 'ns1', {'PROERTY_NAME'=>'PROPERTY_VALUE'}
> + hbase> create_namespace 'ns1', {'PROPERTY_NAME'=>'PROPERTY_VALUE'}
> EOF
>end
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14733) Minor typo in alter_namespace.rb
[
https://issues.apache.org/jira/browse/HBASE-14733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-14733:
--
Attachment: hbase-14733.patch
I've pushed this to 0.98+.
> Minor typo in alter_namespace.rb
>
>
> Key: HBASE-14733
> URL: https://issues.apache.org/jira/browse/HBASE-14733
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Trivial
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: hbase-14733.patch
>
>
> {code}
> diff --git hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
> hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
> index 760bbf7..a16e10d 100644
> --- hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
> +++ hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
> @@ -26,11 +26,11 @@ Alter namespace properties.
>
> To add/modify a property:
>
> - hbase> alter_namespace 'ns1', {METHOD => 'set', 'PROERTY_NAME' =>
> 'PROPERTY_VALUE'}
> + hbase> alter_namespace 'ns1', {METHOD => 'set', 'PROPERTY_NAME' =>
> 'PROPERTY_VALUE'}
>
> To delete a property:
>
> - hbase> alter_namespace 'ns1', {METHOD => 'unset', NAME=>'PROERTY_NAME'}
> + hbase> alter_namespace 'ns1', {METHOD => 'unset', NAME=>'PROPERTY_NAME'}
> EOF
>end
>
> diff --git hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
> hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
> index 3259eb6..adb6897 100644
> --- hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
> +++ hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
> @@ -27,7 +27,7 @@ and optionally a dictionary of namespace configuration.
> Examples:
>
>hbase> create_namespace 'ns1'
> - hbase> create_namespace 'ns1', {'PROERTY_NAME'=>'PROPERTY_VALUE'}
> + hbase> create_namespace 'ns1', {'PROPERTY_NAME'=>'PROPERTY_VALUE'}
> EOF
>end
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14725) Vet categorization of tests so they for sure go into the right small/medium/large buckets
[ https://issues.apache.org/jira/browse/HBASE-14725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14725: -- Attachment: 14725v4.txt Rebase > Vet categorization of tests so they for sure go into the right > small/medium/large buckets > - > > Key: HBASE-14725 > URL: https://issues.apache.org/jira/browse/HBASE-14725 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > Attachments: 14725v2 (1).txt, 14725v2 (1).txt, 14725v2.txt, > 14725v2.txt, 14725v4.txt, categorization.patch > > > I tried doing runSmallTests, runMediumTests, etc., and I noticed that some > tests are larger than our categorization. At least for small tests it means > they area all running in the one JVM. I also noticed that the categorization > only takes effect in hbase-server. > This patch makes it so runSmallTests runs all the small tests only in each > category. Also moves tests that were larger than small out to medium so they > don't run in the one JVM anymore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14540) Write Ahead Log Batching Optimization
[
https://issues.apache.org/jira/browse/HBASE-14540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983720#comment-14983720
]
Hadoop QA commented on HBASE-14540:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12769858/14540.txt
against master branch at commit c6d3dcdea33efa4fa745b323f6ff46b8b0711309.
ATTACHMENT ID: 12769858
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1731 checkstyle errors (more than the master's current 1730 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16322//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16322//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16322//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16322//console
This message is automatically generated.
> Write Ahead Log Batching Optimization
> -
>
> Key: HBASE-14540
> URL: https://issues.apache.org/jira/browse/HBASE-14540
> Project: HBase
> Issue Type: Improvement
>Reporter: John Leach
>Assignee: John Leach
> Attachments: 14540.txt, HBaseWALBlockingWaitStrategy.java, writes.png
>
>
> The new write ahead log mechanism seems to batch too few mutations when
> running inside the disruptor. As we scaled our load up (many threads with
> small writes), we saw the number of hdfs sync operations grow in concert with
> the number of writes. Generally, one would expect the size of the batches to
> grow but the number of actual sync operations to settle.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14575) Reduce scope of compactions holding region lock
[
https://issues.apache.org/jira/browse/HBASE-14575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983719#comment-14983719
]
Hadoop QA commented on HBASE-14575:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12769873/14575-v4.patch
against master branch at commit c6d3dcdea33efa4fa745b323f6ff46b8b0711309.
ATTACHMENT ID: 12769873
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 18 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16323//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16323//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16323//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16323//console
This message is automatically generated.
> Reduce scope of compactions holding region lock
> ---
>
> Key: HBASE-14575
> URL: https://issues.apache.org/jira/browse/HBASE-14575
> Project: HBase
> Issue Type: Sub-task
> Components: Compaction, regionserver
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
> Attachments: 14575-v1.patch, 14575-v2.patch, 14575-v3.patch,
> 14575-v4.patch, 14575.v00.patch
>
>
> Per [~devaraj]'s idea on parent issue, let's see if we can reduce the scope
> of critical section under which compactions hold the region read lock.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14725) Vet categorization of tests so they for sure go into the right small/medium/large buckets
[
https://issues.apache.org/jira/browse/HBASE-14725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983716#comment-14983716
]
Hadoop QA commented on HBASE-14725:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12769896/14725v2%20%281%29.txt
against master branch at commit c6d3dcdea33efa4fa745b323f6ff46b8b0711309.
ATTACHMENT ID: 12769896
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 36 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16326//console
This message is automatically generated.
> Vet categorization of tests so they for sure go into the right
> small/medium/large buckets
> -
>
> Key: HBASE-14725
> URL: https://issues.apache.org/jira/browse/HBASE-14725
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: stack
> Attachments: 14725v2 (1).txt, 14725v2 (1).txt, 14725v2.txt,
> 14725v2.txt, categorization.patch
>
>
> I tried doing runSmallTests, runMediumTests, etc., and I noticed that some
> tests are larger than our categorization. At least for small tests it means
> they area all running in the one JVM. I also noticed that the categorization
> only takes effect in hbase-server.
> This patch makes it so runSmallTests runs all the small tests only in each
> category. Also moves tests that were larger than small out to medium so they
> don't run in the one JVM anymore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14732) Update HBase website skin and CSS
[
https://issues.apache.org/jira/browse/HBASE-14732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983714#comment-14983714
]
Hadoop QA commented on HBASE-14732:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12769886/HBASE-14732.patch
against master branch at commit c6d3dcdea33efa4fa745b323f6ff46b8b0711309.
ATTACHMENT ID: 12769886
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation patch that doesn't require tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16325//console
This message is automatically generated.
> Update HBase website skin and CSS
> -
>
> Key: HBASE-14732
> URL: https://issues.apache.org/jira/browse/HBASE-14732
> Project: HBase
> Issue Type: Task
> Components: documentation
>Reporter: Misty Stanley-Jones
>Assignee: Misty Stanley-Jones
> Fix For: 2.0.0
>
> Attachments: HBASE-14732.patch
>
>
> Use Fluido skin, make things more readable, make improvements to enhance the
> mobile experience.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14687) Un-synchronize BufferedMutator
[ https://issues.apache.org/jira/browse/HBASE-14687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983707#comment-14983707 ] Hudson commented on HBASE-14687: FAILURE: Integrated in HBase-1.2 #328 (See [https://builds.apache.org/job/HBase-1.2/328/]) HBASE-14687 Un-synchronize BufferedMutator (eclark: rev 6308401b6a26551629dfec772fa13408a39936f4) * hbase-client/src/main/java/org/apache/hadoop/hbase/client/BufferedMutatorImpl.java * hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestAsyncProcess.java > Un-synchronize BufferedMutator > -- > > Key: HBASE-14687 > URL: https://issues.apache.org/jira/browse/HBASE-14687 > Project: HBase > Issue Type: Improvement > Components: Client, Performance >Affects Versions: 2.0.0, 1.2.0 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Critical > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14687-v1.patch, HBASE-14687-v2.patch, > HBASE-14687-v3.patch, HBASE-14687-v4.patch, HBASE-14687-v5.patch, > HBASE-14687.patch > > > It should totally be possible to make BufferedMutatorImpl not use much > locking. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14540) Write Ahead Log Batching Optimization
[
https://issues.apache.org/jira/browse/HBASE-14540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983704#comment-14983704
]
Hadoop QA commented on HBASE-14540:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12769858/14540.txt
against master branch at commit ec651bfc7bb004c7a14503bd6ea5dc376975bd3e.
ATTACHMENT ID: 12769858
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1731 checkstyle errors (more than the master's current 1730 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16320//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16320//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16320//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16320//console
This message is automatically generated.
> Write Ahead Log Batching Optimization
> -
>
> Key: HBASE-14540
> URL: https://issues.apache.org/jira/browse/HBASE-14540
> Project: HBase
> Issue Type: Improvement
>Reporter: John Leach
>Assignee: John Leach
> Attachments: 14540.txt, HBaseWALBlockingWaitStrategy.java, writes.png
>
>
> The new write ahead log mechanism seems to batch too few mutations when
> running inside the disruptor. As we scaled our load up (many threads with
> small writes), we saw the number of hdfs sync operations grow in concert with
> the number of writes. Generally, one would expect the size of the batches to
> grow but the number of actual sync operations to settle.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-14733) Minor typo in alter_namespace.rb
Enis Soztutar created HBASE-14733:
-
Summary: Minor typo in alter_namespace.rb
Key: HBASE-14733
URL: https://issues.apache.org/jira/browse/HBASE-14733
Project: HBase
Issue Type: Bug
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Priority: Trivial
{code}
diff --git hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
index 760bbf7..a16e10d 100644
--- hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
+++ hbase-shell/src/main/ruby/shell/commands/alter_namespace.rb
@@ -26,11 +26,11 @@ Alter namespace properties.
To add/modify a property:
- hbase> alter_namespace 'ns1', {METHOD => 'set', 'PROERTY_NAME' =>
'PROPERTY_VALUE'}
+ hbase> alter_namespace 'ns1', {METHOD => 'set', 'PROPERTY_NAME' =>
'PROPERTY_VALUE'}
To delete a property:
- hbase> alter_namespace 'ns1', {METHOD => 'unset', NAME=>'PROERTY_NAME'}
+ hbase> alter_namespace 'ns1', {METHOD => 'unset', NAME=>'PROPERTY_NAME'}
EOF
end
diff --git hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
index 3259eb6..adb6897 100644
--- hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
+++ hbase-shell/src/main/ruby/shell/commands/create_namespace.rb
@@ -27,7 +27,7 @@ and optionally a dictionary of namespace configuration.
Examples:
hbase> create_namespace 'ns1'
- hbase> create_namespace 'ns1', {'PROERTY_NAME'=>'PROPERTY_VALUE'}
+ hbase> create_namespace 'ns1', {'PROPERTY_NAME'=>'PROPERTY_VALUE'}
EOF
end
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14687) Un-synchronize BufferedMutator
[ https://issues.apache.org/jira/browse/HBASE-14687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983692#comment-14983692 ] Hudson commented on HBASE-14687: FAILURE: Integrated in HBase-Trunk_matrix #414 (See [https://builds.apache.org/job/HBase-Trunk_matrix/414/]) HBASE-14687 Un-synchronize BufferedMutator (eclark: rev ec651bfc7bb004c7a14503bd6ea5dc376975bd3e) * hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestAsyncProcess.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/BufferedMutatorImpl.java > Un-synchronize BufferedMutator > -- > > Key: HBASE-14687 > URL: https://issues.apache.org/jira/browse/HBASE-14687 > Project: HBase > Issue Type: Improvement > Components: Client, Performance >Affects Versions: 2.0.0, 1.2.0 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Critical > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14687-v1.patch, HBASE-14687-v2.patch, > HBASE-14687-v3.patch, HBASE-14687-v4.patch, HBASE-14687-v5.patch, > HBASE-14687.patch > > > It should totally be possible to make BufferedMutatorImpl not use much > locking. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-6721) RegionServer Group based Assignment
[
https://issues.apache.org/jira/browse/HBASE-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983663#comment-14983663
]
Enis Soztutar commented on HBASE-6721:
--
Finally got around testing the v15 patch on 1.1 code base with a 7 node
cluster. Here are my test notes. Nothing too concerning, but we have to address
some of these in the patch. This is the configuration to add to enable groups:
{code}
hbase.coprocessor.master.classes
org.apache.hadoop.hbase.group.GroupAdminEndpoint
hbase.master.loadbalancer.class
org.apache.hadoop.hbase.group.GroupBasedLoadBalancer
{code}
1. Need to add this diff, so that new PB files get compiled with
-Pcompile-protobuf command:
{code}
diff --git hbase-protocol/pom.xml hbase-protocol/pom.xml
index 8034576..d352373 100644
--- hbase-protocol/pom.xml
+++ hbase-protocol/pom.xml
@@ -180,6 +180,8 @@
ErrorHandling.proto
Filter.proto
FS.proto
+ Group.proto
+ GroupAdmin.proto
HBase.proto
HFile.proto
LoadBalancer.proto
{code}
2. NPE in group shell commands with nonexisting groups:
{code}
hbase(main):015:0* balance_group 'nonexisting'
ERROR: java.io.IOException
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2156)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hbase.group.GroupAdminServer.groupGetRegionsInTransition(GroupAdminServer.java:412)
at
org.apache.hadoop.hbase.group.GroupAdminServer.balanceGroup(GroupAdminServer.java:348)
at
org.apache.hadoop.hbase.group.GroupAdminEndpoint.balanceGroup(GroupAdminEndpoint.java:229)
at
org.apache.hadoop.hbase.protobuf.generated.GroupAdminProtos$GroupAdminService.callMethod(GroupAdminProtos.java:11156)
at
org.apache.hadoop.hbase.master.MasterRpcServices.execMasterService(MasterRpcServices.java:666)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:51121)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
{code}
and
{code}
hbase(main):030:0> get_group 'nonexisting'
GROUP INFORMATION
Servers:
ERROR: undefined method `getServers' for nil:NilClass
Here is some help for this command:
Get a region server group's information.
Example:
hbase> get_group 'default'
{code}
and
{code}
hbase(main):077:0* move_group_tables 'nonexisting'
ERROR: undefined method `each' for nil:NilClass
Here is some help for this command:
Reassign tables from one group to another.
hbase> move_group_tables 'dest',['table1','table2']
{code}
and
{code}
hbase(main):173:0* move_group_servers 'nonexisting'
ERROR: undefined method `each' for nil:NilClass
Here is some help for this command:
Reassign a region server from one group to another.
hbase> move_group_servers 'dest',['server1:port','server2:port']
{code}
3. Group names should be restricted to alphanumeric only. This one is pretty
easy, but important. This following caused the master to abort, and the master
cannot restart after this point (without manually removing the rsgroup entry
from the table which you cannot do without master). I had to nuke the hdfs and
zk to start over.
{code}
hbase(main):033:0> add_group 'a-/:*'
ERROR: java.io.IOException: Failed to write to groupZNode
at
org.apache.hadoop.hbase.group.GroupInfoManagerImpl.flushConfig(GroupInfoManagerImpl.java:419)
at
org.apache.hadoop.hbase.group.GroupInfoManagerImpl.addGroup(GroupInfoManagerImpl.java:152)
at
org.apache.hadoop.hbase.group.GroupAdminServer.addGroup(GroupAdminServer.java:298)
at
org.apache.hadoop.hbase.group.GroupAdminEndpoint.addGroup(GroupAdminEndpoint.java:197)
at
org.apache.hadoop.hbase.protobuf.generated.GroupAdminProtos$GroupAdminService.callMethod(GroupAdminProtos.java:11146)
at
org.apache.hadoop.hbase.master.MasterRpcServices.execMasterService(MasterRpcServices.java:66
[jira] [Updated] (HBASE-14725) Vet categorization of tests so they for sure go into the right small/medium/large buckets
[ https://issues.apache.org/jira/browse/HBASE-14725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14725: -- Attachment: (was: 14725v3.txt) > Vet categorization of tests so they for sure go into the right > small/medium/large buckets > - > > Key: HBASE-14725 > URL: https://issues.apache.org/jira/browse/HBASE-14725 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > Attachments: 14725v2 (1).txt, 14725v2 (1).txt, 14725v2.txt, > 14725v2.txt, categorization.patch > > > I tried doing runSmallTests, runMediumTests, etc., and I noticed that some > tests are larger than our categorization. At least for small tests it means > they area all running in the one JVM. I also noticed that the categorization > only takes effect in hbase-server. > This patch makes it so runSmallTests runs all the small tests only in each > category. Also moves tests that were larger than small out to medium so they > don't run in the one JVM anymore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14725) Vet categorization of tests so they for sure go into the right small/medium/large buckets
[ https://issues.apache.org/jira/browse/HBASE-14725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14725: -- Attachment: 14725v2 (1).txt Try a run with 2.19 surefire backed out. > Vet categorization of tests so they for sure go into the right > small/medium/large buckets > - > > Key: HBASE-14725 > URL: https://issues.apache.org/jira/browse/HBASE-14725 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > Attachments: 14725v2 (1).txt, 14725v2 (1).txt, 14725v2.txt, > 14725v2.txt, categorization.patch > > > I tried doing runSmallTests, runMediumTests, etc., and I noticed that some > tests are larger than our categorization. At least for small tests it means > they area all running in the one JVM. I also noticed that the categorization > only takes effect in hbase-server. > This patch makes it so runSmallTests runs all the small tests only in each > category. Also moves tests that were larger than small out to medium so they > don't run in the one JVM anymore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14712) MasterProcWALs never clean up
[ https://issues.apache.org/jira/browse/HBASE-14712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983595#comment-14983595 ] Enis Soztutar commented on HBASE-14712: --- Making 1.2.0 as the fixVersion. > MasterProcWALs never clean up > - > > Key: HBASE-14712 > URL: https://issues.apache.org/jira/browse/HBASE-14712 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Assignee: Matteo Bertozzi >Priority: Blocker > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3 > > Attachments: HBASE-14712-v0.patch, state.tar.gz > > > MasterProcWALs directory grows pretty much un-bounded. Because of that when > master failover happens the NN is flooded with connections and everything > grinds to a halt. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14712) MasterProcWALs never clean up
[ https://issues.apache.org/jira/browse/HBASE-14712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-14712: -- Fix Version/s: 1.1.3 1.3.0 1.2.0 2.0.0 > MasterProcWALs never clean up > - > > Key: HBASE-14712 > URL: https://issues.apache.org/jira/browse/HBASE-14712 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Assignee: Matteo Bertozzi >Priority: Blocker > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3 > > Attachments: HBASE-14712-v0.patch, state.tar.gz > > > MasterProcWALs directory grows pretty much un-bounded. Because of that when > master failover happens the NN is flooded with connections and everything > grinds to a halt. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14712) MasterProcWALs never clean up
[ https://issues.apache.org/jira/browse/HBASE-14712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983588#comment-14983588 ] Elliott Clark commented on HBASE-14712: --- +1 on v0. Lets get that in and I can start another test running. > MasterProcWALs never clean up > - > > Key: HBASE-14712 > URL: https://issues.apache.org/jira/browse/HBASE-14712 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Assignee: Matteo Bertozzi >Priority: Blocker > Attachments: HBASE-14712-v0.patch, state.tar.gz > > > MasterProcWALs directory grows pretty much un-bounded. Because of that when > master failover happens the NN is flooded with connections and everything > grinds to a halt. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14712) MasterProcWALs never clean up
[
https://issues.apache.org/jira/browse/HBASE-14712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983586#comment-14983586
]
Elliott Clark commented on HBASE-14712:
---
Here's the entirety of one clusters master proc logs. This one was running an
ITBLL and nothing else.
http://people.apache.org/~eclark/MasterProcWALs-hbaseci002.tar.gz
{code}
Found 15 items
-rw-r--r-- 3 hadoop root 12067862 2015-10-26 17:39
/HBASECI002-HBASE/MasterProcWALs/state-0318.log
-rw-r--r-- 3 hadoop root 33557821 2015-10-26 18:18
/HBASECI002-HBASE/MasterProcWALs/state-0321.log
-rw-r--r-- 3 hadoop root 33561858 2015-10-26 19:07
/HBASECI002-HBASE/MasterProcWALs/state-0325.log
-rw-r--r-- 3 hadoop root 33557309 2015-10-26 19:31
/HBASECI002-HBASE/MasterProcWALs/state-0327.log
-rw-r--r-- 3 hadoop root 989672 2015-10-26 19:32
/HBASECI002-HBASE/MasterProcWALs/state-0328.log
-rw-r--r-- 3 hadoop root 23137180 2015-10-26 20:32
/HBASECI002-HBASE/MasterProcWALs/state-0329.log
-rw-r--r-- 3 hadoop root6212852 2015-10-28 14:19
/HBASECI002-HBASE/MasterProcWALs/state-0373.log
-rw-r--r-- 3 hadoop root5116784 2015-10-29 19:32
/HBASECI002-HBASE/MasterProcWALs/state-0403.log
-rw-r--r-- 3 hadoop root 10176515 2015-10-30 10:46
/HBASECI002-HBASE/MasterProcWALs/state-1048.log
-rw-r--r-- 3 hadoop root 16108983 2015-10-30 11:46
/HBASECI002-HBASE/MasterProcWALs/state-1049.log
-rw-r--r-- 3 hadoop root 10648929 2015-10-30 12:46
/HBASECI002-HBASE/MasterProcWALs/state-1050.log
-rw-r--r-- 3 hadoop root7730706 2015-10-30 13:46
/HBASECI002-HBASE/MasterProcWALs/state-1051.log
-rw-r--r-- 3 hadoop root8072367 2015-10-30 14:46
/HBASECI002-HBASE/MasterProcWALs/state-1052.log
-rw-r--r-- 3 hadoop root5147892 2015-10-30 15:46
/HBASECI002-HBASE/MasterProcWALs/state-1053.log
-rw-r--r-- 3 hadoop root 12515 2015-10-30 15:46
/HBASECI002-HBASE/MasterProcWALs/state-1054.log
{code}
> MasterProcWALs never clean up
> -
>
> Key: HBASE-14712
> URL: https://issues.apache.org/jira/browse/HBASE-14712
> Project: HBase
> Issue Type: Bug
>Reporter: Elliott Clark
>Assignee: Matteo Bertozzi
>Priority: Blocker
> Attachments: HBASE-14712-v0.patch, state.tar.gz
>
>
> MasterProcWALs directory grows pretty much un-bounded. Because of that when
> master failover happens the NN is flooded with connections and everything
> grinds to a halt.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14728) TestRowCounter is broken in master
[
https://issues.apache.org/jira/browse/HBASE-14728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983585#comment-14983585
]
Hadoop QA commented on HBASE-14728:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12769837/HBASE-14728.patch
against master branch at commit b0ad82191cc216c65751b68cbc11280fd5d5cd47.
ATTACHMENT ID: 12769837
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16319//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16319//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16319//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16319//console
This message is automatically generated.
> TestRowCounter is broken in master
> --
>
> Key: HBASE-14728
> URL: https://issues.apache.org/jira/browse/HBASE-14728
> Project: HBase
> Issue Type: Test
>Affects Versions: 2.0.0
>Reporter: Abhishek Singh Chouhan
>Assignee: Abhishek Singh Chouhan
> Fix For: 2.0.0
>
> Attachments: HBASE-14728.patch, HBASE-14728.patch
>
>
> The runRowCount method simply runs the RowCounter and checks if the exit code
> is zero but does not actually check the row count returned
> {noformat}
> private void runRowCount(String[] args, int expectedCount) throws Exception {
> final RowCounter counter = new RowCounter();
> assertEquals("job failed either due to failure or miscount (see log
> output).", 0,
> ToolRunner.run(TEST_UTIL.getConfiguration(), counter, args));
> }
> {noformat}
> This will always give a false positive provided the job ran without errors
> (irrespective of the count it gives).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14725) Vet categorization of tests so they for sure go into the right small/medium/large buckets
[
https://issues.apache.org/jira/browse/HBASE-14725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983548#comment-14983548
]
Hadoop QA commented on HBASE-14725:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12769875/14725v3.txt
against master branch at commit c6d3dcdea33efa4fa745b323f6ff46b8b0711309.
ATTACHMENT ID: 12769875
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 36 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16324//console
This message is automatically generated.
> Vet categorization of tests so they for sure go into the right
> small/medium/large buckets
> -
>
> Key: HBASE-14725
> URL: https://issues.apache.org/jira/browse/HBASE-14725
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: stack
> Attachments: 14725v2 (1).txt, 14725v2.txt, 14725v2.txt, 14725v3.txt,
> categorization.patch
>
>
> I tried doing runSmallTests, runMediumTests, etc., and I noticed that some
> tests are larger than our categorization. At least for small tests it means
> they area all running in the one JVM. I also noticed that the categorization
> only takes effect in hbase-server.
> This patch makes it so runSmallTests runs all the small tests only in each
> category. Also moves tests that were larger than small out to medium so they
> don't run in the one JVM anymore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14732) Update HBase website skin and CSS
[ https://issues.apache.org/jira/browse/HBASE-14732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-14732: Status: Patch Available (was: Open) > Update HBase website skin and CSS > - > > Key: HBASE-14732 > URL: https://issues.apache.org/jira/browse/HBASE-14732 > Project: HBase > Issue Type: Task > Components: documentation >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Fix For: 2.0.0 > > Attachments: HBASE-14732.patch > > > Use Fluido skin, make things more readable, make improvements to enhance the > mobile experience. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14732) Update HBase website skin and CSS
[ https://issues.apache.org/jira/browse/HBASE-14732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-14732: Attachment: HBASE-14732.patch > Update HBase website skin and CSS > - > > Key: HBASE-14732 > URL: https://issues.apache.org/jira/browse/HBASE-14732 > Project: HBase > Issue Type: Task > Components: documentation >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Fix For: 2.0.0 > > Attachments: HBASE-14732.patch > > > Use Fluido skin, make things more readable, make improvements to enhance the > mobile experience. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14732) Update HBase website skin and CSS
Misty Stanley-Jones created HBASE-14732: --- Summary: Update HBase website skin and CSS Key: HBASE-14732 URL: https://issues.apache.org/jira/browse/HBASE-14732 Project: HBase Issue Type: Task Components: documentation Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Fix For: 2.0.0 Use Fluido skin, make things more readable, make improvements to enhance the mobile experience. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HBASE-14722) Update surefire to 2.19
[ https://issues.apache.org/jira/browse/HBASE-14722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack reopened HBASE-14722: --- Reopen. I reverted this for the moment because seeing lots of instances of new message: "ExecutionException Error occurred in starting fork, check output in log" Let me reread the release notes and see if anything new in 2.19 that could bring this on. > Update surefire to 2.19 > --- > > Key: HBASE-14722 > URL: https://issues.apache.org/jira/browse/HBASE-14722 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack >Assignee: stack > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: surefire.patch > > > Bug fixes: > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12317927&version=12331679 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14712) MasterProcWALs never clean up
[ https://issues.apache.org/jira/browse/HBASE-14712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-14712: Attachment: HBASE-14712-v0.patch v0 fixes one issue, where on period roll if there is nothing going on the wals are not deleted. so you accumulate wals up to the next restart. this may be your problem. looking into the storeTracker if there are also bugs in there. in theory every wal entry update/delete call we check the store tracker. if there are no procedures running or all the procs are written in the newest wal (updates) we can drop the previous. > MasterProcWALs never clean up > - > > Key: HBASE-14712 > URL: https://issues.apache.org/jira/browse/HBASE-14712 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Assignee: Matteo Bertozzi >Priority: Blocker > Attachments: HBASE-14712-v0.patch, state.tar.gz > > > MasterProcWALs directory grows pretty much un-bounded. Because of that when > master failover happens the NN is flooded with connections and everything > grinds to a halt. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14725) Vet categorization of tests so they for sure go into the right small/medium/large buckets
[ https://issues.apache.org/jira/browse/HBASE-14725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983474#comment-14983474 ] stack commented on HBASE-14725: --- Hmmm... I don't think it the patch given how frequently the above is happening elsewhere. Let me try undoing 2.19 patch and see if that makes a difference. > Vet categorization of tests so they for sure go into the right > small/medium/large buckets > - > > Key: HBASE-14725 > URL: https://issues.apache.org/jira/browse/HBASE-14725 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > Attachments: 14725v2 (1).txt, 14725v2.txt, 14725v2.txt, 14725v3.txt, > categorization.patch > > > I tried doing runSmallTests, runMediumTests, etc., and I noticed that some > tests are larger than our categorization. At least for small tests it means > they area all running in the one JVM. I also noticed that the categorization > only takes effect in hbase-server. > This patch makes it so runSmallTests runs all the small tests only in each > category. Also moves tests that were larger than small out to medium so they > don't run in the one JVM anymore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14713) Remove simple deprecated-since-1.0 code in hbase-server from hbase 2.0
[
https://issues.apache.org/jira/browse/HBASE-14713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983473#comment-14983473
]
Hadoop QA commented on HBASE-14713:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12769823/hbase-14713.v3.patch
against master branch at commit b0ad82191cc216c65751b68cbc11280fd5d5cd47.
ATTACHMENT ID: 12769823
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 10 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16316//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16316//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16316//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16316//console
This message is automatically generated.
> Remove simple deprecated-since-1.0 code in hbase-server from hbase 2.0
> --
>
> Key: HBASE-14713
> URL: https://issues.apache.org/jira/browse/HBASE-14713
> Project: HBase
> Issue Type: Task
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0
>
> Attachments: hbase-14713.patch, hbase-14713.v2.patch,
> hbase-14713.v3.patch
>
>
> Removed items such as
> * Migration to 96.
> * HFileOutputFormat (HFileOutputFormat2 preferred)
> * MapfilePerformanceEvaluttion
> * getHTable -> getTable()
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14575) Reduce scope of compactions holding region lock
[
https://issues.apache.org/jira/browse/HBASE-14575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983449#comment-14983449
]
Ted Yu commented on HBASE-14575:
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-surefire-plugin:2.19:test
(secondPartTestsExecution) on project hbase-server: ExecutionException Error
occurred in starting fork, check output in log -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :hbase-server
grep: unknown devices method
We're ok: there is no zombie test
{code}
Looks like test environment had issue.
> Reduce scope of compactions holding region lock
> ---
>
> Key: HBASE-14575
> URL: https://issues.apache.org/jira/browse/HBASE-14575
> Project: HBase
> Issue Type: Sub-task
> Components: Compaction, regionserver
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
> Attachments: 14575-v1.patch, 14575-v2.patch, 14575-v3.patch,
> 14575-v4.patch, 14575.v00.patch
>
>
> Per [~devaraj]'s idea on parent issue, let's see if we can reduce the scope
> of critical section under which compactions hold the region read lock.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14711) Remove or annotated deprecated methods in HRegionInfo
[
https://issues.apache.org/jira/browse/HBASE-14711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983446#comment-14983446
]
Hadoop QA commented on HBASE-14711:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12769824/hbase-14711.v4.patch
against master branch at commit b0ad82191cc216c65751b68cbc11280fd5d5cd47.
ATTACHMENT ID: 12769824
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 18 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16315//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16315//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16315//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16315//console
This message is automatically generated.
> Remove or annotated deprecated methods in HRegionInfo
> -
>
> Key: HBASE-14711
> URL: https://issues.apache.org/jira/browse/HBASE-14711
> Project: HBase
> Issue Type: Bug
> Components: hbase
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0
>
> Attachments: hbase-14711.patch, hbase-14711.v2.patch,
> hbase-14711.v3.patch, hbase-14711.v4.patch
>
>
> Several methods were deprecated in 1.0 and moved to MetaTableAccessor. This
> removes them from the 2.0 branch and converts them to the MetaTableAccessor.
> HRegionInfo#getComparator was marked deprecated after 1.0 so needs to
> remain. (HBASE-13501)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14575) Reduce scope of compactions holding region lock
[
https://issues.apache.org/jira/browse/HBASE-14575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983436#comment-14983436
]
Hadoop QA commented on HBASE-14575:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12769839/14575-v3.patch
against master branch at commit b0ad82191cc216c65751b68cbc11280fd5d5cd47.
ATTACHMENT ID: 12769839
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 18 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16318//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16318//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16318//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16318//console
This message is automatically generated.
> Reduce scope of compactions holding region lock
> ---
>
> Key: HBASE-14575
> URL: https://issues.apache.org/jira/browse/HBASE-14575
> Project: HBase
> Issue Type: Sub-task
> Components: Compaction, regionserver
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
> Attachments: 14575-v1.patch, 14575-v2.patch, 14575-v3.patch,
> 14575-v4.patch, 14575.v00.patch
>
>
> Per [~devaraj]'s idea on parent issue, let's see if we can reduce the scope
> of critical section under which compactions hold the region read lock.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14731) Add -DuseMob option to ITBLL
[
https://issues.apache.org/jira/browse/HBASE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983426#comment-14983426
]
Hadoop QA commented on HBASE-14731:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12769840/hbase-14731.patch
against master branch at commit b0ad82191cc216c65751b68cbc11280fd5d5cd47.
ATTACHMENT ID: 12769840
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16317//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16317//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16317//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16317//console
This message is automatically generated.
> Add -DuseMob option to ITBLL
>
>
> Key: HBASE-14731
> URL: https://issues.apache.org/jira/browse/HBASE-14731
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0
>
> Attachments: hbase-14731.patch
>
>
> Previously, I would test mob by creating a table, altering to use mob and
> then running ITBLL. This simplifies it by allowing for a command line arg to
> enable mob on table creation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14725) Vet categorization of tests so they for sure go into the right small/medium/large buckets
[ https://issues.apache.org/jira/browse/HBASE-14725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983370#comment-14983370 ] stack commented on HBASE-14725: --- The last run failed with this too: "ExecutionException Error occurred in starting fork, check output in log" > Vet categorization of tests so they for sure go into the right > small/medium/large buckets > - > > Key: HBASE-14725 > URL: https://issues.apache.org/jira/browse/HBASE-14725 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > Attachments: 14725v2 (1).txt, 14725v2.txt, 14725v2.txt, 14725v3.txt, > categorization.patch > > > I tried doing runSmallTests, runMediumTests, etc., and I noticed that some > tests are larger than our categorization. At least for small tests it means > they area all running in the one JVM. I also noticed that the categorization > only takes effect in hbase-server. > This patch makes it so runSmallTests runs all the small tests only in each > category. Also moves tests that were larger than small out to medium so they > don't run in the one JVM anymore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14725) Vet categorization of tests so they for sure go into the right small/medium/large buckets
[ https://issues.apache.org/jira/browse/HBASE-14725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14725: -- Attachment: 14725v3.txt Let me undo the bit where I scale by CPUs... See if it makes any difference. > Vet categorization of tests so they for sure go into the right > small/medium/large buckets > - > > Key: HBASE-14725 > URL: https://issues.apache.org/jira/browse/HBASE-14725 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack > Attachments: 14725v2 (1).txt, 14725v2.txt, 14725v2.txt, 14725v3.txt, > categorization.patch > > > I tried doing runSmallTests, runMediumTests, etc., and I noticed that some > tests are larger than our categorization. At least for small tests it means > they area all running in the one JVM. I also noticed that the categorization > only takes effect in hbase-server. > This patch makes it so runSmallTests runs all the small tests only in each > category. Also moves tests that were larger than small out to medium so they > don't run in the one JVM anymore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14540) Write Ahead Log Batching Optimization
[ https://issues.apache.org/jira/browse/HBASE-14540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983356#comment-14983356 ] stack commented on HBASE-14540: --- What was the loading like [~jleach] > Write Ahead Log Batching Optimization > - > > Key: HBASE-14540 > URL: https://issues.apache.org/jira/browse/HBASE-14540 > Project: HBase > Issue Type: Improvement >Reporter: John Leach >Assignee: John Leach > Attachments: 14540.txt, HBaseWALBlockingWaitStrategy.java, writes.png > > > The new write ahead log mechanism seems to batch too few mutations when > running inside the disruptor. As we scaled our load up (many threads with > small writes), we saw the number of hdfs sync operations grow in concert with > the number of writes. Generally, one would expect the size of the batches to > grow but the number of actual sync operations to settle. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14540) Write Ahead Log Batching Optimization
[ https://issues.apache.org/jira/browse/HBASE-14540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983350#comment-14983350 ] John Leach commented on HBASE-14540: I did it on Cloudera 5.4.1 for my test... > Write Ahead Log Batching Optimization > - > > Key: HBASE-14540 > URL: https://issues.apache.org/jira/browse/HBASE-14540 > Project: HBase > Issue Type: Improvement >Reporter: John Leach >Assignee: John Leach > Attachments: 14540.txt, HBaseWALBlockingWaitStrategy.java, writes.png > > > The new write ahead log mechanism seems to batch too few mutations when > running inside the disruptor. As we scaled our load up (many threads with > small writes), we saw the number of hdfs sync operations grow in concert with > the number of writes. Generally, one would expect the size of the batches to > grow but the number of actual sync operations to settle. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14575) Reduce scope of compactions holding region lock
[ https://issues.apache.org/jira/browse/HBASE-14575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14575: --- Attachment: 14575-v4.patch Patch v4 parameterizes TestHRegionServerBulkLoad so that the customer observer sleeps 0 and 30 seconds, respectively. This is to preserve the original test scenario where compaction is not slow. > Reduce scope of compactions holding region lock > --- > > Key: HBASE-14575 > URL: https://issues.apache.org/jira/browse/HBASE-14575 > Project: HBase > Issue Type: Sub-task > Components: Compaction, regionserver >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk > Attachments: 14575-v1.patch, 14575-v2.patch, 14575-v3.patch, > 14575-v4.patch, 14575.v00.patch > > > Per [~devaraj]'s idea on parent issue, let's see if we can reduce the scope > of critical section under which compactions hold the region read lock. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14540) Write Ahead Log Batching Optimization
[
https://issues.apache.org/jira/browse/HBASE-14540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983338#comment-14983338
]
stack commented on HBASE-14540:
---
Bumping threads to 100 bumps the write throughput only slightly. Threads are
doing this kinda stuff... waiting:
{code}
Thread 102 (PriorityRpcServer.handler=19,queue=1,port=16020):
State: TIMED_WAITING
Blocked count: 78747
Waited count: 110565
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:167)
org.apache.hadoop.hbase.regionserver.wal.FSHLog.blockOnSync(FSHLog.java:1356)
org.apache.hadoop.hbase.regionserver.wal.FSHLog.publishSyncThenBlockOnCompletion(FSHLog.java:1350)
org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1456)
org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:7901)
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3150)
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2812)
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2754)
org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:779)
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:741)
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2156)
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32462)
org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2179)
org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106)
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
java.lang.Thread.run(Thread.java:745)
Thread 101 (PriorityRpcServer.handler=18,queue=0,port=16020):
State: TIMED_WAITING
Blocked count: 78786
Waited count: 110881
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:167)
org.apache.hadoop.hbase.regionserver.wal.FSHLog.blockOnSync(FSHLog.java:1356)
org.apache.hadoop.hbase.regionserver.wal.FSHLog.publishSyncThenBlockOnCompletion(FSHLog.java:1350)
org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1456)
org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:7901)
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3150)
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2812)
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2754)
org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:779)
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:741)
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2156)
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32462)
org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2179)
org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106)
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
java.lang.Thread.run(Thread.java:745)
Thread 100 (PriorityRpcServer.handler=17,queue=1,port=16020):
State: TIMED_WAITING
Blocked count: 78899
Waited count: 110706
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hbase.regionserver.MultiVersionConcurrencyControl.waitForRead(MultiVersionConcurrencyControl.java:218)
org.apache.hadoop.hbase.regionserver.MultiVersionConcurrencyControl.completeAndWait(MultiVersionConcurrencyControl.java:149)
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3166)
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2812)
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2754)
org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:779)
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:741)
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2156)
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32462)
org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2179)
org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106)
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
java.lang.Thread.run(Thread.java:745)
{code}
I tried to go to 300 threads with 100 c
[jira] [Comment Edited] (HBASE-14540) Write Ahead Log Batching Optimization
[ https://issues.apache.org/jira/browse/HBASE-14540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983306#comment-14983306 ] stack edited comment on HBASE-14540 at 10/30/15 9:11 PM: - I ran a test. Empty table. YCSB all writes going to a single RegionServer sitting on a HDFS of 5 nodes. 25 clients. As is, the new strategy slows writes pretty radically (the strategy is on place for the 5% of the graph that is on the far right side). Let me try but upping the clients a bunch... but would be good if the strategy was a bit smarter and didn't wait when loading was relatively low. was (Author: stack): I ran a test. Empty table. YCSB all writes going to a single RegionServer sitting on a HDFS of 5 nodes. 25 clients. As is, the new strategy slows writes pretty radically (the strategy is on place for the 10% of the graph that is on the far right side). Let me try but upping the clients a bunch... but would be good if the strategy was a bit smarter and didn't wait when loading was relatively low. > Write Ahead Log Batching Optimization > - > > Key: HBASE-14540 > URL: https://issues.apache.org/jira/browse/HBASE-14540 > Project: HBase > Issue Type: Improvement >Reporter: John Leach >Assignee: John Leach > Attachments: 14540.txt, HBaseWALBlockingWaitStrategy.java, writes.png > > > The new write ahead log mechanism seems to batch too few mutations when > running inside the disruptor. As we scaled our load up (many threads with > small writes), we saw the number of hdfs sync operations grow in concert with > the number of writes. Generally, one would expect the size of the batches to > grow but the number of actual sync operations to settle. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14540) Write Ahead Log Batching Optimization
[ https://issues.apache.org/jira/browse/HBASE-14540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14540: -- Attachment: writes.png I ran a test. Empty table. YCSB all writes going to a single RegionServer sitting on a HDFS of 5 nodes. 25 clients. As is, the new strategy slows writes pretty radically (the strategy is on place for the 10% of the graph that is on the far right side). Let me try but upping the clients a bunch... but would be good if the strategy was a bit smarter and didn't wait when loading was relatively low. > Write Ahead Log Batching Optimization > - > > Key: HBASE-14540 > URL: https://issues.apache.org/jira/browse/HBASE-14540 > Project: HBase > Issue Type: Improvement >Reporter: John Leach >Assignee: John Leach > Attachments: 14540.txt, HBaseWALBlockingWaitStrategy.java, writes.png > > > The new write ahead log mechanism seems to batch too few mutations when > running inside the disruptor. As we scaled our load up (many threads with > small writes), we saw the number of hdfs sync operations grow in concert with > the number of writes. Generally, one would expect the size of the batches to > grow but the number of actual sync operations to settle. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14719) Add metric for number of MasterProcWALs
[ https://issues.apache.org/jira/browse/HBASE-14719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983290#comment-14983290 ] stack commented on HBASE-14719: --- Chatting w/ Matteo, procedure metrics could be first place we by-pass hadoop metrics2 > Add metric for number of MasterProcWALs > --- > > Key: HBASE-14719 > URL: https://issues.apache.org/jira/browse/HBASE-14719 > Project: HBase > Issue Type: Sub-task >Reporter: Elliott Clark > > Lets add monitoring to this so that we can see when it starts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14557) MapReduce WALPlayer issue with NoTagsKeyValue
[
https://issues.apache.org/jira/browse/HBASE-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983276#comment-14983276
]
Hudson commented on HBASE-14557:
SUCCESS: Integrated in HBase-1.2-IT #254 (See
[https://builds.apache.org/job/HBase-1.2-IT/254/])
HBASE-14557 MapReduce WALPlayer issue with NoTagsKeyValue. (anoopsamjohn: rev
28dc0abbe140bb1993e8659eef325758ac77c740)
* hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/PutCombiner.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/WALPlayer.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestWALPlayer.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/PutSortReducer.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValueUtil.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TextSortReducer.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java
> MapReduce WALPlayer issue with NoTagsKeyValue
> -
>
> Key: HBASE-14557
> URL: https://issues.apache.org/jira/browse/HBASE-14557
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Jerry He
>Assignee: Anoop Sam John
>Priority: Blocker
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14557.patch, HBASE-14557.patch,
> HBASE-14557_V2.patch, HBASE-14557_branch-1.2.patch,
> HBASE-14557_branch-1.2.patch, HBASE-14557_branch-1.patch
>
>
> Running MapReduce WALPlayer to convert WAL into HFiles:
> {noformat}
> 15/10/05 20:28:08 INFO mapred.JobClient: Task Id :
> attempt_201508031611_0029_m_00_0, Status : FAILED
> java.io.IOException: Type mismatch in value from map: expected
> org.apache.hadoop.hbase.KeyValue, recieved
> org.apache.hadoop.hbase.NoTagsKeyValue
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:997)
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:689)
> at
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
> at
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALKeyValueMapper.map(WALPlayer.java:111)
> at
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALKeyValueMapper.map(WALPlayer.java:96)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:751)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:368)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at
> java.security.AccessController.doPrivileged(AccessController.java:369)
> at javax.security.auth.Subject.doAs(Subject.java:572)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1502)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14721) Memstore add cells - Avoid many garbage
[
https://issues.apache.org/jira/browse/HBASE-14721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983277#comment-14983277
]
Hudson commented on HBASE-14721:
SUCCESS: Integrated in HBase-1.2-IT #254 (See
[https://builds.apache.org/job/HBase-1.2-IT/254/])
HBASE-14721 Memstore add cells - Avoid many garbage. (anoopsamjohn: rev
80e6a9434cf1f9544e26ee8e27f6f78d29322033)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultMemStore.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
> Memstore add cells - Avoid many garbage
> ---
>
> Key: HBASE-14721
> URL: https://issues.apache.org/jira/browse/HBASE-14721
> Project: HBase
> Issue Type: Improvement
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14721.patch, HBASE-14721_branch-1.patch
>
>
> In HRegion# applyFamilyMapToMemstore()
> {code}
> for (int i=0; i < listSize; i++) {
> Cell cell = cells.get(i);
> if (cell.getSequenceId() == 0) {
> CellUtil.setSequenceId(cell, mvccNum);
> }
> Pair ret = store.add(cell);
> size += ret.getFirst();
> if(isInReplay) {
> // set memstore newly added cells with replay mvcc number
> CellUtil.setSequenceId(ret.getSecond(), mvccNum);
> }
> }
> {code}
> Previously we used to pass the added Cell reference to some HLog impl classes
> and later on the seqId used to get assigned. Now it looks totally changed.
> For normal put cells, the assign happen before adding to store.
> After addition we can see in replay mode again the mvccNum assign on returned
> cell.. This is not really needed as the seq of seqId assign and add to
> memstore is changed now.
> For the previous way we had to return added Cell also and that is why we were
> creating Pair object.Now we can avoid this so we can avoid creation of one
> Pair object and a wrapper Long object for every Cell add..
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14687) Un-synchronize BufferedMutator
[ https://issues.apache.org/jira/browse/HBASE-14687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-14687: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 1.3.0 1.2.0 2.0.0 Status: Resolved (was: Patch Available) Thanks for the reviews [~ashu210890] and [~ghelmling] > Un-synchronize BufferedMutator > -- > > Key: HBASE-14687 > URL: https://issues.apache.org/jira/browse/HBASE-14687 > Project: HBase > Issue Type: Improvement > Components: Client, Performance >Affects Versions: 2.0.0, 1.2.0 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Critical > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14687-v1.patch, HBASE-14687-v2.patch, > HBASE-14687-v3.patch, HBASE-14687-v4.patch, HBASE-14687-v5.patch, > HBASE-14687.patch > > > It should totally be possible to make BufferedMutatorImpl not use much > locking. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14540) Write Ahead Log Batching Optimization
[ https://issues.apache.org/jira/browse/HBASE-14540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14540: -- Assignee: John Leach Status: Patch Available (was: Open) > Write Ahead Log Batching Optimization > - > > Key: HBASE-14540 > URL: https://issues.apache.org/jira/browse/HBASE-14540 > Project: HBase > Issue Type: Improvement >Reporter: John Leach >Assignee: John Leach > Attachments: 14540.txt, HBaseWALBlockingWaitStrategy.java > > > The new write ahead log mechanism seems to batch too few mutations when > running inside the disruptor. As we scaled our load up (many threads with > small writes), we saw the number of hdfs sync operations grow in concert with > the number of writes. Generally, one would expect the size of the batches to > grow but the number of actual sync operations to settle. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14540) Write Ahead Log Batching Optimization
[ https://issues.apache.org/jira/browse/HBASE-14540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14540: -- Attachment: 14540.txt Make wait strategy pluggable. Add in Johns' waitFor 2ms as option. > Write Ahead Log Batching Optimization > - > > Key: HBASE-14540 > URL: https://issues.apache.org/jira/browse/HBASE-14540 > Project: HBase > Issue Type: Improvement >Reporter: John Leach > Attachments: 14540.txt, HBaseWALBlockingWaitStrategy.java > > > The new write ahead log mechanism seems to batch too few mutations when > running inside the disruptor. As we scaled our load up (many threads with > small writes), we saw the number of hdfs sync operations grow in concert with > the number of writes. Generally, one would expect the size of the batches to > grow but the number of actual sync operations to settle. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14729) SplitLogManager does not clean files from WALs folder in case of master failover
[
https://issues.apache.org/jira/browse/HBASE-14729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983253#comment-14983253
]
Enis Soztutar commented on HBASE-14729:
---
Thanks Samir. We should still fix HBASE-14223, but I could not test the patch
there yet. Feel free to pick it up if you are interested.
> SplitLogManager does not clean files from WALs folder in case of master
> failover
>
>
> Key: HBASE-14729
> URL: https://issues.apache.org/jira/browse/HBASE-14729
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.0.0
>Reporter: Samir Ahmic
>Assignee: Samir Ahmic
> Fix For: 2.0.0
>
> Attachments: HBASE-14729.patch
>
>
> While i was testing master failover process on master branch (distributed
> cluster setup) i notice following:
> 1. List of dead regionservers was increasing every time active master was
> restarted.
> 2. Number of folders in /hbase/WALs folder was increasing every time active
> master was restarted
> Here is exception from master logs showing why this is happening:
> {code}
> 2015-10-30 09:41:49,238 INFO [ProcedureExecutor-3] master.SplitLogManager:
> finished splitting (more than or equal to) 0 bytes in 0 log files in
> [hdfs://P3cluster/hbase/WALs/hnode1,16000,1446043659224-splitting] in 21ms
> 2015-10-30 09:41:49,235 WARN [ProcedureExecutor-2] master.SplitLogManager:
> Returning success without actually splitting and deleting all the log files
> in path hdfs://P3cluster/hbase/WALs/hnode1,16000,1446046595488-splitting:
> [FileStatus{path=hdfs://P3cluster/hbase/WALs/hnode1,16000,1446046595488-splitting/hnode1%2C16000%2C1446046595488.meta.1446046691314.meta;
> isDirectory=false; length=39944; replication=3; blocksize=268435456;
> modification_time=1446050348104; access_time=1446046691317; owner=hbase;
> group=supergroup; permission=rw-r--r--; isSymlink=false}]
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.fs.PathIsNotEmptyDirectoryException):
> `/hbase/WALs/hnode1,16000,1446046595488-splitting is non empty': Directory
> is not empty
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:3524)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInt(FSNamesystem.java:3479)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:3463)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.delete(NameNodeRpcServer.java:751)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.delete(ClientNamenodeProtocolServerSideTranslatorPB.java:562)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
> at org.apache.hadoop.ipc.Client.call(Client.java:1411)
> at org.apache.hadoop.ipc.Client.call(Client.java:1364)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy15.delete(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:490)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy16.delete(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
> at com.sun.proxy.$Proxy17.delete(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor7.inv
[jira] [Commented] (HBASE-14557) MapReduce WALPlayer issue with NoTagsKeyValue
[
https://issues.apache.org/jira/browse/HBASE-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983234#comment-14983234
]
Hudson commented on HBASE-14557:
FAILURE: Integrated in HBase-Trunk_matrix #413 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/413/])
HBASE-14557 MapReduce WALPlayer issue with NoTagsKeyValue. (anoopsamjohn: rev
b0ad82191cc216c65751b68cbc11280fd5d5cd47)
* hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValueUtil.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/mob/mapreduce/SweepReducer.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/mob/mapreduce/MemStoreWrapper.java
> MapReduce WALPlayer issue with NoTagsKeyValue
> -
>
> Key: HBASE-14557
> URL: https://issues.apache.org/jira/browse/HBASE-14557
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Jerry He
>Assignee: Anoop Sam John
>Priority: Blocker
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14557.patch, HBASE-14557.patch,
> HBASE-14557_V2.patch, HBASE-14557_branch-1.2.patch,
> HBASE-14557_branch-1.2.patch, HBASE-14557_branch-1.patch
>
>
> Running MapReduce WALPlayer to convert WAL into HFiles:
> {noformat}
> 15/10/05 20:28:08 INFO mapred.JobClient: Task Id :
> attempt_201508031611_0029_m_00_0, Status : FAILED
> java.io.IOException: Type mismatch in value from map: expected
> org.apache.hadoop.hbase.KeyValue, recieved
> org.apache.hadoop.hbase.NoTagsKeyValue
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:997)
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:689)
> at
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
> at
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALKeyValueMapper.map(WALPlayer.java:111)
> at
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALKeyValueMapper.map(WALPlayer.java:96)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:751)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:368)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at
> java.security.AccessController.doPrivileged(AccessController.java:369)
> at javax.security.auth.Subject.doAs(Subject.java:572)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1502)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14700) Support a "permissive" mode for secure clusters to allow "simple" auth clients
[ https://issues.apache.org/jira/browse/HBASE-14700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Helmling updated HBASE-14700: -- Fix Version/s: 0.98.16 1.2.0 > Support a "permissive" mode for secure clusters to allow "simple" auth clients > -- > > Key: HBASE-14700 > URL: https://issues.apache.org/jira/browse/HBASE-14700 > Project: HBase > Issue Type: Improvement > Components: security >Reporter: Gary Helmling >Assignee: Gary Helmling > Fix For: 2.0.0, 1.2.0, 0.98.16 > > Attachments: HBASE-14700-v2.patch, HBASE-14700-v3.patch, > HBASE-14700.patch > > > When implementing HBase security for an existing cluster, it can be useful to > support mixed secure and insecure clients while all client configurations are > migrated over to secure authentication. > We currently have an option to allow secure clients to fallback to simple > auth against insecure clusters. By providing an analogous setting for > servers, we would allow a phased rollout of security: > # First, security can be enabled on the cluster servers, with the > "permissive" mode enabled > # Clients can be converting to using secure authentication incrementally > # The server audit logs allow identification of clients still using simple > auth to connect > # Finally, when sufficient clients have been converted to secure operation, > the server-side "permissive" mode can be removed, allowing completely secure > operation. > Obviously with this enabled, there is no effective access control, but this > would still be a useful tool to enable a smooth operational rollout of > security. Permissive mode would of course be disabled by default. Enabling > it should provide a big scary warning in the logs on startup, and possibly be > flagged on relevant UIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14725) Vet categorization of tests so they for sure go into the right small/medium/large buckets
[
https://issues.apache.org/jira/browse/HBASE-14725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983190#comment-14983190
]
Hadoop QA commented on HBASE-14725:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12769819/14725v2%20%281%29.txt
against master branch at commit b0ad82191cc216c65751b68cbc11280fd5d5cd47.
ATTACHMENT ID: 12769819
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 36 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16314//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16314//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16314//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16314//console
This message is automatically generated.
> Vet categorization of tests so they for sure go into the right
> small/medium/large buckets
> -
>
> Key: HBASE-14725
> URL: https://issues.apache.org/jira/browse/HBASE-14725
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: stack
> Attachments: 14725v2 (1).txt, 14725v2.txt, 14725v2.txt,
> categorization.patch
>
>
> I tried doing runSmallTests, runMediumTests, etc., and I noticed that some
> tests are larger than our categorization. At least for small tests it means
> they area all running in the one JVM. I also noticed that the categorization
> only takes effect in hbase-server.
> This patch makes it so runSmallTests runs all the small tests only in each
> category. Also moves tests that were larger than small out to medium so they
> don't run in the one JVM anymore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14729) SplitLogManager does not clean files from WALs folder in case of master failover
[
https://issues.apache.org/jira/browse/HBASE-14729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Samir Ahmic updated HBASE-14729:
Resolution: Duplicate
Status: Resolved (was: Patch Available)
Duplicate of [HBASE-14223| https://issues.apache.org/jira/browse/HBASE-14223]
> SplitLogManager does not clean files from WALs folder in case of master
> failover
>
>
> Key: HBASE-14729
> URL: https://issues.apache.org/jira/browse/HBASE-14729
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.0.0
>Reporter: Samir Ahmic
>Assignee: Samir Ahmic
> Fix For: 2.0.0
>
> Attachments: HBASE-14729.patch
>
>
> While i was testing master failover process on master branch (distributed
> cluster setup) i notice following:
> 1. List of dead regionservers was increasing every time active master was
> restarted.
> 2. Number of folders in /hbase/WALs folder was increasing every time active
> master was restarted
> Here is exception from master logs showing why this is happening:
> {code}
> 2015-10-30 09:41:49,238 INFO [ProcedureExecutor-3] master.SplitLogManager:
> finished splitting (more than or equal to) 0 bytes in 0 log files in
> [hdfs://P3cluster/hbase/WALs/hnode1,16000,1446043659224-splitting] in 21ms
> 2015-10-30 09:41:49,235 WARN [ProcedureExecutor-2] master.SplitLogManager:
> Returning success without actually splitting and deleting all the log files
> in path hdfs://P3cluster/hbase/WALs/hnode1,16000,1446046595488-splitting:
> [FileStatus{path=hdfs://P3cluster/hbase/WALs/hnode1,16000,1446046595488-splitting/hnode1%2C16000%2C1446046595488.meta.1446046691314.meta;
> isDirectory=false; length=39944; replication=3; blocksize=268435456;
> modification_time=1446050348104; access_time=1446046691317; owner=hbase;
> group=supergroup; permission=rw-r--r--; isSymlink=false}]
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.fs.PathIsNotEmptyDirectoryException):
> `/hbase/WALs/hnode1,16000,1446046595488-splitting is non empty': Directory
> is not empty
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:3524)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInt(FSNamesystem.java:3479)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:3463)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.delete(NameNodeRpcServer.java:751)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.delete(ClientNamenodeProtocolServerSideTranslatorPB.java:562)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
> at org.apache.hadoop.ipc.Client.call(Client.java:1411)
> at org.apache.hadoop.ipc.Client.call(Client.java:1364)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy15.delete(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:490)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy16.delete(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
> at com.sun.proxy.$Proxy17.delete(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at
> sun.reflect
[jira] [Commented] (HBASE-14557) MapReduce WALPlayer issue with NoTagsKeyValue
[
https://issues.apache.org/jira/browse/HBASE-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983182#comment-14983182
]
Hudson commented on HBASE-14557:
FAILURE: Integrated in HBase-1.3-IT #283 (See
[https://builds.apache.org/job/HBase-1.3-IT/283/])
HBASE-14557 MapReduce WALPlayer issue with NoTagsKeyValue. (anoopsamjohn: rev
de89172244f3f403ec4ab0c38229c779ac4bb6a1)
* hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/WALPlayer.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TextSortReducer.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/PutSortReducer.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/PutCombiner.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestWALPlayer.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValueUtil.java
> MapReduce WALPlayer issue with NoTagsKeyValue
> -
>
> Key: HBASE-14557
> URL: https://issues.apache.org/jira/browse/HBASE-14557
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Jerry He
>Assignee: Anoop Sam John
>Priority: Blocker
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14557.patch, HBASE-14557.patch,
> HBASE-14557_V2.patch, HBASE-14557_branch-1.2.patch,
> HBASE-14557_branch-1.2.patch, HBASE-14557_branch-1.patch
>
>
> Running MapReduce WALPlayer to convert WAL into HFiles:
> {noformat}
> 15/10/05 20:28:08 INFO mapred.JobClient: Task Id :
> attempt_201508031611_0029_m_00_0, Status : FAILED
> java.io.IOException: Type mismatch in value from map: expected
> org.apache.hadoop.hbase.KeyValue, recieved
> org.apache.hadoop.hbase.NoTagsKeyValue
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:997)
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:689)
> at
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
> at
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALKeyValueMapper.map(WALPlayer.java:111)
> at
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALKeyValueMapper.map(WALPlayer.java:96)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:751)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:368)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at
> java.security.AccessController.doPrivileged(AccessController.java:369)
> at javax.security.auth.Subject.doAs(Subject.java:572)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1502)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14721) Memstore add cells - Avoid many garbage
[
https://issues.apache.org/jira/browse/HBASE-14721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983183#comment-14983183
]
Hudson commented on HBASE-14721:
FAILURE: Integrated in HBase-1.3-IT #283 (See
[https://builds.apache.org/job/HBase-1.3-IT/283/])
HBASE-14721 Memstore add cells - Avoid many garbage. (anoopsamjohn: rev
a3842caf77cfadefee87b5ce3e5397534d15d9e6)
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultMemStore.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
> Memstore add cells - Avoid many garbage
> ---
>
> Key: HBASE-14721
> URL: https://issues.apache.org/jira/browse/HBASE-14721
> Project: HBase
> Issue Type: Improvement
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14721.patch, HBASE-14721_branch-1.patch
>
>
> In HRegion# applyFamilyMapToMemstore()
> {code}
> for (int i=0; i < listSize; i++) {
> Cell cell = cells.get(i);
> if (cell.getSequenceId() == 0) {
> CellUtil.setSequenceId(cell, mvccNum);
> }
> Pair ret = store.add(cell);
> size += ret.getFirst();
> if(isInReplay) {
> // set memstore newly added cells with replay mvcc number
> CellUtil.setSequenceId(ret.getSecond(), mvccNum);
> }
> }
> {code}
> Previously we used to pass the added Cell reference to some HLog impl classes
> and later on the seqId used to get assigned. Now it looks totally changed.
> For normal put cells, the assign happen before adding to store.
> After addition we can see in replay mode again the mvccNum assign on returned
> cell.. This is not really needed as the seq of seqId assign and add to
> memstore is changed now.
> For the previous way we had to return added Cell also and that is why we were
> creating Pair object.Now we can avoid this so we can avoid creation of one
> Pair object and a wrapper Long object for every Cell add..
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14729) SplitLogManager does not clean files from WALs folder in case of master failover
[
https://issues.apache.org/jira/browse/HBASE-14729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983181#comment-14983181
]
Samir Ahmic commented on HBASE-14729:
-
bq. Is there a way we can see whether this is due to HBASE-14223?
I think this issue is a same as [HBASE-14223|
https://issues.apache.org/jira/browse/HBASE-14223] i used similar process for
reproducing it: writting data + killing active master every 30s. Since on
master branch master is also regionserver hosting meta table conditions were
same. Also logs are almost identical to logs in [HBASE-14223|
https://issues.apache.org/jira/browse/HBASE-14223]. I will close this issue as
duplicate.
> SplitLogManager does not clean files from WALs folder in case of master
> failover
>
>
> Key: HBASE-14729
> URL: https://issues.apache.org/jira/browse/HBASE-14729
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.0.0
>Reporter: Samir Ahmic
>Assignee: Samir Ahmic
> Fix For: 2.0.0
>
> Attachments: HBASE-14729.patch
>
>
> While i was testing master failover process on master branch (distributed
> cluster setup) i notice following:
> 1. List of dead regionservers was increasing every time active master was
> restarted.
> 2. Number of folders in /hbase/WALs folder was increasing every time active
> master was restarted
> Here is exception from master logs showing why this is happening:
> {code}
> 2015-10-30 09:41:49,238 INFO [ProcedureExecutor-3] master.SplitLogManager:
> finished splitting (more than or equal to) 0 bytes in 0 log files in
> [hdfs://P3cluster/hbase/WALs/hnode1,16000,1446043659224-splitting] in 21ms
> 2015-10-30 09:41:49,235 WARN [ProcedureExecutor-2] master.SplitLogManager:
> Returning success without actually splitting and deleting all the log files
> in path hdfs://P3cluster/hbase/WALs/hnode1,16000,1446046595488-splitting:
> [FileStatus{path=hdfs://P3cluster/hbase/WALs/hnode1,16000,1446046595488-splitting/hnode1%2C16000%2C1446046595488.meta.1446046691314.meta;
> isDirectory=false; length=39944; replication=3; blocksize=268435456;
> modification_time=1446050348104; access_time=1446046691317; owner=hbase;
> group=supergroup; permission=rw-r--r--; isSymlink=false}]
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.fs.PathIsNotEmptyDirectoryException):
> `/hbase/WALs/hnode1,16000,1446046595488-splitting is non empty': Directory
> is not empty
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:3524)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInt(FSNamesystem.java:3479)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:3463)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.delete(NameNodeRpcServer.java:751)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.delete(ClientNamenodeProtocolServerSideTranslatorPB.java:562)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
> at org.apache.hadoop.ipc.Client.call(Client.java:1411)
> at org.apache.hadoop.ipc.Client.call(Client.java:1364)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy15.delete(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:490)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy16.delete(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unk
[jira] [Commented] (HBASE-14723) Fix IT tests split too many times
[
https://issues.apache.org/jira/browse/HBASE-14723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983168#comment-14983168
]
Hadoop QA commented on HBASE-14723:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12769805/HBASE-14723.patch
against master branch at commit b0ad82191cc216c65751b68cbc11280fd5d5cd47.
ATTACHMENT ID: 12769805
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16313//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16313//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16313//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16313//console
This message is automatically generated.
> Fix IT tests split too many times
> -
>
> Key: HBASE-14723
> URL: https://issues.apache.org/jira/browse/HBASE-14723
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14723.patch
>
>
> Splitting the whole table is happening too often. Lets make this happen less
> frequently as there are more regions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14687) Un-synchronize BufferedMutator
[ https://issues.apache.org/jira/browse/HBASE-14687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983166#comment-14983166 ] Gary Helmling commented on HBASE-14687: --- +1 for v5 > Un-synchronize BufferedMutator > -- > > Key: HBASE-14687 > URL: https://issues.apache.org/jira/browse/HBASE-14687 > Project: HBase > Issue Type: Improvement > Components: Client, Performance >Affects Versions: 2.0.0, 1.2.0 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Critical > Attachments: HBASE-14687-v1.patch, HBASE-14687-v2.patch, > HBASE-14687-v3.patch, HBASE-14687-v4.patch, HBASE-14687-v5.patch, > HBASE-14687.patch > > > It should totally be possible to make BufferedMutatorImpl not use much > locking. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14651) Default minimum compaction size is too high
[ https://issues.apache.org/jira/browse/HBASE-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14651: -- Attachment: bytes.png files.png I applied the patch [~vrodionov] and it does not seem to make any difference. I have a cluster of 5 nodes with one RS on it. I load it using a YCSB all-writes workload. I start with an empty table for both tests. I use 25 clients. The count of files compacted and the number of bytes charts have the same shape (its uncanny). I checked that the patch was indeed applied. > Default minimum compaction size is too high > --- > > Key: HBASE-14651 > URL: https://issues.apache.org/jira/browse/HBASE-14651 > Project: HBase > Issue Type: New Feature >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-14651-v1.patch, HBASE-14651-v2.patch, bytes.png, > files.png > > > *hbase.hstore.compaction.min.size* defines minimum selection size which is > always eligible for minor compaction (no compaction ratio check is performed > on such file selections). Default size is equals to memstore flush size > (128MB). First of all, even this value is too high for some (many) > deployments, especially for write intensive, because of a small sizes of a > memstore flushes, and if user increases memstore flush size (they usually set > it to at least 256MB), they have no idea how will it impact the overall > compaction process efficiency. With 256MB of minimum size to compact, > compactor most of the time skips necessary file ratio checks and this will > result in increased read/write IO during compactions, because of the > unbalanced selections where relatively large files can be mixed with a newly > created small store files. I think we should set this default minimum to > 64MB and not to link it to memstore flush size at all. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13014) Java Tool For Region Moving
[
https://issues.apache.org/jira/browse/HBASE-13014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983147#comment-14983147
]
Hadoop QA commented on HBASE-13014:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12769804/HBASE-13014-master-v3.patch
against master branch at commit b0ad82191cc216c65751b68cbc11280fd5d5cd47.
ATTACHMENT ID: 12769804
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16311//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16311//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16311//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16311//console
This message is automatically generated.
> Java Tool For Region Moving
>
>
> Key: HBASE-13014
> URL: https://issues.apache.org/jira/browse/HBASE-13014
> Project: HBase
> Issue Type: Improvement
>Reporter: Abhishek Singh Chouhan
>Assignee: Abhishek Singh Chouhan
> Fix For: 2.0.0
>
> Attachments: HBASE-13014-master-v2.patch,
> HBASE-13014-master-v3.patch, HBASE-13014-master-v3.patch,
> HBASE-13014-master-v3.patch, HBASE-13014-master.patch, HBASE-13014-v2.patch,
> HBASE-13014-v3.patch, HBASE-13014-v4.patch, HBASE-13014-v5.patch,
> HBASE-13014-v6.patch, HBASE-13014.patch
>
>
> As per discussion on HBASE-12989 we should move the functionality of
> region_mover.rb into a Java tool and use region_mover.rb only only as a
> wrapper around it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14684) Try to remove all MiniMapReduceCluster in unit tests
[
https://issues.apache.org/jira/browse/HBASE-14684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983106#comment-14983106
]
Hadoop QA commented on HBASE-14684:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12769803/HBASE-14684-branch-1.patch
against branch-1 branch at commit b0ad82191cc216c65751b68cbc11280fd5d5cd47.
ATTACHMENT ID: 12769803
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 94 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16312//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16312//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16312//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16312//console
This message is automatically generated.
> Try to remove all MiniMapReduceCluster in unit tests
>
>
> Key: HBASE-14684
> URL: https://issues.apache.org/jira/browse/HBASE-14684
> Project: HBase
> Issue Type: Improvement
> Components: test
>Reporter: Heng Chen
>Assignee: Heng Chen
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: 14684.branch-1.txt, 14684.branch-1.txt,
> 14684.branch-1.txt, HBASE-14684-branch-1.patch, HBASE-14684-branch-1.patch,
> HBASE-14684-branch-1.patch, HBASE-14684.patch, HBASE-14684_v1.patch
>
>
> As discussion in dev list, we will try to do MR job without
> MiniMapReduceCluster.
> Testcases will run faster and more reliable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14575) Reduce scope of compactions holding region lock
[
https://issues.apache.org/jira/browse/HBASE-14575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983101#comment-14983101
]
Ted Yu commented on HBASE-14575:
Without the lock scope reduction, the test fails with:
{code}
testAtomicBulkLoad(org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad)
Time elapsed: 698.467 sec <<< ERROR!
java.lang.RuntimeException: Deferred
at
org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad.runAtomicBulkloadTest(TestHRegionServerBulkLoad.java:357)
at
org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad.testAtomicBulkLoad(TestHRegionServerBulkLoad.java:332)
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException:
Failed after attempts=36, exceptions:
Fri Oct 30 11:57:06 PDT 2015, null, java.net.SocketTimeoutException:
callTimeout=6, callDuration=77951: row '' on table 'atomicBulkLoad' at
region=atomicBulkLoad,,1446231338683.0235425b9d8c662d4839bf99e57d0072.,
hostname=10.22.24.71,53802,1446231335391, seqNum=2
Caused by: java.net.SocketTimeoutException: callTimeout=6,
callDuration=77951: row '' on table 'atomicBulkLoad' at
region=atomicBulkLoad,,1446231338683.0235425b9d8c662d4839bf99e57d0072.,
hostname=10.22.24.71,53802,1446231335391, seqNum=2
Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: callId=6837,
method=Scan, rpcTimeout=1, param {region { type: REGION_NAME value:
"atomicBulkLoad,,1446231338683.0235425b9d8c662d4839bf99e57d0072." } scan {
column { family: "family_" } column { family: "family_0001" } column {
family: "family_0002" } column { family: "family_0003" } column { family:
"family_0004" } column { family: "family_0005" } column { family: "family_0006"
} column { family: "family_0007" } column { family: "family_0008" } column {
family: "family_0009" } max_versions: 1 cache_blocks: true max_result_size:
2097152 caching: 2147483647 } number_of_rows: 0 close_scanner: false
client_handles_partials: true client_handles_heartbeats: true
track_scan_metrics: false}
{code}
> Reduce scope of compactions holding region lock
> ---
>
> Key: HBASE-14575
> URL: https://issues.apache.org/jira/browse/HBASE-14575
> Project: HBase
> Issue Type: Sub-task
> Components: Compaction, regionserver
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
> Attachments: 14575-v1.patch, 14575-v2.patch, 14575-v3.patch,
> 14575.v00.patch
>
>
> Per [~devaraj]'s idea on parent issue, let's see if we can reduce the scope
> of critical section under which compactions hold the region read lock.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14731) Add -DuseMob option to ITBLL
[ https://issues.apache.org/jira/browse/HBASE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-14731: --- Assignee: Jonathan Hsieh Fix Version/s: 2.0.0 Affects Version/s: 2.0.0 Status: Patch Available (was: Open) > Add -DuseMob option to ITBLL > > > Key: HBASE-14731 > URL: https://issues.apache.org/jira/browse/HBASE-14731 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: 2.0.0 > > Attachments: hbase-14731.patch > > > Previously, I would test mob by creating a table, altering to use mob and > then running ITBLL. This simplifies it by allowing for a command line arg to > enable mob on table creation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14557) MapReduce WALPlayer issue with NoTagsKeyValue
[
https://issues.apache.org/jira/browse/HBASE-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983071#comment-14983071
]
Hudson commented on HBASE-14557:
SUCCESS: Integrated in HBase-1.3 #328 (See
[https://builds.apache.org/job/HBase-1.3/328/])
HBASE-14557 MapReduce WALPlayer issue with NoTagsKeyValue. (anoopsamjohn: rev
de89172244f3f403ec4ab0c38229c779ac4bb6a1)
* hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/WALPlayer.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValueUtil.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/PutSortReducer.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestWALPlayer.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TextSortReducer.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/PutCombiner.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
> MapReduce WALPlayer issue with NoTagsKeyValue
> -
>
> Key: HBASE-14557
> URL: https://issues.apache.org/jira/browse/HBASE-14557
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Jerry He
>Assignee: Anoop Sam John
>Priority: Blocker
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14557.patch, HBASE-14557.patch,
> HBASE-14557_V2.patch, HBASE-14557_branch-1.2.patch,
> HBASE-14557_branch-1.2.patch, HBASE-14557_branch-1.patch
>
>
> Running MapReduce WALPlayer to convert WAL into HFiles:
> {noformat}
> 15/10/05 20:28:08 INFO mapred.JobClient: Task Id :
> attempt_201508031611_0029_m_00_0, Status : FAILED
> java.io.IOException: Type mismatch in value from map: expected
> org.apache.hadoop.hbase.KeyValue, recieved
> org.apache.hadoop.hbase.NoTagsKeyValue
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:997)
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:689)
> at
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
> at
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALKeyValueMapper.map(WALPlayer.java:111)
> at
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALKeyValueMapper.map(WALPlayer.java:96)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:751)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:368)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at
> java.security.AccessController.doPrivileged(AccessController.java:369)
> at javax.security.auth.Subject.doAs(Subject.java:572)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1502)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14731) Add -DuseMob option to ITBLL
[ https://issues.apache.org/jira/browse/HBASE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-14731: --- Attachment: hbase-14731.patch > Add -DuseMob option to ITBLL > > > Key: HBASE-14731 > URL: https://issues.apache.org/jira/browse/HBASE-14731 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Jonathan Hsieh > Fix For: 2.0.0 > > Attachments: hbase-14731.patch > > > Previously, I would test mob by creating a table, altering to use mob and > then running ITBLL. This simplifies it by allowing for a command line arg to > enable mob on table creation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14731) Add -DuseMob option to ITBLL
Jonathan Hsieh created HBASE-14731: -- Summary: Add -DuseMob option to ITBLL Key: HBASE-14731 URL: https://issues.apache.org/jira/browse/HBASE-14731 Project: HBase Issue Type: Bug Reporter: Jonathan Hsieh Previously, I would test mob by creating a table, altering to use mob and then running ITBLL. This simplifies it by allowing for a command line arg to enable mob on table creation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14731) Add -DuseMob option to ITBLL
[ https://issues.apache.org/jira/browse/HBASE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-14731: --- Issue Type: Improvement (was: Bug) > Add -DuseMob option to ITBLL > > > Key: HBASE-14731 > URL: https://issues.apache.org/jira/browse/HBASE-14731 > Project: HBase > Issue Type: Improvement >Reporter: Jonathan Hsieh > > Previously, I would test mob by creating a table, altering to use mob and > then running ITBLL. This simplifies it by allowing for a command line arg to > enable mob on table creation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14575) Reduce scope of compactions holding region lock
[ https://issues.apache.org/jira/browse/HBASE-14575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14575: --- Attachment: 14575-v3.patch Patch v3 modifies TestHRegionServerBulkLoad by introducing custom observer which sleeps in preCompact() for 30 seconds. rpc timeout is shortened to 10 seconds meanwhile. > Reduce scope of compactions holding region lock > --- > > Key: HBASE-14575 > URL: https://issues.apache.org/jira/browse/HBASE-14575 > Project: HBase > Issue Type: Sub-task > Components: Compaction, regionserver >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk > Attachments: 14575-v1.patch, 14575-v2.patch, 14575-v3.patch, > 14575.v00.patch > > > Per [~devaraj]'s idea on parent issue, let's see if we can reduce the scope > of critical section under which compactions hold the region read lock. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14728) TestRowCounter is broken in master
[
https://issues.apache.org/jira/browse/HBASE-14728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Singh Chouhan updated HBASE-14728:
---
Attachment: HBASE-14728.patch
> TestRowCounter is broken in master
> --
>
> Key: HBASE-14728
> URL: https://issues.apache.org/jira/browse/HBASE-14728
> Project: HBase
> Issue Type: Test
>Affects Versions: 2.0.0
>Reporter: Abhishek Singh Chouhan
>Assignee: Abhishek Singh Chouhan
> Fix For: 2.0.0
>
> Attachments: HBASE-14728.patch, HBASE-14728.patch
>
>
> The runRowCount method simply runs the RowCounter and checks if the exit code
> is zero but does not actually check the row count returned
> {noformat}
> private void runRowCount(String[] args, int expectedCount) throws Exception {
> final RowCounter counter = new RowCounter();
> assertEquals("job failed either due to failure or miscount (see log
> output).", 0,
> ToolRunner.run(TEST_UTIL.getConfiguration(), counter, args));
> }
> {noformat}
> This will always give a false positive provided the job ran without errors
> (irrespective of the count it gives).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14728) TestRowCounter is broken in master
[
https://issues.apache.org/jira/browse/HBASE-14728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983056#comment-14983056
]
Abhishek Singh Chouhan commented on HBASE-14728:
Test failed because of this:
Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.19:test
(secondPartTestsExecution) on project hbase-server: ExecutionException Error
occurred in starting fork, check output in log -> [Help 1]
Reattaching for another QA run.
> TestRowCounter is broken in master
> --
>
> Key: HBASE-14728
> URL: https://issues.apache.org/jira/browse/HBASE-14728
> Project: HBase
> Issue Type: Test
>Affects Versions: 2.0.0
>Reporter: Abhishek Singh Chouhan
>Assignee: Abhishek Singh Chouhan
> Fix For: 2.0.0
>
> Attachments: HBASE-14728.patch
>
>
> The runRowCount method simply runs the RowCounter and checks if the exit code
> is zero but does not actually check the row count returned
> {noformat}
> private void runRowCount(String[] args, int expectedCount) throws Exception {
> final RowCounter counter = new RowCounter();
> assertEquals("job failed either due to failure or miscount (see log
> output).", 0,
> ToolRunner.run(TEST_UTIL.getConfiguration(), counter, args));
> }
> {noformat}
> This will always give a false positive provided the job ran without errors
> (irrespective of the count it gives).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-14729) SplitLogManager does not clean files from WALs folder in case of master failover
[
https://issues.apache.org/jira/browse/HBASE-14729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14982630#comment-14982630
]
Ted Yu edited comment on HBASE-14729 at 10/30/15 6:44 PM:
--
Edit: we should investigate why the '-splitting' dir was not empty.
The JIRA Enis pointed to could be one cause.
was (Author: yuzhih...@gmail.com):
+1
> SplitLogManager does not clean files from WALs folder in case of master
> failover
>
>
> Key: HBASE-14729
> URL: https://issues.apache.org/jira/browse/HBASE-14729
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.0.0
>Reporter: Samir Ahmic
>Assignee: Samir Ahmic
> Fix For: 2.0.0
>
> Attachments: HBASE-14729.patch
>
>
> While i was testing master failover process on master branch (distributed
> cluster setup) i notice following:
> 1. List of dead regionservers was increasing every time active master was
> restarted.
> 2. Number of folders in /hbase/WALs folder was increasing every time active
> master was restarted
> Here is exception from master logs showing why this is happening:
> {code}
> 2015-10-30 09:41:49,238 INFO [ProcedureExecutor-3] master.SplitLogManager:
> finished splitting (more than or equal to) 0 bytes in 0 log files in
> [hdfs://P3cluster/hbase/WALs/hnode1,16000,1446043659224-splitting] in 21ms
> 2015-10-30 09:41:49,235 WARN [ProcedureExecutor-2] master.SplitLogManager:
> Returning success without actually splitting and deleting all the log files
> in path hdfs://P3cluster/hbase/WALs/hnode1,16000,1446046595488-splitting:
> [FileStatus{path=hdfs://P3cluster/hbase/WALs/hnode1,16000,1446046595488-splitting/hnode1%2C16000%2C1446046595488.meta.1446046691314.meta;
> isDirectory=false; length=39944; replication=3; blocksize=268435456;
> modification_time=1446050348104; access_time=1446046691317; owner=hbase;
> group=supergroup; permission=rw-r--r--; isSymlink=false}]
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.fs.PathIsNotEmptyDirectoryException):
> `/hbase/WALs/hnode1,16000,1446046595488-splitting is non empty': Directory
> is not empty
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:3524)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInt(FSNamesystem.java:3479)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:3463)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.delete(NameNodeRpcServer.java:751)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.delete(ClientNamenodeProtocolServerSideTranslatorPB.java:562)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
> at org.apache.hadoop.ipc.Client.call(Client.java:1411)
> at org.apache.hadoop.ipc.Client.call(Client.java:1364)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy15.delete(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:490)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy16.delete(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
> at com.sun.proxy.$Proxy17.delete(Unknown Source)
[jira] [Updated] (HBASE-14728) TestRowCounter is broken in master
[
https://issues.apache.org/jira/browse/HBASE-14728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Singh Chouhan updated HBASE-14728:
---
Attachment: (was: HBASE-14728.patch)
> TestRowCounter is broken in master
> --
>
> Key: HBASE-14728
> URL: https://issues.apache.org/jira/browse/HBASE-14728
> Project: HBase
> Issue Type: Test
>Affects Versions: 2.0.0
>Reporter: Abhishek Singh Chouhan
>Assignee: Abhishek Singh Chouhan
> Fix For: 2.0.0
>
> Attachments: HBASE-14728.patch
>
>
> The runRowCount method simply runs the RowCounter and checks if the exit code
> is zero but does not actually check the row count returned
> {noformat}
> private void runRowCount(String[] args, int expectedCount) throws Exception {
> final RowCounter counter = new RowCounter();
> assertEquals("job failed either due to failure or miscount (see log
> output).", 0,
> ToolRunner.run(TEST_UTIL.getConfiguration(), counter, args));
> }
> {noformat}
> This will always give a false positive provided the job ran without errors
> (irrespective of the count it gives).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14729) SplitLogManager does not clean files from WALs folder in case of master failover
[
https://issues.apache.org/jira/browse/HBASE-14729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983036#comment-14983036
]
Enis Soztutar commented on HBASE-14729:
---
We should not do this. Agreed that it is not safe to delete the WAL log
directory if it is not empty. Is there a way we can see whether this is due to
HBASE-14223?
> SplitLogManager does not clean files from WALs folder in case of master
> failover
>
>
> Key: HBASE-14729
> URL: https://issues.apache.org/jira/browse/HBASE-14729
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.0.0
>Reporter: Samir Ahmic
>Assignee: Samir Ahmic
> Fix For: 2.0.0
>
> Attachments: HBASE-14729.patch
>
>
> While i was testing master failover process on master branch (distributed
> cluster setup) i notice following:
> 1. List of dead regionservers was increasing every time active master was
> restarted.
> 2. Number of folders in /hbase/WALs folder was increasing every time active
> master was restarted
> Here is exception from master logs showing why this is happening:
> {code}
> 2015-10-30 09:41:49,238 INFO [ProcedureExecutor-3] master.SplitLogManager:
> finished splitting (more than or equal to) 0 bytes in 0 log files in
> [hdfs://P3cluster/hbase/WALs/hnode1,16000,1446043659224-splitting] in 21ms
> 2015-10-30 09:41:49,235 WARN [ProcedureExecutor-2] master.SplitLogManager:
> Returning success without actually splitting and deleting all the log files
> in path hdfs://P3cluster/hbase/WALs/hnode1,16000,1446046595488-splitting:
> [FileStatus{path=hdfs://P3cluster/hbase/WALs/hnode1,16000,1446046595488-splitting/hnode1%2C16000%2C1446046595488.meta.1446046691314.meta;
> isDirectory=false; length=39944; replication=3; blocksize=268435456;
> modification_time=1446050348104; access_time=1446046691317; owner=hbase;
> group=supergroup; permission=rw-r--r--; isSymlink=false}]
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.fs.PathIsNotEmptyDirectoryException):
> `/hbase/WALs/hnode1,16000,1446046595488-splitting is non empty': Directory
> is not empty
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:3524)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInt(FSNamesystem.java:3479)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:3463)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.delete(NameNodeRpcServer.java:751)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.delete(ClientNamenodeProtocolServerSideTranslatorPB.java:562)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
> at org.apache.hadoop.ipc.Client.call(Client.java:1411)
> at org.apache.hadoop.ipc.Client.call(Client.java:1364)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy15.delete(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:490)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy16.delete(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
> at com.sun.proxy.$Proxy17.delete(Unknown Source)
> at sun.reflect
[jira] [Commented] (HBASE-14557) MapReduce WALPlayer issue with NoTagsKeyValue
[
https://issues.apache.org/jira/browse/HBASE-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983033#comment-14983033
]
Hudson commented on HBASE-14557:
SUCCESS: Integrated in HBase-1.2 #327 (See
[https://builds.apache.org/job/HBase-1.2/327/])
HBASE-14557 MapReduce WALPlayer issue with NoTagsKeyValue. (anoopsamjohn: rev
28dc0abbe140bb1993e8659eef325758ac77c740)
* hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValueUtil.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/PutSortReducer.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestWALPlayer.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/WALPlayer.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/PutCombiner.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TextSortReducer.java
> MapReduce WALPlayer issue with NoTagsKeyValue
> -
>
> Key: HBASE-14557
> URL: https://issues.apache.org/jira/browse/HBASE-14557
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Jerry He
>Assignee: Anoop Sam John
>Priority: Blocker
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14557.patch, HBASE-14557.patch,
> HBASE-14557_V2.patch, HBASE-14557_branch-1.2.patch,
> HBASE-14557_branch-1.2.patch, HBASE-14557_branch-1.patch
>
>
> Running MapReduce WALPlayer to convert WAL into HFiles:
> {noformat}
> 15/10/05 20:28:08 INFO mapred.JobClient: Task Id :
> attempt_201508031611_0029_m_00_0, Status : FAILED
> java.io.IOException: Type mismatch in value from map: expected
> org.apache.hadoop.hbase.KeyValue, recieved
> org.apache.hadoop.hbase.NoTagsKeyValue
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:997)
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:689)
> at
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
> at
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALKeyValueMapper.map(WALPlayer.java:111)
> at
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALKeyValueMapper.map(WALPlayer.java:96)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:751)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:368)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at
> java.security.AccessController.doPrivileged(AccessController.java:369)
> at javax.security.auth.Subject.doAs(Subject.java:572)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1502)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14713) Remove simple deprecated-since-1.0 code in hbase-server from hbase 2.0
[ https://issues.apache.org/jira/browse/HBASE-14713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-14713: --- Attachment: hbase-14713.v3.patch v3. cleaned up some checkstyle and javadoc > Remove simple deprecated-since-1.0 code in hbase-server from hbase 2.0 > -- > > Key: HBASE-14713 > URL: https://issues.apache.org/jira/browse/HBASE-14713 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: 2.0.0 > > Attachments: hbase-14713.patch, hbase-14713.v2.patch, > hbase-14713.v3.patch > > > Removed items such as > * Migration to 96. > * HFileOutputFormat (HFileOutputFormat2 preferred) > * MapfilePerformanceEvaluttion > * getHTable -> getTable() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14711) Remove or annotated deprecated methods in HRegionInfo
[ https://issues.apache.org/jira/browse/HBASE-14711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-14711: --- Attachment: hbase-14711.v4.patch v4. cleaned up some checkstyle and javadoc > Remove or annotated deprecated methods in HRegionInfo > - > > Key: HBASE-14711 > URL: https://issues.apache.org/jira/browse/HBASE-14711 > Project: HBase > Issue Type: Bug > Components: hbase >Affects Versions: 2.0.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: 2.0.0 > > Attachments: hbase-14711.patch, hbase-14711.v2.patch, > hbase-14711.v3.patch, hbase-14711.v4.patch > > > Several methods were deprecated in 1.0 and moved to MetaTableAccessor. This > removes them from the 2.0 branch and converts them to the MetaTableAccessor. > HRegionInfo#getComparator was marked deprecated after 1.0 so needs to > remain. (HBASE-13501) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14703) update the per-region stats twice for the call on return
[
https://issues.apache.org/jira/browse/HBASE-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesse Yates updated HBASE-14703:
Attachment: HBASE-14703-start.patch
So I was getting confused with the result returned in the client-small-scan
that, I think, is just used to lookup the region; note that this does _not_
update any stats b/c the form of the result is 'wrong' for stats (its an array
of results).
Adding a second put, I see that only the asyncProcess check is used to collect
stats from the result.
The best I can tell is the StatsTrackingRpcRetryingCaller was there to catch
cases where we don't go through the multi() calls.
I think the "right" way to fix this, following the original intent of the code,
is to add support for unwrapping results from the other, non-put calls.
Currently, things like mutateRows _don't return a result_, even though there is
information there. Thus, the right thing to do would seem to be unwrapping that
result and letting the StatsTrackingRpcCaller handle the result's stats as we
would expect.
I'm attaching a patch that would be *starting point* for doing this - we would
need to add support in a couple more places as well.
If you apply this patch and then drop breakpoints in at
RSRpcServices#mutateRows and StatsTrackingRpcRetryingCaller#callWithRetries.
You'll see that the RS does in fact send along the stats information, but
HTable#mutateRow returns only retuns any kind of result after the changes in
the patch.
> update the per-region stats twice for the call on return
>
>
> Key: HBASE-14703
> URL: https://issues.apache.org/jira/browse/HBASE-14703
> Project: HBase
> Issue Type: Bug
>Reporter: Heng Chen
>Assignee: Heng Chen
> Attachments: HBASE-14703-start.patch, HBASE-14703.patch
>
>
> In {{AsyncProcess.SingleServerRequestRunnable}}, it seems we update
> serverStatistics twice.
> The first one is that we wrapper {{RetryingCallable}} by
> {{StatsTrackingRpcRetryingCaller}}, and do serverStatistics update when we
> call {{callWithRetries}} and {{callWithoutRetries}}. Relates code like below:
> {code}
> @Override
> public T callWithRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> @Override
> public T callWithoutRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> {code}
> The secondary one is after we get response, in {{receiveMultiAction}}, we do
> update again.
> {code}
> // update the stats about the region, if its a user table. We don't want to
> slow down
> // updates to meta tables, especially from internal updates (master, etc).
> if (AsyncProcess.this.connection.getStatisticsTracker() != null) {
> result = ResultStatsUtil.updateStats(result,
> AsyncProcess.this.connection.getStatisticsTracker(), server, regionName);
> }
> {code}
> It seems that {{StatsTrackingRpcRetryingCaller}} is NOT necessary, remove it?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
