[jira] [Created] (HBASE-9158) Serious bug in cyclic replication
Lars Hofhansl created HBASE-9158:
Summary: Serious bug in cyclic replication
Key: HBASE-9158
URL: https://issues.apache.org/jira/browse/HBASE-9158
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.10, 0.95.1, 0.98.0
Reporter: Lars Hofhansl
Priority: Critical
While studying the code for HBASE-7709, I found a serious bug in the current
cyclic replication code. The problem is here in HRegion.doMiniBatchMutation:
{code}
Mutation first = batchOp.operations[firstIndex].getFirst();
txid = this.log.appendNoSync(regionInfo, this.htableDescriptor.getName(),
walEdit, first.getClusterId(), now, this.htableDescriptor);
{code}
Now note that edits replicated from remote cluster and local edits might
interleave in the WAL, we might also receive edit from multiple remote
clusters. Hence that might have edits from many clusters in it, but
all are just labeled with the clusterId of the first Mutation.
Fixing this in doMiniBatchMutation seems tricky to do efficiently (imagine we
get a batch with cluster1, cluster2, cluster1, cluster2, ..., in that case each
edit would have to be its own batch). The coprocessor handling would also be
difficult.
The other option is create batches of Puts grouped by the cluster id in
ReplicationSink.replicateEntries(...), this is not as general, but equally
correct. This is the approach I would favor.
Lastly this is very hard to verify in a unittest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9158) Serious bug in cyclic replication
[
https://issues.apache.org/jira/browse/HBASE-9158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-9158:
-
Fix Version/s: 0.94.11
0.95.2
0.98.0
> Serious bug in cyclic replication
> -
>
> Key: HBASE-9158
> URL: https://issues.apache.org/jira/browse/HBASE-9158
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0, 0.95.1, 0.94.10
>Reporter: Lars Hofhansl
>Priority: Critical
> Fix For: 0.98.0, 0.95.2, 0.94.11
>
>
> While studying the code for HBASE-7709, I found a serious bug in the current
> cyclic replication code. The problem is here in HRegion.doMiniBatchMutation:
> {code}
> Mutation first = batchOp.operations[firstIndex].getFirst();
> txid = this.log.appendNoSync(regionInfo,
> this.htableDescriptor.getName(),
>walEdit, first.getClusterId(), now, this.htableDescriptor);
> {code}
> Now note that edits replicated from remote cluster and local edits might
> interleave in the WAL, we might also receive edit from multiple remote
> clusters. Hence that might have edits from many clusters in it, but
> all are just labeled with the clusterId of the first Mutation.
> Fixing this in doMiniBatchMutation seems tricky to do efficiently (imagine we
> get a batch with cluster1, cluster2, cluster1, cluster2, ..., in that case
> each edit would have to be its own batch). The coprocessor handling would
> also be difficult.
> The other option is create batches of Puts grouped by the cluster id in
> ReplicationSink.replicateEntries(...), this is not as general, but equally
> correct. This is the approach I would favor.
> Lastly this is very hard to verify in a unittest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9158) Serious bug in cyclic replication
[
https://issues.apache.org/jira/browse/HBASE-9158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733216#comment-13733216
]
Lars Hofhansl commented on HBASE-9158:
--
Maybe it can be added to doMiniBatchMutation after all. Will work on a patch
tomorrow.
> Serious bug in cyclic replication
> -
>
> Key: HBASE-9158
> URL: https://issues.apache.org/jira/browse/HBASE-9158
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0, 0.95.1, 0.94.10
>Reporter: Lars Hofhansl
>Priority: Critical
> Fix For: 0.98.0, 0.95.2, 0.94.11
>
>
> While studying the code for HBASE-7709, I found a serious bug in the current
> cyclic replication code. The problem is here in HRegion.doMiniBatchMutation:
> {code}
> Mutation first = batchOp.operations[firstIndex].getFirst();
> txid = this.log.appendNoSync(regionInfo,
> this.htableDescriptor.getName(),
>walEdit, first.getClusterId(), now, this.htableDescriptor);
> {code}
> Now note that edits replicated from remote cluster and local edits might
> interleave in the WAL, we might also receive edit from multiple remote
> clusters. Hence that might have edits from many clusters in it, but
> all are just labeled with the clusterId of the first Mutation.
> Fixing this in doMiniBatchMutation seems tricky to do efficiently (imagine we
> get a batch with cluster1, cluster2, cluster1, cluster2, ..., in that case
> each edit would have to be its own batch). The coprocessor handling would
> also be difficult.
> The other option is create batches of Puts grouped by the cluster id in
> ReplicationSink.replicateEntries(...), this is not as general, but equally
> correct. This is the approach I would favor.
> Lastly this is very hard to verify in a unittest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9158) Serious bug in cyclic replication
[
https://issues.apache.org/jira/browse/HBASE-9158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-9158:
-
Attachment: 9158-0.94.txt
Here's a possibility fixing it in doMiniBatchMutation.
This is probably not correct, as we now have multiple calls to
HLog.appendNoSync(...) and hence log-appending is no longer atomic; what if the
3rd call fails?
> Serious bug in cyclic replication
> -
>
> Key: HBASE-9158
> URL: https://issues.apache.org/jira/browse/HBASE-9158
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0, 0.95.1, 0.94.10
>Reporter: Lars Hofhansl
>Priority: Critical
> Fix For: 0.98.0, 0.95.2, 0.94.11
>
> Attachments: 9158-0.94.txt
>
>
> While studying the code for HBASE-7709, I found a serious bug in the current
> cyclic replication code. The problem is here in HRegion.doMiniBatchMutation:
> {code}
> Mutation first = batchOp.operations[firstIndex].getFirst();
> txid = this.log.appendNoSync(regionInfo,
> this.htableDescriptor.getName(),
>walEdit, first.getClusterId(), now, this.htableDescriptor);
> {code}
> Now note that edits replicated from remote cluster and local edits might
> interleave in the WAL, we might also receive edit from multiple remote
> clusters. Hence that might have edits from many clusters in it, but
> all are just labeled with the clusterId of the first Mutation.
> Fixing this in doMiniBatchMutation seems tricky to do efficiently (imagine we
> get a batch with cluster1, cluster2, cluster1, cluster2, ..., in that case
> each edit would have to be its own batch). The coprocessor handling would
> also be difficult.
> The other option is create batches of Puts grouped by the cluster id in
> ReplicationSink.replicateEntries(...), this is not as general, but equally
> correct. This is the approach I would favor.
> Lastly this is very hard to verify in a unittest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9136) RPC side changes to have a different codec for server to client communication
[ https://issues.apache.org/jira/browse/HBASE-9136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733225#comment-13733225 ] ramkrishna.s.vasudevan commented on HBASE-9136: --- One way to do this is while creating the RPCClient make it aware whether it is on the client side or on the server side. Use this information to initialize a different codec if that new codec is set if not go with the default codec. Another way could be, based on the RPCClient that is created(i.e whether it is on client side or on the server side), set this information to the codec class. So handle this flag inside the codec as what to do if the codec is set for a server or for the client. In case of replication, at least for the tag case the tag part should also be written. > RPC side changes to have a different codec for server to client communication > - > > Key: HBASE-9136 > URL: https://issues.apache.org/jira/browse/HBASE-9136 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 0.98.0 > > > With reference to the mail sent in the dev list, > http://comments.gmane.org/gmane.comp.java.hadoop.hbase.devel/38984 > We should have a provision such that the codec on the server side could be > different from the one on the client side. This would help to remove the > tags for security usecases. > This JIRA is aimed to provide that capability in the codec itself. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8408) Implement namespace
[
https://issues.apache.org/jira/browse/HBASE-8408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733234#comment-13733234
]
Hadoop QA commented on HBASE-8408:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12596781/HBASE-8015.095_21.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 663
new or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/6651//console
This message is automatically generated.
> Implement namespace
> ---
>
> Key: HBASE-8408
> URL: https://issues.apache.org/jira/browse/HBASE-8408
> Project: HBase
> Issue Type: Sub-task
>Reporter: Francis Liu
>Assignee: Francis Liu
> Attachments: HBASE-8015.095_21.txt, HBASE-8015_11.patch,
> HBASE-8015_12.patch, HBASE-8015_13.patch, HBASE-8015_14.patch,
> HBASE-8015_15.patch, HBASE-8015_16.patch, HBASE-8015_17.patch,
> HBASE-8015_18.patch, HBASE-8015_19.patch, HBASE-8015_1.patch,
> HBASE-8015_20.patch, HBASE-8015_2.patch, HBASE-8015_3.patch,
> HBASE-8015_4.patch, HBASE-8015_5.patch, HBASE-8015_6.patch,
> HBASE-8015_7.patch, HBASE-8015_8.patch, HBASE-8015_9.patch, HBASE-8015.patch,
> HBASE-8408.trunk.addendum.txt, TestNamespaceMigration.tgz,
> TestNamespaceUpgrade.tgz
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9098) During recovery use ZK as the source of truth for region state
[ https://issues.apache.org/jira/browse/HBASE-9098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeffrey Zhong updated HBASE-9098: - Attachment: (was: hbase-9098-v3.patch) > During recovery use ZK as the source of truth for region state > --- > > Key: HBASE-9098 > URL: https://issues.apache.org/jira/browse/HBASE-9098 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.95.0 >Reporter: Devaraj Das >Assignee: Jeffrey Zhong >Priority: Blocker > Fix For: 0.95.2 > > Attachments: hbase-9098.patch, hbase-9098-v1.patch, > hbase-9098-v3.patch > > > In HLogSplitter:locateRegionAndRefreshLastFlushedSequenceId(HConnection, > byte[], byte[], String), we talk to the replayee regionserver to figure out > whether a region is in recovery or not. We should look at ZK only for this > piece of information (since that is the source of truth for recovery > otherwise). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9098) During recovery use ZK as the source of truth for region state
[ https://issues.apache.org/jira/browse/HBASE-9098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeffrey Zhong updated HBASE-9098: - Attachment: hbase-9098-v3.patch Reattached the patch to trigger another test run. > During recovery use ZK as the source of truth for region state > --- > > Key: HBASE-9098 > URL: https://issues.apache.org/jira/browse/HBASE-9098 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.95.0 >Reporter: Devaraj Das >Assignee: Jeffrey Zhong >Priority: Blocker > Fix For: 0.95.2 > > Attachments: hbase-9098.patch, hbase-9098-v1.patch, > hbase-9098-v3.patch > > > In HLogSplitter:locateRegionAndRefreshLastFlushedSequenceId(HConnection, > byte[], byte[], String), we talk to the replayee regionserver to figure out > whether a region is in recovery or not. We should look at ZK only for this > piece of information (since that is the source of truth for recovery > otherwise). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9139) Independent timeout configuration for rpc channel between cluster nodes
[ https://issues.apache.org/jira/browse/HBASE-9139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Zhou updated HBASE-9139: --- Assignee: Julian Zhou > Independent timeout configuration for rpc channel between cluster nodes > --- > > Key: HBASE-9139 > URL: https://issues.apache.org/jira/browse/HBASE-9139 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC, regionserver >Affects Versions: 0.94.10, 0.96.0 >Reporter: Julian Zhou >Assignee: Julian Zhou >Priority: Minor > Fix For: 0.94.12, 0.96.0 > > > Default of "hbase.rpc.timeout" is 6 ms (1 min). User sometimes > increase them to a bigger value such as 60 ms (10 mins) for many > concurrent loading application from client. Some user share the same > hbase-site.xml for both client and server. HRegionServer > #tryRegionServerReport via rpc channel to report to live master, but > there was a window for master failover scenario. That region server > attempting to connect to master, which was just killed, backup master > took the active role immediately and put to /hbase/master, but region > server was still waiting for the rpc timeout from connecting to the dead > master. If "hbase.rpc.timeout" is too long, this master failover process > will be long due to long rpc timeout from dead master. > If so, could we separate with 2 options, "hbase.rpc.timeout" is still > for hbase client, while "hbase.rpc.internal.timeout" was for this > regionserver/master rpc channel, which could be set shorted value > without affect real client rpc timeout value? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9139) Independent timeout configuration for rpc channel between cluster nodes
[ https://issues.apache.org/jira/browse/HBASE-9139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Zhou updated HBASE-9139: --- Attachment: 9139-0.94-v0.patch > Independent timeout configuration for rpc channel between cluster nodes > --- > > Key: HBASE-9139 > URL: https://issues.apache.org/jira/browse/HBASE-9139 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC, regionserver >Affects Versions: 0.94.10, 0.96.0 >Reporter: Julian Zhou >Assignee: Julian Zhou >Priority: Minor > Fix For: 0.94.12, 0.96.0 > > Attachments: 9139-0.94-v0.patch > > > Default of "hbase.rpc.timeout" is 6 ms (1 min). User sometimes > increase them to a bigger value such as 60 ms (10 mins) for many > concurrent loading application from client. Some user share the same > hbase-site.xml for both client and server. HRegionServer > #tryRegionServerReport via rpc channel to report to live master, but > there was a window for master failover scenario. That region server > attempting to connect to master, which was just killed, backup master > took the active role immediately and put to /hbase/master, but region > server was still waiting for the rpc timeout from connecting to the dead > master. If "hbase.rpc.timeout" is too long, this master failover process > will be long due to long rpc timeout from dead master. > If so, could we separate with 2 options, "hbase.rpc.timeout" is still > for hbase client, while "hbase.rpc.internal.timeout" was for this > regionserver/master rpc channel, which could be set shorted value > without affect real client rpc timeout value? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9139) Independent timeout configuration for rpc channel between cluster nodes
[ https://issues.apache.org/jira/browse/HBASE-9139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733241#comment-13733241 ] Julian Zhou commented on HBASE-9139: Hi [~nkeywal], agree. I just attached the v0 patch for 0.94, currently, seems that only regionserver's reporting to master needs this short rpc timeout setting. HConnectionManager for client call is w/o this change. Could you help review? If the trunk logic for this was the same, I will attach the version for trunk afterwards. Thanks~ > Independent timeout configuration for rpc channel between cluster nodes > --- > > Key: HBASE-9139 > URL: https://issues.apache.org/jira/browse/HBASE-9139 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC, regionserver >Affects Versions: 0.94.10, 0.96.0 >Reporter: Julian Zhou >Assignee: Julian Zhou >Priority: Minor > Fix For: 0.94.12, 0.96.0 > > Attachments: 9139-0.94-v0.patch > > > Default of "hbase.rpc.timeout" is 6 ms (1 min). User sometimes > increase them to a bigger value such as 60 ms (10 mins) for many > concurrent loading application from client. Some user share the same > hbase-site.xml for both client and server. HRegionServer > #tryRegionServerReport via rpc channel to report to live master, but > there was a window for master failover scenario. That region server > attempting to connect to master, which was just killed, backup master > took the active role immediately and put to /hbase/master, but region > server was still waiting for the rpc timeout from connecting to the dead > master. If "hbase.rpc.timeout" is too long, this master failover process > will be long due to long rpc timeout from dead master. > If so, could we separate with 2 options, "hbase.rpc.timeout" is still > for hbase client, while "hbase.rpc.internal.timeout" was for this > regionserver/master rpc channel, which could be set shorted value > without affect real client rpc timeout value? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9139) Independent timeout configuration for rpc channel between cluster nodes
[ https://issues.apache.org/jira/browse/HBASE-9139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733242#comment-13733242 ] Julian Zhou commented on HBASE-9139: Currently, I just name it as "hbase.rpc.shortoperation.timeout", default is 10s. > Independent timeout configuration for rpc channel between cluster nodes > --- > > Key: HBASE-9139 > URL: https://issues.apache.org/jira/browse/HBASE-9139 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC, regionserver >Affects Versions: 0.94.10, 0.96.0 >Reporter: Julian Zhou >Assignee: Julian Zhou >Priority: Minor > Fix For: 0.94.12, 0.96.0 > > Attachments: 9139-0.94-v0.patch > > > Default of "hbase.rpc.timeout" is 6 ms (1 min). User sometimes > increase them to a bigger value such as 60 ms (10 mins) for many > concurrent loading application from client. Some user share the same > hbase-site.xml for both client and server. HRegionServer > #tryRegionServerReport via rpc channel to report to live master, but > there was a window for master failover scenario. That region server > attempting to connect to master, which was just killed, backup master > took the active role immediately and put to /hbase/master, but region > server was still waiting for the rpc timeout from connecting to the dead > master. If "hbase.rpc.timeout" is too long, this master failover process > will be long due to long rpc timeout from dead master. > If so, could we separate with 2 options, "hbase.rpc.timeout" is still > for hbase client, while "hbase.rpc.internal.timeout" was for this > regionserver/master rpc channel, which could be set shorted value > without affect real client rpc timeout value? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9098) During recovery use ZK as the source of truth for region state
[
https://issues.apache.org/jira/browse/HBASE-9098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733300#comment-13733300
]
Hadoop QA commented on HBASE-9098:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12596790/hbase-9098-v3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
18 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/6652//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6652//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6652//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6652//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6652//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6652//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6652//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6652//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6652//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/6652//console
This message is automatically generated.
> During recovery use ZK as the source of truth for region state
> ---
>
> Key: HBASE-9098
> URL: https://issues.apache.org/jira/browse/HBASE-9098
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 0.95.0
>Reporter: Devaraj Das
>Assignee: Jeffrey Zhong
>Priority: Blocker
> Fix For: 0.95.2
>
> Attachments: hbase-9098.patch, hbase-9098-v1.patch,
> hbase-9098-v3.patch
>
>
> In HLogSplitter:locateRegionAndRefreshLastFlushedSequenceId(HConnection,
> byte[], byte[], String), we talk to the replayee regionserver to figure out
> whether a region is in recovery or not. We should look at ZK only for this
> piece of information (since that is the source of truth for recovery
> otherwise).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8015) Support for Namespaces
[ https://issues.apache.org/jira/browse/HBASE-8015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733303#comment-13733303 ] Hudson commented on HBASE-8015: --- FAILURE: Integrated in HBase-TRUNK #4355 (See [https://builds.apache.org/job/HBase-TRUNK/4355/]) HBASE-8015 implement namespaces; ADDENDUM TWO -- REMOVE FILE REMOVED BY ORIGINAL PATCH (stack: rev 1511589) * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TableName.java > Support for Namespaces > -- > > Key: HBASE-8015 > URL: https://issues.apache.org/jira/browse/HBASE-8015 > Project: HBase > Issue Type: New Feature >Reporter: Francis Liu >Assignee: Francis Liu > Attachments: HBASE-8015_draft_94.patch, Namespace Design.pdf > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8408) Implement namespace
[ https://issues.apache.org/jira/browse/HBASE-8408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733302#comment-13733302 ] Hudson commented on HBASE-8408: --- FAILURE: Integrated in HBase-TRUNK #4355 (See [https://builds.apache.org/job/HBase-TRUNK/4355/]) HBASE-8408 Implement namespace; ADDENDUM (stack: rev 1511587) * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestMergeTool.java > Implement namespace > --- > > Key: HBASE-8408 > URL: https://issues.apache.org/jira/browse/HBASE-8408 > Project: HBase > Issue Type: Sub-task >Reporter: Francis Liu >Assignee: Francis Liu > Attachments: HBASE-8015.095_21.txt, HBASE-8015_11.patch, > HBASE-8015_12.patch, HBASE-8015_13.patch, HBASE-8015_14.patch, > HBASE-8015_15.patch, HBASE-8015_16.patch, HBASE-8015_17.patch, > HBASE-8015_18.patch, HBASE-8015_19.patch, HBASE-8015_1.patch, > HBASE-8015_20.patch, HBASE-8015_2.patch, HBASE-8015_3.patch, > HBASE-8015_4.patch, HBASE-8015_5.patch, HBASE-8015_6.patch, > HBASE-8015_7.patch, HBASE-8015_8.patch, HBASE-8015_9.patch, HBASE-8015.patch, > HBASE-8408.trunk.addendum.txt, TestNamespaceMigration.tgz, > TestNamespaceUpgrade.tgz > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9079) FilterList getNextKeyHint skips rows that should be included in the results
[ https://issues.apache.org/jira/browse/HBASE-9079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1370#comment-1370 ] Viral Bajaria commented on HBASE-9079: -- Sorry for the delay. I did a local test with production data and it looks fine. > FilterList getNextKeyHint skips rows that should be included in the results > --- > > Key: HBASE-9079 > URL: https://issues.apache.org/jira/browse/HBASE-9079 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 0.94.10 >Reporter: Viral Bajaria >Assignee: Viral Bajaria > Fix For: 0.98.0, 0.95.2, 0.94.11 > > Attachments: 9079-0.94-v2.txt, 9079-trunk-v2.txt, > HBASE-9079-0.94.patch, HBASE-9079-trunk.patch > > > I hit a weird issue/bug and am able to reproduce the error consistently. The > problem arises when FilterList has two filters where each implements the > getNextKeyHint method. > The way the current implementation works is, StoreScanner will call > matcher.getNextKeyHint() whenever it gets a SEEK_NEXT_USING_HINT. This in > turn will call filter.getNextKeyHint() which at this stage is of type > FilterList. The implementation in FilterList iterates through all the filters > and keeps the max KeyValue that it sees. All is fine if you wrap filters in > FilterList in which only one of them implements getNextKeyHint. but if > multiple of them implement then that's where things get weird. > For example: > - create two filters: one is FuzzyRowFilter and second is ColumnRangeFilter. > Both of them implement getNextKeyHint > - wrap them in FilterList with MUST_PASS_ALL > - FuzzyRowFilter will seek to the correct first row and then pass it to > ColumnRangeFilter which will return the SEEK_NEXT_USING_HINT code. > - Now in FilterList when getNextKeyHint is called, it calls the one on > FuzzyRow first which basically says what the next row should be. While in > reality we want the ColumnRangeFilter to give the seek hint. > - The above behavior skips data that should be returned, which I have > verified by using a RowFilter with RegexStringComparator. > I updated the FilterList to maintain state on which filter returns the > SEEK_NEXT_USING_HINT and in getNextKeyHint, I invoke the method on the saved > filter and reset that state. I tested it with my current queries and it works > fine but I need to run the entire test suite to make sure I have not > introduced any regression. In addition to that I need to figure out what > should be the behavior when the opeation is MUST_PASS_ONE, but I doubt it > should be any different. > Is my understanding of it being a bug correct ? Or am I trivializing it and > ignoring something very important ? If it's tough to wrap your head around > the explanation, then I can open a JIRA and upload a patch against 0.94 head. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8533) [REST] HBaseAdmin does not ride over cluster restart
[ https://issues.apache.org/jira/browse/HBASE-8533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Zhou updated HBASE-8533: --- Component/s: (was: scripts) (was: REST) > [REST] HBaseAdmin does not ride over cluster restart > > > Key: HBASE-8533 > URL: https://issues.apache.org/jira/browse/HBASE-8533 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.98.0, 0.95.0 >Reporter: Julian Zhou >Assignee: Julian Zhou >Priority: Minor > Fix For: 0.98.0, 0.95.0 > > Attachments: 8533-0.95-v1.patch, 8533-trunk-v1.patch, > hbase-8533-trunk-v0.patch > > > For Restful servlet (org.apache.hadoop.hbase.rest.Main (0.94), > org.apache.hadoop.hbase.rest.RESTServer (trunk)) on Jetty, we need to first > explicitly start the service (% ./bin/hbase-daemon.sh start rest -p 8000 ) > for application running. Here is a scenario, sometimes, HBase cluster are > stopped/started for maintanence, but rest is a seperated standalone process, > which binds the HBaseAdmin at construction method. > HBase stop/start cause this binding lost for existing rest servlet. Rest > servlet still exist to trying on old bound HBaseAdmin until a long time > duration later with an "Unavailable" caught via an IOException caught in > such as RootResource. > Could we pairwise the HBase service with HBase rest service with some > start/stop options? since seems no reason to still keep the rest servlet > process after HBase stopped? When HBase restarts, original rest service could > not resume to bind to the new HBase service via its old HBaseAdmin reference? > So may we stop the rest when hbase stopped, or even if hbase was killed by > acident, restart hbase with rest option could detect the old rest process, > kill it and start to bind a new one? > From this point of view, application rely on rest api in previous scenario > could immediately detect it when setting up http connection session instead > of wasting a long time to fail back from IOException with "Unavailable" from > rest servlet. > Put current options from the discussion history here from Andrew, Stack and > Jean-Daniel, > 1) create an HBaseAdmin on demand in rest servlet instead of keeping > singleton instance; (another possible enhancement for HBase client: automatic > reconnection of an open HBaseAdmin handle after a cluster bounce?) > 2) pairwise the rest webapp with hbase webui so the rest is always on with > HBase serive; > 3) add an option for rest service (such as HBASE_MANAGES_REST) in > hbase-env.sh, set HBASE_MANAGES_REST to true, the scripts will start/stop the > REST server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8533) HBaseAdmin does not ride over cluster restart
[ https://issues.apache.org/jira/browse/HBASE-8533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Zhou updated HBASE-8533: --- Summary: HBaseAdmin does not ride over cluster restart (was: [REST] HBaseAdmin does not ride over cluster restart) > HBaseAdmin does not ride over cluster restart > - > > Key: HBASE-8533 > URL: https://issues.apache.org/jira/browse/HBASE-8533 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.98.0, 0.95.0 >Reporter: Julian Zhou >Assignee: Julian Zhou >Priority: Minor > Fix For: 0.98.0, 0.95.0 > > Attachments: 8533-0.95-v1.patch, 8533-trunk-v1.patch, > hbase-8533-trunk-v0.patch > > > For Restful servlet (org.apache.hadoop.hbase.rest.Main (0.94), > org.apache.hadoop.hbase.rest.RESTServer (trunk)) on Jetty, we need to first > explicitly start the service (% ./bin/hbase-daemon.sh start rest -p 8000 ) > for application running. Here is a scenario, sometimes, HBase cluster are > stopped/started for maintanence, but rest is a seperated standalone process, > which binds the HBaseAdmin at construction method. > HBase stop/start cause this binding lost for existing rest servlet. Rest > servlet still exist to trying on old bound HBaseAdmin until a long time > duration later with an "Unavailable" caught via an IOException caught in > such as RootResource. > Could we pairwise the HBase service with HBase rest service with some > start/stop options? since seems no reason to still keep the rest servlet > process after HBase stopped? When HBase restarts, original rest service could > not resume to bind to the new HBase service via its old HBaseAdmin reference? > So may we stop the rest when hbase stopped, or even if hbase was killed by > acident, restart hbase with rest option could detect the old rest process, > kill it and start to bind a new one? > From this point of view, application rely on rest api in previous scenario > could immediately detect it when setting up http connection session instead > of wasting a long time to fail back from IOException with "Unavailable" from > rest servlet. > Put current options from the discussion history here from Andrew, Stack and > Jean-Daniel, > 1) create an HBaseAdmin on demand in rest servlet instead of keeping > singleton instance; (another possible enhancement for HBase client: automatic > reconnection of an open HBaseAdmin handle after a cluster bounce?) > 2) pairwise the rest webapp with hbase webui so the rest is always on with > HBase serive; > 3) add an option for rest service (such as HBASE_MANAGES_REST) in > hbase-env.sh, set HBASE_MANAGES_REST to true, the scripts will start/stop the > REST server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8533) HBaseAdmin does not ride over cluster restart
[ https://issues.apache.org/jira/browse/HBASE-8533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Zhou updated HBASE-8533: --- Component/s: Client > HBaseAdmin does not ride over cluster restart > - > > Key: HBASE-8533 > URL: https://issues.apache.org/jira/browse/HBASE-8533 > Project: HBase > Issue Type: Improvement > Components: Client >Affects Versions: 0.98.0, 0.95.0 >Reporter: Julian Zhou >Assignee: Julian Zhou >Priority: Minor > Fix For: 0.98.0, 0.95.0 > > Attachments: 8533-0.95-v1.patch, 8533-trunk-v1.patch, > hbase-8533-trunk-v0.patch > > > For Restful servlet (org.apache.hadoop.hbase.rest.Main (0.94), > org.apache.hadoop.hbase.rest.RESTServer (trunk)) on Jetty, we need to first > explicitly start the service (% ./bin/hbase-daemon.sh start rest -p 8000 ) > for application running. Here is a scenario, sometimes, HBase cluster are > stopped/started for maintanence, but rest is a seperated standalone process, > which binds the HBaseAdmin at construction method. > HBase stop/start cause this binding lost for existing rest servlet. Rest > servlet still exist to trying on old bound HBaseAdmin until a long time > duration later with an "Unavailable" caught via an IOException caught in > such as RootResource. > Could we pairwise the HBase service with HBase rest service with some > start/stop options? since seems no reason to still keep the rest servlet > process after HBase stopped? When HBase restarts, original rest service could > not resume to bind to the new HBase service via its old HBaseAdmin reference? > So may we stop the rest when hbase stopped, or even if hbase was killed by > acident, restart hbase with rest option could detect the old rest process, > kill it and start to bind a new one? > From this point of view, application rely on rest api in previous scenario > could immediately detect it when setting up http connection session instead > of wasting a long time to fail back from IOException with "Unavailable" from > rest servlet. > Put current options from the discussion history here from Andrew, Stack and > Jean-Daniel, > 1) create an HBaseAdmin on demand in rest servlet instead of keeping > singleton instance; (another possible enhancement for HBase client: automatic > reconnection of an open HBaseAdmin handle after a cluster bounce?) > 2) pairwise the rest webapp with hbase webui so the rest is always on with > HBase serive; > 3) add an option for rest service (such as HBASE_MANAGES_REST) in > hbase-env.sh, set HBASE_MANAGES_REST to true, the scripts will start/stop the > REST server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-7391) Review/improve HLog compression's memory consumption
[
https://issues.apache.org/jira/browse/HBASE-7391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733363#comment-13733363
]
ramkrishna.s.vasudevan edited comment on HBASE-7391 at 8/8/13 10:44 AM:
Patch for trunk. Once HBASE-8615 is fixed this can be checked for.
Ran the test suite. Got one failure in TestWALReplay but running it once again
it passed.
{Edit}HBASE-8315 to HBASE-8615 {Edit}
was (Author: ram_krish):
Patch for trunk. Once HBASE-8315 is fixed this can be checked for.
Ran the test suite. Got one failure in TestWALReplay but running it once again
it passed.
> Review/improve HLog compression's memory consumption
>
>
> Key: HBASE-7391
> URL: https://issues.apache.org/jira/browse/HBASE-7391
> Project: HBase
> Issue Type: Bug
>Reporter: Jean-Daniel Cryans
>Assignee: ramkrishna.s.vasudevan
> Fix For: 0.95.2
>
> Attachments: HBASE-7391_1.patch
>
>
> From Ram in
> http://mail-archives.apache.org/mod_mbox/hbase-dev/201205.mbox/%3C00bc01cd31e6$7caf1320$760d3960$%25vasude...@huawei.com%3E:
> {quote}
> One small observation after giving +1 on the RC.
> The WAL compression feature causes OOME and causes Full GC.
> The problem is, if we have 1500 regions and I need to create recovered.edits
> for each of the region (I don’t have much data in the regions (~300MB)).
> Now when I try to build the dictionary there is a Node object getting
> created.
> Each node object occupies 32 bytes.
> We have 5 such dictionaries.
> Initially we create indexToNodes array and its size is 32767.
> So now we have 32*5*32767 = ~5MB.
> Now I have 1500 regions.
> So 5MB*1500 = ~7GB.(Excluding actual data). This seems to a very high
> initial memory foot print and this never allows me to split the logs and I
> am not able to make the cluster up at all.
> Our configured heap size was 8GB, tested in 3 node cluster with 5000
> regions, very less data( 1GB in hdfs cluster including replication), some
> small data is spread evenly across all regions.
> The formula is 32(Node object size)*5(No of dictionary)*32767(no of node
> objects)*noofregions.
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7391) Review/improve HLog compression's memory consumption
[
https://issues.apache.org/jira/browse/HBASE-7391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-7391:
--
Status: Patch Available (was: Open)
> Review/improve HLog compression's memory consumption
>
>
> Key: HBASE-7391
> URL: https://issues.apache.org/jira/browse/HBASE-7391
> Project: HBase
> Issue Type: Bug
>Reporter: Jean-Daniel Cryans
>Assignee: ramkrishna.s.vasudevan
> Fix For: 0.95.2
>
> Attachments: HBASE-7391_1.patch
>
>
> From Ram in
> http://mail-archives.apache.org/mod_mbox/hbase-dev/201205.mbox/%3C00bc01cd31e6$7caf1320$760d3960$%25vasude...@huawei.com%3E:
> {quote}
> One small observation after giving +1 on the RC.
> The WAL compression feature causes OOME and causes Full GC.
> The problem is, if we have 1500 regions and I need to create recovered.edits
> for each of the region (I don’t have much data in the regions (~300MB)).
> Now when I try to build the dictionary there is a Node object getting
> created.
> Each node object occupies 32 bytes.
> We have 5 such dictionaries.
> Initially we create indexToNodes array and its size is 32767.
> So now we have 32*5*32767 = ~5MB.
> Now I have 1500 regions.
> So 5MB*1500 = ~7GB.(Excluding actual data). This seems to a very high
> initial memory foot print and this never allows me to split the logs and I
> am not able to make the cluster up at all.
> Our configured heap size was 8GB, tested in 3 node cluster with 5000
> regions, very less data( 1GB in hdfs cluster including replication), some
> small data is spread evenly across all regions.
> The formula is 32(Node object size)*5(No of dictionary)*32767(no of node
> objects)*noofregions.
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7391) Review/improve HLog compression's memory consumption
[
https://issues.apache.org/jira/browse/HBASE-7391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-7391:
--
Attachment: HBASE-7391_1.patch
Patch for trunk. Once HBASE-8315 is fixed this can be checked for.
Ran the test suite. Got one failure in TestWALReplay but running it once again
it passed.
> Review/improve HLog compression's memory consumption
>
>
> Key: HBASE-7391
> URL: https://issues.apache.org/jira/browse/HBASE-7391
> Project: HBase
> Issue Type: Bug
>Reporter: Jean-Daniel Cryans
>Assignee: ramkrishna.s.vasudevan
> Fix For: 0.95.2
>
> Attachments: HBASE-7391_1.patch
>
>
> From Ram in
> http://mail-archives.apache.org/mod_mbox/hbase-dev/201205.mbox/%3C00bc01cd31e6$7caf1320$760d3960$%25vasude...@huawei.com%3E:
> {quote}
> One small observation after giving +1 on the RC.
> The WAL compression feature causes OOME and causes Full GC.
> The problem is, if we have 1500 regions and I need to create recovered.edits
> for each of the region (I don’t have much data in the regions (~300MB)).
> Now when I try to build the dictionary there is a Node object getting
> created.
> Each node object occupies 32 bytes.
> We have 5 such dictionaries.
> Initially we create indexToNodes array and its size is 32767.
> So now we have 32*5*32767 = ~5MB.
> Now I have 1500 regions.
> So 5MB*1500 = ~7GB.(Excluding actual data). This seems to a very high
> initial memory foot print and this never allows me to split the logs and I
> am not able to make the cluster up at all.
> Our configured heap size was 8GB, tested in 3 node cluster with 5000
> regions, very less data( 1GB in hdfs cluster including replication), some
> small data is spread evenly across all regions.
> The formula is 32(Node object size)*5(No of dictionary)*32767(no of node
> objects)*noofregions.
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8408) Implement namespace
[ https://issues.apache.org/jira/browse/HBASE-8408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733376#comment-13733376 ] Hudson commented on HBASE-8408: --- FAILURE: Integrated in hbase-0.95 #415 (See [https://builds.apache.org/job/hbase-0.95/415/]) HBASE-8408 implement Namespaces (stack: rev 1511591) * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/CoprocessorEnvironment.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/TableExistsException.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/TableNotDisabledException.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/TableNotEnabledException.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/TableNotFoundException.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/catalog/MetaReader.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnection.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnectionWrapper.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTablePool.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/MultiServerCallable.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/RegionServerCallable.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Registry.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/UnmodifyableHTableDescriptor.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ZooKeeperRegistry.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/coprocessor/SecureBulkLoadClient.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/RegionCoprocessorRpcChannel.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/security/access/TablePermission.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/security/access/UserPermission.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/snapshot/ClientSnapshotDescriptionUtils.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/snapshot/TablePartiallyOpenException.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKTable.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKTableReadOnly.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java * /hbase/branches/0.95/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestAsyncProcess.java * /hbase/branches/0.95/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestClientNoCluster.java * /hbase/branches/0.95/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotFromAdmin.java * /hbase/branches/0.95/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java * /hbase/branches/0.95/hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java * /hbase/branches/0.95/hbase-common/src/main/java/org/apache/hadoop/hbase/NamespaceDescriptor.java * /hbase/branches/0.95/hbase-common/src/main/java/org/apache/hadoop/hbase/TableName.java * /hbase/branches/0.95/hbase-common/src/main/java/org/apache/hadoop/hbase/util/Bytes.java * /hbase/branches/0.95/hbase-common/src/test/java/org/apache/hadoop/
[jira] [Updated] (HBASE-9142) Mutation#getFamilyMap() return type change between HBase 94 and 96 breaks downstream apps
[ https://issues.apache.org/jira/browse/HBASE-9142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-9142: -- Attachment: hbase-9142-0.95.patch > Mutation#getFamilyMap() return type change between HBase 94 and 96 breaks > downstream apps > - > > Key: HBASE-9142 > URL: https://issues.apache.org/jira/browse/HBASE-9142 > Project: HBase > Issue Type: Bug >Reporter: Hari Shreedharan >Assignee: Jonathan Hsieh >Priority: Critical > Fix For: 0.95.2 > > Attachments: hbase-9142-0.95.patch, hbase-9142-trunk.patch > > > In Hbase-94, Mutation#getFamilyMap() had signature: > public Map> getFamilyMap() > In Hbase-96 it is: > public NavigableMap> getFamilyMap() > I understand this might not be an easy fix or even a possible one - but it > breaks downstream apps in a nasty way. If the app needs to process the > individual columns, then the whole logic is now different. Is there a way to > work around this, if this cannot be fixed? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9142) Mutation#getFamilyMap() return type change between HBase 94 and 96 breaks downstream apps
[
https://issues.apache.org/jira/browse/HBASE-9142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733415#comment-13733415
]
Hadoop QA commented on HBASE-9142:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12596823/hbase-9142-0.95.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 18 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/6654//console
This message is automatically generated.
> Mutation#getFamilyMap() return type change between HBase 94 and 96 breaks
> downstream apps
> -
>
> Key: HBASE-9142
> URL: https://issues.apache.org/jira/browse/HBASE-9142
> Project: HBase
> Issue Type: Bug
>Reporter: Hari Shreedharan
>Assignee: Jonathan Hsieh
>Priority: Critical
> Fix For: 0.95.2
>
> Attachments: hbase-9142-0.95.patch, hbase-9142-trunk.patch
>
>
> In Hbase-94, Mutation#getFamilyMap() had signature:
> public Map> getFamilyMap()
> In Hbase-96 it is:
> public NavigableMap> getFamilyMap()
> I understand this might not be an easy fix or even a possible one - but it
> breaks downstream apps in a nasty way. If the app needs to process the
> individual columns, then the whole logic is now different. Is there a way to
> work around this, if this cannot be fixed?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7391) Review/improve HLog compression's memory consumption
[
https://issues.apache.org/jira/browse/HBASE-7391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733419#comment-13733419
]
Hadoop QA commented on HBASE-7391:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12596819/HBASE-7391_1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 12 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
18 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/6653//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6653//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6653//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6653//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6653//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6653//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6653//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6653//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6653//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/6653//console
This message is automatically generated.
> Review/improve HLog compression's memory consumption
>
>
> Key: HBASE-7391
> URL: https://issues.apache.org/jira/browse/HBASE-7391
> Project: HBase
> Issue Type: Bug
>Reporter: Jean-Daniel Cryans
>Assignee: ramkrishna.s.vasudevan
> Fix For: 0.95.2
>
> Attachments: HBASE-7391_1.patch
>
>
> From Ram in
> http://mail-archives.apache.org/mod_mbox/hbase-dev/201205.mbox/%3C00bc01cd31e6$7caf1320$760d3960$%25vasude...@huawei.com%3E:
> {quote}
> One small observation after giving +1 on the RC.
> The WAL compression feature causes OOME and causes Full GC.
> The problem is, if we have 1500 regions and I need to create recovered.edits
> for each of the region (I don’t have much data in the regions (~300MB)).
> Now when I try to build the dictionary there is a Node object getting
> created.
> Each node object occupies 32 bytes.
> We have 5 such dictionaries.
> Initially we create indexToNodes array and its size is 32767.
> So now we have 32*5*32767 = ~5MB.
> Now I have 1500 regions.
> So 5MB*1500 = ~7GB.(Excluding actual data). This seems to a very high
> initial memory foot print and this never allows me to split the logs and I
> am not able to make the cluster up at all.
> Our configured heap size was 8GB, tested in 3 node cluster with 5000
> regions, very less data( 1GB in hdfs cluster including replication), some
> small data is spread evenly across all regions.
> The formula is 32(Node object size)*5(No of dictionary)*32767(no of node
> objects)*noofregions.
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7391) Review/improve HLog compression's memory consumption
[
https://issues.apache.org/jira/browse/HBASE-7391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733424#comment-13733424
]
ramkrishna.s.vasudevan commented on HBASE-7391:
---
There are no testcase failures in the testReport attached.
> Review/improve HLog compression's memory consumption
>
>
> Key: HBASE-7391
> URL: https://issues.apache.org/jira/browse/HBASE-7391
> Project: HBase
> Issue Type: Bug
>Reporter: Jean-Daniel Cryans
>Assignee: ramkrishna.s.vasudevan
> Fix For: 0.95.2
>
> Attachments: HBASE-7391_1.patch
>
>
> From Ram in
> http://mail-archives.apache.org/mod_mbox/hbase-dev/201205.mbox/%3C00bc01cd31e6$7caf1320$760d3960$%25vasude...@huawei.com%3E:
> {quote}
> One small observation after giving +1 on the RC.
> The WAL compression feature causes OOME and causes Full GC.
> The problem is, if we have 1500 regions and I need to create recovered.edits
> for each of the region (I don’t have much data in the regions (~300MB)).
> Now when I try to build the dictionary there is a Node object getting
> created.
> Each node object occupies 32 bytes.
> We have 5 such dictionaries.
> Initially we create indexToNodes array and its size is 32767.
> So now we have 32*5*32767 = ~5MB.
> Now I have 1500 regions.
> So 5MB*1500 = ~7GB.(Excluding actual data). This seems to a very high
> initial memory foot print and this never allows me to split the logs and I
> am not able to make the cluster up at all.
> Our configured heap size was 8GB, tested in 3 node cluster with 5000
> regions, very less data( 1GB in hdfs cluster including replication), some
> small data is spread evenly across all regions.
> The formula is 32(Node object size)*5(No of dictionary)*32767(no of node
> objects)*noofregions.
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9142) Mutation#getFamilyMap() return type change between HBase 94 and 96 breaks downstream apps
[ https://issues.apache.org/jira/browse/HBASE-9142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-9142: -- Attachment: hbase-9142-trunk.patch reupload to have jira test test the correct patch. > Mutation#getFamilyMap() return type change between HBase 94 and 96 breaks > downstream apps > - > > Key: HBASE-9142 > URL: https://issues.apache.org/jira/browse/HBASE-9142 > Project: HBase > Issue Type: Bug >Reporter: Hari Shreedharan >Assignee: Jonathan Hsieh >Priority: Critical > Fix For: 0.95.2 > > Attachments: hbase-9142-0.95.patch, hbase-9142-trunk.patch > > > In Hbase-94, Mutation#getFamilyMap() had signature: > public Map> getFamilyMap() > In Hbase-96 it is: > public NavigableMap> getFamilyMap() > I understand this might not be an easy fix or even a possible one - but it > breaks downstream apps in a nasty way. If the app needs to process the > individual columns, then the whole logic is now different. Is there a way to > work around this, if this cannot be fixed? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9142) Mutation#getFamilyMap() return type change between HBase 94 and 96 breaks downstream apps
[ https://issues.apache.org/jira/browse/HBASE-9142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-9142: -- Attachment: (was: hbase-9142-trunk.patch) > Mutation#getFamilyMap() return type change between HBase 94 and 96 breaks > downstream apps > - > > Key: HBASE-9142 > URL: https://issues.apache.org/jira/browse/HBASE-9142 > Project: HBase > Issue Type: Bug >Reporter: Hari Shreedharan >Assignee: Jonathan Hsieh >Priority: Critical > Fix For: 0.95.2 > > Attachments: hbase-9142-0.95.patch, hbase-9142-trunk.patch > > > In Hbase-94, Mutation#getFamilyMap() had signature: > public Map> getFamilyMap() > In Hbase-96 it is: > public NavigableMap> getFamilyMap() > I understand this might not be an easy fix or even a possible one - but it > breaks downstream apps in a nasty way. If the app needs to process the > individual columns, then the whole logic is now different. Is there a way to > work around this, if this cannot be fixed? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9142) Mutation#getFamilyMap() return type change between HBase 94 and 96 breaks downstream apps
[
https://issues.apache.org/jira/browse/HBASE-9142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733439#comment-13733439
]
Hadoop QA commented on HBASE-9142:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12596833/hbase-9142-trunk.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 18 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/6655//console
This message is automatically generated.
> Mutation#getFamilyMap() return type change between HBase 94 and 96 breaks
> downstream apps
> -
>
> Key: HBASE-9142
> URL: https://issues.apache.org/jira/browse/HBASE-9142
> Project: HBase
> Issue Type: Bug
>Reporter: Hari Shreedharan
>Assignee: Jonathan Hsieh
>Priority: Critical
> Fix For: 0.95.2
>
> Attachments: hbase-9142-0.95.patch, hbase-9142-trunk.patch
>
>
> In Hbase-94, Mutation#getFamilyMap() had signature:
> public Map> getFamilyMap()
> In Hbase-96 it is:
> public NavigableMap> getFamilyMap()
> I understand this might not be an easy fix or even a possible one - but it
> breaks downstream apps in a nasty way. If the app needs to process the
> individual columns, then the whole logic is now different. Is there a way to
> work around this, if this cannot be fixed?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8408) Implement namespace
[ https://issues.apache.org/jira/browse/HBASE-8408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733487#comment-13733487 ] Hudson commented on HBASE-8408: --- FAILURE: Integrated in hbase-0.95-on-hadoop2 #224 (See [https://builds.apache.org/job/hbase-0.95-on-hadoop2/224/]) HBASE-8408 implement Namespaces (stack: rev 1511591) * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/CoprocessorEnvironment.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/TableExistsException.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/TableNotDisabledException.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/TableNotEnabledException.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/TableNotFoundException.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/catalog/MetaReader.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnection.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnectionWrapper.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTablePool.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/MultiServerCallable.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/RegionServerCallable.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Registry.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/UnmodifyableHTableDescriptor.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ZooKeeperRegistry.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/coprocessor/SecureBulkLoadClient.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/RegionCoprocessorRpcChannel.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/security/access/TablePermission.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/security/access/UserPermission.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/snapshot/ClientSnapshotDescriptionUtils.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/snapshot/TablePartiallyOpenException.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKTable.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKTableReadOnly.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java * /hbase/branches/0.95/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestAsyncProcess.java * /hbase/branches/0.95/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestClientNoCluster.java * /hbase/branches/0.95/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotFromAdmin.java * /hbase/branches/0.95/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java * /hbase/branches/0.95/hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java * /hbase/branches/0.95/hbase-common/src/main/java/org/apache/hadoop/hbase/NamespaceDescriptor.java * /hbase/branches/0.95/hbase-common/src/main/java/org/apache/hadoop/hbase/TableName.java * /hbase/branches/0.95/hbase-common/src/main/java/org/apache/hadoop/hbase/util/Bytes.java * /hbase/branches/0.95/hbase-common/src/test/j
[jira] [Commented] (HBASE-6580) Deprecate HTablePool in favor of HConnection.getTable(...)
[ https://issues.apache.org/jira/browse/HBASE-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733486#comment-13733486 ] Hudson commented on HBASE-6580: --- FAILURE: Integrated in hbase-0.95-on-hadoop2 #224 (See [https://builds.apache.org/job/hbase-0.95-on-hadoop2/224/]) HBASE-6580 Deprecate HTablePool in favor of HConnection.getTable(...) (larsh: rev 1511544) * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnection.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnectionWrapper.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTablePool.java * /hbase/branches/0.95/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestClientNoCluster.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHCM.java * /hbase/branches/0.95/src/main/docbkx/book.xml > Deprecate HTablePool in favor of HConnection.getTable(...) > -- > > Key: HBASE-6580 > URL: https://issues.apache.org/jira/browse/HBASE-6580 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.94.6, 0.95.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.98.0, 0.95.2, 0.94.11 > > Attachments: 6580-0.94.txt, 6580-0.94-v2.txt, 6580-trunk.txt, > 6580-trunk-v2.txt, 6580-trunk-v3.txt, 6580-trunk-v4.txt, 6580-trunk-v5.txt, > 6580-trunk-v6.txt, HBASE-6580_v1.patch, HBASE-6580_v2.patch > > > Update: > I now propose deprecating HTablePool and instead introduce a getTable method > on HConnection and allow HConnection to manage the ThreadPool. > Initial proposal: > Here I propose a very simple TablePool. > It could be called LightHTablePool (or something - if you have a better name). > Internally it would maintain an HConnection and an Executor service and each > invocation of getTable(...) would create a new HTable and close() would just > close it. > In testing I find this more light weight than HTablePool and easier to > monitor in terms of resources used. > It would hardly be more than a few dozen lines of code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8408) Implement namespace
[ https://issues.apache.org/jira/browse/HBASE-8408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733500#comment-13733500 ] Hudson commented on HBASE-8408: --- FAILURE: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #657 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/657/]) HBASE-8408 Implement namespace; ADDENDUM (stack: rev 1511587) * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestMergeTool.java HBASE-8408 Implement namespace (stack: rev 1511577) * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/CoprocessorEnvironment.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/TableExistsException.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/TableNotDisabledException.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/TableNotEnabledException.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/TableNotFoundException.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/catalog/MetaReader.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnection.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnectionWrapper.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTablePool.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/MultiServerCallable.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/RegionServerCallable.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Registry.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/UnmodifyableHTableDescriptor.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ZooKeeperRegistry.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/coprocessor/AggregationClient.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/coprocessor/SecureBulkLoadClient.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/RegionCoprocessorRpcChannel.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/security/access/TablePermission.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/security/access/UserPermission.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/snapshot/ClientSnapshotDescriptionUtils.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/snapshot/TablePartiallyOpenException.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKTable.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKTableReadOnly.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java * /hbase/trunk/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestAsyncProcess.java * /hbase/trunk/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestClientNoCluster.java * /hbase/trunk/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotFromAdmin.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/NamespaceDescriptor.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/TableName.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/Bytes.java * /hbase/trunk/hbase-common/src/test
[jira] [Commented] (HBASE-8015) Support for Namespaces
[ https://issues.apache.org/jira/browse/HBASE-8015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733501#comment-13733501 ] Hudson commented on HBASE-8015: --- FAILURE: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #657 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/657/]) HBASE-8015 implement namespaces; ADDENDUM TWO -- REMOVE FILE REMOVED BY ORIGINAL PATCH (stack: rev 1511589) * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TableName.java > Support for Namespaces > -- > > Key: HBASE-8015 > URL: https://issues.apache.org/jira/browse/HBASE-8015 > Project: HBase > Issue Type: New Feature >Reporter: Francis Liu >Assignee: Francis Liu > Attachments: HBASE-8015_draft_94.patch, Namespace Design.pdf > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6580) Deprecate HTablePool in favor of HConnection.getTable(...)
[ https://issues.apache.org/jira/browse/HBASE-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733499#comment-13733499 ] Hudson commented on HBASE-6580: --- FAILURE: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #657 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/657/]) HBASE-6580 Deprecate HTablePool in favor of HConnection.getTable(...) (larsh: rev 1511543) * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnection.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnectionWrapper.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTablePool.java * /hbase/trunk/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestClientNoCluster.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHCM.java * /hbase/trunk/src/main/docbkx/book.xml > Deprecate HTablePool in favor of HConnection.getTable(...) > -- > > Key: HBASE-6580 > URL: https://issues.apache.org/jira/browse/HBASE-6580 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.94.6, 0.95.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.98.0, 0.95.2, 0.94.11 > > Attachments: 6580-0.94.txt, 6580-0.94-v2.txt, 6580-trunk.txt, > 6580-trunk-v2.txt, 6580-trunk-v3.txt, 6580-trunk-v4.txt, 6580-trunk-v5.txt, > 6580-trunk-v6.txt, HBASE-6580_v1.patch, HBASE-6580_v2.patch > > > Update: > I now propose deprecating HTablePool and instead introduce a getTable method > on HConnection and allow HConnection to manage the ThreadPool. > Initial proposal: > Here I propose a very simple TablePool. > It could be called LightHTablePool (or something - if you have a better name). > Internally it would maintain an HConnection and an Executor service and each > invocation of getTable(...) would create a new HTable and close() would just > close it. > In testing I find this more light weight than HTablePool and easier to > monitor in terms of resources used. > It would hardly be more than a few dozen lines of code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8408) Implement namespace
[ https://issues.apache.org/jira/browse/HBASE-8408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-8408: - Resolution: Fixed Fix Version/s: 0.95.2 0.98.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Resolving. Committed to trunk and 0.95. There are some unit test failures post-commit. Will address in follow-ups. Nice contrib Francis. Lets get the remaining ns stuff in. > Implement namespace > --- > > Key: HBASE-8408 > URL: https://issues.apache.org/jira/browse/HBASE-8408 > Project: HBase > Issue Type: Sub-task >Reporter: Francis Liu >Assignee: Francis Liu > Fix For: 0.98.0, 0.95.2 > > Attachments: HBASE-8015.095_21.txt, HBASE-8015_11.patch, > HBASE-8015_12.patch, HBASE-8015_13.patch, HBASE-8015_14.patch, > HBASE-8015_15.patch, HBASE-8015_16.patch, HBASE-8015_17.patch, > HBASE-8015_18.patch, HBASE-8015_19.patch, HBASE-8015_1.patch, > HBASE-8015_20.patch, HBASE-8015_2.patch, HBASE-8015_3.patch, > HBASE-8015_4.patch, HBASE-8015_5.patch, HBASE-8015_6.patch, > HBASE-8015_7.patch, HBASE-8015_8.patch, HBASE-8015_9.patch, HBASE-8015.patch, > HBASE-8408.trunk.addendum.txt, TestNamespaceMigration.tgz, > TestNamespaceUpgrade.tgz > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9158) Serious bug in cyclic replication
[
https://issues.apache.org/jira/browse/HBASE-9158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733608#comment-13733608
]
Lars Hofhansl commented on HBASE-9158:
--
Here is why this is serious. Say we have the following cycle setup: A -> B -> C
-> A.
Now image on cluster C we get a batch of edits from B. Now also say it just so
happens that the first edit in the batch is from A and the following ones are
from B.
When we execute the Put all edits will be tagged with A as he source, and thus
none of B's edit will make it to A when it is C's turn to replicate to A.
> Serious bug in cyclic replication
> -
>
> Key: HBASE-9158
> URL: https://issues.apache.org/jira/browse/HBASE-9158
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0, 0.95.1, 0.94.10
>Reporter: Lars Hofhansl
>Priority: Critical
> Fix For: 0.98.0, 0.95.2, 0.94.11
>
> Attachments: 9158-0.94.txt
>
>
> While studying the code for HBASE-7709, I found a serious bug in the current
> cyclic replication code. The problem is here in HRegion.doMiniBatchMutation:
> {code}
> Mutation first = batchOp.operations[firstIndex].getFirst();
> txid = this.log.appendNoSync(regionInfo,
> this.htableDescriptor.getName(),
>walEdit, first.getClusterId(), now, this.htableDescriptor);
> {code}
> Now note that edits replicated from remote cluster and local edits might
> interleave in the WAL, we might also receive edit from multiple remote
> clusters. Hence that might have edits from many clusters in it, but
> all are just labeled with the clusterId of the first Mutation.
> Fixing this in doMiniBatchMutation seems tricky to do efficiently (imagine we
> get a batch with cluster1, cluster2, cluster1, cluster2, ..., in that case
> each edit would have to be its own batch). The coprocessor handling would
> also be difficult.
> The other option is create batches of Puts grouped by the cluster id in
> ReplicationSink.replicateEntries(...), this is not as general, but equally
> correct. This is the approach I would favor.
> Lastly this is very hard to verify in a unittest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7709) Infinite loop possible in Master/Master replication
[ https://issues.apache.org/jira/browse/HBASE-7709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733611#comment-13733611 ] Lars Hofhansl commented on HBASE-7709: -- We also need to keep HBASE-9158 (a bug I just discovered). Here we need to group the edits by path and apply them strictly in these groups. > Infinite loop possible in Master/Master replication > --- > > Key: HBASE-7709 > URL: https://issues.apache.org/jira/browse/HBASE-7709 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 0.94.6, 0.95.1 >Reporter: Lars Hofhansl > Fix For: 0.98.0, 0.95.2, 0.94.12 > > > We just discovered the following scenario: > # Cluster A and B are setup in master/master replication > # By accident we had Cluster C replicate to Cluster A. > Now all edit originating from C will be bouncing between A and B. Forever! > The reason is that when the edit come in from C the cluster ID is already set > and won't be reset. > We have a couple of options here: > # Optionally only support master/master (not cycles of more than two > clusters). In that case we can always reset the cluster ID in the > ReplicationSource. That means that now cycles > 2 will have the data cycle > forever. This is the only option that requires no changes in the HLog format. > # Instead of a single cluster id per edit maintain a (unordered) set of > cluster id that have seen this edit. Then in ReplicationSource we drop any > edit that the sink has seen already. The is the cleanest approach, but it > might need a lot of data stored per edit if there are many clusters involved. > # Maintain a configurable counter of the maximum cycle side we want to > support. Could default to 10 (even maybe even just). Store a hop-count in the > WAL and the ReplicationSource increases that hop-count on each hop. If we're > over the max, just drop the edit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9123) Filter protobuf generated code from long line warning
[ https://issues.apache.org/jira/browse/HBASE-9123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733634#comment-13733634 ] Ted Yu commented on HBASE-9123: --- Integrated to trunk. Thanks for the review, Jesse. > Filter protobuf generated code from long line warning > - > > Key: HBASE-9123 > URL: https://issues.apache.org/jira/browse/HBASE-9123 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Minor > Fix For: 0.98.0 > > Attachments: 9123.patch, 9123-v2.patch > > > For big patch, such as the one for namespace, there would be many changes in > the protobuf generated code. > See example here: > https://builds.apache.org/job/PreCommit-HBASE-Build/6569/console > We should filter protobuf generated code from long line warning -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9156) 2000ms timeout for some client calls
[ https://issues.apache.org/jira/browse/HBASE-9156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9156: --- Resolution: Fixed Fix Version/s: 0.95.2 0.98.0 Status: Resolved (was: Patch Available) > 2000ms timeout for some client calls > > > Key: HBASE-9156 > URL: https://issues.apache.org/jira/browse/HBASE-9156 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.98.0, 0.95.2 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon >Priority: Blocker > Fix For: 0.98.0, 0.95.2 > > Attachments: 9156.v1.patch > > > RpcRetryingCaller#callTimeout is not initialized for all paths, so when > called from the AsyncProcess we end up with a timeout of 2 seconds. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9158) Serious bug in cyclic replication
[
https://issues.apache.org/jira/browse/HBASE-9158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-9158:
-
Attachment: 9158-0.94-v2.txt
Patch that does the grouping at the ReplicationSink. Since replication sink is
the only part that ever gets edits from different clusters, this is correct.
(Also made uses the new API from HBASE-6580)
Please let know what you think. I'll make a trunk patch and would to commit
this in time for 0.94.11.
> Serious bug in cyclic replication
> -
>
> Key: HBASE-9158
> URL: https://issues.apache.org/jira/browse/HBASE-9158
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0, 0.95.1, 0.94.10
>Reporter: Lars Hofhansl
>Priority: Critical
> Fix For: 0.98.0, 0.95.2, 0.94.11
>
> Attachments: 9158-0.94.txt, 9158-0.94-v2.txt
>
>
> While studying the code for HBASE-7709, I found a serious bug in the current
> cyclic replication code. The problem is here in HRegion.doMiniBatchMutation:
> {code}
> Mutation first = batchOp.operations[firstIndex].getFirst();
> txid = this.log.appendNoSync(regionInfo,
> this.htableDescriptor.getName(),
>walEdit, first.getClusterId(), now, this.htableDescriptor);
> {code}
> Now note that edits replicated from remote cluster and local edits might
> interleave in the WAL, we might also receive edit from multiple remote
> clusters. Hence that might have edits from many clusters in it, but
> all are just labeled with the clusterId of the first Mutation.
> Fixing this in doMiniBatchMutation seems tricky to do efficiently (imagine we
> get a batch with cluster1, cluster2, cluster1, cluster2, ..., in that case
> each edit would have to be its own batch). The coprocessor handling would
> also be difficult.
> The other option is create batches of Puts grouped by the cluster id in
> ReplicationSink.replicateEntries(...), this is not as general, but equally
> correct. This is the approach I would favor.
> Lastly this is very hard to verify in a unittest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7391) Review/improve HLog compression's memory consumption
[
https://issues.apache.org/jira/browse/HBASE-7391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733658#comment-13733658
]
Ted Yu commented on HBASE-7391:
---
ReaderBase#isRecoveredEdits() doesn't have to be public, right ?
I think you can unify ReaderBase#isRecoveredEdits() with the one from
WriterBase by putting it in some utility class.
Please add class javadoc for WriterBase.
> Review/improve HLog compression's memory consumption
>
>
> Key: HBASE-7391
> URL: https://issues.apache.org/jira/browse/HBASE-7391
> Project: HBase
> Issue Type: Bug
>Reporter: Jean-Daniel Cryans
>Assignee: ramkrishna.s.vasudevan
> Fix For: 0.95.2
>
> Attachments: HBASE-7391_1.patch
>
>
> From Ram in
> http://mail-archives.apache.org/mod_mbox/hbase-dev/201205.mbox/%3C00bc01cd31e6$7caf1320$760d3960$%25vasude...@huawei.com%3E:
> {quote}
> One small observation after giving +1 on the RC.
> The WAL compression feature causes OOME and causes Full GC.
> The problem is, if we have 1500 regions and I need to create recovered.edits
> for each of the region (I don’t have much data in the regions (~300MB)).
> Now when I try to build the dictionary there is a Node object getting
> created.
> Each node object occupies 32 bytes.
> We have 5 such dictionaries.
> Initially we create indexToNodes array and its size is 32767.
> So now we have 32*5*32767 = ~5MB.
> Now I have 1500 regions.
> So 5MB*1500 = ~7GB.(Excluding actual data). This seems to a very high
> initial memory foot print and this never allows me to split the logs and I
> am not able to make the cluster up at all.
> Our configured heap size was 8GB, tested in 3 node cluster with 5000
> regions, very less data( 1GB in hdfs cluster including replication), some
> small data is spread evenly across all regions.
> The formula is 32(Node object size)*5(No of dictionary)*32767(no of node
> objects)*noofregions.
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9158) Serious bug in cyclic replication
[
https://issues.apache.org/jira/browse/HBASE-9158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-9158:
-
Attachment: 9158-0.94-v3.txt
Cleaned up 0.94 patch.
> Serious bug in cyclic replication
> -
>
> Key: HBASE-9158
> URL: https://issues.apache.org/jira/browse/HBASE-9158
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0, 0.95.1, 0.94.10
>Reporter: Lars Hofhansl
>Priority: Critical
> Fix For: 0.98.0, 0.95.2, 0.94.11
>
> Attachments: 9158-0.94.txt, 9158-0.94-v2.txt, 9158-0.94-v3.txt
>
>
> While studying the code for HBASE-7709, I found a serious bug in the current
> cyclic replication code. The problem is here in HRegion.doMiniBatchMutation:
> {code}
> Mutation first = batchOp.operations[firstIndex].getFirst();
> txid = this.log.appendNoSync(regionInfo,
> this.htableDescriptor.getName(),
>walEdit, first.getClusterId(), now, this.htableDescriptor);
> {code}
> Now note that edits replicated from remote cluster and local edits might
> interleave in the WAL, we might also receive edit from multiple remote
> clusters. Hence that might have edits from many clusters in it, but
> all are just labeled with the clusterId of the first Mutation.
> Fixing this in doMiniBatchMutation seems tricky to do efficiently (imagine we
> get a batch with cluster1, cluster2, cluster1, cluster2, ..., in that case
> each edit would have to be its own batch). The coprocessor handling would
> also be difficult.
> The other option is create batches of Puts grouped by the cluster id in
> ReplicationSink.replicateEntries(...), this is not as general, but equally
> correct. This is the approach I would favor.
> Lastly this is very hard to verify in a unittest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8408) Implement namespace
[ https://issues.apache.org/jira/browse/HBASE-8408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733669#comment-13733669 ] Lars Hofhansl commented on HBASE-8408: -- This introduces a new TableName class and added API to HTable. But it did not add the API to the new HConnection.getTable API. Lemme do a quick addendum here. > Implement namespace > --- > > Key: HBASE-8408 > URL: https://issues.apache.org/jira/browse/HBASE-8408 > Project: HBase > Issue Type: Sub-task >Reporter: Francis Liu >Assignee: Francis Liu > Fix For: 0.98.0, 0.95.2 > > Attachments: HBASE-8015.095_21.txt, HBASE-8015_11.patch, > HBASE-8015_12.patch, HBASE-8015_13.patch, HBASE-8015_14.patch, > HBASE-8015_15.patch, HBASE-8015_16.patch, HBASE-8015_17.patch, > HBASE-8015_18.patch, HBASE-8015_19.patch, HBASE-8015_1.patch, > HBASE-8015_20.patch, HBASE-8015_2.patch, HBASE-8015_3.patch, > HBASE-8015_4.patch, HBASE-8015_5.patch, HBASE-8015_6.patch, > HBASE-8015_7.patch, HBASE-8015_8.patch, HBASE-8015_9.patch, HBASE-8015.patch, > HBASE-8408.trunk.addendum.txt, TestNamespaceMigration.tgz, > TestNamespaceUpgrade.tgz > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8408) Implement namespace
[ https://issues.apache.org/jira/browse/HBASE-8408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733672#comment-13733672 ] Lars Hofhansl commented on HBASE-8408: -- Now the question is: Should we remove anything but TableName from the new API in HConnection? > Implement namespace > --- > > Key: HBASE-8408 > URL: https://issues.apache.org/jira/browse/HBASE-8408 > Project: HBase > Issue Type: Sub-task >Reporter: Francis Liu >Assignee: Francis Liu > Fix For: 0.98.0, 0.95.2 > > Attachments: HBASE-8015.095_21.txt, HBASE-8015_11.patch, > HBASE-8015_12.patch, HBASE-8015_13.patch, HBASE-8015_14.patch, > HBASE-8015_15.patch, HBASE-8015_16.patch, HBASE-8015_17.patch, > HBASE-8015_18.patch, HBASE-8015_19.patch, HBASE-8015_1.patch, > HBASE-8015_20.patch, HBASE-8015_2.patch, HBASE-8015_3.patch, > HBASE-8015_4.patch, HBASE-8015_5.patch, HBASE-8015_6.patch, > HBASE-8015_7.patch, HBASE-8015_8.patch, HBASE-8015_9.patch, HBASE-8015.patch, > HBASE-8408.trunk.addendum.txt, TestNamespaceMigration.tgz, > TestNamespaceUpgrade.tgz > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9139) Independent timeout configuration for rpc channel between cluster nodes
[ https://issues.apache.org/jira/browse/HBASE-9139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733675#comment-13733675 ] Nicolas Liochon commented on HBASE-9139: [~julian.zhou], I'm ok with the patch. It simpler than I though. Existing retry logic seems ok. I'm also ok with the naming and the default value. Yes, we need a patch for trunk/0.95 before applying it to 0.94 (it should be very similar). For 0.94, it's better to have a go from [~lhofhansl] > Independent timeout configuration for rpc channel between cluster nodes > --- > > Key: HBASE-9139 > URL: https://issues.apache.org/jira/browse/HBASE-9139 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC, regionserver >Affects Versions: 0.94.10, 0.96.0 >Reporter: Julian Zhou >Assignee: Julian Zhou >Priority: Minor > Fix For: 0.94.12, 0.96.0 > > Attachments: 9139-0.94-v0.patch > > > Default of "hbase.rpc.timeout" is 6 ms (1 min). User sometimes > increase them to a bigger value such as 60 ms (10 mins) for many > concurrent loading application from client. Some user share the same > hbase-site.xml for both client and server. HRegionServer > #tryRegionServerReport via rpc channel to report to live master, but > there was a window for master failover scenario. That region server > attempting to connect to master, which was just killed, backup master > took the active role immediately and put to /hbase/master, but region > server was still waiting for the rpc timeout from connecting to the dead > master. If "hbase.rpc.timeout" is too long, this master failover process > will be long due to long rpc timeout from dead master. > If so, could we separate with 2 options, "hbase.rpc.timeout" is still > for hbase client, while "hbase.rpc.internal.timeout" was for this > regionserver/master rpc channel, which could be set shorted value > without affect real client rpc timeout value? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8408) Implement namespace
[
https://issues.apache.org/jira/browse/HBASE-8408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733683#comment-13733683
]
Francis Liu commented on HBASE-8408:
Thanks a lot for your help [~saint@gmail.com]!
Thanks as well [~enis], [~te...@apache.org] and [~mbertozzi]!
{quote}
Lets get the remaining ns stuff in.
{quote}
Will work on security first not unless you'd like something addressed first?
> Implement namespace
> ---
>
> Key: HBASE-8408
> URL: https://issues.apache.org/jira/browse/HBASE-8408
> Project: HBase
> Issue Type: Sub-task
>Reporter: Francis Liu
>Assignee: Francis Liu
> Fix For: 0.98.0, 0.95.2
>
> Attachments: HBASE-8015.095_21.txt, HBASE-8015_11.patch,
> HBASE-8015_12.patch, HBASE-8015_13.patch, HBASE-8015_14.patch,
> HBASE-8015_15.patch, HBASE-8015_16.patch, HBASE-8015_17.patch,
> HBASE-8015_18.patch, HBASE-8015_19.patch, HBASE-8015_1.patch,
> HBASE-8015_20.patch, HBASE-8015_2.patch, HBASE-8015_3.patch,
> HBASE-8015_4.patch, HBASE-8015_5.patch, HBASE-8015_6.patch,
> HBASE-8015_7.patch, HBASE-8015_8.patch, HBASE-8015_9.patch, HBASE-8015.patch,
> HBASE-8408.trunk.addendum.txt, TestNamespaceMigration.tgz,
> TestNamespaceUpgrade.tgz
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8408) Implement namespace
[ https://issues.apache.org/jira/browse/HBASE-8408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-8408: - Attachment: 8408-hconnection-add.txt Addendum. Will commit unless I hear objections. > Implement namespace > --- > > Key: HBASE-8408 > URL: https://issues.apache.org/jira/browse/HBASE-8408 > Project: HBase > Issue Type: Sub-task >Reporter: Francis Liu >Assignee: Francis Liu > Fix For: 0.98.0, 0.95.2 > > Attachments: 8408-hconnection-add.txt, HBASE-8015.095_21.txt, > HBASE-8015_11.patch, HBASE-8015_12.patch, HBASE-8015_13.patch, > HBASE-8015_14.patch, HBASE-8015_15.patch, HBASE-8015_16.patch, > HBASE-8015_17.patch, HBASE-8015_18.patch, HBASE-8015_19.patch, > HBASE-8015_1.patch, HBASE-8015_20.patch, HBASE-8015_2.patch, > HBASE-8015_3.patch, HBASE-8015_4.patch, HBASE-8015_5.patch, > HBASE-8015_6.patch, HBASE-8015_7.patch, HBASE-8015_8.patch, > HBASE-8015_9.patch, HBASE-8015.patch, HBASE-8408.trunk.addendum.txt, > TestNamespaceMigration.tgz, TestNamespaceUpgrade.tgz > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9141) Replication Znodes Backup Tool
[ https://issues.apache.org/jira/browse/HBASE-9141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Himanshu Vashishtha updated HBASE-9141: --- Attachment: (was: HBase-9141-v1.patch) > Replication Znodes Backup Tool > -- > > Key: HBASE-9141 > URL: https://issues.apache.org/jira/browse/HBASE-9141 > Project: HBase > Issue Type: Improvement > Components: migration, Replication >Affects Versions: 0.94.10 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Fix For: 0.95.2 > > Attachments: HBase-9141.patch > > > While migrating to 0.96, we recommend deleting old znodes so users not face > issues like HBASE-7766, and let HBase create them out of box. > Though HBase tends to store only ephemeral data in zookeeper, replication has > a different approach. Almost all of its data (state, peer info, logs, etc) is > present in zookeeper. We would like to preserve them in order to not do > re-adding of peers, and ensuring complete replication after we have migrated > to 0.96. > This jira adds a tool to serialize/de-serialize replication znodes to the > underlying filesystem. This could be used while migrating to 0.96.0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9141) Replication Znodes Backup Tool
[ https://issues.apache.org/jira/browse/HBASE-9141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733693#comment-13733693 ] Himanshu Vashishtha commented on HBASE-9141: The attached patch has the following changes: 1) Include a test cases where it create-->serializes-->delete existing--> restores znodes. It checks whether the data is restored or not, etc. 2) The znodes are saved with relative path, starting from replication base znode (by default, from replication/). This is useful in case a user wants to restore under a different hbase root znode. 3) Options to delete existing znode (after taking the backup, or before restoring the znodes); option to delete the backup file after successful restoration. 4) Other nits suggested by Ted. More help documentation on "moving" znodes. > Replication Znodes Backup Tool > -- > > Key: HBASE-9141 > URL: https://issues.apache.org/jira/browse/HBASE-9141 > Project: HBase > Issue Type: Improvement > Components: migration, Replication >Affects Versions: 0.94.10 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Fix For: 0.95.2 > > Attachments: HBase-9141.patch > > > While migrating to 0.96, we recommend deleting old znodes so users not face > issues like HBASE-7766, and let HBase create them out of box. > Though HBase tends to store only ephemeral data in zookeeper, replication has > a different approach. Almost all of its data (state, peer info, logs, etc) is > present in zookeeper. We would like to preserve them in order to not do > re-adding of peers, and ensuring complete replication after we have migrated > to 0.96. > This jira adds a tool to serialize/de-serialize replication znodes to the > underlying filesystem. This could be used while migrating to 0.96.0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8089) Add type support
[ https://issues.apache.org/jira/browse/HBASE-8089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733695#comment-13733695 ] Nick Dimiduk commented on HBASE-8089: - Hi Eli, You're right, I've left this ticket untouched while working through the initial subtasks. The order-preserving serialization is a critical component of the work. I think this is a feature that HBase absolutely must provide if there's to be any hope for interoperability. I also think the serialization format is necessary but not sufficient. An HBase that ships with an API for describing data types and implementations of a set of common definitions takes the next step in interoperability. By defining the type interface, type implementations provided by 3rd parties become pluggable -- it becomes feasible for a user to plug a type from Phoenix into their Kiji application. Systems like Phoenix, Kiji, and HCatalog are all choices for defining and managing schema. It may be the case that HBase should define the schema interfaces as well, but that's definitely beyond the scope here. But if those tools are going to interoperate, they need a common language of types with which to do so. Serialization, IMHO, is insufficient. I don't know if there's a new project to be built out of this work. I see no need to create such a thing when the needs and use are not yet proven. The introduction of types in HBase will shake things up enough as it is, let's see how people and projects use them before promoting this stuff to its own project. Yes, the serialization formats defined in HBASE-8201 are designed to be language agnostic. It's highly likely that I've missed some critical details here or there in the specification. Time will tell :) -n > Add type support > > > Key: HBASE-8089 > URL: https://issues.apache.org/jira/browse/HBASE-8089 > Project: HBase > Issue Type: New Feature > Components: Client >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk > Fix For: 0.98.0 > > Attachments: HBASE-8089-types.txt, HBASE-8089-types.txt, > HBASE-8089-types.txt, HBASE-8089-types.txt, hbase data types WIP.pdf > > > This proposal outlines an improvement to HBase that provides for a set of > types, above and beyond the existing "byte-bucket" strategy. This is intended > to reduce user-level duplication of effort, provide better support for > 3rd-party integration, and provide an overall improved experience for > developers using HBase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9141) Replication Znodes Backup Tool
[ https://issues.apache.org/jira/browse/HBASE-9141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Himanshu Vashishtha updated HBASE-9141: --- Attachment: HBase-9141-v1.patch > Replication Znodes Backup Tool > -- > > Key: HBASE-9141 > URL: https://issues.apache.org/jira/browse/HBASE-9141 > Project: HBase > Issue Type: Improvement > Components: migration, Replication >Affects Versions: 0.94.10 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Fix For: 0.95.2 > > Attachments: HBase-9141.patch, HBase-9141-v1.patch > > > While migrating to 0.96, we recommend deleting old znodes so users not face > issues like HBASE-7766, and let HBase create them out of box. > Though HBase tends to store only ephemeral data in zookeeper, replication has > a different approach. Almost all of its data (state, peer info, logs, etc) is > present in zookeeper. We would like to preserve them in order to not do > re-adding of peers, and ensuring complete replication after we have migrated > to 0.96. > This jira adds a tool to serialize/de-serialize replication znodes to the > underlying filesystem. This could be used while migrating to 0.96.0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9141) Replication Znodes Backup Tool
[ https://issues.apache.org/jira/browse/HBASE-9141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Himanshu Vashishtha updated HBASE-9141: --- Attachment: HBase-9141-v1.patch > Replication Znodes Backup Tool > -- > > Key: HBASE-9141 > URL: https://issues.apache.org/jira/browse/HBASE-9141 > Project: HBase > Issue Type: Improvement > Components: migration, Replication >Affects Versions: 0.94.10 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Fix For: 0.95.2 > > Attachments: HBase-9141.patch, HBase-9141-v1.patch > > > While migrating to 0.96, we recommend deleting old znodes so users not face > issues like HBASE-7766, and let HBase create them out of box. > Though HBase tends to store only ephemeral data in zookeeper, replication has > a different approach. Almost all of its data (state, peer info, logs, etc) is > present in zookeeper. We would like to preserve them in order to not do > re-adding of peers, and ensuring complete replication after we have migrated > to 0.96. > This jira adds a tool to serialize/de-serialize replication znodes to the > underlying filesystem. This could be used while migrating to 0.96.0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9141) Replication Znodes Backup Tool
[ https://issues.apache.org/jira/browse/HBASE-9141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Himanshu Vashishtha updated HBASE-9141: --- Status: Patch Available (was: Open) > Replication Znodes Backup Tool > -- > > Key: HBASE-9141 > URL: https://issues.apache.org/jira/browse/HBASE-9141 > Project: HBase > Issue Type: Improvement > Components: migration, Replication >Affects Versions: 0.94.10 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Fix For: 0.95.2 > > Attachments: HBase-9141.patch, HBase-9141-v1.patch > > > While migrating to 0.96, we recommend deleting old znodes so users not face > issues like HBASE-7766, and let HBase create them out of box. > Though HBase tends to store only ephemeral data in zookeeper, replication has > a different approach. Almost all of its data (state, peer info, logs, etc) is > present in zookeeper. We would like to preserve them in order to not do > re-adding of peers, and ensuring complete replication after we have migrated > to 0.96. > This jira adds a tool to serialize/de-serialize replication znodes to the > underlying filesystem. This could be used while migrating to 0.96.0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9108) LoadTestTool need to have a way to ignore keys which were failed during write.
[
https://issues.apache.org/jira/browse/HBASE-9108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733715#comment-13733715
]
Jeffrey Zhong commented on HBASE-9108:
--
I looked the code and it looks good to me except the following code
{code}
if (allowedExceptions != null && allowedExceptions.size() > 0) {
{code}
It's better to use isEmpty instead of .size(). Thanks.
> LoadTestTool need to have a way to ignore keys which were failed during
> write.
> ---
>
> Key: HBASE-9108
> URL: https://issues.apache.org/jira/browse/HBASE-9108
> Project: HBase
> Issue Type: Improvement
> Components: test
>Affects Versions: 0.95.0, 0.95.1, 0.94.9, 0.94.10
>Reporter: gautam
>Assignee: gautam
>Priority: Critical
> Attachments: 9108.patch._trunk.5, HBASE-9108.patch._trunk.2,
> HBASE-9108.patch._trunk.3, HBASE-9108.patch._trunk.4
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> While running the chaosmonkey integration tests, it is found that write
> sometimes fails when the cluster components are restarted/stopped/killed etc..
> The data key which was being put, using the LoadTestTool, is added to the
> failed key set, and at the end of the test, this failed key set is checked
> for any entries to assert failures.
> While doing fail-over testing, it is expected that some of the keys may go
> un-written. The point here is to validate that whatever gets into hbase for
> an unstable cluster really goes in, and hence read should be 100% for
> whatever keys went in successfully.
> Currently LoadTestTool has strict checks to validate every key being written
> or not. In case any keys is not written, it fails.
> I wanted to loosen this constraint by allowing users to pass in a set of
> exceptions they expect when doing put/write operations over hbase. If one of
> these expected exception set is thrown while writing key to hbase, the failed
> key would be ignored, and hence wont even be considered again for subsequent
> write as well as read.
> This can be passed to the load test tool as csv list parameter
> -allowed_write_exceptions, or it can be passed through hbase-site.xml by
> writing a value for "test.ignore.exceptions.during.write"
> Here is the usage:
> -allowed_write_exceptions
> "java.io.EOFException,org.apache.hadoop.hbase.NotServingRegionException,org.apache.hadoop.hbase.client.NoServerForRegionException,org.apache.hadoop.hbase.ipc.ServerNotRunningYetException"
> Hence, by doing this the existing integration tests can also make use of this
> change by passing it as property in hbase-site.xml, as well.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8408) Implement namespace
[ https://issues.apache.org/jira/browse/HBASE-8408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733714#comment-13733714 ] Lars Hofhansl commented on HBASE-8408: -- I committed the addendum. > Implement namespace > --- > > Key: HBASE-8408 > URL: https://issues.apache.org/jira/browse/HBASE-8408 > Project: HBase > Issue Type: Sub-task >Reporter: Francis Liu >Assignee: Francis Liu > Fix For: 0.98.0, 0.95.2 > > Attachments: 8408-hconnection-add.txt, HBASE-8015.095_21.txt, > HBASE-8015_11.patch, HBASE-8015_12.patch, HBASE-8015_13.patch, > HBASE-8015_14.patch, HBASE-8015_15.patch, HBASE-8015_16.patch, > HBASE-8015_17.patch, HBASE-8015_18.patch, HBASE-8015_19.patch, > HBASE-8015_1.patch, HBASE-8015_20.patch, HBASE-8015_2.patch, > HBASE-8015_3.patch, HBASE-8015_4.patch, HBASE-8015_5.patch, > HBASE-8015_6.patch, HBASE-8015_7.patch, HBASE-8015_8.patch, > HBASE-8015_9.patch, HBASE-8015.patch, HBASE-8408.trunk.addendum.txt, > TestNamespaceMigration.tgz, TestNamespaceUpgrade.tgz > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9098) During recovery use ZK as the source of truth for region state
[
https://issues.apache.org/jira/browse/HBASE-9098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733716#comment-13733716
]
Ted Yu commented on HBASE-9098:
---
{code}
+|| op == Operation.COMPACT_REGION) {
+ // split, merge or compact region doesn't need to check the
closing/closed state or lock the
{code}
Why was COMPACT_REGION added to the above check ?
{code}
+ // use one element array in order for waitUntilRegionOnline to pass
inRecovering state out
+ boolean[] isRecovering = new boolean[] { true };
{code}
You can also use AtomicBoolean above.
{code}
+// been being moved to somewhere before hosting RS fails
{code}
'being' is redundant.
> During recovery use ZK as the source of truth for region state
> ---
>
> Key: HBASE-9098
> URL: https://issues.apache.org/jira/browse/HBASE-9098
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 0.95.0
>Reporter: Devaraj Das
>Assignee: Jeffrey Zhong
>Priority: Blocker
> Fix For: 0.95.2
>
> Attachments: hbase-9098.patch, hbase-9098-v1.patch,
> hbase-9098-v3.patch
>
>
> In HLogSplitter:locateRegionAndRefreshLastFlushedSequenceId(HConnection,
> byte[], byte[], String), we talk to the replayee regionserver to figure out
> whether a region is in recovery or not. We should look at ZK only for this
> piece of information (since that is the source of truth for recovery
> otherwise).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9158) Serious bug in cyclic replication
[
https://issues.apache.org/jira/browse/HBASE-9158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-9158:
-
Attachment: 9158-trunk-v1.txt
Same patch for trunk. (Needed addendum in HBASE-8408)
> Serious bug in cyclic replication
> -
>
> Key: HBASE-9158
> URL: https://issues.apache.org/jira/browse/HBASE-9158
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0, 0.95.1, 0.94.10
>Reporter: Lars Hofhansl
>Priority: Critical
> Fix For: 0.98.0, 0.95.2, 0.94.11
>
> Attachments: 9158-0.94.txt, 9158-0.94-v2.txt, 9158-0.94-v3.txt,
> 9158-trunk-v1.txt
>
>
> While studying the code for HBASE-7709, I found a serious bug in the current
> cyclic replication code. The problem is here in HRegion.doMiniBatchMutation:
> {code}
> Mutation first = batchOp.operations[firstIndex].getFirst();
> txid = this.log.appendNoSync(regionInfo,
> this.htableDescriptor.getName(),
>walEdit, first.getClusterId(), now, this.htableDescriptor);
> {code}
> Now note that edits replicated from remote cluster and local edits might
> interleave in the WAL, we might also receive edit from multiple remote
> clusters. Hence that might have edits from many clusters in it, but
> all are just labeled with the clusterId of the first Mutation.
> Fixing this in doMiniBatchMutation seems tricky to do efficiently (imagine we
> get a batch with cluster1, cluster2, cluster1, cluster2, ..., in that case
> each edit would have to be its own batch). The coprocessor handling would
> also be difficult.
> The other option is create batches of Puts grouped by the cluster id in
> ReplicationSink.replicateEntries(...), this is not as general, but equally
> correct. This is the approach I would favor.
> Lastly this is very hard to verify in a unittest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9158) Serious bug in cyclic replication
[
https://issues.apache.org/jira/browse/HBASE-9158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-9158:
-
Status: Open (was: Patch Available)
> Serious bug in cyclic replication
> -
>
> Key: HBASE-9158
> URL: https://issues.apache.org/jira/browse/HBASE-9158
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.10, 0.95.1, 0.98.0
>Reporter: Lars Hofhansl
>Priority: Critical
> Fix For: 0.98.0, 0.95.2, 0.94.11
>
> Attachments: 9158-0.94.txt, 9158-0.94-v2.txt, 9158-0.94-v3.txt,
> 9158-trunk-v1.txt
>
>
> While studying the code for HBASE-7709, I found a serious bug in the current
> cyclic replication code. The problem is here in HRegion.doMiniBatchMutation:
> {code}
> Mutation first = batchOp.operations[firstIndex].getFirst();
> txid = this.log.appendNoSync(regionInfo,
> this.htableDescriptor.getName(),
>walEdit, first.getClusterId(), now, this.htableDescriptor);
> {code}
> Now note that edits replicated from remote cluster and local edits might
> interleave in the WAL, we might also receive edit from multiple remote
> clusters. Hence that might have edits from many clusters in it, but
> all are just labeled with the clusterId of the first Mutation.
> Fixing this in doMiniBatchMutation seems tricky to do efficiently (imagine we
> get a batch with cluster1, cluster2, cluster1, cluster2, ..., in that case
> each edit would have to be its own batch). The coprocessor handling would
> also be difficult.
> The other option is create batches of Puts grouped by the cluster id in
> ReplicationSink.replicateEntries(...), this is not as general, but equally
> correct. This is the approach I would favor.
> Lastly this is very hard to verify in a unittest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9158) Serious bug in cyclic replication
[
https://issues.apache.org/jira/browse/HBASE-9158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-9158:
-
Status: Patch Available (was: Open)
Let's get a hadoop qa run.
> Serious bug in cyclic replication
> -
>
> Key: HBASE-9158
> URL: https://issues.apache.org/jira/browse/HBASE-9158
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.10, 0.95.1, 0.98.0
>Reporter: Lars Hofhansl
>Priority: Critical
> Fix For: 0.98.0, 0.95.2, 0.94.11
>
> Attachments: 9158-0.94.txt, 9158-0.94-v2.txt, 9158-0.94-v3.txt,
> 9158-trunk-v1.txt
>
>
> While studying the code for HBASE-7709, I found a serious bug in the current
> cyclic replication code. The problem is here in HRegion.doMiniBatchMutation:
> {code}
> Mutation first = batchOp.operations[firstIndex].getFirst();
> txid = this.log.appendNoSync(regionInfo,
> this.htableDescriptor.getName(),
>walEdit, first.getClusterId(), now, this.htableDescriptor);
> {code}
> Now note that edits replicated from remote cluster and local edits might
> interleave in the WAL, we might also receive edit from multiple remote
> clusters. Hence that might have edits from many clusters in it, but
> all are just labeled with the clusterId of the first Mutation.
> Fixing this in doMiniBatchMutation seems tricky to do efficiently (imagine we
> get a batch with cluster1, cluster2, cluster1, cluster2, ..., in that case
> each edit would have to be its own batch). The coprocessor handling would
> also be difficult.
> The other option is create batches of Puts grouped by the cluster id in
> ReplicationSink.replicateEntries(...), this is not as general, but equally
> correct. This is the approach I would favor.
> Lastly this is very hard to verify in a unittest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9158) Serious bug in cyclic replication
[
https://issues.apache.org/jira/browse/HBASE-9158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-9158:
-
Status: Patch Available (was: Open)
> Serious bug in cyclic replication
> -
>
> Key: HBASE-9158
> URL: https://issues.apache.org/jira/browse/HBASE-9158
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.10, 0.95.1, 0.98.0
>Reporter: Lars Hofhansl
>Priority: Critical
> Fix For: 0.98.0, 0.95.2, 0.94.11
>
> Attachments: 9158-0.94.txt, 9158-0.94-v2.txt, 9158-0.94-v3.txt,
> 9158-trunk-v1.txt, 9158-trunk-v2.txt
>
>
> While studying the code for HBASE-7709, I found a serious bug in the current
> cyclic replication code. The problem is here in HRegion.doMiniBatchMutation:
> {code}
> Mutation first = batchOp.operations[firstIndex].getFirst();
> txid = this.log.appendNoSync(regionInfo,
> this.htableDescriptor.getName(),
>walEdit, first.getClusterId(), now, this.htableDescriptor);
> {code}
> Now note that edits replicated from remote cluster and local edits might
> interleave in the WAL, we might also receive edit from multiple remote
> clusters. Hence that might have edits from many clusters in it, but
> all are just labeled with the clusterId of the first Mutation.
> Fixing this in doMiniBatchMutation seems tricky to do efficiently (imagine we
> get a batch with cluster1, cluster2, cluster1, cluster2, ..., in that case
> each edit would have to be its own batch). The coprocessor handling would
> also be difficult.
> The other option is create batches of Puts grouped by the cluster id in
> ReplicationSink.replicateEntries(...), this is not as general, but equally
> correct. This is the approach I would favor.
> Lastly this is very hard to verify in a unittest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7709) Infinite loop possible in Master/Master replication
[ https://issues.apache.org/jira/browse/HBASE-7709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-7709: - Status: Open (was: Patch Available) > Infinite loop possible in Master/Master replication > --- > > Key: HBASE-7709 > URL: https://issues.apache.org/jira/browse/HBASE-7709 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 0.95.1, 0.94.6 >Reporter: Lars Hofhansl > Fix For: 0.98.0, 0.95.2, 0.94.12 > > > We just discovered the following scenario: > # Cluster A and B are setup in master/master replication > # By accident we had Cluster C replicate to Cluster A. > Now all edit originating from C will be bouncing between A and B. Forever! > The reason is that when the edit come in from C the cluster ID is already set > and won't be reset. > We have a couple of options here: > # Optionally only support master/master (not cycles of more than two > clusters). In that case we can always reset the cluster ID in the > ReplicationSource. That means that now cycles > 2 will have the data cycle > forever. This is the only option that requires no changes in the HLog format. > # Instead of a single cluster id per edit maintain a (unordered) set of > cluster id that have seen this edit. Then in ReplicationSource we drop any > edit that the sink has seen already. The is the cleanest approach, but it > might need a lot of data stored per edit if there are many clusters involved. > # Maintain a configurable counter of the maximum cycle side we want to > support. Could default to 10 (even maybe even just). Store a hop-count in the > WAL and the ReplicationSource increases that hop-count on each hop. If we're > over the max, just drop the edit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7709) Infinite loop possible in Master/Master replication
[ https://issues.apache.org/jira/browse/HBASE-7709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-7709: - Status: Patch Available (was: Open) > Infinite loop possible in Master/Master replication > --- > > Key: HBASE-7709 > URL: https://issues.apache.org/jira/browse/HBASE-7709 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 0.95.1, 0.94.6 >Reporter: Lars Hofhansl > Fix For: 0.98.0, 0.95.2, 0.94.12 > > > We just discovered the following scenario: > # Cluster A and B are setup in master/master replication > # By accident we had Cluster C replicate to Cluster A. > Now all edit originating from C will be bouncing between A and B. Forever! > The reason is that when the edit come in from C the cluster ID is already set > and won't be reset. > We have a couple of options here: > # Optionally only support master/master (not cycles of more than two > clusters). In that case we can always reset the cluster ID in the > ReplicationSource. That means that now cycles > 2 will have the data cycle > forever. This is the only option that requires no changes in the HLog format. > # Instead of a single cluster id per edit maintain a (unordered) set of > cluster id that have seen this edit. Then in ReplicationSource we drop any > edit that the sink has seen already. The is the cleanest approach, but it > might need a lot of data stored per edit if there are many clusters involved. > # Maintain a configurable counter of the maximum cycle side we want to > support. Could default to 10 (even maybe even just). Store a hop-count in the > WAL and the ReplicationSource increases that hop-count on each hop. If we're > over the max, just drop the edit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9158) Serious bug in cyclic replication
[
https://issues.apache.org/jira/browse/HBASE-9158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-9158:
-
Attachment: 9158-trunk-v2.txt
This is the right one.
> Serious bug in cyclic replication
> -
>
> Key: HBASE-9158
> URL: https://issues.apache.org/jira/browse/HBASE-9158
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0, 0.95.1, 0.94.10
>Reporter: Lars Hofhansl
>Priority: Critical
> Fix For: 0.98.0, 0.95.2, 0.94.11
>
> Attachments: 9158-0.94.txt, 9158-0.94-v2.txt, 9158-0.94-v3.txt,
> 9158-trunk-v1.txt, 9158-trunk-v2.txt
>
>
> While studying the code for HBASE-7709, I found a serious bug in the current
> cyclic replication code. The problem is here in HRegion.doMiniBatchMutation:
> {code}
> Mutation first = batchOp.operations[firstIndex].getFirst();
> txid = this.log.appendNoSync(regionInfo,
> this.htableDescriptor.getName(),
>walEdit, first.getClusterId(), now, this.htableDescriptor);
> {code}
> Now note that edits replicated from remote cluster and local edits might
> interleave in the WAL, we might also receive edit from multiple remote
> clusters. Hence that might have edits from many clusters in it, but
> all are just labeled with the clusterId of the first Mutation.
> Fixing this in doMiniBatchMutation seems tricky to do efficiently (imagine we
> get a batch with cluster1, cluster2, cluster1, cluster2, ..., in that case
> each edit would have to be its own batch). The coprocessor handling would
> also be difficult.
> The other option is create batches of Puts grouped by the cluster id in
> ReplicationSink.replicateEntries(...), this is not as general, but equally
> correct. This is the approach I would favor.
> Lastly this is very hard to verify in a unittest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9141) Replication Znodes Backup Tool
[
https://issues.apache.org/jira/browse/HBASE-9141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733765#comment-13733765
]
Hadoop QA commented on HBASE-9141:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12596879/HBase-9141-v1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
18 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.TestCheckTestClasses
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/6656//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6656//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6656//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6656//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6656//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6656//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6656//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6656//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6656//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/6656//console
This message is automatically generated.
> Replication Znodes Backup Tool
> --
>
> Key: HBASE-9141
> URL: https://issues.apache.org/jira/browse/HBASE-9141
> Project: HBase
> Issue Type: Improvement
> Components: migration, Replication
>Affects Versions: 0.94.10
>Reporter: Himanshu Vashishtha
>Assignee: Himanshu Vashishtha
> Fix For: 0.95.2
>
> Attachments: HBase-9141.patch, HBase-9141-v1.patch
>
>
> While migrating to 0.96, we recommend deleting old znodes so users not face
> issues like HBASE-7766, and let HBase create them out of box.
> Though HBase tends to store only ephemeral data in zookeeper, replication has
> a different approach. Almost all of its data (state, peer info, logs, etc) is
> present in zookeeper. We would like to preserve them in order to not do
> re-adding of peers, and ensuring complete replication after we have migrated
> to 0.96.
> This jira adds a tool to serialize/de-serialize replication znodes to the
> underlying filesystem. This could be used while migrating to 0.96.0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8408) Implement namespace
[ https://issues.apache.org/jira/browse/HBASE-8408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733788#comment-13733788 ] stack commented on HBASE-8408: -- [~lhofhansl] Thanks. I missed that. [~toffer] Yeah, security is critical now. I'm on some failing tests at the mo. > Implement namespace > --- > > Key: HBASE-8408 > URL: https://issues.apache.org/jira/browse/HBASE-8408 > Project: HBase > Issue Type: Sub-task >Reporter: Francis Liu >Assignee: Francis Liu > Fix For: 0.98.0, 0.95.2 > > Attachments: 8408-hconnection-add.txt, HBASE-8015.095_21.txt, > HBASE-8015_11.patch, HBASE-8015_12.patch, HBASE-8015_13.patch, > HBASE-8015_14.patch, HBASE-8015_15.patch, HBASE-8015_16.patch, > HBASE-8015_17.patch, HBASE-8015_18.patch, HBASE-8015_19.patch, > HBASE-8015_1.patch, HBASE-8015_20.patch, HBASE-8015_2.patch, > HBASE-8015_3.patch, HBASE-8015_4.patch, HBASE-8015_5.patch, > HBASE-8015_6.patch, HBASE-8015_7.patch, HBASE-8015_8.patch, > HBASE-8015_9.patch, HBASE-8015.patch, HBASE-8408.trunk.addendum.txt, > TestNamespaceMigration.tgz, TestNamespaceUpgrade.tgz > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8836) Separate reader and writer thread pool in RegionServer, so that write throughput will not be impacted when the read load is very high
[ https://issues.apache.org/jira/browse/HBASE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733797#comment-13733797 ] Devaraj Das commented on HBASE-8836: Overall, seems like a good improvement. Some high level comments. 1. We need a patch for trunk before we can commit this in 0.94.x 2. Can we look at annotations (similar to QoS annotation) for marking which methods should be treated as READ vs WRITE. That way we can keep the RPC code intact and have to only play with annotations on methods in the future. 3. Other comments are around code duplication that others have talked about already (in trunk it's better since the SecureRPCEngine doesn't exist there), and maybe we should keep the original getServer method, and it should do what it does currently (without the patch). > Separate reader and writer thread pool in RegionServer, so that write > throughput will not be impacted when the read load is very high > - > > Key: HBASE-8836 > URL: https://issues.apache.org/jira/browse/HBASE-8836 > Project: HBase > Issue Type: New Feature > Components: Performance, regionserver >Affects Versions: 0.94.8 >Reporter: Tianying Chang >Assignee: Tianying Chang > Fix For: 0.98.0, 0.94.12 > > Attachments: hbase-8836.patch, Hbase-8836-perfNumber.pdf > > > We found that when the read load on a specific RS is high, the write > throughput also get impacted dramatically, and even cause write data loss > sometimes. We want to prioritize the write by putting them in a separate > queue from the read request, so that slower read will not make fast write > wait nu-necessarily long. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9113) Expose region statistics on table.jsp
[ https://issues.apache.org/jira/browse/HBASE-9113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733809#comment-13733809 ] samar commented on HBASE-9113: -- [~bbeaudreault] Write now I have not introduced any new APIs.. I think its a good Idea to expose TableStatus/RegionStatus as client side api. We can take it up as a separate JIRA. As suggested by the team. > Expose region statistics on table.jsp > - > > Key: HBASE-9113 > URL: https://issues.apache.org/jira/browse/HBASE-9113 > Project: HBase > Issue Type: New Feature > Components: Admin, UI >Reporter: Bryan Beaudreault >Assignee: samar >Priority: Minor > Attachments: Screen Shot-table-details-V1.png > > > While Hannibal (https://github.com/sentric/hannibal) is great, the goal > should be to eventually make it obsolete by providing the same features in > the main HBase web UI (and HBaseAdmin API). > The first step for that is region statistics on the table.jsp. Please > provide the same statistics per-region on table.jsp as in rs-status.jsp. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9098) During recovery use ZK as the source of truth for region state
[
https://issues.apache.org/jira/browse/HBASE-9098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeffrey Zhong updated HBASE-9098:
-
Attachment: hbase-9098-v4.patch
Thanks Ted for comments!
{quote}
Why was COMPACT_REGION added to the above check ?
{quote}
In tests, I found major compaction requests drag recovery process. Current
recovered edits recovery doesn't allow compaction. So I disable compaction
request for log replay recovery as well.
I incorporated the rest comments into v4 patch. Thanks.
> During recovery use ZK as the source of truth for region state
> ---
>
> Key: HBASE-9098
> URL: https://issues.apache.org/jira/browse/HBASE-9098
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 0.95.0
>Reporter: Devaraj Das
>Assignee: Jeffrey Zhong
>Priority: Blocker
> Fix For: 0.95.2
>
> Attachments: hbase-9098.patch, hbase-9098-v1.patch,
> hbase-9098-v3.patch, hbase-9098-v4.patch
>
>
> In HLogSplitter:locateRegionAndRefreshLastFlushedSequenceId(HConnection,
> byte[], byte[], String), we talk to the replayee regionserver to figure out
> whether a region is in recovery or not. We should look at ZK only for this
> piece of information (since that is the source of truth for recovery
> otherwise).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9141) Replication Znodes Backup Tool
[ https://issues.apache.org/jira/browse/HBASE-9141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Himanshu Vashishtha updated HBASE-9141: --- Attachment: (was: HBase-9141-v1.patch) > Replication Znodes Backup Tool > -- > > Key: HBASE-9141 > URL: https://issues.apache.org/jira/browse/HBASE-9141 > Project: HBase > Issue Type: Improvement > Components: migration, Replication >Affects Versions: 0.94.10 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Fix For: 0.95.2 > > Attachments: HBase-9141.patch, HBase-9141-v1.patch > > > While migrating to 0.96, we recommend deleting old znodes so users not face > issues like HBASE-7766, and let HBase create them out of box. > Though HBase tends to store only ephemeral data in zookeeper, replication has > a different approach. Almost all of its data (state, peer info, logs, etc) is > present in zookeeper. We would like to preserve them in order to not do > re-adding of peers, and ensuring complete replication after we have migrated > to 0.96. > This jira adds a tool to serialize/de-serialize replication znodes to the > underlying filesystem. This could be used while migrating to 0.96.0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9141) Replication Znodes Backup Tool
[ https://issues.apache.org/jira/browse/HBASE-9141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Himanshu Vashishtha updated HBASE-9141: --- Attachment: HBase-9141-v1.patch Added category to pass the tests. > Replication Znodes Backup Tool > -- > > Key: HBASE-9141 > URL: https://issues.apache.org/jira/browse/HBASE-9141 > Project: HBase > Issue Type: Improvement > Components: migration, Replication >Affects Versions: 0.94.10 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Fix For: 0.95.2 > > Attachments: HBase-9141.patch, HBase-9141-v1.patch > > > While migrating to 0.96, we recommend deleting old znodes so users not face > issues like HBASE-7766, and let HBase create them out of box. > Though HBase tends to store only ephemeral data in zookeeper, replication has > a different approach. Almost all of its data (state, peer info, logs, etc) is > present in zookeeper. We would like to preserve them in order to not do > re-adding of peers, and ensuring complete replication after we have migrated > to 0.96. > This jira adds a tool to serialize/de-serialize replication znodes to the > underlying filesystem. This could be used while migrating to 0.96.0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-9159) TestSplitTransaction.testFailAfterPONR fails
stack created HBASE-9159:
Summary: TestSplitTransaction.testFailAfterPONR fails
Key: HBASE-9159
URL: https://issues.apache.org/jira/browse/HBASE-9159
Project: HBase
Issue Type: Bug
Components: test
Reporter: stack
Failed here
http://54.241.6.143/job/HBase-0.95/org.apache.hbase$hbase-server/776/testReport/junit/org.apache.hadoop.hbase.regionserver/TestSplitTransaction/testFailAfterPONR/
Failure is that mockito did not throw the expected exception:
{code}
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.hbase.regionserver.TestSplitTransaction.testFailAfterPONR(TestSplitTransaction.java:133)
{code}
I've not seen this before that I remember. Keeping open a while. Adding ns
has destabilized our clean test runs. Working on restabilization.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9115) HTableInterface.append operation may overwrites values
[
https://issues.apache.org/jira/browse/HBASE-9115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733842#comment-13733842
]
Aleksandr B commented on HBASE-9115:
Hej Ted,
Please excuse me for not answering for so long. Thank you all very
much for the fast responce and the provided patch. I have tried 95.2
and everything works :)
Best regards,
Aleksandr Bodriagov
> HTableInterface.append operation may overwrites values
> --
>
> Key: HBASE-9115
> URL: https://issues.apache.org/jira/browse/HBASE-9115
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.10
> Environment: MAC OS X 10.8.4, Hbase in the pseudo-distributed mode,
> hadoop v1.2.0, Hbase Java API based client.
> *hdfs-site.xml*:
> {code:xml}
>
>
> dfs.replication
> 1
>
>
> dfs.support.append
> true
>
>
> {code}
> *hbase-site.xml*:
> {code:xml}
>
>
> hbase.rootdir
> hdfs://localhost:9000/hbase
>
>
> hbase.cluster.distributed
> true
>
>
> hbase.zookeeper.quorum
> localhost
>
>
> dfs.support.append
> true
>
>
> {code}
>Reporter: Aleksandr B
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 0.98.0, 0.95.2, 0.94.11
>
> Attachments: 9115-0.94.add, 9115-0.94.txt, 9115-0.94-v2.txt,
> 9115-trunk.addendum, 9115-trunk.addendum2, 9115-trunk.addendum3,
> 9115-trunk.txt
>
>
> I use Hbase Java API and I try to append values Bytes.toBytes("one two") and
> Bytes.toBytes(" three") in 3 columns.
> Only for 2 out of these 3 columns the result is "one two three".
> *Output from the hbase shell:*
> {noformat}
> hbase(main):008:0* scan "mytesttable"
> ROWCOLUMN+CELL
>
> mytestRowKey column=TestA:dlbytes,
> timestamp=1375436156140, value=one two three
>
> mytestRowKey column=TestA:tbytes,
> timestamp=1375436156140, value=one two three
>
> mytestRowKey column=TestA:ulbytes,
> timestamp=1375436156140, value= three
>
> 1 row(s) in 0.0280 seconds
> {noformat}
> *My test code:*
> {code:title=Database.java|borderStyle=solid}
> import static org.junit.Assert.*;
> import java.io.IOException;
>
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.hbase.HBaseConfiguration;
> import org.apache.hadoop.hbase.HColumnDescriptor;
> import org.apache.hadoop.hbase.HTableDescriptor;
> import org.apache.hadoop.hbase.client.HBaseAdmin;
> import org.apache.hadoop.hbase.client.HTableInterface;
> import org.apache.hadoop.hbase.client.HTablePool;
> import org.apache.hadoop.hbase.client.Append;
> import org.apache.hadoop.hbase.client.Result;
> import org.apache.hadoop.hbase.util.Bytes;
> import org.junit.Test;
> ...
> @Test
> public void testAppend() throws IOException {
> byte [] rowKey = Bytes.toBytes("mytestRowKey");
> byte [] column1 = Bytes.toBytes("ulbytes");
> byte [] column2 = Bytes.toBytes("dlbytes");
> byte [] column3 = Bytes.toBytes("tbytes");
> String part11 = "one two";
> String part12 = " three";
> String cFamily = "TestA";
> String TABLE = "mytesttable";
> Configuration conf = HBaseConfiguration.create();
> HTablePool pool = new HTablePool(conf, 10);
> HBaseAdmin admin = new HBaseAdmin(conf);
>
> if(admin.tableExists(TABLE)){
> admin.disableTable(TABLE);
> admin.deleteTable(TABLE);
> }
>
> HTableDescriptor tableDescriptor = new HTableDescriptor(TABLE);
> HColumnDescriptor hcd = new HColumnDescriptor(cFamily);
> hcd.setMaxVersions(1);
> tableDescriptor.addFamily(hcd);
> admin.createTable(tableDescriptor);
> HTableInterface table = pool.getTable(TABLE);
>
> Append a = new Append(rowKey);
> a.setReturnResults(false);
> a.add(Bytes.toBytes(cFamily), column1, Bytes.toBytes(part11));
> a.add(Bytes.toBytes(cFamily), column2, Bytes.toBytes(part11));
> a.add(Bytes.toBytes(cFamily), column3, Bytes.toBytes(part11));
> table.append(a);
> a = new Append(rowKey);
> a.add(Bytes.toBytes(cFamily), column1, Bytes.toBytes(part12));
> a.add(Bytes.toBytes(cFamily), column2, Bytes.toBytes(part12));
> a.add(Bytes.toBytes(cFamily), column3, Bytes.toBytes(part12));
> Result result = table.appen
[jira] [Updated] (HBASE-9141) Replication Znodes Backup Tool
[ https://issues.apache.org/jira/browse/HBASE-9141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Himanshu Vashishtha updated HBASE-9141: --- Attachment: HBase-9141-v1.patch Trying it again. > Replication Znodes Backup Tool > -- > > Key: HBASE-9141 > URL: https://issues.apache.org/jira/browse/HBASE-9141 > Project: HBase > Issue Type: Improvement > Components: migration, Replication >Affects Versions: 0.94.10 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Fix For: 0.95.2 > > Attachments: HBase-9141.patch, HBase-9141-v1.patch, > HBase-9141-v1.patch > > > While migrating to 0.96, we recommend deleting old znodes so users not face > issues like HBASE-7766, and let HBase create them out of box. > Though HBase tends to store only ephemeral data in zookeeper, replication has > a different approach. Almost all of its data (state, peer info, logs, etc) is > present in zookeeper. We would like to preserve them in order to not do > re-adding of peers, and ensuring complete replication after we have migrated > to 0.96. > This jira adds a tool to serialize/de-serialize replication znodes to the > underlying filesystem. This could be used while migrating to 0.96.0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9158) Serious bug in cyclic replication
[
https://issues.apache.org/jira/browse/HBASE-9158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733853#comment-13733853
]
Hadoop QA commented on HBASE-9158:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12596887/9158-trunk-v2.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
18 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.replication.TestReplicationKillMasterRS
org.apache.hadoop.hbase.replication.TestReplicationKillSlaveRS
org.apache.hadoop.hbase.replication.TestReplicationSmallTests
org.apache.hadoop.hbase.replication.TestMultiSlaveReplication
org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/6657//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6657//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6657//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6657//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6657//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6657//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6657//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6657//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6657//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/6657//console
This message is automatically generated.
> Serious bug in cyclic replication
> -
>
> Key: HBASE-9158
> URL: https://issues.apache.org/jira/browse/HBASE-9158
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0, 0.95.1, 0.94.10
>Reporter: Lars Hofhansl
>Priority: Critical
> Fix For: 0.98.0, 0.95.2, 0.94.11
>
> Attachments: 9158-0.94.txt, 9158-0.94-v2.txt, 9158-0.94-v3.txt,
> 9158-trunk-v1.txt, 9158-trunk-v2.txt
>
>
> While studying the code for HBASE-7709, I found a serious bug in the current
> cyclic replication code. The problem is here in HRegion.doMiniBatchMutation:
> {code}
> Mutation first = batchOp.operations[firstIndex].getFirst();
> txid = this.log.appendNoSync(regionInfo,
> this.htableDescriptor.getName(),
>walEdit, first.getClusterId(), now, this.htableDescriptor);
> {code}
> Now note that edits replicated from remote cluster and local edits might
> interleave in the WAL, we might also receive edit from multiple remote
> clusters. Hence that might have edits from many clusters in it, but
> all are just labeled with the clusterId of the first Mutation.
> Fixing this in doMiniBatchMutation seems tricky to do efficiently (imagine we
> get a batch with cluster1, cluster2, cluster1, cluster2, ..., in that case
> each edit would have to be its own batch). The coprocessor handling wo
[jira] [Updated] (HBASE-9141) Replication Znodes Backup Tool
[ https://issues.apache.org/jira/browse/HBASE-9141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Himanshu Vashishtha updated HBASE-9141: --- Attachment: (was: HBase-9141-v1.patch) > Replication Znodes Backup Tool > -- > > Key: HBASE-9141 > URL: https://issues.apache.org/jira/browse/HBASE-9141 > Project: HBase > Issue Type: Improvement > Components: migration, Replication >Affects Versions: 0.94.10 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Fix For: 0.95.2 > > Attachments: HBase-9141.patch, HBase-9141-v1.patch > > > While migrating to 0.96, we recommend deleting old znodes so users not face > issues like HBASE-7766, and let HBase create them out of box. > Though HBase tends to store only ephemeral data in zookeeper, replication has > a different approach. Almost all of its data (state, peer info, logs, etc) is > present in zookeeper. We would like to preserve them in order to not do > re-adding of peers, and ensuring complete replication after we have migrated > to 0.96. > This jira adds a tool to serialize/de-serialize replication znodes to the > underlying filesystem. This could be used while migrating to 0.96.0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9141) Replication Znodes Backup Tool
[ https://issues.apache.org/jira/browse/HBASE-9141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733856#comment-13733856 ] Himanshu Vashishtha commented on HBASE-9141: I am sorry for re-adding the attachment folks; I figured out lately that I didn't attach the right one earlier. Thanks. > Replication Znodes Backup Tool > -- > > Key: HBASE-9141 > URL: https://issues.apache.org/jira/browse/HBASE-9141 > Project: HBase > Issue Type: Improvement > Components: migration, Replication >Affects Versions: 0.94.10 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Fix For: 0.95.2 > > Attachments: HBase-9141.patch, HBase-9141-v1.patch > > > While migrating to 0.96, we recommend deleting old znodes so users not face > issues like HBASE-7766, and let HBase create them out of box. > Though HBase tends to store only ephemeral data in zookeeper, replication has > a different approach. Almost all of its data (state, peer info, logs, etc) is > present in zookeeper. We would like to preserve them in order to not do > re-adding of peers, and ensuring complete replication after we have migrated > to 0.96. > This jira adds a tool to serialize/de-serialize replication znodes to the > underlying filesystem. This could be used while migrating to 0.96.0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9156) 2000ms timeout for some client calls
[ https://issues.apache.org/jira/browse/HBASE-9156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733858#comment-13733858 ] Hudson commented on HBASE-9156: --- FAILURE: Integrated in hbase-0.95 #416 (See [https://builds.apache.org/job/hbase-0.95/416/]) HBASE-9156 2000ms timeout for some client calls (nkeywal: rev 1511854) * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCaller.java > 2000ms timeout for some client calls > > > Key: HBASE-9156 > URL: https://issues.apache.org/jira/browse/HBASE-9156 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.98.0, 0.95.2 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon >Priority: Blocker > Fix For: 0.98.0, 0.95.2 > > Attachments: 9156.v1.patch > > > RpcRetryingCaller#callTimeout is not initialized for all paths, so when > called from the AsyncProcess we end up with a timeout of 2 seconds. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9158) Serious bug in cyclic replication
[
https://issues.apache.org/jira/browse/HBASE-9158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733861#comment-13733861
]
Lars Hofhansl commented on HBASE-9158:
--
Yeah... That does not look good.
> Serious bug in cyclic replication
> -
>
> Key: HBASE-9158
> URL: https://issues.apache.org/jira/browse/HBASE-9158
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0, 0.95.1, 0.94.10
>Reporter: Lars Hofhansl
>Priority: Critical
> Fix For: 0.98.0, 0.95.2, 0.94.11
>
> Attachments: 9158-0.94.txt, 9158-0.94-v2.txt, 9158-0.94-v3.txt,
> 9158-trunk-v1.txt, 9158-trunk-v2.txt
>
>
> While studying the code for HBASE-7709, I found a serious bug in the current
> cyclic replication code. The problem is here in HRegion.doMiniBatchMutation:
> {code}
> Mutation first = batchOp.operations[firstIndex].getFirst();
> txid = this.log.appendNoSync(regionInfo,
> this.htableDescriptor.getName(),
>walEdit, first.getClusterId(), now, this.htableDescriptor);
> {code}
> Now note that edits replicated from remote cluster and local edits might
> interleave in the WAL, we might also receive edit from multiple remote
> clusters. Hence that might have edits from many clusters in it, but
> all are just labeled with the clusterId of the first Mutation.
> Fixing this in doMiniBatchMutation seems tricky to do efficiently (imagine we
> get a batch with cluster1, cluster2, cluster1, cluster2, ..., in that case
> each edit would have to be its own batch). The coprocessor handling would
> also be difficult.
> The other option is create batches of Puts grouped by the cluster id in
> ReplicationSink.replicateEntries(...), this is not as general, but equally
> correct. This is the approach I would favor.
> Lastly this is very hard to verify in a unittest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9158) Serious bug in cyclic replication
[
https://issues.apache.org/jira/browse/HBASE-9158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-9158:
-
Status: Open (was: Patch Available)
> Serious bug in cyclic replication
> -
>
> Key: HBASE-9158
> URL: https://issues.apache.org/jira/browse/HBASE-9158
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.10, 0.95.1, 0.98.0
>Reporter: Lars Hofhansl
>Priority: Critical
> Fix For: 0.98.0, 0.95.2, 0.94.11
>
> Attachments: 9158-0.94.txt, 9158-0.94-v2.txt, 9158-0.94-v3.txt,
> 9158-0.94-v4.txt, 9158-trunk-v1.txt, 9158-trunk-v2.txt
>
>
> While studying the code for HBASE-7709, I found a serious bug in the current
> cyclic replication code. The problem is here in HRegion.doMiniBatchMutation:
> {code}
> Mutation first = batchOp.operations[firstIndex].getFirst();
> txid = this.log.appendNoSync(regionInfo,
> this.htableDescriptor.getName(),
>walEdit, first.getClusterId(), now, this.htableDescriptor);
> {code}
> Now note that edits replicated from remote cluster and local edits might
> interleave in the WAL, we might also receive edit from multiple remote
> clusters. Hence that might have edits from many clusters in it, but
> all are just labeled with the clusterId of the first Mutation.
> Fixing this in doMiniBatchMutation seems tricky to do efficiently (imagine we
> get a batch with cluster1, cluster2, cluster1, cluster2, ..., in that case
> each edit would have to be its own batch). The coprocessor handling would
> also be difficult.
> The other option is create batches of Puts grouped by the cluster id in
> ReplicationSink.replicateEntries(...), this is not as general, but equally
> correct. This is the approach I would favor.
> Lastly this is very hard to verify in a unittest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9158) Serious bug in cyclic replication
[
https://issues.apache.org/jira/browse/HBASE-9158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-9158:
-
Attachment: 9158-0.94-v4.txt
Should fix the tests (0.94)
> Serious bug in cyclic replication
> -
>
> Key: HBASE-9158
> URL: https://issues.apache.org/jira/browse/HBASE-9158
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0, 0.95.1, 0.94.10
>Reporter: Lars Hofhansl
>Priority: Critical
> Fix For: 0.98.0, 0.95.2, 0.94.11
>
> Attachments: 9158-0.94.txt, 9158-0.94-v2.txt, 9158-0.94-v3.txt,
> 9158-0.94-v4.txt, 9158-trunk-v1.txt, 9158-trunk-v2.txt
>
>
> While studying the code for HBASE-7709, I found a serious bug in the current
> cyclic replication code. The problem is here in HRegion.doMiniBatchMutation:
> {code}
> Mutation first = batchOp.operations[firstIndex].getFirst();
> txid = this.log.appendNoSync(regionInfo,
> this.htableDescriptor.getName(),
>walEdit, first.getClusterId(), now, this.htableDescriptor);
> {code}
> Now note that edits replicated from remote cluster and local edits might
> interleave in the WAL, we might also receive edit from multiple remote
> clusters. Hence that might have edits from many clusters in it, but
> all are just labeled with the clusterId of the first Mutation.
> Fixing this in doMiniBatchMutation seems tricky to do efficiently (imagine we
> get a batch with cluster1, cluster2, cluster1, cluster2, ..., in that case
> each edit would have to be its own batch). The coprocessor handling would
> also be difficult.
> The other option is create batches of Puts grouped by the cluster id in
> ReplicationSink.replicateEntries(...), this is not as general, but equally
> correct. This is the approach I would favor.
> Lastly this is very hard to verify in a unittest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9158) Serious bug in cyclic replication
[
https://issues.apache.org/jira/browse/HBASE-9158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-9158:
-
Status: Patch Available (was: Open)
> Serious bug in cyclic replication
> -
>
> Key: HBASE-9158
> URL: https://issues.apache.org/jira/browse/HBASE-9158
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.10, 0.95.1, 0.98.0
>Reporter: Lars Hofhansl
>Priority: Critical
> Fix For: 0.98.0, 0.95.2, 0.94.11
>
> Attachments: 9158-0.94.txt, 9158-0.94-v2.txt, 9158-0.94-v3.txt,
> 9158-0.94-v4.txt, 9158-trunk-v1.txt, 9158-trunk-v2.txt, 9158-trunk-v3.txt
>
>
> While studying the code for HBASE-7709, I found a serious bug in the current
> cyclic replication code. The problem is here in HRegion.doMiniBatchMutation:
> {code}
> Mutation first = batchOp.operations[firstIndex].getFirst();
> txid = this.log.appendNoSync(regionInfo,
> this.htableDescriptor.getName(),
>walEdit, first.getClusterId(), now, this.htableDescriptor);
> {code}
> Now note that edits replicated from remote cluster and local edits might
> interleave in the WAL, we might also receive edit from multiple remote
> clusters. Hence that might have edits from many clusters in it, but
> all are just labeled with the clusterId of the first Mutation.
> Fixing this in doMiniBatchMutation seems tricky to do efficiently (imagine we
> get a batch with cluster1, cluster2, cluster1, cluster2, ..., in that case
> each edit would have to be its own batch). The coprocessor handling would
> also be difficult.
> The other option is create batches of Puts grouped by the cluster id in
> ReplicationSink.replicateEntries(...), this is not as general, but equally
> correct. This is the approach I would favor.
> Lastly this is very hard to verify in a unittest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9158) Serious bug in cyclic replication
[
https://issues.apache.org/jira/browse/HBASE-9158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-9158:
-
Attachment: 9158-trunk-v3.txt
And for trunk.
(just one character change :) )
> Serious bug in cyclic replication
> -
>
> Key: HBASE-9158
> URL: https://issues.apache.org/jira/browse/HBASE-9158
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0, 0.95.1, 0.94.10
>Reporter: Lars Hofhansl
>Priority: Critical
> Fix For: 0.98.0, 0.95.2, 0.94.11
>
> Attachments: 9158-0.94.txt, 9158-0.94-v2.txt, 9158-0.94-v3.txt,
> 9158-0.94-v4.txt, 9158-trunk-v1.txt, 9158-trunk-v2.txt, 9158-trunk-v3.txt
>
>
> While studying the code for HBASE-7709, I found a serious bug in the current
> cyclic replication code. The problem is here in HRegion.doMiniBatchMutation:
> {code}
> Mutation first = batchOp.operations[firstIndex].getFirst();
> txid = this.log.appendNoSync(regionInfo,
> this.htableDescriptor.getName(),
>walEdit, first.getClusterId(), now, this.htableDescriptor);
> {code}
> Now note that edits replicated from remote cluster and local edits might
> interleave in the WAL, we might also receive edit from multiple remote
> clusters. Hence that might have edits from many clusters in it, but
> all are just labeled with the clusterId of the first Mutation.
> Fixing this in doMiniBatchMutation seems tricky to do efficiently (imagine we
> get a batch with cluster1, cluster2, cluster1, cluster2, ..., in that case
> each edit would have to be its own batch). The coprocessor handling would
> also be difficult.
> The other option is create batches of Puts grouped by the cluster id in
> ReplicationSink.replicateEntries(...), this is not as general, but equally
> correct. This is the approach I would favor.
> Lastly this is very hard to verify in a unittest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5478) [0.92] TestMetaReaderEditor intermittently hangs
[
https://issues.apache.org/jira/browse/HBASE-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733878#comment-13733878
]
Andrew Purtell commented on HBASE-5478:
---
After patching 0.92 with HBASE-5114 I also observe no more failures or hangs of
TestMetaReaderEditor after 200 local invocations or 5 more runs up on EC2
jenkins. Going to commit and resolve.
> [0.92] TestMetaReaderEditor intermittently hangs
>
>
> Key: HBASE-5478
> URL: https://issues.apache.org/jira/browse/HBASE-5478
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.92.1
>Reporter: Ted Yu
>Assignee: Andrew Purtell
>
> From 0.92 build #304:
> {code}
> Running org.apache.hadoop.hbase.catalog.TestMetaReaderEditor
> Running org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 149.104 sec
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9123) Filter protobuf generated code from long line warning
[ https://issues.apache.org/jira/browse/HBASE-9123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733890#comment-13733890 ] Hudson commented on HBASE-9123: --- FAILURE: Integrated in HBase-TRUNK #4356 (See [https://builds.apache.org/job/HBase-TRUNK/4356/]) HBASE-9123 Filter protobuf generated code from long line warning (Ted Yu) (tedyu: rev 1511846) * /hbase/trunk/dev-support/test-patch.sh > Filter protobuf generated code from long line warning > - > > Key: HBASE-9123 > URL: https://issues.apache.org/jira/browse/HBASE-9123 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Minor > Fix For: 0.98.0 > > Attachments: 9123.patch, 9123-v2.patch > > > For big patch, such as the one for namespace, there would be many changes in > the protobuf generated code. > See example here: > https://builds.apache.org/job/PreCommit-HBASE-Build/6569/console > We should filter protobuf generated code from long line warning -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8408) Implement namespace
[ https://issues.apache.org/jira/browse/HBASE-8408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733891#comment-13733891 ] Hudson commented on HBASE-8408: --- FAILURE: Integrated in HBase-TRUNK #4356 (See [https://builds.apache.org/job/HBase-TRUNK/4356/]) HBASE-8408; addendum for new HConnection API (larsh: rev 1511885) * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnection.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnectionWrapper.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHCM.java > Implement namespace > --- > > Key: HBASE-8408 > URL: https://issues.apache.org/jira/browse/HBASE-8408 > Project: HBase > Issue Type: Sub-task >Reporter: Francis Liu >Assignee: Francis Liu > Fix For: 0.98.0, 0.95.2 > > Attachments: 8408-hconnection-add.txt, HBASE-8015.095_21.txt, > HBASE-8015_11.patch, HBASE-8015_12.patch, HBASE-8015_13.patch, > HBASE-8015_14.patch, HBASE-8015_15.patch, HBASE-8015_16.patch, > HBASE-8015_17.patch, HBASE-8015_18.patch, HBASE-8015_19.patch, > HBASE-8015_1.patch, HBASE-8015_20.patch, HBASE-8015_2.patch, > HBASE-8015_3.patch, HBASE-8015_4.patch, HBASE-8015_5.patch, > HBASE-8015_6.patch, HBASE-8015_7.patch, HBASE-8015_8.patch, > HBASE-8015_9.patch, HBASE-8015.patch, HBASE-8408.trunk.addendum.txt, > TestNamespaceMigration.tgz, TestNamespaceUpgrade.tgz > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9156) 2000ms timeout for some client calls
[ https://issues.apache.org/jira/browse/HBASE-9156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733889#comment-13733889 ] Hudson commented on HBASE-9156: --- FAILURE: Integrated in HBase-TRUNK #4356 (See [https://builds.apache.org/job/HBase-TRUNK/4356/]) HBASE-9156 2000ms timeout for some client calls (nkeywal: rev 1511853) * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCaller.java > 2000ms timeout for some client calls > > > Key: HBASE-9156 > URL: https://issues.apache.org/jira/browse/HBASE-9156 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.98.0, 0.95.2 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon >Priority: Blocker > Fix For: 0.98.0, 0.95.2 > > Attachments: 9156.v1.patch > > > RpcRetryingCaller#callTimeout is not initialized for all paths, so when > called from the AsyncProcess we end up with a timeout of 2 seconds. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8670) [0.94] Backport HBASE-8449,HBASE-8204 and HBASE-8699 to 0.94 (Refactor recoverLease retries and pauses)
[ https://issues.apache.org/jira/browse/HBASE-8670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733894#comment-13733894 ] Himanshu Vashishtha commented on HBASE-8670: Any specific reason why the first pause is 3sec, and not 4sec as in trunk (little bit higher than default dfs heartbeat)? https://github.com/apache/hbase/blob/0.94/src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java#L93 https://github.com/apache/hbase/blob/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java#L93 > [0.94] Backport HBASE-8449,HBASE-8204 and HBASE-8699 to 0.94 (Refactor > recoverLease retries and pauses) > --- > > Key: HBASE-8670 > URL: https://issues.apache.org/jira/browse/HBASE-8670 > Project: HBase > Issue Type: Bug > Components: Filesystem Integration, master, wal >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 0.94.11 > > Attachments: hbase-8670_v1.patch, hbase-8670_v2.patch > > > Some history: > Up until 0.94.8, Hbase did not check the result of recoverLease() call, but > things kind of worked since we are checking for 0-length files in distributed > log split tasks from region servers. If lease recovery is not finished, the > log file will report 0 length, and the task will fail, and master will then > re-call recoverLease() and reassign the task. This scheme might fail for log > files that are larger than 1 hdfs block though. > In 0.94.8, we committed (HBASE-8354, which is backport of HBASE-7878) and > later increased the sleep time to 4 secs in HBASE-8389. > However, the proper solution arrived in trunk in HBASE-8449 which uses a > backoff sleep policy + isFileClosed() api. We should backport this patch to > 0.94 as well. > isFileClosed() is released in Hadoop 1.2.0 (HDFS-4774) and 2.0.5(HDFS-4525). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8089) Add type support
[ https://issues.apache.org/jira/browse/HBASE-8089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733898#comment-13733898 ] stack commented on HBASE-8089: -- bq. An HBase that ships with an API for describing data types and implementations of a set of common definitions takes the next step in interoperability. So you are thinking more than just a client-side utility lib but an actual facade that does typing (though it is all client-side) as in for example TypedHTable that does something like typedHTable.put(new Put(row).addInteger(12345)) and int i = typedHTable.get(new Get(row).getInteger());? HBase internally is still all byte arrays but it'd have this new class that made it look like we could exploit this typing info server-side (e.g. better compression)? I suppose I'd have to register a serializer w/ this new TypedHTable too? Would the serializer be per table? > Add type support > > > Key: HBASE-8089 > URL: https://issues.apache.org/jira/browse/HBASE-8089 > Project: HBase > Issue Type: New Feature > Components: Client >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk > Fix For: 0.98.0 > > Attachments: HBASE-8089-types.txt, HBASE-8089-types.txt, > HBASE-8089-types.txt, HBASE-8089-types.txt, hbase data types WIP.pdf > > > This proposal outlines an improvement to HBase that provides for a set of > types, above and beyond the existing "byte-bucket" strategy. This is intended > to reduce user-level duplication of effort, provide better support for > 3rd-party integration, and provide an overall improved experience for > developers using HBase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9141) Replication Znodes Backup Tool
[ https://issues.apache.org/jira/browse/HBASE-9141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733907#comment-13733907 ] Jean-Daniel Cryans commented on HBASE-9141: --- Good progress here. I'd be ok committing although thinking more about it might be good to not put the znodes in place right away, so create them elsewhere and then move them in place all at once? Or am I just being over-cautious? Also the unit test could cover more than the normal path. Right now if anything goes wrong you get into untested code paths. What do you think Himanshu? > Replication Znodes Backup Tool > -- > > Key: HBASE-9141 > URL: https://issues.apache.org/jira/browse/HBASE-9141 > Project: HBase > Issue Type: Improvement > Components: migration, Replication >Affects Versions: 0.94.10 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Fix For: 0.95.2 > > Attachments: HBase-9141.patch, HBase-9141-v1.patch > > > While migrating to 0.96, we recommend deleting old znodes so users not face > issues like HBASE-7766, and let HBase create them out of box. > Though HBase tends to store only ephemeral data in zookeeper, replication has > a different approach. Almost all of its data (state, peer info, logs, etc) is > present in zookeeper. We would like to preserve them in order to not do > re-adding of peers, and ensuring complete replication after we have migrated > to 0.96. > This jira adds a tool to serialize/de-serialize replication znodes to the > underlying filesystem. This could be used while migrating to 0.96.0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-9160) TestNamespaceUpgrade fails on hadoop 2.1 due to existence of .snapshot directory in tar ball
Ted Yu created HBASE-9160:
-
Summary: TestNamespaceUpgrade fails on hadoop 2.1 due to existence
of .snapshot directory in tar ball
Key: HBASE-9160
URL: https://issues.apache.org/jira/browse/HBASE-9160
Project: HBase
Issue Type: Test
Reporter: Ted Yu
Here is partial listing of contents for TestNamespaceUpgrade.tgz :
{code}
-rwxrwxrwx 0 fcliu staff 0 May 13 10:50
hbase/.snapshot/ns.two.foo_snapshot2/7112077ccfba8a46c0694ca8c0a4bc2d/f/136799c89e244b9bad79281b37650927
-rwxrwxrwx 0 fcliu staff 0 May 13 10:50
hbase/.snapshot/ns.two.foo_snapshot2/7112077ccfba8a46c0694ca8c0a4bc2d/f/98093488150c42229927fd2a1e8c5d69
-rw-r--r-- 0 fcliu staff 12 May 13 10:50
hbase/.snapshot/ns.two.foo_snapshot1/..snapshotinfo.crc
{code}
Running the test on hadoop 2.1, I saw the following in test output:
{code}
2013-08-08 19:59:29,834 WARN [IPC Server handler 3 on 47143]
ipc.Server$Handler(2044): IPC Server handler 3 on 47143, call
org.apache.hadoop.hdfs.protocol.ClientProtocol.mkdirs from 127.0.0.1:46287
Call#264 Retry#0: error: org.apache.hadoop.HadoopIllegalArgumentException:
".snapshot" is a reserved name.
org.apache.hadoop.HadoopIllegalArgumentException: ".snapshot" is a reserved
name.
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyINodeName(FSDirectory.java:2108)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.addChild(FSDirectory.java:2204)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedMkdir(FSDirectory.java:2003)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.mkdirs(FSDirectory.java:1956)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:3379)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:3338)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3310)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:694)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:502)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:48089)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2028)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2024)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2022)
put: ".snapshot" is a reserved name.
{code}
Looks like the tar ball was generated using an old version of 0.94
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8615) HLog Compression fails in mysterious ways (working title)
[
https://issues.apache.org/jira/browse/HBASE-8615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733918#comment-13733918
]
Jean-Daniel Cryans commented on HBASE-8615:
---
Thanks for the patch Ted. To commit we should at least fix the unit test though
because what I did was kind of a hack, TestReplicationHLogReaderManager isn't
supposed to run on compressed data. Maybe do a
TestReplicationHLogReaderManagerCompressed that just enables it?
Then we could also test more than just one failure mode in there, easy refactor
where you just have to pass the two ints to a method, then get rid of most of
the comments. Right now it's just dirty.
Finally, if we are fixing HLog compression for real, we need to also put
TestReplicationKillMasterRSCompressed back AKA HBASE-9061.
> HLog Compression fails in mysterious ways (working title)
> -
>
> Key: HBASE-8615
> URL: https://issues.apache.org/jira/browse/HBASE-8615
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 0.98.0, 0.96.0
>
> Attachments: 172.21.3.117%2C60020%2C1375222888304.1375222894855.zip,
> 8615-v2.txt, 8615-v3.txt, HBASE-8615-test.patch,
> org.apache.hadoop.hbase.replication.TestReplicationQueueFailoverCompressed-output.txt
>
>
> In a recent test run, I noticed the following in test output:
> {code}
> 2013-05-24 22:01:02,424 DEBUG
> [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2]
> fs.HFileSystem$ReorderWALBlocks(327):
> /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432840428
> is an HLog file, so reordering blocks, last hostname will
> be:kiyo.gq1.ygridcore.net
> 2013-05-24 22:01:02,429 DEBUG
> [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2]
> wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset:
> 132235, fileLength: 132243, trailerPresent: true
> 2013-05-24 22:01:02,438 ERROR
> [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2]
> wal.ProtobufLogReader(236): Error while reading 691 WAL KVs; started
> reading at 53272 and read up to 65538
> 2013-05-24 22:01:02,438 WARN
> [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2]
> regionserver.ReplicationSource(324): 2 Got:
> java.io.IOException: Error while reading 691 WAL KVs; started reading at
> 53272 and read up to 65538
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:237)
> at
> org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:96)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.readNextAndSetPosition(ReplicationHLogReaderManager.java:89)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.readAllEntriesToReplicateOrNextFile(ReplicationSource.java:404)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:320)
> Caused by: java.lang.IndexOutOfBoundsException: index (30062) must be less
> than size (1)
> at
> com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305)
> at
> com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284)
> at
> org.apache.hadoop.hbase.regionserver.wal.LRUDictionary$BidirectionalLRUMap.get(LRUDictionary.java:124)
> at
> org.apache.hadoop.hbase.regionserver.wal.LRUDictionary$BidirectionalLRUMap.access$000(LRUDictionary.java:71)
> at
> org.apache.hadoop.hbase.regionserver.wal.LRUDictionary.getEntry(LRUDictionary.java:42)
> at
> org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$CompressedKvDecoder.readIntoArray(WALCellCodec.java:210)
> at
> org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$CompressedKvDecoder.parseCell(WALCellCodec.java:184)
> at
> org.apache.hadoop.hbase.codec.BaseDecoder.advance(BaseDecoder.java:46)
> at
> org.apache.hadoop.hbase.regionserver.wal.WALEdit.readFromCells(WALEdit.java:213)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:217)
> ... 4 more
> 2013-05-24 22:01:02,439 DEBUG
> [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2]
> regionserver.ReplicationSource(583): Nothing to replicate, sleeping 100
> times 10
> {code}
> Will attach test output.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atla
[jira] [Updated] (HBASE-9061) Put back TestReplicationKillMasterRSCompressed when fixed over in HBASE-8615
[ https://issues.apache.org/jira/browse/HBASE-9061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-9061: -- Summary: Put back TestReplicationKillMasterRSCompressed when fixed over in HBASE-8615 (was: Put back TestReplicationKillMasterRSCompressedwhen fixed over in HBASE-8615) > Put back TestReplicationKillMasterRSCompressed when fixed over in HBASE-8615 > > > Key: HBASE-9061 > URL: https://issues.apache.org/jira/browse/HBASE-9061 > Project: HBase > Issue Type: Task > Components: test >Reporter: stack >Priority: Critical > Fix For: 0.96.0 > > > The suite of TestReplicationKillRs* tests were removed temporarily. Put them > back after they've been fixed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9158) Serious bug in cyclic replication
[
https://issues.apache.org/jira/browse/HBASE-9158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-9158:
-
Assignee: Lars Hofhansl
> Serious bug in cyclic replication
> -
>
> Key: HBASE-9158
> URL: https://issues.apache.org/jira/browse/HBASE-9158
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0, 0.95.1, 0.94.10
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
>Priority: Critical
> Fix For: 0.98.0, 0.95.2, 0.94.11
>
> Attachments: 9158-0.94.txt, 9158-0.94-v2.txt, 9158-0.94-v3.txt,
> 9158-0.94-v4.txt, 9158-trunk-v1.txt, 9158-trunk-v2.txt, 9158-trunk-v3.txt
>
>
> While studying the code for HBASE-7709, I found a serious bug in the current
> cyclic replication code. The problem is here in HRegion.doMiniBatchMutation:
> {code}
> Mutation first = batchOp.operations[firstIndex].getFirst();
> txid = this.log.appendNoSync(regionInfo,
> this.htableDescriptor.getName(),
>walEdit, first.getClusterId(), now, this.htableDescriptor);
> {code}
> Now note that edits replicated from remote cluster and local edits might
> interleave in the WAL, we might also receive edit from multiple remote
> clusters. Hence that might have edits from many clusters in it, but
> all are just labeled with the clusterId of the first Mutation.
> Fixing this in doMiniBatchMutation seems tricky to do efficiently (imagine we
> get a batch with cluster1, cluster2, cluster1, cluster2, ..., in that case
> each edit would have to be its own batch). The coprocessor handling would
> also be difficult.
> The other option is create batches of Puts grouped by the cluster id in
> ReplicationSink.replicateEntries(...), this is not as general, but equally
> correct. This is the approach I would favor.
> Lastly this is very hard to verify in a unittest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8615) HLog Compression fails in mysterious ways (working title)
[
https://issues.apache.org/jira/browse/HBASE-8615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733935#comment-13733935
]
Ted Yu commented on HBASE-8615:
---
The inclusion of changes to TestReplicationHLogReaderManager was to show that
the problem has been solved.
Since restoring TestReplicationKillMasterRSCompressed would be done in
HBASE-9061, how about dropping the changes to TestReplicationHLogReaderManager
in patch v4 ?
Please comment.
> HLog Compression fails in mysterious ways (working title)
> -
>
> Key: HBASE-8615
> URL: https://issues.apache.org/jira/browse/HBASE-8615
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 0.98.0, 0.96.0
>
> Attachments: 172.21.3.117%2C60020%2C1375222888304.1375222894855.zip,
> 8615-v2.txt, 8615-v3.txt, HBASE-8615-test.patch,
> org.apache.hadoop.hbase.replication.TestReplicationQueueFailoverCompressed-output.txt
>
>
> In a recent test run, I noticed the following in test output:
> {code}
> 2013-05-24 22:01:02,424 DEBUG
> [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2]
> fs.HFileSystem$ReorderWALBlocks(327):
> /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432840428
> is an HLog file, so reordering blocks, last hostname will
> be:kiyo.gq1.ygridcore.net
> 2013-05-24 22:01:02,429 DEBUG
> [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2]
> wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset:
> 132235, fileLength: 132243, trailerPresent: true
> 2013-05-24 22:01:02,438 ERROR
> [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2]
> wal.ProtobufLogReader(236): Error while reading 691 WAL KVs; started
> reading at 53272 and read up to 65538
> 2013-05-24 22:01:02,438 WARN
> [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2]
> regionserver.ReplicationSource(324): 2 Got:
> java.io.IOException: Error while reading 691 WAL KVs; started reading at
> 53272 and read up to 65538
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:237)
> at
> org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:96)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.readNextAndSetPosition(ReplicationHLogReaderManager.java:89)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.readAllEntriesToReplicateOrNextFile(ReplicationSource.java:404)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:320)
> Caused by: java.lang.IndexOutOfBoundsException: index (30062) must be less
> than size (1)
> at
> com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305)
> at
> com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284)
> at
> org.apache.hadoop.hbase.regionserver.wal.LRUDictionary$BidirectionalLRUMap.get(LRUDictionary.java:124)
> at
> org.apache.hadoop.hbase.regionserver.wal.LRUDictionary$BidirectionalLRUMap.access$000(LRUDictionary.java:71)
> at
> org.apache.hadoop.hbase.regionserver.wal.LRUDictionary.getEntry(LRUDictionary.java:42)
> at
> org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$CompressedKvDecoder.readIntoArray(WALCellCodec.java:210)
> at
> org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$CompressedKvDecoder.parseCell(WALCellCodec.java:184)
> at
> org.apache.hadoop.hbase.codec.BaseDecoder.advance(BaseDecoder.java:46)
> at
> org.apache.hadoop.hbase.regionserver.wal.WALEdit.readFromCells(WALEdit.java:213)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:217)
> ... 4 more
> 2013-05-24 22:01:02,439 DEBUG
> [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2]
> regionserver.ReplicationSource(583): Nothing to replicate, sleeping 100
> times 10
> {code}
> Will attach test output.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9098) During recovery use ZK as the source of truth for region state
[
https://issues.apache.org/jira/browse/HBASE-9098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733938#comment-13733938
]
Hadoop QA commented on HBASE-9098:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12596903/hbase-9098-v4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
18 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/6658//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6658//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6658//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6658//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6658//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6658//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6658//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6658//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6658//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/6658//console
This message is automatically generated.
> During recovery use ZK as the source of truth for region state
> ---
>
> Key: HBASE-9098
> URL: https://issues.apache.org/jira/browse/HBASE-9098
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 0.95.0
>Reporter: Devaraj Das
>Assignee: Jeffrey Zhong
>Priority: Blocker
> Fix For: 0.95.2
>
> Attachments: hbase-9098.patch, hbase-9098-v1.patch,
> hbase-9098-v3.patch, hbase-9098-v4.patch
>
>
> In HLogSplitter:locateRegionAndRefreshLastFlushedSequenceId(HConnection,
> byte[], byte[], String), we talk to the replayee regionserver to figure out
> whether a region is in recovery or not. We should look at ZK only for this
> piece of information (since that is the source of truth for recovery
> otherwise).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8960) TestDistributedLogSplitting.testLogReplayForDisablingTable fails sometimes
[
https://issues.apache.org/jira/browse/HBASE-8960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733939#comment-13733939
]
Jeffrey Zhong commented on HBASE-8960:
--
[~saint@gmail.com] The changes in hbase-8960 make the
TestDistributedLogSplitting test case complete within two mins(by not killing &
restarting dfscluster and zkcluster for each test case) so I think there is no
need to split the TestDistributedLogSplitting test suite. I'll try to enhance
the failure detection tool to include the skipped ones. Thanks.
> TestDistributedLogSplitting.testLogReplayForDisablingTable fails sometimes
> --
>
> Key: HBASE-8960
> URL: https://issues.apache.org/jira/browse/HBASE-8960
> Project: HBase
> Issue Type: Task
> Components: test
>Reporter: Jimmy Xiang
>Assignee: Jeffrey Zhong
>Priority: Minor
> Fix For: 0.95.2
>
> Attachments: hbase-8960-addendum-2.patch, hbase-8960-addendum.patch,
> hbase-8960.patch
>
>
> http://54.241.6.143/job/HBase-0.95-Hadoop-2/org.apache.hbase$hbase-server/634/testReport/junit/org.apache.hadoop.hbase.master/TestDistributedLogSplitting/testLogReplayForDisablingTable/
> {noformat}
> java.lang.AssertionError: expected:<1000> but was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting.testLogReplayForDisablingTable(TestDistributedLogSplitting.java:797)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9079) FilterList getNextKeyHint skips rows that should be included in the results
[ https://issues.apache.org/jira/browse/HBASE-9079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-9079: - Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to 0.94, 0.95, and trunk. Thanks for the patch, Viral. > FilterList getNextKeyHint skips rows that should be included in the results > --- > > Key: HBASE-9079 > URL: https://issues.apache.org/jira/browse/HBASE-9079 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 0.94.10 >Reporter: Viral Bajaria >Assignee: Viral Bajaria > Fix For: 0.98.0, 0.95.2, 0.94.11 > > Attachments: 9079-0.94-v2.txt, 9079-trunk-v2.txt, > HBASE-9079-0.94.patch, HBASE-9079-trunk.patch > > > I hit a weird issue/bug and am able to reproduce the error consistently. The > problem arises when FilterList has two filters where each implements the > getNextKeyHint method. > The way the current implementation works is, StoreScanner will call > matcher.getNextKeyHint() whenever it gets a SEEK_NEXT_USING_HINT. This in > turn will call filter.getNextKeyHint() which at this stage is of type > FilterList. The implementation in FilterList iterates through all the filters > and keeps the max KeyValue that it sees. All is fine if you wrap filters in > FilterList in which only one of them implements getNextKeyHint. but if > multiple of them implement then that's where things get weird. > For example: > - create two filters: one is FuzzyRowFilter and second is ColumnRangeFilter. > Both of them implement getNextKeyHint > - wrap them in FilterList with MUST_PASS_ALL > - FuzzyRowFilter will seek to the correct first row and then pass it to > ColumnRangeFilter which will return the SEEK_NEXT_USING_HINT code. > - Now in FilterList when getNextKeyHint is called, it calls the one on > FuzzyRow first which basically says what the next row should be. While in > reality we want the ColumnRangeFilter to give the seek hint. > - The above behavior skips data that should be returned, which I have > verified by using a RowFilter with RegexStringComparator. > I updated the FilterList to maintain state on which filter returns the > SEEK_NEXT_USING_HINT and in getNextKeyHint, I invoke the method on the saved > filter and reset that state. I tested it with my current queries and it works > fine but I need to run the entire test suite to make sure I have not > introduced any regression. In addition to that I need to figure out what > should be the behavior when the opeation is MUST_PASS_ONE, but I doubt it > should be any different. > Is my understanding of it being a bug correct ? Or am I trivializing it and > ignoring something very important ? If it's tough to wrap your head around > the explanation, then I can open a JIRA and upload a patch against 0.94 head. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7391) Review/improve HLog compression's memory consumption
[
https://issues.apache.org/jira/browse/HBASE-7391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733941#comment-13733941
]
Jean-Daniel Cryans commented on HBASE-7391:
---
Why is this not pulled out of the if-else statement?
bq. rowDict.init(Short.MAX_VALUE);
Is the HLog change necessary?
The new blank line in Dictionary has white spaces. Same in ProtobufLogWriter.
And ReaderBase.
ReaderBase.isRecoveredEdits() should just be one line, returning the statement,
don't "if () return true else false".
And I see the same method in WriterBase, can we have a more centralized way of
telling if a path contains recovered edits?
> Review/improve HLog compression's memory consumption
>
>
> Key: HBASE-7391
> URL: https://issues.apache.org/jira/browse/HBASE-7391
> Project: HBase
> Issue Type: Bug
>Reporter: Jean-Daniel Cryans
>Assignee: ramkrishna.s.vasudevan
> Fix For: 0.95.2
>
> Attachments: HBASE-7391_1.patch
>
>
> From Ram in
> http://mail-archives.apache.org/mod_mbox/hbase-dev/201205.mbox/%3C00bc01cd31e6$7caf1320$760d3960$%25vasude...@huawei.com%3E:
> {quote}
> One small observation after giving +1 on the RC.
> The WAL compression feature causes OOME and causes Full GC.
> The problem is, if we have 1500 regions and I need to create recovered.edits
> for each of the region (I don’t have much data in the regions (~300MB)).
> Now when I try to build the dictionary there is a Node object getting
> created.
> Each node object occupies 32 bytes.
> We have 5 such dictionaries.
> Initially we create indexToNodes array and its size is 32767.
> So now we have 32*5*32767 = ~5MB.
> Now I have 1500 regions.
> So 5MB*1500 = ~7GB.(Excluding actual data). This seems to a very high
> initial memory foot print and this never allows me to split the logs and I
> am not able to make the cluster up at all.
> Our configured heap size was 8GB, tested in 3 node cluster with 5000
> regions, very less data( 1GB in hdfs cluster including replication), some
> small data is spread evenly across all regions.
> The formula is 32(Node object size)*5(No of dictionary)*32767(no of node
> objects)*noofregions.
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8615) HLog Compression fails in mysterious ways (working title)
[
https://issues.apache.org/jira/browse/HBASE-8615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13733944#comment-13733944
]
Jean-Daniel Cryans commented on HBASE-8615:
---
I understand why you put it there and I think it's a valuable test because it's
a micro-er (???) test than TestReplicationKillMasterRSCompressed. It just needs
to be cleaned up.
> HLog Compression fails in mysterious ways (working title)
> -
>
> Key: HBASE-8615
> URL: https://issues.apache.org/jira/browse/HBASE-8615
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 0.98.0, 0.96.0
>
> Attachments: 172.21.3.117%2C60020%2C1375222888304.1375222894855.zip,
> 8615-v2.txt, 8615-v3.txt, HBASE-8615-test.patch,
> org.apache.hadoop.hbase.replication.TestReplicationQueueFailoverCompressed-output.txt
>
>
> In a recent test run, I noticed the following in test output:
> {code}
> 2013-05-24 22:01:02,424 DEBUG
> [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2]
> fs.HFileSystem$ReorderWALBlocks(327):
> /user/hortonzy/hbase/.logs/kiyo.gq1.ygridcore.net,42690,1369432806911/kiyo.gq1.ygridcore.net%2C42690%2C1369432806911.1369432840428
> is an HLog file, so reordering blocks, last hostname will
> be:kiyo.gq1.ygridcore.net
> 2013-05-24 22:01:02,429 DEBUG
> [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2]
> wal.ProtobufLogReader(118): After reading the trailer: walEditsStopOffset:
> 132235, fileLength: 132243, trailerPresent: true
> 2013-05-24 22:01:02,438 ERROR
> [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2]
> wal.ProtobufLogReader(236): Error while reading 691 WAL KVs; started
> reading at 53272 and read up to 65538
> 2013-05-24 22:01:02,438 WARN
> [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2]
> regionserver.ReplicationSource(324): 2 Got:
> java.io.IOException: Error while reading 691 WAL KVs; started reading at
> 53272 and read up to 65538
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:237)
> at
> org.apache.hadoop.hbase.regionserver.wal.ReaderBase.next(ReaderBase.java:96)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.readNextAndSetPosition(ReplicationHLogReaderManager.java:89)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.readAllEntriesToReplicateOrNextFile(ReplicationSource.java:404)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:320)
> Caused by: java.lang.IndexOutOfBoundsException: index (30062) must be less
> than size (1)
> at
> com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305)
> at
> com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284)
> at
> org.apache.hadoop.hbase.regionserver.wal.LRUDictionary$BidirectionalLRUMap.get(LRUDictionary.java:124)
> at
> org.apache.hadoop.hbase.regionserver.wal.LRUDictionary$BidirectionalLRUMap.access$000(LRUDictionary.java:71)
> at
> org.apache.hadoop.hbase.regionserver.wal.LRUDictionary.getEntry(LRUDictionary.java:42)
> at
> org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$CompressedKvDecoder.readIntoArray(WALCellCodec.java:210)
> at
> org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$CompressedKvDecoder.parseCell(WALCellCodec.java:184)
> at
> org.apache.hadoop.hbase.codec.BaseDecoder.advance(BaseDecoder.java:46)
> at
> org.apache.hadoop.hbase.regionserver.wal.WALEdit.readFromCells(WALEdit.java:213)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.readNext(ProtobufLogReader.java:217)
> ... 4 more
> 2013-05-24 22:01:02,439 DEBUG
> [RegionServer:0;kiyo.gq1.ygridcore.net,42690,1369432806911.replicationSource,2]
> regionserver.ReplicationSource(583): Nothing to replicate, sleeping 100
> times 10
> {code}
> Will attach test output.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
