[jira] [Commented] (HBASE-7258) Hbase needs to create baseZNode recursively
[
https://issues.apache.org/jira/browse/HBASE-7258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13541047#comment-13541047
]
ramkrishna.s.vasudevan commented on HBASE-7258:
---
{code}
I think they have relations with the patch.
{code}
Oh..so we need to see what is exactly wrong?
> Hbase needs to create baseZNode recursively
> ---
>
> Key: HBASE-7258
> URL: https://issues.apache.org/jira/browse/HBASE-7258
> Project: HBase
> Issue Type: Bug
> Components: Zookeeper
>Affects Versions: 0.94.2
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Minor
> Labels: zookeeper, zookeeper.znode.parent
> Fix For: 0.96.0
>
> Attachments: HBASE-7258-0.94.patch, HBASE-7258.diff, HBASE-7258.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> In deploy env, multi small hbase clusters may share a same zk cluster. So,
> for hbase cluster1, its parent znode is /hbase/cluster1. But in hbase version
> 0.94.1, hbase use ZKUtil.createAndFailSilent(this, baseZNode) to create
> parent path and it will throw a NoNode exception if znode /hbase donot exist.
> We want to change it to ZKUtil.createWithParents(this, baseZNode); to suport
> create baseZNode recursivly.
> The NoNode exception is:
> java.lang.RuntimeException: Failed construction of Master: class
> org.apache.hadoop.hbase.master.HMaster
> at
> org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1792)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:146)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:103)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:77)
> at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1806)
> Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode = NoNode for /hbase/cluster1
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:778)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:420)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:402)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:905)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.createBaseZNodes(ZooKeeperWatcher.java:166)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.(ZooKeeperWatcher.java:159)
> at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:282)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at org.apache.hadoop.hbase.master.H
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13541040#comment-13541040
]
Hadoop QA commented on HBASE-7403:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12562704/hbase-7403-trunkv6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.TestDrainingServer
{color:red}-1 core zombie tests{color}. There are 3 zombie test(s):
at
org.apache.hadoop.hbase.io.encoding.TestUpgradeFromHFileV1ToEncoding.testUpgrade(TestUpgradeFromHFileV1ToEncoding.java:83)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3762//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3762//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3762//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3762//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3762//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3762//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3762//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3762//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3762//console
This message is automatically generated.
> Online Merge
>
>
> Key: HBASE-7403
> URL: https://issues.apache.org/jira/browse/HBASE-7403
> Project: HBase
> Issue Type: New Feature
>Affects Versions: 0.94.3
>Reporter: chunhui shen
>Assignee: chunhui shen
> Fix For: 0.96.0, 0.94.5
>
> Attachments: 7403-trunkv5.patch, 7403v5.diff, 7403-v5.txt,
> 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv1.patch,
> hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, merge region.pdf
>
>
> The feature of this online merge:
> 1.Online,no necessary to disable table
> 2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90
> 3.Easy to call merege request, no need to input a long region name, only
> encoded name enough
> 4.No limit when operation, you don't need to care event like Server Dead,
> Balance, Split, Disabing/Enabing table, no need to care whether you send a
> wrong merge request, it alread have done for you
> 5.Only little offline time for two merging regions
> We need merge in the following cases：
> 1.Region hole or region overlap, can’t be fix by hbck
> 2.Region become empty because of TTL and not reasonable Rowkey design
> 3.Region is always empty or very small because of presplit when create table
> 4.Too many empty or small regions would reduce the system performance(e.g.
> mslab)
> Current merge tools only support offline and are not able to redo if
> exception is thrown in the process of merging, causing a dirty data
> For online system, we need a online merge.
> This implement logic of this patch for Online Merge is :
> For example, merge regionA and regionB into regionC
> 1.Offline the two regions A and B
> 2.Merge the two regions in the HDFS(Create regionC’s directory, move
> regionA’s and regionB’s file to regionC’s directory, delete regionA’s and
> regionB’s directory)
> 3.Add the merged regionC to .META.
> 4.Assign the merged regionC
> As design of this patch , once we do the merge work in the HDFS,we could redo
> it until successful if it throws excep
[jira] [Commented] (HBASE-7460) Cleanup client connection layers
[
https://issues.apache.org/jira/browse/HBASE-7460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13541037#comment-13541037
]
Gary Helmling commented on HBASE-7460:
--
[~saint@gmail.com] Yes, it would be good to sync up in this area and align
our plans. If going through with exposing the protobuf Service implementations
proves out, dropping the dynamic proxying would definitely simplify some
things, and I also got the impression that there was a lot of accumulated cruft
to weed out, ClientCache being a prime example.
Let's streamline the intermediate code first, then look at cutting out the wire
transfer overhead. Benoit's docs make good reading. There are a lot of gains
to be made there as well.
> Cleanup client connection layers
>
>
> Key: HBASE-7460
> URL: https://issues.apache.org/jira/browse/HBASE-7460
> Project: HBase
> Issue Type: Improvement
> Components: Client, IPC/RPC
>Reporter: Gary Helmling
>
> This issue originated from a discussion over in HBASE-7442. We currently
> have a broken abstraction with {{HBaseClient}}, where it is bound to a single
> {{Configuration}} instance at time of construction, but then reused for all
> connections to all clusters. This is combined with multiple, overlapping
> layers of connection caching.
> Going through this code, it seems like we have a lot of mismatch between the
> higher layers and the lower layers, with too much abstraction in between. At
> the lower layers, most of the {{ClientCache}} stuff seems completely unused.
> We currently effectively have an {{HBaseClient}} singleton (for
> {{SecureClient}} as well in 0.92/0.94) in the client code, as I don't see
> anything that calls the constructor or {{RpcEngine.getProxy()}} versions with
> a non-default socket factory. So a lot of the code around this seems like
> built up waste.
> The fact that a single Configuration is fixed in the {{HBaseClient}} seems
> like a broken abstraction as it currently stands. In addition to cluster ID,
> other configuration parameters (max retries, retry sleep) are fixed at time
> of construction. The more I look at the code, the more it looks like the
> {{ClientCache}} and sharing the {{HBaseClient}} instance is an unnecessary
> complication. Why cache the {{HBaseClient}} instances at all? In
> {{HConnectionManager}}, we already have a mapping from {{Configuration}} to
> {{HConnection}}. It seems to me like each {{HConnection(Implementation)}}
> instance should have it's own {{HBaseClient}} instance, doing away with the
> {{ClientCache}} mapping. This would keep each {{HBaseClient}} associated with
> a single cluster/configuration and fix the current breakage from reusing the
> same {{HBaseClient}} against different clusters.
> We need a refactoring of some of the interactions of
> {{HConnection(Implementation)}}, {{HBaseRPC/RpcEngine}}, and {{HBaseClient}}.
> Off hand, we might want to expose a separate {{RpcEngine.getClient()}} method
> that returns a new {{RpcClient}} interface (implemented by {{HBaseClient}})
> and move the {{RpcEngine.getProxy()}}/{{stopProxy()}} implementations into
> the client. So all proxy invocations can go through the same client, without
> requiring the static client cache. I haven't fully thought this through, so I
> could be missing other important aspects. But that approach at least seems
> like a step in the right direction for fixing the client abstractions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chunhui shen updated HBASE-7403: Attachment: hbase-7403-trunkv6.patch Attaching patchV6: Short long lines and fix javadoc warnings > Online Merge > > > Key: HBASE-7403 > URL: https://issues.apache.org/jira/browse/HBASE-7403 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.94.3 >Reporter: chunhui shen >Assignee: chunhui shen > Fix For: 0.96.0, 0.94.5 > > Attachments: 7403-trunkv5.patch, 7403v5.diff, 7403-v5.txt, > 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv1.patch, > hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, merge region.pdf > > > The feature of this online merge: > 1.Online,no necessary to disable table > 2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90 > 3.Easy to call merege request, no need to input a long region name, only > encoded name enough > 4.No limit when operation, you don't need to care event like Server Dead, > Balance, Split, Disabing/Enabing table, no need to care whether you send a > wrong merge request, it alread have done for you > 5.Only little offline time for two merging regions > We need merge in the following cases： > 1.Region hole or region overlap, can’t be fix by hbck > 2.Region become empty because of TTL and not reasonable Rowkey design > 3.Region is always empty or very small because of presplit when create table > 4.Too many empty or small regions would reduce the system performance(e.g. > mslab) > Current merge tools only support offline and are not able to redo if > exception is thrown in the process of merging, causing a dirty data > For online system, we need a online merge. > This implement logic of this patch for Online Merge is : > For example, merge regionA and regionB into regionC > 1.Offline the two regions A and B > 2.Merge the two regions in the HDFS(Create regionC’s directory, move > regionA’s and regionB’s file to regionC’s directory, delete regionA’s and > regionB’s directory) > 3.Add the merged regionC to .META. > 4.Assign the merged regionC > As design of this patch , once we do the merge work in the HDFS,we could redo > it until successful if it throws exception or abort or server restart, but > couldn’t be rolled back. > It depends on > Use zookeeper to record the transaction journal state, make redo easier > Use zookeeper to send/receive merge request > Merge transaction is executed on the master > Support calling merge request through API or shell tool > About the merge process, please see the attachment and patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7459) NPE in HMaster TestlocalHBaseCluster
[
https://issues.apache.org/jira/browse/HBASE-7459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13541028#comment-13541028
]
Hudson commented on HBASE-7459:
---
Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #319 (See
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/319/])
HBASE-7459 NPE in HMaster TestLocalHBaseCluster (Revision 1426789)
Result = FAILURE
jmhsieh :
Files :
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
> NPE in HMaster TestlocalHBaseCluster
>
>
> Key: HBASE-7459
> URL: https://issues.apache.org/jira/browse/HBASE-7459
> Project: HBase
> Issue Type: Bug
> Components: master, test
>Affects Versions: hbase-6055, 0.96.0, hbase-7290
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: hbase-6055, 0.96.0, hbase-7290
>
> Attachments: hbase-7459.patch
>
>
> TestLocalHBaseCluster has failed intermittently on a unit test run with this
> exception stack.
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.master.HMaster.shutdown(HMaster.java:2064)
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.shutdown(JVMClusterUtil.java:238)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.shutdown(LocalHBaseCluster.java:430)
> at
> org.apache.hadoop.hbase.TestLocalHBaseCluster.testLocalHBaseCluster(TestLocalHBaseCluster.java:71)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
>
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7321) Simple Flush Snapshot
[
https://issues.apache.org/jira/browse/HBASE-7321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13541017#comment-13541017
]
Hadoop QA commented on HBASE-7321:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12562699/pre-hbase-7321.v3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 88 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3760//console
This message is automatically generated.
> Simple Flush Snapshot
> -
>
> Key: HBASE-7321
> URL: https://issues.apache.org/jira/browse/HBASE-7321
> Project: HBase
> Issue Type: Sub-task
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Attachments: hbase-7321.v2.patch, hbase-7321.v3.patch,
> pre-hbase-7321.v2.patch, pre-hbase-7321.v3.patch
>
>
> This snapshot style just issues a region flush and then "snapshots" the
> region.
> This is a simple implementation that gives the equivalent of copytable
> consistency. While by most definitions of consistency if a client writes A
> and then write B to different region servers, only neither, only A, or both
> A+B writes should be present, this one allows the only B case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6864) Online snapshots scaffolding
[
https://issues.apache.org/jira/browse/HBASE-6864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13541016#comment-13541016
]
Hadoop QA commented on HBASE-6864:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12562697/pre-hbase-6864.v3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 88 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3761//console
This message is automatically generated.
> Online snapshots scaffolding
>
>
> Key: HBASE-6864
> URL: https://issues.apache.org/jira/browse/HBASE-6864
> Project: HBase
> Issue Type: Sub-task
>Reporter: Jesse Yates
>Assignee: Jonathan Hsieh
> Fix For: hbase-6055
>
> Attachments: hbase-6864.v1.patch, hbase-6864.v2.patch,
> hbase-6864.v3.patch, pre-hbase-6864.v1.patch, pre-hbase-6864.v2.patch,
> pre-hbase-6864.v3.patch
>
>
> Basic scaffolding for taking a snapshot of an offline table. This includes
> the basic work on both the regionserver and master to support (but not
> implement) timestamp and globally consistent snapshots.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6864) Online snapshots scaffolding
[ https://issues.apache.org/jira/browse/HBASE-6864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-6864: -- Attachment: pre-hbase-6864.v3.patch hbase-6864.v3.patch Updates from reviews > Online snapshots scaffolding > > > Key: HBASE-6864 > URL: https://issues.apache.org/jira/browse/HBASE-6864 > Project: HBase > Issue Type: Sub-task >Reporter: Jesse Yates >Assignee: Jonathan Hsieh > Fix For: hbase-6055 > > Attachments: hbase-6864.v1.patch, hbase-6864.v2.patch, > hbase-6864.v3.patch, pre-hbase-6864.v1.patch, pre-hbase-6864.v2.patch, > pre-hbase-6864.v3.patch > > > Basic scaffolding for taking a snapshot of an offline table. This includes > the basic work on both the regionserver and master to support (but not > implement) timestamp and globally consistent snapshots. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7321) Simple Flush Snapshot
[ https://issues.apache.org/jira/browse/HBASE-7321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-7321: -- Attachment: pre-hbase-7321.v3.patch hbase-7321.v3.patch updates from reviews > Simple Flush Snapshot > - > > Key: HBASE-7321 > URL: https://issues.apache.org/jira/browse/HBASE-7321 > Project: HBase > Issue Type: Sub-task >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Attachments: hbase-7321.v2.patch, hbase-7321.v3.patch, > pre-hbase-7321.v2.patch, pre-hbase-7321.v3.patch > > > This snapshot style just issues a region flush and then "snapshots" the > region. > This is a simple implementation that gives the equivalent of copytable > consistency. While by most definitions of consistency if a client writes A > and then write B to different region servers, only neither, only A, or both > A+B writes should be present, this one allows the only B case. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7461) Cleanup stoppable/abortable/closeable in the online snapshot cases.
Jonathan Hsieh created HBASE-7461: - Summary: Cleanup stoppable/abortable/closeable in the online snapshot cases. Key: HBASE-7461 URL: https://issues.apache.org/jira/browse/HBASE-7461 Project: HBase Issue Type: Sub-task Reporter: Jonathan Hsieh The RegionServer managers implement abort, close, and stop -- although the interfaces are similar and all their meanings are muddled. The conventions in hbase are gernally: abort == passed into managers so they can trigger a suicide kill (for rs or hmaster) stop == *managers for on the way to cleanup cancel == operations that don't kill long running processes but bail out of the current attempt. close == files or network resources. This patch brings the naming into line. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7460) Cleanup client connection layers
[
https://issues.apache.org/jira/browse/HBASE-7460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13541010#comment-13541010
]
stack commented on HBASE-7460:
--
I am in this area at the moment, at a level just above HBaseClient trying to
make use of it. I'm playing with trying to use protobuf Service and hooking it
up on either end to use our RPC. There are pros but a bunch of cons with the
main one being mostly the amount of refactoring that would have to do in this
area if we were to go this route.
My first impression submerging below the level of HBaseClientRPC is that there
is a bunch of cruft in here, stuff that has been accumulating over time and
that we've probably been afraid to apply the compressed air can too.
I want to make use of clients. Was going to copy what is going on in Invoker
not knowing any better. I want to use something else than "protocol" as the
key getting the client.
In my investigations, the first thing to jettison would be the proxy stuff. In
my case it is in the way (I'd use the protobuf Service.Stub instead). Getting
a proxy has a bunch of overrides. A bunch look unused, as you say. Also,
protocol 'versioning' and protocol 'fingerprinting' -- VersionedProtocol and
ProtocolSignature -- are in the former case not hooked up, and in the latter, a
facility that is incomplete and unused so all this code needs finishing or we
need to just throw it out.
bq. It seems to me like each HConnection(Implementation) instance should have
it's own HBaseClient instance, doing away with the ClientCache mapping
Sounds imminently sensible.
I'd be up for sketching something out if you had a few minutes to hang G.
Still to do, though not directly related here but it is in this realm only at a
lower level, is the back and forth over RPC, what we put on the wire. As is
where we create a pb from an already made request pb -- with the former making
a copy of the latter -- needs fixing and we should take the opportunity to
address some of the criticisms' Benoît/Tsuna raised in Unofficial Hadoop /
HBase RPC protocol documentation
(http://www.google.com/url?q=https%3A%2F%2Fgithub.com%2FOpenTSDB%2Fasynchbase%2Fblob%2Fmaster%2Fsrc%2FHBaseRpc.java%23L164&sa=D&sntz=1&usg=AFQjCNEy00ZQVclIR7BaBJYBdRV-i7QGTg)
> Cleanup client connection layers
>
>
> Key: HBASE-7460
> URL: https://issues.apache.org/jira/browse/HBASE-7460
> Project: HBase
> Issue Type: Improvement
> Components: Client, IPC/RPC
>Reporter: Gary Helmling
>
> This issue originated from a discussion over in HBASE-7442. We currently
> have a broken abstraction with {{HBaseClient}}, where it is bound to a single
> {{Configuration}} instance at time of construction, but then reused for all
> connections to all clusters. This is combined with multiple, overlapping
> layers of connection caching.
> Going through this code, it seems like we have a lot of mismatch between the
> higher layers and the lower layers, with too much abstraction in between. At
> the lower layers, most of the {{ClientCache}} stuff seems completely unused.
> We currently effectively have an {{HBaseClient}} singleton (for
> {{SecureClient}} as well in 0.92/0.94) in the client code, as I don't see
> anything that calls the constructor or {{RpcEngine.getProxy()}} versions with
> a non-default socket factory. So a lot of the code around this seems like
> built up waste.
> The fact that a single Configuration is fixed in the {{HBaseClient}} seems
> like a broken abstraction as it currently stands. In addition to cluster ID,
> other configuration parameters (max retries, retry sleep) are fixed at time
> of construction. The more I look at the code, the more it looks like the
> {{ClientCache}} and sharing the {{HBaseClient}} instance is an unnecessary
> complication. Why cache the {{HBaseClient}} instances at all? In
> {{HConnectionManager}}, we already have a mapping from {{Configuration}} to
> {{HConnection}}. It seems to me like each {{HConnection(Implementation)}}
> instance should have it's own {{HBaseClient}} instance, doing away with the
> {{ClientCache}} mapping. This would keep each {{HBaseClient}} associated with
> a single cluster/configuration and fix the current breakage from reusing the
> same {{HBaseClient}} against different clusters.
> We need a refactoring of some of the interactions of
> {{HConnection(Implementation)}}, {{HBaseRPC/RpcEngine}}, and {{HBaseClient}}.
> Off hand, we might want to expose a separate {{RpcEngine.getClient()}} method
> that returns a new {{RpcClient}} interface (implemented by {{HBaseClient}})
> and move the {{RpcEngine.getProxy()}}/{{stopProxy()}} implementations into
> the client. So all proxy invocations can go through the same client, without
> requiring the static client cache. I
[jira] [Updated] (HBASE-7442) HBase remote CopyTable not working when security enabled
[
https://issues.apache.org/jira/browse/HBASE-7442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Kinley updated HBASE-7442:
Attachment: HBASE-7442-0.94.patch
It works :)
You're correct, the 0.94 patch is identical (attached). Trunk is different so
this will take me a little longer.
> HBase remote CopyTable not working when security enabled
>
>
> Key: HBASE-7442
> URL: https://issues.apache.org/jira/browse/HBASE-7442
> Project: HBase
> Issue Type: Bug
> Components: IPC/RPC, mapreduce, security
>Affects Versions: 0.92.1
>Reporter: James Kinley
> Fix For: 0.92.3, 0.96.0, 0.94.5
>
> Attachments: attempt_201212271546_0001_m_00_0.log,
> HBASE-7442-0.92.1.patch, HBASE-7442-0.94.patch
>
>
> When security is enabled, HBase CopyTable fails with Kerberos exception:
> {code}
> FATAL org.apache.hadoop.ipc.SecureClient: SASL authentication failed. The
> most likely cause is missing or invalid credentials. Consider 'kinit'.
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> {code}
> This is only when copying to remote HBase cluster (using either MRv1 or
> YARN), local copy works fine.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7321) Simple Flush Snapshot
[ https://issues.apache.org/jira/browse/HBASE-7321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13541006#comment-13541006 ] Jonathan Hsieh commented on HBASE-7321: --- I've addressed ted's comments on the snapshot-work-1229 version. I'll file a follow on for the TODO about refactoring TestFlushSnapshotFromClient. Currently the order of patches has changed, and this is the first online version to make it. The refactor makes sense for when we add a 3rd flavor (offline, flush, + the next one). > Simple Flush Snapshot > - > > Key: HBASE-7321 > URL: https://issues.apache.org/jira/browse/HBASE-7321 > Project: HBase > Issue Type: Sub-task >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Attachments: hbase-7321.v2.patch, pre-hbase-7321.v2.patch > > > This snapshot style just issues a region flush and then "snapshots" the > region. > This is a simple implementation that gives the equivalent of copytable > consistency. While by most definitions of consistency if a client writes A > and then write B to different region servers, only neither, only A, or both > A+B writes should be present, this one allows the only B case. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-7207) Consolidate snapshot related classes into fewer packages
[ https://issues.apache.org/jira/browse/HBASE-7207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh resolved HBASE-7207. --- Resolution: Fixed Fix Version/s: (was: 0.96.0) hbase-7290 Hadoop Flags: Reviewed > Consolidate snapshot related classes into fewer packages > > > Key: HBASE-7207 > URL: https://issues.apache.org/jira/browse/HBASE-7207 > Project: HBase > Issue Type: Sub-task > Components: snapshots >Affects Versions: hbase-6055 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: hbase-6055, hbase-7290 > > Attachments: hbase-7207.patch, pre-hbase-7207.patch > > > The snapshot branch seems to have more packages with fewer classes present in > each. We should consolidate them down to a core set. I'm suggesting > limiting it to: > o.a.h.h.errorhandling (move o.a.h.h.server.errorhandling.** to this package) > o.a.h.h.procedure (eliminate procedure.member, procedure.coordinator, > possibly add zk for the zk implementation) > o.a.h.h.snapshot (move o.a.h.h.server.snapshots.** to this package) > o.a.h.h.master.snapshot (fold subpackages in) > o.a.h.h.regionserver.snapshot (fold subpackges in) > Likely move all TestSnapshotFrom* to o.a.h.h.snapshot. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7459) NPE in HMaster TestlocalHBaseCluster
[
https://issues.apache.org/jira/browse/HBASE-7459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540975#comment-13540975
]
Hudson commented on HBASE-7459:
---
Integrated in HBase-TRUNK #3674 (See
[https://builds.apache.org/job/HBase-TRUNK/3674/])
HBASE-7459 NPE in HMaster TestLocalHBaseCluster (Revision 1426789)
Result = FAILURE
jmhsieh :
Files :
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
> NPE in HMaster TestlocalHBaseCluster
>
>
> Key: HBASE-7459
> URL: https://issues.apache.org/jira/browse/HBASE-7459
> Project: HBase
> Issue Type: Bug
> Components: master, test
>Affects Versions: hbase-6055, 0.96.0, hbase-7290
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: hbase-6055, 0.96.0, hbase-7290
>
> Attachments: hbase-7459.patch
>
>
> TestLocalHBaseCluster has failed intermittently on a unit test run with this
> exception stack.
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.master.HMaster.shutdown(HMaster.java:2064)
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.shutdown(JVMClusterUtil.java:238)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.shutdown(LocalHBaseCluster.java:430)
> at
> org.apache.hadoop.hbase.TestLocalHBaseCluster.testLocalHBaseCluster(TestLocalHBaseCluster.java:71)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
>
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7352) clone operation from HBaseAdmin can hang forever.
[
https://issues.apache.org/jira/browse/HBASE-7352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-7352:
--
Resolution: Fixed
Fix Version/s: (was: 0.96.0)
hbase-7290
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
> clone operation from HBaseAdmin can hang forever.
> -
>
> Key: HBASE-7352
> URL: https://issues.apache.org/jira/browse/HBASE-7352
> Project: HBase
> Issue Type: Sub-task
> Components: Client, master, regionserver, snapshots, Zookeeper
>Reporter: Jonathan Hsieh
>Assignee: Matteo Bertozzi
> Fix For: hbase-6055, hbase-7290
>
> Attachments: HBASE-7352-v0.patch, HBASE-7352-v1.patch,
> HBASE-7352-v2.patch, hbase-7352.v3.patch
>
>
> Sometimes the clone operation from the hbase shell can hang. The table has
> been created (it shows up in the web ui), but does not have any entries in
> META.
> There don't seem to be any clone, snapshot, enable or disable found in the
> master's jstack.
> Here's a trace from the HBaseAdmin:
> {code}
> "main" prio=10 tid=0x7f782800d000 nid=0x25c waiting on condition
> [0x7f782f9bf000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.cloneSnapshot(HBaseAdmin.java:2413)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.cloneSnapshot(HBaseAdmin.java:2393)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:465)
> at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:323)
> at
> org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:69)
> at
> org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:201)
> at org.jruby.ast.CallTwoArgNode.interpret(CallTwoArgNode.java:59)
> at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
> ... (more jruby stack) ...
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-7352) clone operation from HBaseAdmin can hang forever.
[
https://issues.apache.org/jira/browse/HBASE-7352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540972#comment-13540972
]
Jonathan Hsieh edited comment on HBASE-7352 at 12/29/12 7:05 PM:
-
passed with failures on known flakys. Thanks matteo and ted. Committing to
online and offline snapshots branches.
was (Author: jmhsieh):
passed with failures on known flakys. committing to online and offline
snapshots branches.
> clone operation from HBaseAdmin can hang forever.
> -
>
> Key: HBASE-7352
> URL: https://issues.apache.org/jira/browse/HBASE-7352
> Project: HBase
> Issue Type: Sub-task
> Components: Client, master, regionserver, snapshots, Zookeeper
>Reporter: Jonathan Hsieh
>Assignee: Matteo Bertozzi
> Fix For: hbase-6055, 0.96.0
>
> Attachments: HBASE-7352-v0.patch, HBASE-7352-v1.patch,
> HBASE-7352-v2.patch, hbase-7352.v3.patch
>
>
> Sometimes the clone operation from the hbase shell can hang. The table has
> been created (it shows up in the web ui), but does not have any entries in
> META.
> There don't seem to be any clone, snapshot, enable or disable found in the
> master's jstack.
> Here's a trace from the HBaseAdmin:
> {code}
> "main" prio=10 tid=0x7f782800d000 nid=0x25c waiting on condition
> [0x7f782f9bf000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.cloneSnapshot(HBaseAdmin.java:2413)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.cloneSnapshot(HBaseAdmin.java:2393)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:465)
> at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:323)
> at
> org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:69)
> at
> org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:201)
> at org.jruby.ast.CallTwoArgNode.interpret(CallTwoArgNode.java:59)
> at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
> ... (more jruby stack) ...
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7352) clone operation from HBaseAdmin can hang forever.
[
https://issues.apache.org/jira/browse/HBASE-7352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540972#comment-13540972
]
Jonathan Hsieh commented on HBASE-7352:
---
passed with failures on known flakys. committing to online and offline
snapshots branches.
> clone operation from HBaseAdmin can hang forever.
> -
>
> Key: HBASE-7352
> URL: https://issues.apache.org/jira/browse/HBASE-7352
> Project: HBase
> Issue Type: Sub-task
> Components: Client, master, regionserver, snapshots, Zookeeper
>Reporter: Jonathan Hsieh
>Assignee: Matteo Bertozzi
> Fix For: hbase-6055, 0.96.0
>
> Attachments: HBASE-7352-v0.patch, HBASE-7352-v1.patch,
> HBASE-7352-v2.patch, hbase-7352.v3.patch
>
>
> Sometimes the clone operation from the hbase shell can hang. The table has
> been created (it shows up in the web ui), but does not have any entries in
> META.
> There don't seem to be any clone, snapshot, enable or disable found in the
> master's jstack.
> Here's a trace from the HBaseAdmin:
> {code}
> "main" prio=10 tid=0x7f782800d000 nid=0x25c waiting on condition
> [0x7f782f9bf000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.cloneSnapshot(HBaseAdmin.java:2413)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.cloneSnapshot(HBaseAdmin.java:2393)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:465)
> at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:323)
> at
> org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:69)
> at
> org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:201)
> at org.jruby.ast.CallTwoArgNode.interpret(CallTwoArgNode.java:59)
> at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
> ... (more jruby stack) ...
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7294) Check for snapshot file cleaners on start
[
https://issues.apache.org/jira/browse/HBASE-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-7294:
--
Resolution: Fixed
Fix Version/s: (was: 0.96.0)
hbase-7290
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
> Check for snapshot file cleaners on start
> -
>
> Key: HBASE-7294
> URL: https://issues.apache.org/jira/browse/HBASE-7294
> Project: HBase
> Issue Type: Sub-task
> Components: Client, master, regionserver, snapshots, Zookeeper
>Affects Versions: hbase-6055
>Reporter: Jesse Yates
>Assignee: Matteo Bertozzi
> Fix For: hbase-6055, hbase-7290
>
> Attachments: HBASE-7294-v1.patch, HBASE-7294-v2.patch,
> HBASE-7294-v3.patch, HBASE-7294-v4.patch, HBASE-7294-v5.patch,
> HBASE-7294-v6.patch, HBASE-7294-v7.patch
>
>
> Snapshots currently use the SnaphotHfileCleaner and SnapshotHLogCleaner to
> ensure that any hfiles or hlogs (respectively) that are currently part of a
> snapshot are not removed from their respective archive directories (.archive
> and .oldlogs).
> From Matteo Bertozzi:
> {quote}
> currently the snapshot cleaner is not in hbase-default.xml
> and there's no warning/exception on snapshot/restore operation, if not
> enabled.
> even if we add the cleaner to the hbase-default.xml how do we ensure that the
> user doesn't remove it?
> Do we want to hardcode the cleaner at master startup?
> Do we want to add a check in snapshot/restore that throws an exception if the
> cleaner is not enabled?
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7294) Check for snapshot file cleaners on start
[
https://issues.apache.org/jira/browse/HBASE-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540970#comment-13540970
]
Jonathan Hsieh commented on HBASE-7294:
---
+1. Thanks Matteo, Jesse. Committed to offline and online snapshots branches.
> Check for snapshot file cleaners on start
> -
>
> Key: HBASE-7294
> URL: https://issues.apache.org/jira/browse/HBASE-7294
> Project: HBase
> Issue Type: Sub-task
> Components: Client, master, regionserver, snapshots, Zookeeper
>Affects Versions: hbase-6055
>Reporter: Jesse Yates
>Assignee: Matteo Bertozzi
> Fix For: hbase-6055, 0.96.0
>
> Attachments: HBASE-7294-v1.patch, HBASE-7294-v2.patch,
> HBASE-7294-v3.patch, HBASE-7294-v4.patch, HBASE-7294-v5.patch,
> HBASE-7294-v6.patch, HBASE-7294-v7.patch
>
>
> Snapshots currently use the SnaphotHfileCleaner and SnapshotHLogCleaner to
> ensure that any hfiles or hlogs (respectively) that are currently part of a
> snapshot are not removed from their respective archive directories (.archive
> and .oldlogs).
> From Matteo Bertozzi:
> {quote}
> currently the snapshot cleaner is not in hbase-default.xml
> and there's no warning/exception on snapshot/restore operation, if not
> enabled.
> even if we add the cleaner to the hbase-default.xml how do we ensure that the
> user doesn't remove it?
> Do we want to hardcode the cleaner at master startup?
> Do we want to add a check in snapshot/restore that throws an exception if the
> cleaner is not enabled?
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7354) Snapshot Info/Debug Tool
[
https://issues.apache.org/jira/browse/HBASE-7354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540963#comment-13540963
]
Jonathan Hsieh commented on HBASE-7354:
---
I like. Please add the comments (comments near where some of these questions
are would be good) and please post what an example run's output looks like
(instead of tests)?
change to o.a.h.h.snapshot? (my fault, HBASE-7207)
{code}
package org.apache.hadoop.hbase.snapshot.tool;
{code}
class Javadoc? what is this for (for someone looking only at the code?)
Log when dir doesn't exist (more helpful to have dir name here -- calling site
doesn't have it)
{code}
+snapshotDir =
SnapshotDescriptionUtils.getCompletedSnapshotDir(snapshotName, rootDir);
+if (!fs.exists(snapshotDir)) return false;
{code}
Worth showing if in archive or in original file location? (could be follow on)
{code}
+ if (showFiles) {
+System.out.printf("%8s %s/%s/%s/%s\n",
+ StringUtils.humanReadableInt(size), table, region, family,
hfile);
+ }
{code}
The storefile case had if with two branches -- this logfile link doesn't need
to do this?
{code}
public void logFile (final String server, final String logfile)
+throws IOException {
+ HLogLink logLink = new HLogLink(conf, server, logfile);
+ long size = logLink.getFileStatus(fs).getLen();
+ logSize.addAndGet(size);
+ logsCount.addAndGet(1);
+
{code}
> Snapshot Info/Debug Tool
>
>
> Key: HBASE-7354
> URL: https://issues.apache.org/jira/browse/HBASE-7354
> Project: HBase
> Issue Type: Sub-task
> Components: Client, master, regionserver, snapshots, Zookeeper
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
>Priority: Minor
> Fix For: hbase-6055, 0.96.0
>
> Attachments: HBASE-7354-v0.patch
>
>
> Tool that show snapshot metadata, hfiles and logs retained.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7352) clone operation from HBaseAdmin can hang forever.
[
https://issues.apache.org/jira/browse/HBASE-7352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540953#comment-13540953
]
Hadoop QA commented on HBASE-7352:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12562683/hbase-7352.v3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3759//console
This message is automatically generated.
> clone operation from HBaseAdmin can hang forever.
> -
>
> Key: HBASE-7352
> URL: https://issues.apache.org/jira/browse/HBASE-7352
> Project: HBase
> Issue Type: Sub-task
> Components: Client, master, regionserver, snapshots, Zookeeper
>Reporter: Jonathan Hsieh
>Assignee: Matteo Bertozzi
> Fix For: hbase-6055, 0.96.0
>
> Attachments: HBASE-7352-v0.patch, HBASE-7352-v1.patch,
> HBASE-7352-v2.patch, hbase-7352.v3.patch
>
>
> Sometimes the clone operation from the hbase shell can hang. The table has
> been created (it shows up in the web ui), but does not have any entries in
> META.
> There don't seem to be any clone, snapshot, enable or disable found in the
> master's jstack.
> Here's a trace from the HBaseAdmin:
> {code}
> "main" prio=10 tid=0x7f782800d000 nid=0x25c waiting on condition
> [0x7f782f9bf000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.cloneSnapshot(HBaseAdmin.java:2413)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.cloneSnapshot(HBaseAdmin.java:2393)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:465)
> at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:323)
> at
> org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:69)
> at
> org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:201)
> at org.jruby.ast.CallTwoArgNode.interpret(CallTwoArgNode.java:59)
> at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
> ... (more jruby stack) ...
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7459) NPE in HMaster TestlocalHBaseCluster
[
https://issues.apache.org/jira/browse/HBASE-7459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540950#comment-13540950
]
Hadoop QA commented on HBASE-7459:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12562679/hbase-7459.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 3 zombie test(s):
at
org.apache.hadoop.hbase.io.encoding.TestUpgradeFromHFileV1ToEncoding.testUpgrade(TestUpgradeFromHFileV1ToEncoding.java:83)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3758//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3758//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3758//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3758//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3758//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3758//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3758//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3758//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3758//console
This message is automatically generated.
> NPE in HMaster TestlocalHBaseCluster
>
>
> Key: HBASE-7459
> URL: https://issues.apache.org/jira/browse/HBASE-7459
> Project: HBase
> Issue Type: Bug
> Components: master, test
>Affects Versions: hbase-6055, 0.96.0, hbase-7290
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: hbase-6055, 0.96.0, hbase-7290
>
> Attachments: hbase-7459.patch
>
>
> TestLocalHBaseCluster has failed intermittently on a unit test run with this
> exception stack.
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.master.HMaster.shutdown(HMaster.java:2064)
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.shutdown(JVMClusterUtil.java:238)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.shutdown(LocalHBaseCluster.java:430)
> at
> org.apache.hadoop.hbase.TestLocalHBaseCluster.testLocalHBaseCluster(TestLocalHBaseCluster.java:71)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
>
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7352) clone operation from HBaseAdmin can hang forever.
[
https://issues.apache.org/jira/browse/HBASE-7352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-7352:
--
Attachment: hbase-7352.v3.patch
I've made ted's suggested fix. v3 is what I will commit if after tests run.
> clone operation from HBaseAdmin can hang forever.
> -
>
> Key: HBASE-7352
> URL: https://issues.apache.org/jira/browse/HBASE-7352
> Project: HBase
> Issue Type: Sub-task
> Components: Client, master, regionserver, snapshots, Zookeeper
>Reporter: Jonathan Hsieh
>Assignee: Matteo Bertozzi
> Fix For: hbase-6055, 0.96.0
>
> Attachments: HBASE-7352-v0.patch, HBASE-7352-v1.patch,
> HBASE-7352-v2.patch, hbase-7352.v3.patch
>
>
> Sometimes the clone operation from the hbase shell can hang. The table has
> been created (it shows up in the web ui), but does not have any entries in
> META.
> There don't seem to be any clone, snapshot, enable or disable found in the
> master's jstack.
> Here's a trace from the HBaseAdmin:
> {code}
> "main" prio=10 tid=0x7f782800d000 nid=0x25c waiting on condition
> [0x7f782f9bf000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.cloneSnapshot(HBaseAdmin.java:2413)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.cloneSnapshot(HBaseAdmin.java:2393)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:465)
> at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:323)
> at
> org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:69)
> at
> org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:201)
> at org.jruby.ast.CallTwoArgNode.interpret(CallTwoArgNode.java:59)
> at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
> ... (more jruby stack) ...
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7212) Globally Barriered Procedure mechanism
[ https://issues.apache.org/jira/browse/HBASE-7212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-7212: -- Fix Version/s: (was: hbase-6055) hbase-7290 > Globally Barriered Procedure mechanism > -- > > Key: HBASE-7212 > URL: https://issues.apache.org/jira/browse/HBASE-7212 > Project: HBase > Issue Type: Sub-task > Components: snapshots >Affects Versions: hbase-6055 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: hbase-7290 > > Attachments: 121127-global-barrier-proc.pdf, hbase-7212.patch, > hbase-7212.v5.patch, hbase-7212.v8.patch, pre-hbase-7212.patch, > pre-hbase-7212.v5.patch, pre-hbase-7212.v8.patch > > > This is a simplified version of what was proposed in HBASE-6573. Instead of > claiming to be a 2pc or 3pc implementation (which implies logging at each > actor, and recovery operations) this is just provides a best effort global > barrier mechanism called a Procedure. > Users need only to implement a methods to acquireBarrier, to act when > insideBarrier, and to releaseBarrier that use the ExternalException > cooperative error checking mechanism. > Globally consistent snapshots require the ability to quiesce writes to a set > of region servers before a the snapshot operation is executed. Also if any > node fails, it needs to be able to notify them so that they abort. > The first cut of other online snapshots don't need the fully barrier but may > still use this for its error propagation mechanisms. > This version removes the extra layer incurred in the previous implementation > due to the use of generics, separates the coordinator and members, and > reduces the amount of inheritance used in favor of composition. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7459) NPE in HMaster TestlocalHBaseCluster
[
https://issues.apache.org/jira/browse/HBASE-7459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540946#comment-13540946
]
Jonathan Hsieh commented on HBASE-7459:
---
The other managers are not forced to null, so it seems ok to keep it this way.
> NPE in HMaster TestlocalHBaseCluster
>
>
> Key: HBASE-7459
> URL: https://issues.apache.org/jira/browse/HBASE-7459
> Project: HBase
> Issue Type: Bug
> Components: master, test
>Affects Versions: hbase-6055, 0.96.0, hbase-7290
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: hbase-6055, 0.96.0, hbase-7290
>
> Attachments: hbase-7459.patch
>
>
> TestLocalHBaseCluster has failed intermittently on a unit test run with this
> exception stack.
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.master.HMaster.shutdown(HMaster.java:2064)
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.shutdown(JVMClusterUtil.java:238)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.shutdown(LocalHBaseCluster.java:430)
> at
> org.apache.hadoop.hbase.TestLocalHBaseCluster.testLocalHBaseCluster(TestLocalHBaseCluster.java:71)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
>
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7459) NPE in HMaster TestlocalHBaseCluster
[
https://issues.apache.org/jira/browse/HBASE-7459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540945#comment-13540945
]
Matteo Bertozzi commented on HBASE-7459:
spanReceiverHost is private and only the master uses that, so I'd say no need
to assign to null since is not exposed and we're in the shutdown phase. (not
that also the others assignmentManager, serverManager & co are not assigned to
null after shutdown)
> NPE in HMaster TestlocalHBaseCluster
>
>
> Key: HBASE-7459
> URL: https://issues.apache.org/jira/browse/HBASE-7459
> Project: HBase
> Issue Type: Bug
> Components: master, test
>Affects Versions: hbase-6055, 0.96.0, hbase-7290
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: hbase-6055, 0.96.0, hbase-7290
>
> Attachments: hbase-7459.patch
>
>
> TestLocalHBaseCluster has failed intermittently on a unit test run with this
> exception stack.
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.master.HMaster.shutdown(HMaster.java:2064)
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.shutdown(JVMClusterUtil.java:238)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.shutdown(LocalHBaseCluster.java:430)
> at
> org.apache.hadoop.hbase.TestLocalHBaseCluster.testLocalHBaseCluster(TestLocalHBaseCluster.java:71)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
>
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7459) NPE in HMaster TestlocalHBaseCluster
[
https://issues.apache.org/jira/browse/HBASE-7459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540943#comment-13540943
]
Ted Yu commented on HBASE-7459:
---
{code}
+if (spanReceiverHost != null) {
+ spanReceiverHost.closeReceivers();
{code}
Should spanReceiverHost be assigned null at the end of the if block ?
> NPE in HMaster TestlocalHBaseCluster
>
>
> Key: HBASE-7459
> URL: https://issues.apache.org/jira/browse/HBASE-7459
> Project: HBase
> Issue Type: Bug
> Components: master, test
>Affects Versions: hbase-6055, 0.96.0, hbase-7290
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: hbase-6055, 0.96.0, hbase-7290
>
> Attachments: hbase-7459.patch
>
>
> TestLocalHBaseCluster has failed intermittently on a unit test run with this
> exception stack.
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.master.HMaster.shutdown(HMaster.java:2064)
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.shutdown(JVMClusterUtil.java:238)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.shutdown(LocalHBaseCluster.java:430)
> at
> org.apache.hadoop.hbase.TestLocalHBaseCluster.testLocalHBaseCluster(TestLocalHBaseCluster.java:71)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
>
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7460) Cleanup client connection layers
Gary Helmling created HBASE-7460:
Summary: Cleanup client connection layers
Key: HBASE-7460
URL: https://issues.apache.org/jira/browse/HBASE-7460
Project: HBase
Issue Type: Improvement
Components: Client, IPC/RPC
Reporter: Gary Helmling
This issue originated from a discussion over in HBASE-7442. We currently have
a broken abstraction with {{HBaseClient}}, where it is bound to a single
{{Configuration}} instance at time of construction, but then reused for all
connections to all clusters. This is combined with multiple, overlapping
layers of connection caching.
Going through this code, it seems like we have a lot of mismatch between the
higher layers and the lower layers, with too much abstraction in between. At
the lower layers, most of the {{ClientCache}} stuff seems completely unused. We
currently effectively have an {{HBaseClient}} singleton (for {{SecureClient}}
as well in 0.92/0.94) in the client code, as I don't see anything that calls
the constructor or {{RpcEngine.getProxy()}} versions with a non-default socket
factory. So a lot of the code around this seems like built up waste.
The fact that a single Configuration is fixed in the {{HBaseClient}} seems like
a broken abstraction as it currently stands. In addition to cluster ID, other
configuration parameters (max retries, retry sleep) are fixed at time of
construction. The more I look at the code, the more it looks like the
{{ClientCache}} and sharing the {{HBaseClient}} instance is an unnecessary
complication. Why cache the {{HBaseClient}} instances at all? In
{{HConnectionManager}}, we already have a mapping from {{Configuration}} to
{{HConnection}}. It seems to me like each {{HConnection(Implementation)}}
instance should have it's own {{HBaseClient}} instance, doing away with the
{{ClientCache}} mapping. This would keep each {{HBaseClient}} associated with a
single cluster/configuration and fix the current breakage from reusing the same
{{HBaseClient}} against different clusters.
We need a refactoring of some of the interactions of
{{HConnection(Implementation)}}, {{HBaseRPC/RpcEngine}}, and {{HBaseClient}}.
Off hand, we might want to expose a separate {{RpcEngine.getClient()}} method
that returns a new {{RpcClient}} interface (implemented by {{HBaseClient}}) and
move the {{RpcEngine.getProxy()}}/{{stopProxy()}} implementations into the
client. So all proxy invocations can go through the same client, without
requiring the static client cache. I haven't fully thought this through, so I
could be missing other important aspects. But that approach at least seems like
a step in the right direction for fixing the client abstractions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7400) ExportSnapshot mapper closes the FileSystem
[ https://issues.apache.org/jira/browse/HBASE-7400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-7400: -- Affects Version/s: hbase-7290 hbase-6055 Fix Version/s: hbase-7290 > ExportSnapshot mapper closes the FileSystem > --- > > Key: HBASE-7400 > URL: https://issues.apache.org/jira/browse/HBASE-7400 > Project: HBase > Issue Type: Sub-task > Components: Client, master, regionserver, snapshots, Zookeeper >Affects Versions: hbase-6055, hbase-7290 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Blocker > Fix For: hbase-6055, hbase-7290 > > Attachments: HBASE-7400-v0.patch, HBASE-7400-v1.patch, > HBASE-7400-v2.patch > > > The ExportSnapshotMapper closes the FileSystem on cleanup, > and in case of shared instead, since there's no ref count, the job fails. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7459) NPE in HMaster TestlocalHBaseCluster
[
https://issues.apache.org/jira/browse/HBASE-7459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-7459:
--
Affects Version/s: hbase-7290
Fix Version/s: hbase-7290
> NPE in HMaster TestlocalHBaseCluster
>
>
> Key: HBASE-7459
> URL: https://issues.apache.org/jira/browse/HBASE-7459
> Project: HBase
> Issue Type: Bug
> Components: master, test
>Affects Versions: hbase-6055, 0.96.0, hbase-7290
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: hbase-6055, 0.96.0, hbase-7290
>
> Attachments: hbase-7459.patch
>
>
> TestLocalHBaseCluster has failed intermittently on a unit test run with this
> exception stack.
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.master.HMaster.shutdown(HMaster.java:2064)
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.shutdown(JVMClusterUtil.java:238)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.shutdown(LocalHBaseCluster.java:430)
> at
> org.apache.hadoop.hbase.TestLocalHBaseCluster.testLocalHBaseCluster(TestLocalHBaseCluster.java:71)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
>
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7459) NPE in HMaster TestlocalHBaseCluster
[
https://issues.apache.org/jira/browse/HBASE-7459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-7459:
--
Resolution: Fixed
Fix Version/s: 0.96.0
hbase-6055
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
> NPE in HMaster TestlocalHBaseCluster
>
>
> Key: HBASE-7459
> URL: https://issues.apache.org/jira/browse/HBASE-7459
> Project: HBase
> Issue Type: Bug
> Components: master, test
>Affects Versions: hbase-6055, 0.96.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: hbase-6055, 0.96.0
>
> Attachments: hbase-7459.patch
>
>
> TestLocalHBaseCluster has failed intermittently on a unit test run with this
> exception stack.
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.master.HMaster.shutdown(HMaster.java:2064)
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.shutdown(JVMClusterUtil.java:238)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.shutdown(LocalHBaseCluster.java:430)
> at
> org.apache.hadoop.hbase.TestLocalHBaseCluster.testLocalHBaseCluster(TestLocalHBaseCluster.java:71)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
>
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7459) NPE in HMaster TestlocalHBaseCluster
[
https://issues.apache.org/jira/browse/HBASE-7459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540941#comment-13540941
]
Jonathan Hsieh commented on HBASE-7459:
---
committed to trunk, offline and online snapshot branches.
> NPE in HMaster TestlocalHBaseCluster
>
>
> Key: HBASE-7459
> URL: https://issues.apache.org/jira/browse/HBASE-7459
> Project: HBase
> Issue Type: Bug
> Components: master, test
>Affects Versions: hbase-6055, 0.96.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Attachments: hbase-7459.patch
>
>
> TestLocalHBaseCluster has failed intermittently on a unit test run with this
> exception stack.
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.master.HMaster.shutdown(HMaster.java:2064)
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.shutdown(JVMClusterUtil.java:238)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.shutdown(LocalHBaseCluster.java:430)
> at
> org.apache.hadoop.hbase.TestLocalHBaseCluster.testLocalHBaseCluster(TestLocalHBaseCluster.java:71)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
>
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7400) ExportSnapshot mapper closes the FileSystem
[ https://issues.apache.org/jira/browse/HBASE-7400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-7400: -- Resolution: Fixed Fix Version/s: (was: 0.96.0) Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) > ExportSnapshot mapper closes the FileSystem > --- > > Key: HBASE-7400 > URL: https://issues.apache.org/jira/browse/HBASE-7400 > Project: HBase > Issue Type: Sub-task > Components: Client, master, regionserver, snapshots, Zookeeper >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Blocker > Fix For: hbase-6055 > > Attachments: HBASE-7400-v0.patch, HBASE-7400-v1.patch, > HBASE-7400-v2.patch > > > The ExportSnapshotMapper closes the FileSystem on cleanup, > and in case of shared instead, since there's no ref count, the job fails. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7400) ExportSnapshot mapper closes the FileSystem
[ https://issues.apache.org/jira/browse/HBASE-7400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540940#comment-13540940 ] Jonathan Hsieh commented on HBASE-7400: --- thanks ted, matteo. committing to the offline and online snapshots branches. > ExportSnapshot mapper closes the FileSystem > --- > > Key: HBASE-7400 > URL: https://issues.apache.org/jira/browse/HBASE-7400 > Project: HBase > Issue Type: Sub-task > Components: Client, master, regionserver, snapshots, Zookeeper >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Blocker > Fix For: hbase-6055, 0.96.0 > > Attachments: HBASE-7400-v0.patch, HBASE-7400-v1.patch, > HBASE-7400-v2.patch > > > The ExportSnapshotMapper closes the FileSystem on cleanup, > and in case of shared instead, since there's no ref count, the job fails. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7352) clone operation from HBaseAdmin can hang forever.
[
https://issues.apache.org/jira/browse/HBASE-7352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540939#comment-13540939
]
Ted Yu commented on HBASE-7352:
---
Another nit:
{code}
+ * If the table exceeded the retry period, an exception is thrown.
{code}
'table' is not an action. You can say 'enabling table exceeds'
Looks like you should rebase your patch.
> clone operation from HBaseAdmin can hang forever.
> -
>
> Key: HBASE-7352
> URL: https://issues.apache.org/jira/browse/HBASE-7352
> Project: HBase
> Issue Type: Sub-task
> Components: Client, master, regionserver, snapshots, Zookeeper
>Reporter: Jonathan Hsieh
>Assignee: Matteo Bertozzi
> Fix For: hbase-6055, 0.96.0
>
> Attachments: HBASE-7352-v0.patch, HBASE-7352-v1.patch,
> HBASE-7352-v2.patch
>
>
> Sometimes the clone operation from the hbase shell can hang. The table has
> been created (it shows up in the web ui), but does not have any entries in
> META.
> There don't seem to be any clone, snapshot, enable or disable found in the
> master's jstack.
> Here's a trace from the HBaseAdmin:
> {code}
> "main" prio=10 tid=0x7f782800d000 nid=0x25c waiting on condition
> [0x7f782f9bf000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.cloneSnapshot(HBaseAdmin.java:2413)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.cloneSnapshot(HBaseAdmin.java:2393)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:465)
> at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:323)
> at
> org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:69)
> at
> org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:201)
> at org.jruby.ast.CallTwoArgNode.interpret(CallTwoArgNode.java:59)
> at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
> ... (more jruby stack) ...
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7459) NPE in HMaster TestlocalHBaseCluster
[
https://issues.apache.org/jira/browse/HBASE-7459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540938#comment-13540938
]
Jonathan Hsieh commented on HBASE-7459:
---
yup, here's the offending method
{code}
public void shutdown() throws IOException {
spanReceiverHost.closeReceivers();
if (cpHost != null) {
cpHost.preShutdown();
}
if (mxBean != null) {
MBeanUtil.unregisterMBean(mxBean);
mxBean = null;
}
if (this.assignmentManager != null) this.assignmentManager.shutdown();
if (this.serverManager != null) this.serverManager.shutdownCluster();
try {
if (this.clusterStatusTracker != null){
this.clusterStatusTracker.setClusterDown();
}
} catch (KeeperException e) {
LOG.error("ZooKeeper exception trying to set cluster as down in ZK", e);
}
}
{code}
> NPE in HMaster TestlocalHBaseCluster
>
>
> Key: HBASE-7459
> URL: https://issues.apache.org/jira/browse/HBASE-7459
> Project: HBase
> Issue Type: Bug
> Components: master, test
>Affects Versions: hbase-6055, 0.96.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Attachments: hbase-7459.patch
>
>
> TestLocalHBaseCluster has failed intermittently on a unit test run with this
> exception stack.
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.master.HMaster.shutdown(HMaster.java:2064)
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.shutdown(JVMClusterUtil.java:238)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.shutdown(LocalHBaseCluster.java:430)
> at
> org.apache.hadoop.hbase.TestLocalHBaseCluster.testLocalHBaseCluster(TestLocalHBaseCluster.java:71)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
>
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7442) HBase remote CopyTable not working when security enabled
[
https://issues.apache.org/jira/browse/HBASE-7442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540937#comment-13540937
]
Gary Helmling commented on HBASE-7442:
--
Agree on both the need for a short term fix and moving the bigger refactoring
into another JIRA. I'll open up a new issue.
On the short term fix, I agree with Andy that the change to key the ClientCache
by cluster ID isn't the cleanest fix, but is the most expedient approach for
now.
> HBase remote CopyTable not working when security enabled
>
>
> Key: HBASE-7442
> URL: https://issues.apache.org/jira/browse/HBASE-7442
> Project: HBase
> Issue Type: Bug
> Components: IPC/RPC, mapreduce, security
>Affects Versions: 0.92.1
>Reporter: James Kinley
> Fix For: 0.92.3, 0.96.0, 0.94.5
>
> Attachments: attempt_201212271546_0001_m_00_0.log,
> HBASE-7442-0.92.1.patch
>
>
> When security is enabled, HBase CopyTable fails with Kerberos exception:
> {code}
> FATAL org.apache.hadoop.ipc.SecureClient: SASL authentication failed. The
> most likely cause is missing or invalid credentials. Consider 'kinit'.
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> {code}
> This is only when copying to remote HBase cluster (using either MRv1 or
> YARN), local copy works fine.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7459) NPE in HMaster TestlocalHBaseCluster
[
https://issues.apache.org/jira/browse/HBASE-7459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540936#comment-13540936
]
Matteo Bertozzi commented on HBASE-7459:
+1, spanReceiverHost is the only unprotected in the shutdown() method
> NPE in HMaster TestlocalHBaseCluster
>
>
> Key: HBASE-7459
> URL: https://issues.apache.org/jira/browse/HBASE-7459
> Project: HBase
> Issue Type: Bug
> Components: master, test
>Affects Versions: hbase-6055, 0.96.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Attachments: hbase-7459.patch
>
>
> TestLocalHBaseCluster has failed intermittently on a unit test run with this
> exception stack.
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.master.HMaster.shutdown(HMaster.java:2064)
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.shutdown(JVMClusterUtil.java:238)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.shutdown(LocalHBaseCluster.java:430)
> at
> org.apache.hadoop.hbase.TestLocalHBaseCluster.testLocalHBaseCluster(TestLocalHBaseCluster.java:71)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
>
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7459) NPE in HMaster TestlocalHBaseCluster
[
https://issues.apache.org/jira/browse/HBASE-7459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-7459:
--
Attachment: hbase-7459.patch
> NPE in HMaster TestlocalHBaseCluster
>
>
> Key: HBASE-7459
> URL: https://issues.apache.org/jira/browse/HBASE-7459
> Project: HBase
> Issue Type: Bug
> Components: master, test
>Affects Versions: hbase-6055, 0.96.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Attachments: hbase-7459.patch
>
>
> TestLocalHBaseCluster has failed on a unit test run with this exception stack.
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.master.HMaster.shutdown(HMaster.java:2064)
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.shutdown(JVMClusterUtil.java:238)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.shutdown(LocalHBaseCluster.java:430)
> at
> org.apache.hadoop.hbase.TestLocalHBaseCluster.testLocalHBaseCluster(TestLocalHBaseCluster.java:71)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
>
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7459) NPE in HMaster TestlocalHBaseCluster
[
https://issues.apache.org/jira/browse/HBASE-7459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-7459:
--
Description:
TestLocalHBaseCluster has failed intermittently on a unit test run with this
exception stack.
{code}
java.lang.NullPointerException
at org.apache.hadoop.hbase.master.HMaster.shutdown(HMaster.java:2064)
at
org.apache.hadoop.hbase.util.JVMClusterUtil.shutdown(JVMClusterUtil.java:238)
at
org.apache.hadoop.hbase.LocalHBaseCluster.shutdown(LocalHBaseCluster.java:430)
at
org.apache.hadoop.hbase.TestLocalHBaseCluster.testLocalHBaseCluster(TestLocalHBaseCluster.java:71)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
{code}
was:
TestLocalHBaseCluster has failed on a unit test run with this exception stack.
{code}
java.lang.NullPointerException
at org.apache.hadoop.hbase.master.HMaster.shutdown(HMaster.java:2064)
at
org.apache.hadoop.hbase.util.JVMClusterUtil.shutdown(JVMClusterUtil.java:238)
at
org.apache.hadoop.hbase.LocalHBaseCluster.shutdown(LocalHBaseCluster.java:430)
at
org.apache.hadoop.hbase.TestLocalHBaseCluster.testLocalHBaseCluster(TestLocalHBaseCluster.java:71)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
{code}
> NPE in HMaster TestlocalHBaseCluster
>
>
> Key: HBASE-7459
> URL: https://issues.apache.org/jira/browse/HBASE-7459
> Project: HBase
> Issue Type: Bug
> Components: master, test
>Affects Versions: hbase-6055, 0.96.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Attachments: hbase-7459.patch
>
>
> TestLocalHBaseCluster has failed intermittently on a unit test run with this
> exception stack.
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.master.HMaster.shutdown(HMaster.java:2064)
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.shutdown(JVMClusterUtil.java:238)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.shutdown(LocalHBaseCluster.java:430)
> at
> org.apache.hadoop.hbase.TestLocalHBaseCluster.testLocalHBaseCluster(TestLocalHBaseCluster.java:71)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
>
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7459) NPE in HMaster TestlocalHBaseCluster
[
https://issues.apache.org/jira/browse/HBASE-7459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-7459:
--
Status: Patch Available (was: Open)
> NPE in HMaster TestlocalHBaseCluster
>
>
> Key: HBASE-7459
> URL: https://issues.apache.org/jira/browse/HBASE-7459
> Project: HBase
> Issue Type: Bug
> Components: master, test
>Affects Versions: hbase-6055, 0.96.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Attachments: hbase-7459.patch
>
>
> TestLocalHBaseCluster has failed intermittently on a unit test run with this
> exception stack.
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.master.HMaster.shutdown(HMaster.java:2064)
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.shutdown(JVMClusterUtil.java:238)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.shutdown(LocalHBaseCluster.java:430)
> at
> org.apache.hadoop.hbase.TestLocalHBaseCluster.testLocalHBaseCluster(TestLocalHBaseCluster.java:71)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
>
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7459) NPE in HMaster TestlocalHBaseCluster
Jonathan Hsieh created HBASE-7459:
-
Summary: NPE in HMaster TestlocalHBaseCluster
Key: HBASE-7459
URL: https://issues.apache.org/jira/browse/HBASE-7459
Project: HBase
Issue Type: Bug
Components: master, test
Affects Versions: hbase-6055, 0.96.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
TestLocalHBaseCluster has failed on a unit test run with this exception stack.
{code}
java.lang.NullPointerException
at org.apache.hadoop.hbase.master.HMaster.shutdown(HMaster.java:2064)
at
org.apache.hadoop.hbase.util.JVMClusterUtil.shutdown(JVMClusterUtil.java:238)
at
org.apache.hadoop.hbase.LocalHBaseCluster.shutdown(LocalHBaseCluster.java:430)
at
org.apache.hadoop.hbase.TestLocalHBaseCluster.testLocalHBaseCluster(TestLocalHBaseCluster.java:71)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7352) clone operation from HBaseAdmin can hang forever.
[
https://issues.apache.org/jira/browse/HBASE-7352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540926#comment-13540926
]
Hadoop QA commented on HBASE-7352:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12562676/HBASE-7352-v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3757//console
This message is automatically generated.
> clone operation from HBaseAdmin can hang forever.
> -
>
> Key: HBASE-7352
> URL: https://issues.apache.org/jira/browse/HBASE-7352
> Project: HBase
> Issue Type: Sub-task
> Components: Client, master, regionserver, snapshots, Zookeeper
>Reporter: Jonathan Hsieh
>Assignee: Matteo Bertozzi
> Fix For: hbase-6055, 0.96.0
>
> Attachments: HBASE-7352-v0.patch, HBASE-7352-v1.patch,
> HBASE-7352-v2.patch
>
>
> Sometimes the clone operation from the hbase shell can hang. The table has
> been created (it shows up in the web ui), but does not have any entries in
> META.
> There don't seem to be any clone, snapshot, enable or disable found in the
> master's jstack.
> Here's a trace from the HBaseAdmin:
> {code}
> "main" prio=10 tid=0x7f782800d000 nid=0x25c waiting on condition
> [0x7f782f9bf000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.cloneSnapshot(HBaseAdmin.java:2413)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.cloneSnapshot(HBaseAdmin.java:2393)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:465)
> at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:323)
> at
> org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:69)
> at
> org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:201)
> at org.jruby.ast.CallTwoArgNode.interpret(CallTwoArgNode.java:59)
> at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
> ... (more jruby stack) ...
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7352) clone operation from HBaseAdmin can hang forever.
[
https://issues.apache.org/jira/browse/HBASE-7352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matteo Bertozzi updated HBASE-7352:
---
Attachment: HBASE-7352-v2.patch
> clone operation from HBaseAdmin can hang forever.
> -
>
> Key: HBASE-7352
> URL: https://issues.apache.org/jira/browse/HBASE-7352
> Project: HBase
> Issue Type: Sub-task
> Components: Client, master, regionserver, snapshots, Zookeeper
>Reporter: Jonathan Hsieh
>Assignee: Matteo Bertozzi
> Fix For: hbase-6055, 0.96.0
>
> Attachments: HBASE-7352-v0.patch, HBASE-7352-v1.patch,
> HBASE-7352-v2.patch
>
>
> Sometimes the clone operation from the hbase shell can hang. The table has
> been created (it shows up in the web ui), but does not have any entries in
> META.
> There don't seem to be any clone, snapshot, enable or disable found in the
> master's jstack.
> Here's a trace from the HBaseAdmin:
> {code}
> "main" prio=10 tid=0x7f782800d000 nid=0x25c waiting on condition
> [0x7f782f9bf000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.cloneSnapshot(HBaseAdmin.java:2413)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.cloneSnapshot(HBaseAdmin.java:2393)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:465)
> at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:323)
> at
> org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:69)
> at
> org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:201)
> at org.jruby.ast.CallTwoArgNode.interpret(CallTwoArgNode.java:59)
> at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
> ... (more jruby stack) ...
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4955) Use the official versions of surefire & junit
[
https://issues.apache.org/jira/browse/HBASE-4955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540925#comment-13540925
]
nkeywal commented on HBASE-4955:
Surefire 2.13 is nearly out. I tried it, it seems to work for us, with the
restriction mentioned above: we will have to accept some partial output on
screen or to recategorize the tests. It's anyway better to wait for the
official release and see what the other users find: we don't need to be the
first users. But 2.13 is likely the winner.
I tried again with JUnit, and here the news are not so good: we still have
errors:
{noformat}
TestServerCustomProtocol.testSingleProxy:219 null
TestMultiParallel.testFlushCommitsWithAbort:226->doTestFlushCommits:267
Server count=2, abort=true expected:<1> but was:<2>
TestAdmin.testDeleteEditUnknownColumnFamilyAndOrTable:203 null
TestReplication.setUp:180 Waited too much time for truncate
Tests in error:
TestZooKeeper.testZNodeDeletes:291 » NoAuth KeeperErrorCode = NoAuth for /l1
{noformat}
I checked the JUnit mailing lists and bug tracking, there is nothing. So it's
likely us and we will have to investigate I guess.
> Use the official versions of surefire & junit
> -
>
> Key: HBASE-4955
> URL: https://issues.apache.org/jira/browse/HBASE-4955
> Project: HBase
> Issue Type: Improvement
> Components: test
>Affects Versions: 0.94.0
> Environment: all
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Minor
> Fix For: 0.96.0
>
> Attachments: 4955.v1.patch, 4955.v2.patch, 4955.v2.patch
>
>
> We currently use private versions for Surefire & JUnit since HBASE-4763.
> This JIRA traks what we need to move to official versions.
> Surefire 2.11 is just out, but, after some tests, it does not contain all
> what we need.
> JUnit. Could be for JUnit 4.11. Issue to monitor:
> https://github.com/KentBeck/junit/issues/359: fixed in our version, no
> feedback for an integration on trunk
> Surefire: Could be for Surefire 2.12. Issues to monitor are:
> 329 (category support): fixed, we use the official implementation from the
> trunk
> 786 (@Category with forkMode=always): fixed, we use the official
> implementation from the trunk
> 791 (incorrect elapsed time on test failure): fixed, we use the official
> implementation from the trunk
> 793 (incorrect time in the XML report): Not fixed (reopen) on trunk, fixed on
> our version.
> 760 (does not take into account the test method): fixed in trunk, not fixed
> in our version
> 798 (print immediately the test class name): not fixed in trunk, not fixed in
> our version
> 799 (Allow test parallelization when forkMode=always): not fixed in trunk,
> not fixed in our version
> 800 (redirectTestOutputToFile not taken into account): not yet fix on trunk,
> fixed on our version
> 800 & 793 are the more important to monitor, it's the only ones that are
> fixed in our version but not on trunk.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7458) TestReplicationWithCompression fails intermittently in both PreCommit and trunk builds
[ https://issues.apache.org/jira/browse/HBASE-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-7458: -- Description: TestReplicationWithCompression has been failing often. Here are few examples: https://builds.apache.org/job/PreCommit-HBASE-Build/3755/testReport/ https://builds.apache.org/job/HBase-TRUNK/3672/testReport/org.apache.hadoop.hbase.replication/TestReplicationWithCompression/testDeleteTypes/ https://builds.apache.org/job/HBase-0.94/677/testReport/junit/org.apache.hadoop.hbase.replication/TestReplicationWithCompression/queueFailover/ was: TestReplicationWithCompression has been failing often. Here are few examples: https://builds.apache.org/job/PreCommit-HBASE-Build/3755/testReport/ https://builds.apache.org/job/HBase-TRUNK/3672/testReport/org.apache.hadoop.hbase.replication/TestReplicationWithCompression/testDeleteTypes/ > TestReplicationWithCompression fails intermittently in both PreCommit and > trunk builds > -- > > Key: HBASE-7458 > URL: https://issues.apache.org/jira/browse/HBASE-7458 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Priority: Critical > Fix For: 0.96.0 > > > TestReplicationWithCompression has been failing often. > Here are few examples: > https://builds.apache.org/job/PreCommit-HBASE-Build/3755/testReport/ > https://builds.apache.org/job/HBase-TRUNK/3672/testReport/org.apache.hadoop.hbase.replication/TestReplicationWithCompression/testDeleteTypes/ > https://builds.apache.org/job/HBase-0.94/677/testReport/junit/org.apache.hadoop.hbase.replication/TestReplicationWithCompression/queueFailover/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7458) TestReplicationWithCompression fails in both PreCommit and trunk builds
Ted Yu created HBASE-7458: - Summary: TestReplicationWithCompression fails in both PreCommit and trunk builds Key: HBASE-7458 URL: https://issues.apache.org/jira/browse/HBASE-7458 Project: HBase Issue Type: Bug Reporter: Ted Yu Priority: Critical Fix For: 0.96.0 TestReplicationWithCompression has been failing often. Here are few examples: https://builds.apache.org/job/PreCommit-HBASE-Build/3755/testReport/ https://builds.apache.org/job/HBase-TRUNK/3672/testReport/org.apache.hadoop.hbase.replication/TestReplicationWithCompression/testDeleteTypes/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7458) TestReplicationWithCompression fails intermittently in both PreCommit and trunk builds
[ https://issues.apache.org/jira/browse/HBASE-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-7458: -- Summary: TestReplicationWithCompression fails intermittently in both PreCommit and trunk builds (was: TestReplicationWithCompression fails in both PreCommit and trunk builds) > TestReplicationWithCompression fails intermittently in both PreCommit and > trunk builds > -- > > Key: HBASE-7458 > URL: https://issues.apache.org/jira/browse/HBASE-7458 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Priority: Critical > Fix For: 0.96.0 > > > TestReplicationWithCompression has been failing often. > Here are few examples: > https://builds.apache.org/job/PreCommit-HBASE-Build/3755/testReport/ > https://builds.apache.org/job/HBase-TRUNK/3672/testReport/org.apache.hadoop.hbase.replication/TestReplicationWithCompression/testDeleteTypes/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540912#comment-13540912 ] Ted Yu commented on HBASE-7403: --- There are lines longer than 100 characters in patch v5. Please shorten them. Thanks > Online Merge > > > Key: HBASE-7403 > URL: https://issues.apache.org/jira/browse/HBASE-7403 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.94.3 >Reporter: chunhui shen >Assignee: chunhui shen > Fix For: 0.96.0, 0.94.5 > > Attachments: 7403-trunkv5.patch, 7403v5.diff, 7403-v5.txt, > 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv1.patch, > hbase-7403-trunkv5.patch, merge region.pdf > > > The feature of this online merge: > 1.Online,no necessary to disable table > 2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90 > 3.Easy to call merege request, no need to input a long region name, only > encoded name enough > 4.No limit when operation, you don't need to care event like Server Dead, > Balance, Split, Disabing/Enabing table, no need to care whether you send a > wrong merge request, it alread have done for you > 5.Only little offline time for two merging regions > We need merge in the following cases： > 1.Region hole or region overlap, can’t be fix by hbck > 2.Region become empty because of TTL and not reasonable Rowkey design > 3.Region is always empty or very small because of presplit when create table > 4.Too many empty or small regions would reduce the system performance(e.g. > mslab) > Current merge tools only support offline and are not able to redo if > exception is thrown in the process of merging, causing a dirty data > For online system, we need a online merge. > This implement logic of this patch for Online Merge is : > For example, merge regionA and regionB into regionC > 1.Offline the two regions A and B > 2.Merge the two regions in the HDFS(Create regionC’s directory, move > regionA’s and regionB’s file to regionC’s directory, delete regionA’s and > regionB’s directory) > 3.Add the merged regionC to .META. > 4.Assign the merged regionC > As design of this patch , once we do the merge work in the HDFS,we could redo > it until successful if it throws exception or abort or server restart, but > couldn’t be rolled back. > It depends on > Use zookeeper to record the transaction journal state, make redo easier > Use zookeeper to send/receive merge request > Merge transaction is executed on the master > Support calling merge request through API or shell tool > About the merge process, please see the attachment and patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-4210) Allow coprocessor to interact with batches per region sent from a client(?)
[
https://issues.apache.org/jira/browse/HBASE-4210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John reassigned HBASE-4210:
-
Assignee: Anoop Sam John
> Allow coprocessor to interact with batches per region sent from a client(?)
> ---
>
> Key: HBASE-4210
> URL: https://issues.apache.org/jira/browse/HBASE-4210
> Project: HBase
> Issue Type: New Feature
>Reporter: Lars Hofhansl
>Assignee: Anoop Sam John
>Priority: Minor
>
> Currently the coprocessor write hooks - {pre|post}{Put|Delete} - are strictly
> one row|cell operations.
> It might be a good idea to allow a coprocessor to deal with batches of puts
> and deletes as they arrive from the client.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540908#comment-13540908
]
Hadoop QA commented on HBASE-7403:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12562667/7403v5.diff
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 8
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestMultiParallel
org.apache.hadoop.hbase.replication.TestReplicationWithCompression
{color:red}-1 core zombie tests{color}. There are 2 zombie test(s):
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3755//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3755//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3755//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3755//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3755//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3755//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3755//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3755//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3755//console
This message is automatically generated.
> Online Merge
>
>
> Key: HBASE-7403
> URL: https://issues.apache.org/jira/browse/HBASE-7403
> Project: HBase
> Issue Type: New Feature
>Affects Versions: 0.94.3
>Reporter: chunhui shen
>Assignee: chunhui shen
> Fix For: 0.96.0, 0.94.5
>
> Attachments: 7403-trunkv5.patch, 7403v5.diff, 7403-v5.txt,
> 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv1.patch,
> hbase-7403-trunkv5.patch, merge region.pdf
>
>
> The feature of this online merge:
> 1.Online,no necessary to disable table
> 2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90
> 3.Easy to call merege request, no need to input a long region name, only
> encoded name enough
> 4.No limit when operation, you don't need to care event like Server Dead,
> Balance, Split, Disabing/Enabing table, no need to care whether you send a
> wrong merge request, it alread have done for you
> 5.Only little offline time for two merging regions
> We need merge in the following cases：
> 1.Region hole or region overlap, can’t be fix by hbck
> 2.Region become empty because of TTL and not reasonable Rowkey design
> 3.Region is always empty or very small because of presplit when create table
> 4.Too many empty or small regions would reduce the system performance(e.g.
> mslab)
> Current merge tools only support offline and are not able to redo if
> exception is thrown in the process of merging, causing a dirty data
> For online system, we need a online merge.
> This implement logic of this patch for Online Merge is :
> For example, merge regionA and regionB into regionC
> 1.Offline the two regions A and B
> 2.Merge the two regions in the HDFS(Create regionC’s directory, move
> regionA’s and regionB’s file to regionC’s directory, delete regionA’s and
> regionB’s directory)
> 3.Add the merged regionC to .META.
> 4.Assign the merged regionC
> As design of this patch , once we do the merge work in the HDFS,we could redo
> it until successful if it throws exception or abort or server restart, but
> couldn’t be rolled back.
> It depends on
> Us
[jira] [Commented] (HBASE-7400) ExportSnapshot mapper closes the FileSystem
[
https://issues.apache.org/jira/browse/HBASE-7400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540894#comment-13540894
]
Hadoop QA commented on HBASE-7400:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12562670/HBASE-7400-v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3756//console
This message is automatically generated.
> ExportSnapshot mapper closes the FileSystem
> ---
>
> Key: HBASE-7400
> URL: https://issues.apache.org/jira/browse/HBASE-7400
> Project: HBase
> Issue Type: Sub-task
> Components: Client, master, regionserver, snapshots, Zookeeper
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
>Priority: Blocker
> Fix For: hbase-6055, 0.96.0
>
> Attachments: HBASE-7400-v0.patch, HBASE-7400-v1.patch,
> HBASE-7400-v2.patch
>
>
> The ExportSnapshotMapper closes the FileSystem on cleanup,
> and in case of shared instead, since there's no ref count, the job fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7400) ExportSnapshot mapper closes the FileSystem
[ https://issues.apache.org/jira/browse/HBASE-7400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-7400: --- Attachment: HBASE-7400-v2.patch added v2 that applies cleanly after HBASE-7207 (massive renaming) > ExportSnapshot mapper closes the FileSystem > --- > > Key: HBASE-7400 > URL: https://issues.apache.org/jira/browse/HBASE-7400 > Project: HBase > Issue Type: Sub-task > Components: Client, master, regionserver, snapshots, Zookeeper >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Blocker > Fix For: hbase-6055, 0.96.0 > > Attachments: HBASE-7400-v0.patch, HBASE-7400-v1.patch, > HBASE-7400-v2.patch > > > The ExportSnapshotMapper closes the FileSystem on cleanup, > and in case of shared instead, since there's no ref count, the job fails. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7455) Increase timeouts in TestReplication and TestSplitLogWorker
[ https://issues.apache.org/jira/browse/HBASE-7455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540878#comment-13540878 ] Hudson commented on HBASE-7455: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #318 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/318/]) HBASE-7455 Increase timeouts in TestReplication and TestSplitLogWorker (Revision 1426710) Result = FAILURE larsh : Files : * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplication.java > Increase timeouts in TestReplication and TestSplitLogWorker > --- > > Key: HBASE-7455 > URL: https://issues.apache.org/jira/browse/HBASE-7455 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl > Fix For: 0.96.0, 0.94.4 > > Attachments: 7455-0.94.txt, 7455-0.96.txt > > > When I measure the times in TestReplication.queueFailover, it takes about 15s > on my (reasonably fast) Laptop. > The timeout in queueFailover currently is 1500*2*15 = 45000ms. > For setup before each test (which truncates the table and waits for the > changes to replicate) it is 1500*15 = 22500ms. > Interestingly I see queueFailover failures where the wait time is measured as > 64260ms and some at 72316ms. > Since these numbers are not even close to 45000ms, the machine or JVM must > have been stuck for 15 or almost 30s (otherwise we'd get a timeout and the > total time spent should be close to the timeout). > So I would suggest that we increase the timeouts further. > We could set SLEEP_TIME to 2000 and retries to 20. Would lead to 2000*2*20 = > 8ms. > Any objections? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7412) Fix how HTableDescriptor handles default max file size and flush size
[ https://issues.apache.org/jira/browse/HBASE-7412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540879#comment-13540879 ] Hudson commented on HBASE-7412: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #318 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/318/]) HBASE-7412 Fix how HTableDescriptor handles default max file size and flush size - ADDENDUM (Revision 1426683) Result = FAILURE jxiang : Files : * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/backup/TestHFileArchiving.java > Fix how HTableDescriptor handles default max file size and flush size > - > > Key: HBASE-7412 > URL: https://issues.apache.org/jira/browse/HBASE-7412 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Fix For: 0.96.0, 0.94.4 > > Attachments: 0.94-7412-addendum.patch, 0.94-7412.patch, > 0.94-7412_v2.patch, trunk-7412.patch, trunk-7412_v2.patch > > > If the region flush size is not set in the table, > IncreasingToUpperBoundRegionSplitPolicy will most likely always use the > default value: 128MB, even if the flush size is set to a different value in > hbase-site.xml. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7224) Remove references to Writable in the ipc package
[ https://issues.apache.org/jira/browse/HBASE-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540876#comment-13540876 ] Hudson commented on HBASE-7224: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #318 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/318/]) HBASE-7224 Remove references to Writable in the ipc package (Revision 1426729) Result = FAILURE stack : Files : * /hbase/trunk/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClientProtos.java * /hbase/trunk/hbase-protocol/src/main/protobuf/Client.proto * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/CodeToClassAndBack.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/Reference.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/TimeRange.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/BlockingRpcCallback.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/CallerDisconnectedException.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/ClientCache.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/Delayable.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/MasterCoprocessorRpcChannel.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/ProtobufRpcClientEngine.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/ProtobufRpcServerEngine.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/UnknownProtocolException.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/TimeRangeTracker.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/CodeToClassAndBackFor96Migration.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/HbaseObjectWritableFor96Migration.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/ZKPermissionWatcher.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TestSerialization.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestHbaseObjectWritable.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestTablePermissions.java > Remove references to Writable in the ipc package > > > Key: HBASE-7224 > URL: https://issues.apache.org/jira/browse/HBASE-7224 > Project: HBase > Issue Type: Sub-task > Components: IPC/RPC, Protobufs >Reporter: Devaraj Das >Assignee: stack >Priority: Blocker > Fix For: 0.96.0 > > Attachments: 7224.txt, 7224v2.txt, 7224v3.txt, 7224v4.txt, > 7224v4.txt, 7224v4.txt, 7224v5.txt, purge_more_writables.txt > > > I see references to Writable in the ipc package, most notably in the > Invocation class. This class is not being used that much in the core ipc > package but used in the coprocessor protocol implementations (there are some > coprocessor protocols that are Writable based still). This jira is to track > removing those references and the Invocation class (once HBASE-6895 is > resolved). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7457) Fix javadoc warnings in hadoopqa tool, it complains about unsafe accesses
[
https://issues.apache.org/jira/browse/HBASE-7457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540877#comment-13540877
]
Hudson commented on HBASE-7457:
---
Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #318 (See
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/318/])
HBASE-7457 Fix javadoc warnings in hadoopqa tool, it complains about unsafe
accesses (Revision 1426728)
Result = FAILURE
stack :
Files :
* /hbase/trunk/dev-support/test-patch.properties
> Fix javadoc warnings in hadoopqa tool, it complains about unsafe accesses
> -
>
> Key: HBASE-7457
> URL: https://issues.apache.org/jira/browse/HBASE-7457
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: stack
> Attachments: 7457.txt
>
>
> I see this in hadoopqa output and it seems to be causing the two warnings we
> currently see in hadoopqa reports:
> {code}
> 2 warnings
> [WARNING] Javadoc Warnings
> [WARNING]
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/Bytes.java:43:
> warning: sun.misc.Unsafe is Sun proprietary API and may be removed in a
> future release
> [WARNING] import sun.misc.Unsafe;
> [WARNING] ^
> [WARNING]
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/Bytes.java:1032:
> warning: sun.misc.Unsafe is Sun proprietary API and may be removed in a
> future release
> [WARNING] static final Unsafe theUnsafe;
> [WARNING] ^
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7440) ReplicationZookeeper#addPeer is racy
[
https://issues.apache.org/jira/browse/HBASE-7440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540875#comment-13540875
]
Hudson commented on HBASE-7440:
---
Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #318 (See
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/318/])
HBASE-7440 ReplicationZookeeper#addPeer is racy (Himanshu) (Revision
1426702)
Result = FAILURE
larsh :
Files :
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java
> ReplicationZookeeper#addPeer is racy
>
>
> Key: HBASE-7440
> URL: https://issues.apache.org/jira/browse/HBASE-7440
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 0.94.3
>Reporter: Himanshu Vashishtha
>Assignee: Himanshu Vashishtha
> Fix For: 0.96.0, 0.94.4
>
> Attachments: HBASE-7440-trunk-v0.patch, HBASE-7440-trunk-v1.patch,
> HBASE-7440-v0.patch, HBASE-7440-v1.patch, HBASE-7440-v2.patch
>
>
> While adding a peer, ReplicationZK does the znodes creation in three
> transactions. Create :
> a) peers znode
> b) peerId specific znode, and
> c) peerState znode
> There is a PeerWatcher which invokes getPeer() (after steps b) and c)). If it
> happens that while adding a peer, the control flows to getPeer() and step c)
> has not been processed, it may results in a state where the peer will not be
> added. This happens while running
> TestMasterReplication#testCyclicReplication().
> {code}
> 2012-12-26 07:36:35,187 INFO
> [RegionServer:0;p0120.X,38423,1356536179470-EventThread]
> zookeeper.RecoverableZooKeeper(447): Node /2/replication/peers/1/peer-state
> already exists and this is not a retry
> 2012-12-26 07:36:35,188 ERROR
> [RegionServer:0;p0120.X,38423,1356536179470-EventThread]
> regionserver.ReplicationSourceManager$PeersWatcher(527): Error while adding a
> new peer
> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
> NodeExists for /2/replication/peers/1/peer-state
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:428)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:410)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndWatch(ZKUtil.java:1044)
> at
> org.apache.hadoop.hbase.replication.ReplicationPeer.startStateTracker(ReplicationPeer.java:82)
> at
> org.apache.hadoop.hbase.replication.ReplicationZookeeper.getPeer(ReplicationZookeeper.java:344)
> at
> org.apache.hadoop.hbase.replication.ReplicationZookeeper.connectToPeer(ReplicationZookeeper.java:307)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager$PeersWatcher.nodeChildrenChanged(ReplicationSourceManager.java:519)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:315)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
> 2012-12-26 07:36:35,188 DEBUG
> [RegionServer:0;p0120.X,55742,1356536171947-EventThread]
> zookeeper.ZKUtil(1545): regionserver:55742-0x13bd7db39580004 Retrieved 36
> byte(s) of data from znode /1/hbaseid; data=9ce66123-d3e8-4ae9-a249-afe03...
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7258) Hbase needs to create baseZNode recursively
[ https://issues.apache.org/jira/browse/HBASE-7258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540874#comment-13540874 ] Hudson commented on HBASE-7258: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #318 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/318/]) HBASE-7258 Hbase needs to create baseZNode recursively (Liu Shaohui via Ram) (Revision 1426707) Result = FAILURE ramkrishna : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java > Hbase needs to create baseZNode recursively > --- > > Key: HBASE-7258 > URL: https://issues.apache.org/jira/browse/HBASE-7258 > Project: HBase > Issue Type: Bug > Components: Zookeeper >Affects Versions: 0.94.2 >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Minor > Labels: zookeeper, zookeeper.znode.parent > Fix For: 0.96.0 > > Attachments: HBASE-7258-0.94.patch, HBASE-7258.diff, HBASE-7258.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > In deploy env, multi small hbase clusters may share a same zk cluster. So, > for hbase cluster1, its parent znode is /hbase/cluster1. But in hbase version > 0.94.1, hbase use ZKUtil.createAndFailSilent(this, baseZNode) to create > parent path and it will throw a NoNode exception if znode /hbase donot exist. > We want to change it to ZKUtil.createWithParents(this, baseZNode); to suport > create baseZNode recursivly. > The NoNode exception is: > java.lang.RuntimeException: Failed construction of Master: class > org.apache.hadoop.hbase.master.HMaster > at > org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1792) > at > org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:146) > at > org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:103) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at > org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:77) > at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1806) > Caused by: org.apache.zookeeper.KeeperException$NoNodeException: > KeeperErrorCode = NoNode for /hbase/cluster1 > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:111) > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:778) > at > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:420) > at > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:402) > at > org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:905) > at > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.createBaseZNodes(ZooKeeperWatcher.java:166) > at > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.(ZooKeeperWatcher.java:159) > at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:282) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at org.apache.hadoop.hbase.master.H -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4210) Allow coprocessor to interact with batches per region sent from a client(?)
[
https://issues.apache.org/jira/browse/HBASE-4210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540873#comment-13540873
]
Anoop Sam John commented on HBASE-4210:
---
Oh I missed this issue till now...
We have implemented some thing like this while working with the secondary index.
I will give more details and a patch that we had...
We will discuss further based on that then :)
More useful CP hooks we can give...
> Allow coprocessor to interact with batches per region sent from a client(?)
> ---
>
> Key: HBASE-4210
> URL: https://issues.apache.org/jira/browse/HBASE-4210
> Project: HBase
> Issue Type: New Feature
>Reporter: Lars Hofhansl
>Priority: Minor
>
> Currently the coprocessor write hooks - {pre|post}{Put|Delete} - are strictly
> one row|cell operations.
> It might be a good idea to allow a coprocessor to deal with batches of puts
> and deletes as they arrive from the client.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chunhui shen updated HBASE-7403: Attachment: 7403v5.diff > Online Merge > > > Key: HBASE-7403 > URL: https://issues.apache.org/jira/browse/HBASE-7403 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.94.3 >Reporter: chunhui shen >Assignee: chunhui shen > Fix For: 0.96.0, 0.94.5 > > Attachments: 7403-trunkv5.patch, 7403v5.diff, 7403-v5.txt, > 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv1.patch, > hbase-7403-trunkv5.patch, merge region.pdf > > > The feature of this online merge: > 1.Online,no necessary to disable table > 2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90 > 3.Easy to call merege request, no need to input a long region name, only > encoded name enough > 4.No limit when operation, you don't need to care event like Server Dead, > Balance, Split, Disabing/Enabing table, no need to care whether you send a > wrong merge request, it alread have done for you > 5.Only little offline time for two merging regions > We need merge in the following cases： > 1.Region hole or region overlap, can’t be fix by hbck > 2.Region become empty because of TTL and not reasonable Rowkey design > 3.Region is always empty or very small because of presplit when create table > 4.Too many empty or small regions would reduce the system performance(e.g. > mslab) > Current merge tools only support offline and are not able to redo if > exception is thrown in the process of merging, causing a dirty data > For online system, we need a online merge. > This implement logic of this patch for Online Merge is : > For example, merge regionA and regionB into regionC > 1.Offline the two regions A and B > 2.Merge the two regions in the HDFS(Create regionC’s directory, move > regionA’s and regionB’s file to regionC’s directory, delete regionA’s and > regionB’s directory) > 3.Add the merged regionC to .META. > 4.Assign the merged regionC > As design of this patch , once we do the merge work in the HDFS,we could redo > it until successful if it throws exception or abort or server restart, but > couldn’t be rolled back. > It depends on > Use zookeeper to record the transaction journal state, make redo easier > Use zookeeper to send/receive merge request > Merge transaction is executed on the master > Support calling merge request through API or shell tool > About the merge process, please see the attachment and patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6666) Allow RegionObserver to access other regions on the same server.
[ https://issues.apache.org/jira/browse/HBASE-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540864#comment-13540864 ] Anoop Sam John commented on HBASE-: --- Within a RegionObserver we can access any other region in that RS via the HRegionServer#getOnlineRegion ?? e.getEnvironment().getRegionServerServices().getOnlineRegions(tableName) Or you mean some thing else here? > Allow RegionObserver to access other regions on the same server. > > > Key: HBASE- > URL: https://issues.apache.org/jira/browse/HBASE- > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl > > This is an idea I had some time back. It would be nice if a RegionObserver > (or Endpoint) could get access to all other regions on the same RegionServer > to efficiently make updates to those regions as well (instead of going > through the standard HTable path). > Together with a smart region placement strategy this can lead to much better > performance for some coprocessor tasks. > Maybe it could be abstracted in a special HTable implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4435) Add Group By functionality using Coprocessors
[
https://issues.apache.org/jira/browse/HBASE-4435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540856#comment-13540856
]
Anoop Sam John commented on HBASE-4435:
---
May be good to add to the CP examples section. Someone working with this?
> Add Group By functionality using Coprocessors
> -
>
> Key: HBASE-4435
> URL: https://issues.apache.org/jira/browse/HBASE-4435
> Project: HBase
> Issue Type: Improvement
> Components: Coprocessors
>Reporter: Nichole Treadway
>Priority: Minor
> Labels: by, coprocessors, group, hbase
> Attachments: HBase-4435.patch, HBASE-4435-v2.patch
>
>
> Adds in a Group By -like functionality to HBase, using the Coprocessor
> framework.
> It provides the ability to group the result set on one or more columns
> (groupBy families). It computes statistics (max, min, sum, count, sum of
> squares, number missing) for a second column, called the stats column.
> To use, I've provided two implementations.
> 1. In the first, you specify a single group-by column and a stats field:
> statsMap = gbc.getStats(tableName, scan, groupByFamily,
> groupByQualifier, statsFamily, statsQualifier, statsFieldColumnInterpreter);
> The result is a map with the Group By column value (as a String) to a
> GroupByStatsValues object. The GroupByStatsValues object has max,min,sum etc.
> of the stats column for that group.
> 2. The second implementation allows you to specify a list of group-by columns
> and a stats field. The List of group-by columns is expected to contain lists
> of {column family, qualifier} pairs.
> statsMap = gbc.getStats(tableName, scan, listOfGroupByColumns,
> statsFamily, statsQualifier, statsFieldColumnInterpreter);
> The GroupByStatsValues code is adapted from the Solr Stats component.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7457) Fix javadoc warnings in hadoopqa tool, it complains about unsafe accesses
[
https://issues.apache.org/jira/browse/HBASE-7457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540852#comment-13540852
]
Hudson commented on HBASE-7457:
---
Integrated in HBase-TRUNK #3673 (See
[https://builds.apache.org/job/HBase-TRUNK/3673/])
HBASE-7457 Fix javadoc warnings in hadoopqa tool, it complains about unsafe
accesses (Revision 1426728)
Result = FAILURE
stack :
Files :
* /hbase/trunk/dev-support/test-patch.properties
> Fix javadoc warnings in hadoopqa tool, it complains about unsafe accesses
> -
>
> Key: HBASE-7457
> URL: https://issues.apache.org/jira/browse/HBASE-7457
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: stack
> Attachments: 7457.txt
>
>
> I see this in hadoopqa output and it seems to be causing the two warnings we
> currently see in hadoopqa reports:
> {code}
> 2 warnings
> [WARNING] Javadoc Warnings
> [WARNING]
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/Bytes.java:43:
> warning: sun.misc.Unsafe is Sun proprietary API and may be removed in a
> future release
> [WARNING] import sun.misc.Unsafe;
> [WARNING] ^
> [WARNING]
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/Bytes.java:1032:
> warning: sun.misc.Unsafe is Sun proprietary API and may be removed in a
> future release
> [WARNING] static final Unsafe theUnsafe;
> [WARNING] ^
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7224) Remove references to Writable in the ipc package
[ https://issues.apache.org/jira/browse/HBASE-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540851#comment-13540851 ] Hudson commented on HBASE-7224: --- Integrated in HBase-TRUNK #3673 (See [https://builds.apache.org/job/HBase-TRUNK/3673/]) HBASE-7224 Remove references to Writable in the ipc package (Revision 1426729) Result = FAILURE stack : Files : * /hbase/trunk/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClientProtos.java * /hbase/trunk/hbase-protocol/src/main/protobuf/Client.proto * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/CodeToClassAndBack.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/Reference.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/TimeRange.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/BlockingRpcCallback.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/CallerDisconnectedException.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/ClientCache.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/Delayable.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/MasterCoprocessorRpcChannel.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/ProtobufRpcClientEngine.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/ProtobufRpcServerEngine.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/UnknownProtocolException.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/TimeRangeTracker.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/CodeToClassAndBackFor96Migration.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/HbaseObjectWritableFor96Migration.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/ZKPermissionWatcher.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TestSerialization.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestHbaseObjectWritable.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestTablePermissions.java > Remove references to Writable in the ipc package > > > Key: HBASE-7224 > URL: https://issues.apache.org/jira/browse/HBASE-7224 > Project: HBase > Issue Type: Sub-task > Components: IPC/RPC, Protobufs >Reporter: Devaraj Das >Assignee: stack >Priority: Blocker > Fix For: 0.96.0 > > Attachments: 7224.txt, 7224v2.txt, 7224v3.txt, 7224v4.txt, > 7224v4.txt, 7224v4.txt, 7224v5.txt, purge_more_writables.txt > > > I see references to Writable in the ipc package, most notably in the > Invocation class. This class is not being used that much in the core ipc > package but used in the coprocessor protocol implementations (there are some > coprocessor protocols that are Writable based still). This jira is to track > removing those references and the Invocation class (once HBASE-6895 is > resolved). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7294) Check for snapshot file cleaners on start
[
https://issues.apache.org/jira/browse/HBASE-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540849#comment-13540849
]
Hadoop QA commented on HBASE-7294:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12562664/HBASE-7294-v7.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 15 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3754//console
This message is automatically generated.
> Check for snapshot file cleaners on start
> -
>
> Key: HBASE-7294
> URL: https://issues.apache.org/jira/browse/HBASE-7294
> Project: HBase
> Issue Type: Sub-task
> Components: Client, master, regionserver, snapshots, Zookeeper
>Affects Versions: hbase-6055
>Reporter: Jesse Yates
>Assignee: Matteo Bertozzi
> Fix For: hbase-6055, 0.96.0
>
> Attachments: HBASE-7294-v1.patch, HBASE-7294-v2.patch,
> HBASE-7294-v3.patch, HBASE-7294-v4.patch, HBASE-7294-v5.patch,
> HBASE-7294-v6.patch, HBASE-7294-v7.patch
>
>
> Snapshots currently use the SnaphotHfileCleaner and SnapshotHLogCleaner to
> ensure that any hfiles or hlogs (respectively) that are currently part of a
> snapshot are not removed from their respective archive directories (.archive
> and .oldlogs).
> From Matteo Bertozzi:
> {quote}
> currently the snapshot cleaner is not in hbase-default.xml
> and there's no warning/exception on snapshot/restore operation, if not
> enabled.
> even if we add the cleaner to the hbase-default.xml how do we ensure that the
> user doesn't remove it?
> Do we want to hardcode the cleaner at master startup?
> Do we want to add a check in snapshot/restore that throws an exception if the
> cleaner is not enabled?
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7294) Check for snapshot file cleaners on start
[
https://issues.apache.org/jira/browse/HBASE-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matteo Bertozzi updated HBASE-7294:
---
Attachment: HBASE-7294-v7.patch
TestInfoServers got an exception generated from the snapshots not enabled.
TestSnapshotFromMaster had two problems:
The TTL on by default after the new snapshot.enabled property introduced (so
files are not removed immediately after the first cleaner call)
and enableTable() trigger a compaction (async) and sometimes (on jenkins) is
slow and the files are still there, and one of the assert fail.
> Check for snapshot file cleaners on start
> -
>
> Key: HBASE-7294
> URL: https://issues.apache.org/jira/browse/HBASE-7294
> Project: HBase
> Issue Type: Sub-task
> Components: Client, master, regionserver, snapshots, Zookeeper
>Affects Versions: hbase-6055
>Reporter: Jesse Yates
>Assignee: Matteo Bertozzi
> Fix For: hbase-6055, 0.96.0
>
> Attachments: HBASE-7294-v1.patch, HBASE-7294-v2.patch,
> HBASE-7294-v3.patch, HBASE-7294-v4.patch, HBASE-7294-v5.patch,
> HBASE-7294-v6.patch, HBASE-7294-v7.patch
>
>
> Snapshots currently use the SnaphotHfileCleaner and SnapshotHLogCleaner to
> ensure that any hfiles or hlogs (respectively) that are currently part of a
> snapshot are not removed from their respective archive directories (.archive
> and .oldlogs).
> From Matteo Bertozzi:
> {quote}
> currently the snapshot cleaner is not in hbase-default.xml
> and there's no warning/exception on snapshot/restore operation, if not
> enabled.
> even if we add the cleaner to the hbase-default.xml how do we ensure that the
> user doesn't remove it?
> Do we want to hardcode the cleaner at master startup?
> Do we want to add a check in snapshot/restore that throws an exception if the
> cleaner is not enabled?
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7224) Remove references to Writable in the ipc package
[
https://issues.apache.org/jira/browse/HBASE-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540837#comment-13540837
]
Hadoop QA commented on HBASE-7224:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12562661/7224v5.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 17 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3753//console
This message is automatically generated.
> Remove references to Writable in the ipc package
>
>
> Key: HBASE-7224
> URL: https://issues.apache.org/jira/browse/HBASE-7224
> Project: HBase
> Issue Type: Sub-task
> Components: IPC/RPC, Protobufs
>Reporter: Devaraj Das
>Assignee: stack
>Priority: Blocker
> Fix For: 0.96.0
>
> Attachments: 7224.txt, 7224v2.txt, 7224v3.txt, 7224v4.txt,
> 7224v4.txt, 7224v4.txt, 7224v5.txt, purge_more_writables.txt
>
>
> I see references to Writable in the ipc package, most notably in the
> Invocation class. This class is not being used that much in the core ipc
> package but used in the coprocessor protocol implementations (there are some
> coprocessor protocols that are Writable based still). This jira is to track
> removing those references and the Invocation class (once HBASE-6895 is
> resolved).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7457) Fix javadoc warnings in hadoopqa tool, it complains about unsafe accesses
[
https://issues.apache.org/jira/browse/HBASE-7457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540838#comment-13540838
]
Hadoop QA commented on HBASE-7457:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12562660/7457.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 20 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3752//console
This message is automatically generated.
> Fix javadoc warnings in hadoopqa tool, it complains about unsafe accesses
> -
>
> Key: HBASE-7457
> URL: https://issues.apache.org/jira/browse/HBASE-7457
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: stack
> Attachments: 7457.txt
>
>
> I see this in hadoopqa output and it seems to be causing the two warnings we
> currently see in hadoopqa reports:
> {code}
> 2 warnings
> [WARNING] Javadoc Warnings
> [WARNING]
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/Bytes.java:43:
> warning: sun.misc.Unsafe is Sun proprietary API and may be removed in a
> future release
> [WARNING] import sun.misc.Unsafe;
> [WARNING] ^
> [WARNING]
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/Bytes.java:1032:
> warning: sun.misc.Unsafe is Sun proprietary API and may be removed in a
> future release
> [WARNING] static final Unsafe theUnsafe;
> [WARNING] ^
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7441) Make ClusterManager in IntegrationTestingUtility pluggable
[
https://issues.apache.org/jira/browse/HBASE-7441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540836#comment-13540836
]
Hadoop QA commented on HBASE-7441:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12562656/HBASE-7441-trunk-v1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 2
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.replication.TestReplication
{color:red}-1 core zombie tests{color}. There are 3 zombie test(s):
at
org.apache.hadoop.hbase.io.encoding.TestUpgradeFromHFileV1ToEncoding.testUpgrade(TestUpgradeFromHFileV1ToEncoding.java:83)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3750//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3750//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3750//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3750//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3750//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3750//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3750//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3750//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3750//console
This message is automatically generated.
> Make ClusterManager in IntegrationTestingUtility pluggable
> --
>
> Key: HBASE-7441
> URL: https://issues.apache.org/jira/browse/HBASE-7441
> Project: HBase
> Issue Type: Improvement
> Components: test
>Affects Versions: 0.94.3
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Minor
> Labels: newbie, patch
> Fix For: 0.96.0, 0.94.5
>
> Attachments: HBASE-7441-0.94-v1.patch, HBASE-7441-trunk-v1.patch
>
> Original Estimate: 3h
> Remaining Estimate: 3h
>
> After the patch HBASE-7009, we can use ChaosMonkey to test the Hbase cluster.
> The ClusterManager use ssh to stop/start the rs or master without passwd. To
> support other cluster manager tool, we need to make clusterManager in
> IntegrationTestingUtility pluggable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7224) Remove references to Writable in the ipc package
[ https://issues.apache.org/jira/browse/HBASE-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540832#comment-13540832 ] stack commented on HBASE-7224: -- Oh, the two javadoc warnings are complains about our unsafe access. Addressed in another issue. > Remove references to Writable in the ipc package > > > Key: HBASE-7224 > URL: https://issues.apache.org/jira/browse/HBASE-7224 > Project: HBase > Issue Type: Sub-task > Components: IPC/RPC, Protobufs >Reporter: Devaraj Das >Assignee: stack >Priority: Blocker > Fix For: 0.96.0 > > Attachments: 7224.txt, 7224v2.txt, 7224v3.txt, 7224v4.txt, > 7224v4.txt, 7224v4.txt, 7224v5.txt, purge_more_writables.txt > > > I see references to Writable in the ipc package, most notably in the > Invocation class. This class is not being used that much in the core ipc > package but used in the coprocessor protocol implementations (there are some > coprocessor protocols that are Writable based still). This jira is to track > removing those references and the Invocation class (once HBASE-6895 is > resolved). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7299) TestMultiParallel fails intermittently in trunk builds
[
https://issues.apache.org/jira/browse/HBASE-7299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540831#comment-13540831
]
Hadoop QA commented on HBASE-7299:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12562657/HBASE-7299v3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 2
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.ipc.TestProtoBufRpc
org.apache.hadoop.hbase.ipc.TestDelayedRpc
org.apache.hadoop.hbase.TestClusterBootOrder
org.apache.hadoop.hbase.TestDrainingServer
org.apache.hadoop.hbase.TestZooKeeper
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3751//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3751//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3751//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3751//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3751//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3751//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3751//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3751//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3751//console
This message is automatically generated.
> TestMultiParallel fails intermittently in trunk builds
> --
>
> Key: HBASE-7299
> URL: https://issues.apache.org/jira/browse/HBASE-7299
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: chunhui shen
>Priority: Critical
> Fix For: 0.96.0
>
> Attachments: HBASE-7299.patch, HBASE-7299v2.patch, HBASE-7299v3.patch
>
>
> From trunk build #3598:
> {code}
> testFlushCommitsNoAbort(org.apache.hadoop.hbase.client.TestMultiParallel):
> Count of regions=8
> {code}
> It failed in 3595 as well:
> {code}
> java.lang.AssertionError: Server count=2, abort=true expected:<1> but was:<2>
> at org.junit.Assert.fail(Assert.java:93)
> at org.junit.Assert.failNotEquals(Assert.java:647)
> at org.junit.Assert.assertEquals(Assert.java:128)
> at org.junit.Assert.assertEquals(Assert.java:472)
> at
> org.apache.hadoop.hbase.client.TestMultiParallel.doTestFlushCommits(TestMultiParallel.java:267)
> at
> org.apache.hadoop.hbase.client.TestMultiParallel.testFlushCommitsWithAbort(TestMultiParallel.java:226)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7224) Remove references to Writable in the ipc package
[ https://issues.apache.org/jira/browse/HBASE-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-7224: - Attachment: 7224v5.txt Here is what I committed. Includes feedback by lads from up on rb (white space and some extra comments). > Remove references to Writable in the ipc package > > > Key: HBASE-7224 > URL: https://issues.apache.org/jira/browse/HBASE-7224 > Project: HBase > Issue Type: Sub-task > Components: IPC/RPC, Protobufs >Reporter: Devaraj Das >Assignee: stack >Priority: Blocker > Fix For: 0.96.0 > > Attachments: 7224.txt, 7224v2.txt, 7224v3.txt, 7224v4.txt, > 7224v4.txt, 7224v4.txt, 7224v5.txt, purge_more_writables.txt > > > I see references to Writable in the ipc package, most notably in the > Invocation class. This class is not being used that much in the core ipc > package but used in the coprocessor protocol implementations (there are some > coprocessor protocols that are Writable based still). This jira is to track > removing those references and the Invocation class (once HBASE-6895 is > resolved). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7457) Fix javadoc warnings in hadoopqa tool, it complains about unsafe accesses
[
https://issues.apache.org/jira/browse/HBASE-7457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-7457:
-
Assignee: stack
Status: Patch Available (was: Open)
Submit. See if warnings.
> Fix javadoc warnings in hadoopqa tool, it complains about unsafe accesses
> -
>
> Key: HBASE-7457
> URL: https://issues.apache.org/jira/browse/HBASE-7457
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: stack
> Attachments: 7457.txt
>
>
> I see this in hadoopqa output and it seems to be causing the two warnings we
> currently see in hadoopqa reports:
> {code}
> 2 warnings
> [WARNING] Javadoc Warnings
> [WARNING]
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/Bytes.java:43:
> warning: sun.misc.Unsafe is Sun proprietary API and may be removed in a
> future release
> [WARNING] import sun.misc.Unsafe;
> [WARNING] ^
> [WARNING]
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/Bytes.java:1032:
> warning: sun.misc.Unsafe is Sun proprietary API and may be removed in a
> future release
> [WARNING] static final Unsafe theUnsafe;
> [WARNING] ^
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7457) Fix javadoc warnings in hadoopqa tool, it complains about unsafe accesses
[
https://issues.apache.org/jira/browse/HBASE-7457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-7457:
-
Attachment: 7457.txt
Here is what I committed... allowing the two unsafe access warnings.
> Fix javadoc warnings in hadoopqa tool, it complains about unsafe accesses
> -
>
> Key: HBASE-7457
> URL: https://issues.apache.org/jira/browse/HBASE-7457
> Project: HBase
> Issue Type: Bug
>Reporter: stack
> Attachments: 7457.txt
>
>
> I see this in hadoopqa output and it seems to be causing the two warnings we
> currently see in hadoopqa reports:
> {code}
> 2 warnings
> [WARNING] Javadoc Warnings
> [WARNING]
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/Bytes.java:43:
> warning: sun.misc.Unsafe is Sun proprietary API and may be removed in a
> future release
> [WARNING] import sun.misc.Unsafe;
> [WARNING] ^
> [WARNING]
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/Bytes.java:1032:
> warning: sun.misc.Unsafe is Sun proprietary API and may be removed in a
> future release
> [WARNING] static final Unsafe theUnsafe;
> [WARNING] ^
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7457) Fix javadoc warnings in hadoopqa tool, it complains about unsafe accesses
stack created HBASE-7457:
Summary: Fix javadoc warnings in hadoopqa tool, it complains about
unsafe accesses
Key: HBASE-7457
URL: https://issues.apache.org/jira/browse/HBASE-7457
Project: HBase
Issue Type: Bug
Reporter: stack
I see this in hadoopqa output and it seems to be causing the two warnings we
currently see in hadoopqa reports:
{code}
2 warnings
[WARNING] Javadoc Warnings
[WARNING]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/Bytes.java:43:
warning: sun.misc.Unsafe is Sun proprietary API and may be removed in a future
release
[WARNING] import sun.misc.Unsafe;
[WARNING] ^
[WARNING]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/Bytes.java:1032:
warning: sun.misc.Unsafe is Sun proprietary API and may be removed in a future
release
[WARNING] static final Unsafe theUnsafe;
[WARNING] ^
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7299) TestMultiParallel fails intermittently in trunk builds
[
https://issues.apache.org/jira/browse/HBASE-7299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540826#comment-13540826
]
stack commented on HBASE-7299:
--
I think it is these two outputs:
{code}
2 warnings
[WARNING] Javadoc Warnings
[WARNING]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/Bytes.java:43:
warning: sun.misc.Unsafe is Sun proprietary API and may be removed in a future
release
[WARNING] import sun.misc.Unsafe;
[WARNING] ^
[WARNING]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/Bytes.java:1032:
warning: sun.misc.Unsafe is Sun proprietary API and may be removed in a future
release
[WARNING] static final Unsafe theUnsafe;
[WARNING] ^
{code}
Let me fix in new issue.
> TestMultiParallel fails intermittently in trunk builds
> --
>
> Key: HBASE-7299
> URL: https://issues.apache.org/jira/browse/HBASE-7299
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: chunhui shen
>Priority: Critical
> Fix For: 0.96.0
>
> Attachments: HBASE-7299.patch, HBASE-7299v2.patch, HBASE-7299v3.patch
>
>
> From trunk build #3598:
> {code}
> testFlushCommitsNoAbort(org.apache.hadoop.hbase.client.TestMultiParallel):
> Count of regions=8
> {code}
> It failed in 3595 as well:
> {code}
> java.lang.AssertionError: Server count=2, abort=true expected:<1> but was:<2>
> at org.junit.Assert.fail(Assert.java:93)
> at org.junit.Assert.failNotEquals(Assert.java:647)
> at org.junit.Assert.assertEquals(Assert.java:128)
> at org.junit.Assert.assertEquals(Assert.java:472)
> at
> org.apache.hadoop.hbase.client.TestMultiParallel.doTestFlushCommits(TestMultiParallel.java:267)
> at
> org.apache.hadoop.hbase.client.TestMultiParallel.testFlushCommitsWithAbort(TestMultiParallel.java:226)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540823#comment-13540823
]
Hadoop QA commented on HBASE-7403:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12562655/7403v5.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
10 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestFromClientSide
{color:red}-1 core zombie tests{color}. There are 2 zombie test(s):
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3749//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3749//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3749//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3749//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3749//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3749//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3749//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3749//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3749//console
This message is automatically generated.
> Online Merge
>
>
> Key: HBASE-7403
> URL: https://issues.apache.org/jira/browse/HBASE-7403
> Project: HBase
> Issue Type: New Feature
>Affects Versions: 0.94.3
>Reporter: chunhui shen
>Assignee: chunhui shen
> Fix For: 0.96.0, 0.94.5
>
> Attachments: 7403-trunkv5.patch, 7403-v5.txt, 7403v5.txt,
> hbase-7403-94v1.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch,
> merge region.pdf
>
>
> The feature of this online merge:
> 1.Online,no necessary to disable table
> 2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90
> 3.Easy to call merege request, no need to input a long region name, only
> encoded name enough
> 4.No limit when operation, you don't need to care event like Server Dead,
> Balance, Split, Disabing/Enabing table, no need to care whether you send a
> wrong merge request, it alread have done for you
> 5.Only little offline time for two merging regions
> We need merge in the following cases：
> 1.Region hole or region overlap, can’t be fix by hbck
> 2.Region become empty because of TTL and not reasonable Rowkey design
> 3.Region is always empty or very small because of presplit when create table
> 4.Too many empty or small regions would reduce the system performance(e.g.
> mslab)
> Current merge tools only support offline and are not able to redo if
> exception is thrown in the process of merging, causing a dirty data
> For online system, we need a online merge.
> This implement logic of this patch for Online Merge is :
> For example, merge regionA and regionB into regionC
> 1.Offline the two regions A and B
> 2.Merge the two regions in the HDFS(Create regionC’s directory, move
> regionA’s and regionB’s file to regionC’s directory, delete regionA’s and
> regionB’s directory)
> 3.Add the merged regionC to .META.
> 4.Assign the merged regionC
> As design of this patch , once we do the merge work in the HDFS,we could redo
> it until successful if it throws exception or abort or server restart, but
> couldn’t be rolled back.
> It depends on
> Use zookeeper to record the transaction journal state, make redo easier
> Use zookeeper to send/rece
[jira] [Updated] (HBASE-7299) TestMultiParallel fails intermittently in trunk builds
[
https://issues.apache.org/jira/browse/HBASE-7299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chunhui shen updated HBASE-7299:
Attachment: HBASE-7299v3.patch
I think it's because of no annotation for the method beforeClass() and method
doTestFlushCommits(boolean doAbort)
> TestMultiParallel fails intermittently in trunk builds
> --
>
> Key: HBASE-7299
> URL: https://issues.apache.org/jira/browse/HBASE-7299
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Critical
> Fix For: 0.96.0
>
> Attachments: HBASE-7299.patch, HBASE-7299v2.patch, HBASE-7299v3.patch
>
>
> From trunk build #3598:
> {code}
> testFlushCommitsNoAbort(org.apache.hadoop.hbase.client.TestMultiParallel):
> Count of regions=8
> {code}
> It failed in 3595 as well:
> {code}
> java.lang.AssertionError: Server count=2, abort=true expected:<1> but was:<2>
> at org.junit.Assert.fail(Assert.java:93)
> at org.junit.Assert.failNotEquals(Assert.java:647)
> at org.junit.Assert.assertEquals(Assert.java:128)
> at org.junit.Assert.assertEquals(Assert.java:472)
> at
> org.apache.hadoop.hbase.client.TestMultiParallel.doTestFlushCommits(TestMultiParallel.java:267)
> at
> org.apache.hadoop.hbase.client.TestMultiParallel.testFlushCommitsWithAbort(TestMultiParallel.java:226)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-7299) TestMultiParallel fails intermittently in trunk builds
[
https://issues.apache.org/jira/browse/HBASE-7299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chunhui shen reassigned HBASE-7299:
---
Assignee: chunhui shen
> TestMultiParallel fails intermittently in trunk builds
> --
>
> Key: HBASE-7299
> URL: https://issues.apache.org/jira/browse/HBASE-7299
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: chunhui shen
>Priority: Critical
> Fix For: 0.96.0
>
> Attachments: HBASE-7299.patch, HBASE-7299v2.patch, HBASE-7299v3.patch
>
>
> From trunk build #3598:
> {code}
> testFlushCommitsNoAbort(org.apache.hadoop.hbase.client.TestMultiParallel):
> Count of regions=8
> {code}
> It failed in 3595 as well:
> {code}
> java.lang.AssertionError: Server count=2, abort=true expected:<1> but was:<2>
> at org.junit.Assert.fail(Assert.java:93)
> at org.junit.Assert.failNotEquals(Assert.java:647)
> at org.junit.Assert.assertEquals(Assert.java:128)
> at org.junit.Assert.assertEquals(Assert.java:472)
> at
> org.apache.hadoop.hbase.client.TestMultiParallel.doTestFlushCommits(TestMultiParallel.java:267)
> at
> org.apache.hadoop.hbase.client.TestMultiParallel.testFlushCommitsWithAbort(TestMultiParallel.java:226)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7455) Increase timeouts in TestReplication and TestSplitLogWorker
[ https://issues.apache.org/jira/browse/HBASE-7455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540815#comment-13540815 ] Hudson commented on HBASE-7455: --- Integrated in HBase-0.94-security #90 (See [https://builds.apache.org/job/HBase-0.94-security/90/]) HBASE-7455 Increase timeouts in TestReplication and TestSplitLogWorker (Revision 1426711) Result = FAILURE larsh : Files : * /hbase/branches/0.94/CHANGES.txt * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitLogWorker.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/replication/TestReplication.java > Increase timeouts in TestReplication and TestSplitLogWorker > --- > > Key: HBASE-7455 > URL: https://issues.apache.org/jira/browse/HBASE-7455 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl > Fix For: 0.96.0, 0.94.4 > > Attachments: 7455-0.94.txt, 7455-0.96.txt > > > When I measure the times in TestReplication.queueFailover, it takes about 15s > on my (reasonably fast) Laptop. > The timeout in queueFailover currently is 1500*2*15 = 45000ms. > For setup before each test (which truncates the table and waits for the > changes to replicate) it is 1500*15 = 22500ms. > Interestingly I see queueFailover failures where the wait time is measured as > 64260ms and some at 72316ms. > Since these numbers are not even close to 45000ms, the machine or JVM must > have been stuck for 15 or almost 30s (otherwise we'd get a timeout and the > total time spent should be close to the timeout). > So I would suggest that we increase the timeouts further. > We could set SLEEP_TIME to 2000 and retries to 20. Would lead to 2000*2*20 = > 8ms. > Any objections? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7412) Fix how HTableDescriptor handles default max file size and flush size
[ https://issues.apache.org/jira/browse/HBASE-7412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540816#comment-13540816 ] Hudson commented on HBASE-7412: --- Integrated in HBase-0.94-security #90 (See [https://builds.apache.org/job/HBase-0.94-security/90/]) HBASE-7412 Fix how HTableDescriptor handles default max file size and flush size - ADDENDUM (Revision 1426682) HBASE-7412 Fix how HTableDescriptor handles default max file size and flush size (Revision 1426658) Result = FAILURE jxiang : Files : * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/backup/TestHFileArchiving.java jxiang : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/IncreasingToUpperBoundRegionSplitPolicy.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/TestHTableDescriptor.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java > Fix how HTableDescriptor handles default max file size and flush size > - > > Key: HBASE-7412 > URL: https://issues.apache.org/jira/browse/HBASE-7412 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Fix For: 0.96.0, 0.94.4 > > Attachments: 0.94-7412-addendum.patch, 0.94-7412.patch, > 0.94-7412_v2.patch, trunk-7412.patch, trunk-7412_v2.patch > > > If the region flush size is not set in the table, > IncreasingToUpperBoundRegionSplitPolicy will most likely always use the > default value: 128MB, even if the flush size is set to a different value in > hbase-site.xml. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7438) TestSplitTransactionOnCluster has too many infinite loops
[ https://issues.apache.org/jira/browse/HBASE-7438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540814#comment-13540814 ] Hudson commented on HBASE-7438: --- Integrated in HBase-0.94-security #90 (See [https://builds.apache.org/job/HBase-0.94-security/90/]) HBASE-7438 TestSplitTransactionOnCluster has too many infinite loops (Revision 1426067) Result = FAILURE larsh : Files : * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java > TestSplitTransactionOnCluster has too many infinite loops > - > > Key: HBASE-7438 > URL: https://issues.apache.org/jira/browse/HBASE-7438 > Project: HBase > Issue Type: Sub-task >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0, 0.94.4 > > Attachments: 7438-0.94.txt, 7438-0.96.txt > > > There are many cases in these test where we loop until a condition happens. > If that condition never occurs we'll wait forever, and the test will time out > instead of failing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3776) Add Bloom Filter Support to HFileOutputFormat
[ https://issues.apache.org/jira/browse/HBASE-3776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540813#comment-13540813 ] Hudson commented on HBASE-3776: --- Integrated in HBase-0.94-security #90 (See [https://builds.apache.org/job/HBase-0.94-security/90/]) HBASE-3776 Add Bloom Filter Support to HFileOutputFormat (Anoop Sam John) (Revision 1426415) Result = FAILURE larsh : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java > Add Bloom Filter Support to HFileOutputFormat > - > > Key: HBASE-3776 > URL: https://issues.apache.org/jira/browse/HBASE-3776 > Project: HBase > Issue Type: Sub-task > Components: mapreduce >Reporter: Nicolas Spiegelberg >Assignee: Anoop Sam John >Priority: Critical > Labels: hbase > Fix For: 0.96.0, 0.94.4 > > Attachments: 3776-0.94.txt, 3776-trunk.addendum, HBASE-3776.patch, > HBASE-3776_v2.patch > > > Add Bloom Filter support for bulk imports. Lacking a bloom filter, even on a > single imported file, can cause perf degradation. Since we now set our > compression type based on the HBase CF configuration, it would be good to > follow this path for the bloom filter addition. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7440) ReplicationZookeeper#addPeer is racy
[
https://issues.apache.org/jira/browse/HBASE-7440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540812#comment-13540812
]
Hudson commented on HBASE-7440:
---
Integrated in HBase-0.94-security #90 (See
[https://builds.apache.org/job/HBase-0.94-security/90/])
HBASE-7440 ReplicationZookeeper#addPeer is racy (Himanshu) (Revision
1426704)
Result = FAILURE
larsh :
Files :
* /hbase/branches/0.94/CHANGES.txt
*
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/replication/ReplicationPeer.java
*
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java
> ReplicationZookeeper#addPeer is racy
>
>
> Key: HBASE-7440
> URL: https://issues.apache.org/jira/browse/HBASE-7440
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 0.94.3
>Reporter: Himanshu Vashishtha
>Assignee: Himanshu Vashishtha
> Fix For: 0.96.0, 0.94.4
>
> Attachments: HBASE-7440-trunk-v0.patch, HBASE-7440-trunk-v1.patch,
> HBASE-7440-v0.patch, HBASE-7440-v1.patch, HBASE-7440-v2.patch
>
>
> While adding a peer, ReplicationZK does the znodes creation in three
> transactions. Create :
> a) peers znode
> b) peerId specific znode, and
> c) peerState znode
> There is a PeerWatcher which invokes getPeer() (after steps b) and c)). If it
> happens that while adding a peer, the control flows to getPeer() and step c)
> has not been processed, it may results in a state where the peer will not be
> added. This happens while running
> TestMasterReplication#testCyclicReplication().
> {code}
> 2012-12-26 07:36:35,187 INFO
> [RegionServer:0;p0120.X,38423,1356536179470-EventThread]
> zookeeper.RecoverableZooKeeper(447): Node /2/replication/peers/1/peer-state
> already exists and this is not a retry
> 2012-12-26 07:36:35,188 ERROR
> [RegionServer:0;p0120.X,38423,1356536179470-EventThread]
> regionserver.ReplicationSourceManager$PeersWatcher(527): Error while adding a
> new peer
> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
> NodeExists for /2/replication/peers/1/peer-state
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:428)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:410)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndWatch(ZKUtil.java:1044)
> at
> org.apache.hadoop.hbase.replication.ReplicationPeer.startStateTracker(ReplicationPeer.java:82)
> at
> org.apache.hadoop.hbase.replication.ReplicationZookeeper.getPeer(ReplicationZookeeper.java:344)
> at
> org.apache.hadoop.hbase.replication.ReplicationZookeeper.connectToPeer(ReplicationZookeeper.java:307)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager$PeersWatcher.nodeChildrenChanged(ReplicationSourceManager.java:519)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:315)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
> 2012-12-26 07:36:35,188 DEBUG
> [RegionServer:0;p0120.X,55742,1356536171947-EventThread]
> zookeeper.ZKUtil(1545): regionserver:55742-0x13bd7db39580004 Retrieved 36
> byte(s) of data from znode /1/hbaseid; data=9ce66123-d3e8-4ae9-a249-afe03...
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7456) Stargate's HTablePool maxSize is hard-coded at 10, too small for heavy loads
Chip Salzenberg created HBASE-7456: -- Summary: Stargate's HTablePool maxSize is hard-coded at 10, too small for heavy loads Key: HBASE-7456 URL: https://issues.apache.org/jira/browse/HBASE-7456 Project: HBase Issue Type: Bug Components: REST Reporter: Chip Salzenberg Please allow the Configuration to override the hard-coded maxSize of 10 for its HTablePool. Under high loads, 10 is too small. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7441) Make ClusterManager in IntegrationTestingUtility pluggable
[ https://issues.apache.org/jira/browse/HBASE-7441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui updated HBASE-7441: --- Attachment: HBASE-7441-trunk-v1.patch The hbase 7441 patch for trunk. [~lhofhansl] > Make ClusterManager in IntegrationTestingUtility pluggable > -- > > Key: HBASE-7441 > URL: https://issues.apache.org/jira/browse/HBASE-7441 > Project: HBase > Issue Type: Improvement > Components: test >Affects Versions: 0.94.3 >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Minor > Labels: newbie, patch > Fix For: 0.96.0, 0.94.5 > > Attachments: HBASE-7441-0.94-v1.patch, HBASE-7441-trunk-v1.patch > > Original Estimate: 3h > Remaining Estimate: 3h > > After the patch HBASE-7009, we can use ChaosMonkey to test the Hbase cluster. > The ClusterManager use ssh to stop/start the rs or master without passwd. To > support other cluster manager tool, we need to make clusterManager in > IntegrationTestingUtility pluggable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7441) Make ClusterManager in IntegrationTestingUtility pluggable
[ https://issues.apache.org/jira/browse/HBASE-7441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui updated HBASE-7441: --- Assignee: Liu Shaohui > Make ClusterManager in IntegrationTestingUtility pluggable > -- > > Key: HBASE-7441 > URL: https://issues.apache.org/jira/browse/HBASE-7441 > Project: HBase > Issue Type: Improvement > Components: test >Affects Versions: 0.94.3 >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Minor > Labels: newbie, patch > Fix For: 0.96.0, 0.94.5 > > Attachments: HBASE-7441-0.94-v1.patch > > Original Estimate: 3h > Remaining Estimate: 3h > > After the patch HBASE-7009, we can use ChaosMonkey to test the Hbase cluster. > The ClusterManager use ssh to stop/start the rs or master without passwd. To > support other cluster manager tool, we need to make clusterManager in > IntegrationTestingUtility pluggable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7258) Hbase needs to create baseZNode recursively
[ https://issues.apache.org/jira/browse/HBASE-7258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540807#comment-13540807 ] Liu Shaohui commented on HBASE-7258: [~ram_krish] I have tested the change in hbase version 0.94. The patch passed all the tests except some randomly failed test cases. I think they have relations with the patch. > Hbase needs to create baseZNode recursively > --- > > Key: HBASE-7258 > URL: https://issues.apache.org/jira/browse/HBASE-7258 > Project: HBase > Issue Type: Bug > Components: Zookeeper >Affects Versions: 0.94.2 >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Minor > Labels: zookeeper, zookeeper.znode.parent > Fix For: 0.96.0 > > Attachments: HBASE-7258-0.94.patch, HBASE-7258.diff, HBASE-7258.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > In deploy env, multi small hbase clusters may share a same zk cluster. So, > for hbase cluster1, its parent znode is /hbase/cluster1. But in hbase version > 0.94.1, hbase use ZKUtil.createAndFailSilent(this, baseZNode) to create > parent path and it will throw a NoNode exception if znode /hbase donot exist. > We want to change it to ZKUtil.createWithParents(this, baseZNode); to suport > create baseZNode recursivly. > The NoNode exception is: > java.lang.RuntimeException: Failed construction of Master: class > org.apache.hadoop.hbase.master.HMaster > at > org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1792) > at > org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:146) > at > org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:103) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at > org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:77) > at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1806) > Caused by: org.apache.zookeeper.KeeperException$NoNodeException: > KeeperErrorCode = NoNode for /hbase/cluster1 > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:111) > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:778) > at > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:420) > at > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:402) > at > org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:905) > at > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.createBaseZNodes(ZooKeeperWatcher.java:166) > at > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.(ZooKeeperWatcher.java:159) > at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:282) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at org.apache.hadoop.hbase.master.H -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chunhui shen updated HBASE-7403: Attachment: 7403v5.txt > Online Merge > > > Key: HBASE-7403 > URL: https://issues.apache.org/jira/browse/HBASE-7403 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.94.3 >Reporter: chunhui shen >Assignee: chunhui shen > Fix For: 0.96.0, 0.94.5 > > Attachments: 7403-trunkv5.patch, 7403-v5.txt, 7403v5.txt, > hbase-7403-94v1.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch, > merge region.pdf > > > The feature of this online merge: > 1.Online,no necessary to disable table > 2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90 > 3.Easy to call merege request, no need to input a long region name, only > encoded name enough > 4.No limit when operation, you don't need to care event like Server Dead, > Balance, Split, Disabing/Enabing table, no need to care whether you send a > wrong merge request, it alread have done for you > 5.Only little offline time for two merging regions > We need merge in the following cases： > 1.Region hole or region overlap, can’t be fix by hbck > 2.Region become empty because of TTL and not reasonable Rowkey design > 3.Region is always empty or very small because of presplit when create table > 4.Too many empty or small regions would reduce the system performance(e.g. > mslab) > Current merge tools only support offline and are not able to redo if > exception is thrown in the process of merging, causing a dirty data > For online system, we need a online merge. > This implement logic of this patch for Online Merge is : > For example, merge regionA and regionB into regionC > 1.Offline the two regions A and B > 2.Merge the two regions in the HDFS(Create regionC’s directory, move > regionA’s and regionB’s file to regionC’s directory, delete regionA’s and > regionB’s directory) > 3.Add the merged regionC to .META. > 4.Assign the merged regionC > As design of this patch , once we do the merge work in the HDFS,we could redo > it until successful if it throws exception or abort or server restart, but > couldn’t be rolled back. > It depends on > Use zookeeper to record the transaction journal state, make redo easier > Use zookeeper to send/receive merge request > Merge transaction is executed on the master > Support calling merge request through API or shell tool > About the merge process, please see the attachment and patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7299) TestMultiParallel fails intermittently in trunk builds
[
https://issues.apache.org/jira/browse/HBASE-7299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540806#comment-13540806
]
stack commented on HBASE-7299:
--
Are you responsible for the two new javadoc warnings Chunhui?
-1 javadoc. The javadoc tool appears to have generated 2 warning messages.
> TestMultiParallel fails intermittently in trunk builds
> --
>
> Key: HBASE-7299
> URL: https://issues.apache.org/jira/browse/HBASE-7299
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Critical
> Fix For: 0.96.0
>
> Attachments: HBASE-7299.patch, HBASE-7299v2.patch
>
>
> From trunk build #3598:
> {code}
> testFlushCommitsNoAbort(org.apache.hadoop.hbase.client.TestMultiParallel):
> Count of regions=8
> {code}
> It failed in 3595 as well:
> {code}
> java.lang.AssertionError: Server count=2, abort=true expected:<1> but was:<2>
> at org.junit.Assert.fail(Assert.java:93)
> at org.junit.Assert.failNotEquals(Assert.java:647)
> at org.junit.Assert.assertEquals(Assert.java:128)
> at org.junit.Assert.assertEquals(Assert.java:472)
> at
> org.apache.hadoop.hbase.client.TestMultiParallel.doTestFlushCommits(TestMultiParallel.java:267)
> at
> org.apache.hadoop.hbase.client.TestMultiParallel.testFlushCommitsWithAbort(TestMultiParallel.java:226)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7455) Increase timeouts in TestReplication and TestSplitLogWorker
[ https://issues.apache.org/jira/browse/HBASE-7455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540803#comment-13540803 ] Hudson commented on HBASE-7455: --- Integrated in HBase-TRUNK #3672 (See [https://builds.apache.org/job/HBase-TRUNK/3672/]) HBASE-7455 Increase timeouts in TestReplication and TestSplitLogWorker (Revision 1426710) Result = FAILURE larsh : Files : * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplication.java > Increase timeouts in TestReplication and TestSplitLogWorker > --- > > Key: HBASE-7455 > URL: https://issues.apache.org/jira/browse/HBASE-7455 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl > Fix For: 0.96.0, 0.94.4 > > Attachments: 7455-0.94.txt, 7455-0.96.txt > > > When I measure the times in TestReplication.queueFailover, it takes about 15s > on my (reasonably fast) Laptop. > The timeout in queueFailover currently is 1500*2*15 = 45000ms. > For setup before each test (which truncates the table and waits for the > changes to replicate) it is 1500*15 = 22500ms. > Interestingly I see queueFailover failures where the wait time is measured as > 64260ms and some at 72316ms. > Since these numbers are not even close to 45000ms, the machine or JVM must > have been stuck for 15 or almost 30s (otherwise we'd get a timeout and the > total time spent should be close to the timeout). > So I would suggest that we increase the timeouts further. > We could set SLEEP_TIME to 2000 and retries to 20. Would lead to 2000*2*20 = > 8ms. > Any objections? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
