[
https://issues.apache.org/jira/browse/HBASE-7352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13531857#comment-13531857
]
Jonathan Hsieh edited comment on HBASE-7352 at 12/14/12 1:27 AM:
-----------------------------------------------------------------
The snapshot can be cloned from another shell instance but must write to a
different name.
If you attempt to clone the snapshot (pe-11) to the same target table
(pe-11-table), you get:
{code}
hbase(main):006:0> clone_snapshot 'pe-11', 'pe-11-table'
ERROR: org.apache.hadoop.hbase.snapshot.exception.RestoreSnapshotException:
org.apache.hadoop.hbase.snapshot.exception.RestoreSnapshotExcept
ion: Couldn't clone the snapshot=name: "pe-11"
table: "TestTable"
creationTime: 1355441918484
type: FLUSH
version: 0
on table=pe-11-table
at
org.apache.hadoop.hbase.master.snapshot.manage.SnapshotManager.cloneSnapshot(SnapshotManager.java:558)
at
org.apache.hadoop.hbase.master.snapshot.manage.SnapshotManager.restoreSnapshot(SnapshotManager.java:597)
at
org.apache.hadoop.hbase.master.HMaster.restoreSnapshot(HMaster.java:2528)
at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.ProtobufRpcEngine$Server.call(ProtobufRpcEngine.java:356)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1816)
Caused by: org.apache.hadoop.hbase.TableExistsException: pe-11-table
at
org.apache.hadoop.hbase.master.handler.CreateTableHandler.<init>(CreateTableHandler.java:96)
at
org.apache.hadoop.hbase.master.snapshot.CloneSnapshotHandler.<init>(CloneSnapshotHandler.java:65)
at
org.apache.hadoop.hbase.master.snapshot.manage.SnapshotManager.cloneSnapshot(SnapshotManager.java:551)
at
org.apache.hadoop.hbase.master.snapshot.manage.SnapshotManager.cloneSnapshot(SnapshotManager.java:558)
at
org.apache.hadoop.hbase.master.snapshot.manage.SnapshotManager.restoreSnapshot(SnapshotManager.java:597)
at
org.apache.hadoop.hbase.master.HMaster.restoreSnapshot(HMaster.java:2528)
at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.ProtobufRpcEngine$Server.call(ProtobufRpcEngine.java:356)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1816)
Caused by: org.apache.hadoop.hbase.TableExistsException: pe-11-table
at
org.apache.hadoop.hbase.master.handler.CreateTableHandler.<init>(CreateTableHandler.java:96)
at
org.apache.hadoop.hbase.master.snapshot.CloneSnapshotHandler.<init>(CloneSnapshotHandler.java:65)
at
org.apache.hadoop.hbase.master.snapshot.manage.SnapshotManager.cloneSnapshot(SnapshotManager.java:551)
... 7 more
{code}
The dir for seems to be present for table pe-11, but there seems to be a large
number of missing files.
In this particular case, the snapshot has 16 regions, while the failed attempt
to restore has 12 regions moved into real table position. This suggests that
something failed internally but was allowed to continue to do the dir rename at
some point.
If one removes the data bad data from the hdfs we still cannot clone to the
same pe-11-table name because there is some in memory state that blocks this.
(grammar fixes)
was (Author: jmhsieh):
The snapshot can be cloned from another shell instance but must write to a
different name.
If you attempt to clone the snapshot (pe-11) to the same target table
(pe-11-table), you get:
{code}
hbase(main):006:0> clone_snapshot 'pe-11', 'pe-11-table'
ERROR: org.apache.hadoop.hbase.snapshot.exception.RestoreSnapshotException:
org.apache.hadoop.hbase.snapshot.exception.RestoreSnapshotExcept
ion: Couldn't clone the snapshot=name: "pe-11"
table: "TestTable"
creationTime: 1355441918484
type: FLUSH
version: 0
on table=pe-11-table
at
org.apache.hadoop.hbase.master.snapshot.manage.SnapshotManager.cloneSnapshot(SnapshotManager.java:558)
at
org.apache.hadoop.hbase.master.snapshot.manage.SnapshotManager.restoreSnapshot(SnapshotManager.java:597)
at
org.apache.hadoop.hbase.master.HMaster.restoreSnapshot(HMaster.java:2528)
at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.ProtobufRpcEngine$Server.call(ProtobufRpcEngine.java:356)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1816)
Caused by: org.apache.hadoop.hbase.TableExistsException: pe-11-table
at
org.apache.hadoop.hbase.master.handler.CreateTableHandler.<init>(CreateTableHandler.java:96)
at
org.apache.hadoop.hbase.master.snapshot.CloneSnapshotHandler.<init>(CloneSnapshotHandler.java:65)
at
org.apache.hadoop.hbase.master.snapshot.manage.SnapshotManager.cloneSnapshot(SnapshotManager.java:551)
at
org.apache.hadoop.hbase.master.snapshot.manage.SnapshotManager.cloneSnapshot(SnapshotManager.java:558)
at
org.apache.hadoop.hbase.master.snapshot.manage.SnapshotManager.restoreSnapshot(SnapshotManager.java:597)
at
org.apache.hadoop.hbase.master.HMaster.restoreSnapshot(HMaster.java:2528)
at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.ProtobufRpcEngine$Server.call(ProtobufRpcEngine.java:356)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1816)
Caused by: org.apache.hadoop.hbase.TableExistsException: pe-11-table
at
org.apache.hadoop.hbase.master.handler.CreateTableHandler.<init>(CreateTableHandler.java:96)
at
org.apache.hadoop.hbase.master.snapshot.CloneSnapshotHandler.<init>(CloneSnapshotHandler.java:65)
at
org.apache.hadoop.hbase.master.snapshot.manage.SnapshotManager.cloneSnapshot(SnapshotManager.java:551)
... 7 more
{code}
The dir for seems to be present for table pe-11, but there seems to be a large
number of missing files.
In this particular case, the snapshot has 16 regions, while the failed attempt
to restore has 12 regions moved into real table position. This suggest that
something failed internally but was allowed to continue to do the dir rename at
some point.
If one removing the data bad data from the hdfs we still cannot clone to the
same pe-11-table name because there is some in memory state that blocks this.
> clone operation from HBaseAdmin can hang forever.
> -------------------------------------------------
>
> Key: HBASE-7352
> URL: https://issues.apache.org/jira/browse/HBASE-7352
> Project: HBase
> Issue Type: Sub-task
> Components: Client, master, regionserver, snapshots, Zookeeper
> Reporter: Jonathan Hsieh
> Fix For: hbase-6055, 0.96.0
>
>
> Sometimes the clone operation from the hbase shell can hang. The table has
> been created (it shows up in the web ui), but does not have any entries in
> META.
> There don't seem to be any clone, snapshot, enable or disable found in the
> master's jstack.
> Here's a trace from the HBaseAdmin:
> {code}
> "main" prio=10 tid=0x00007f782800d000 nid=0x25c waiting on condition
> [0x00007f782f9bf000]
> java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.cloneSnapshot(HBaseAdmin.java:2413)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.cloneSnapshot(HBaseAdmin.java:2393)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:465)
> at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:323)
> at
> org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:69)
> at
> org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:201)
> at org.jruby.ast.CallTwoArgNode.interpret(CallTwoArgNode.java:59)
> at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
> ... (more jruby stack) ...
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
