restoring old hbase backup in new hbase
Hi, We are planning to move our cluster from hbase 0.94.14 to 0.98.2. Our query is - If we have backup taken(using snapshots) on hbase 0.94.14, can we restore the same backup on newer version of hbase 0.98.2? Thanks oc.tsdb
how to reduce hbase-regionserver connection retries to zookeeper
Hi, if the namenode (or zookeeper) is down for some reason, hbase-regionserver(on datanode) keep retrying to connect to zookeeper continuously for long time. We tried by setting hbase.client.retries.number to 1 in hbase-site.xml, But no effect. Can someone please let us know, hbase configuration to stop hbase-regionserver in 5 sec, in case, if zookeeper is not running. Thanks oc.tsdb
incremental cluster backup using snapshots
Hi, Does new hbase version(0.99) supports incremental backup using snapshots? If not supported in current releases, is it planned to add in future releases? can we export snapshots to local file system directly? Thanks oc.tsdb
Re: HBase 0.98 rpm
Hi, That means first we should setup bigtop work space and then build rpm? Can you please point me, if there are any links describing in details about building rpms from scratch? Thanks oc.tsdb On Wed, Jun 4, 2014 at 4:52 AM, Ted Yu yuzhih...@gmail.com wrote: This is what I did: clone bigtop master branch under root of bigtop workspace, issue the following command: make hbase-rpm You should see this at the tail of output: + exit 0 cp -r /homes/hortonzy/bigtop/build/hbase//rpm/RPMS/*/* /homes/hortonzy/bigtop/output/hbase/ touch /homes/hortonzy/bigtop/build/hbase//.rpm You can find rpm's under output/hbase/ Cheers On Mon, Jun 2, 2014 at 10:52 PM, oc tsdb oc.t...@gmail.com wrote: Hi, To build rpm, we downloaded HBase 0.98.2 src from apache download link( http://mirror.sdunix.com/apache/hbase/hbase-0.98.2/hbase-0.98.2-src.tar.gz ) and other files (including specs) from bigtop ( https://github.com/apache/bigtop/tree/master/bigtop-packages/src/common/hbase ) But while building rpm we are getting the following error. *hbase/bigtop.bom: No such file or directory* i see following line in hbase/do-component-build *. `dirname $0`/bigtop.bom* Please let us know when do we get this error? and how to resolve. Thanks oc.tsdb On Tue, May 27, 2014 at 3:07 PM, Bharath Vissapragada bhara...@cloudera.com wrote: On Tue, May 27, 2014 at 2:38 PM, oc tsdb oc.t...@gmail.com wrote: Hi , 1) Where can we find hbase.098 tar with hbase 0.98 directory structure similar to hbase-0.94.19 ? i,e .. Directory structure of hbase 0.98 # ls bin/ dev-support/ hbase-common/ hbase-hadoop2-compat/ hbase-prefix-tree/ hbase-shell/ LICENSE.txt README.txt CHANGES.txt hbase-assembly/ hbase-examples/ hbase-hadoop-compat/ hbase-protocol/ hbase-testing-util/ NOTICE.txt src/ conf/ hbase-client/hbase-hadoop1-compat/ hbase-it/ hbase-server/ hbase-thrift/ pom.xml target/ where as directory structure of hbase-0.94.14 # ls ../hbase-0.94.14 bin/ CHANGES.txt conf/ docs/ hbase-0.94.14.jar hbase-0.94.14-tests.jar hbase-webapps/ lib/ LICENSE.txt NOTICE.txt pom.xml README.txt sbin/ security/ src/ we downloaded hbase 0.98 and hbase 0.94 from http://www.webhostingjams.com/mirror/apache/hbase/hbase-0.98.2/hbase-0.98.2-src.tar.gzand http://www.webhostingjams.com/mirror/apache/hbase/stable/hbase-0.94.19.tar.gz respectively . This was done in 0.95 release to shift from traditional single src/ directory to a multi-module layout as it makes more sense and easy to manage. More discussion about it in this jira https://issues.apache.org/jira/browse/HBASE-4336 2) we are trying to build an rpm out of hbase 0.98 version , where can i get the spec file or srpm for the same if available ? Does this work? Bigtop manages it now IIRC. https://github.com/apache/bigtop/blob/master/bigtop-packages/src/rpm/hbase/SPECS/hbase.spec Thanks oc.tsdb -- Bharath Vissapragada http://www.cloudera.com
Re: HBase 0.98 rpm
Hi, To build rpm, we downloaded HBase 0.98.2 src from apache download link( http://mirror.sdunix.com/apache/hbase/hbase-0.98.2/hbase-0.98.2-src.tar.gz) and other files (including specs) from bigtop ( https://github.com/apache/bigtop/tree/master/bigtop-packages/src/common/hbase ) But while building rpm we are getting the following error. *hbase/bigtop.bom: No such file or directory* i see following line in hbase/do-component-build *. `dirname $0`/bigtop.bom* Please let us know when do we get this error? and how to resolve. Thanks oc.tsdb On Tue, May 27, 2014 at 3:07 PM, Bharath Vissapragada bhara...@cloudera.com wrote: On Tue, May 27, 2014 at 2:38 PM, oc tsdb oc.t...@gmail.com wrote: Hi , 1) Where can we find hbase.098 tar with hbase 0.98 directory structure similar to hbase-0.94.19 ? i,e .. Directory structure of hbase 0.98 # ls bin/ dev-support/ hbase-common/ hbase-hadoop2-compat/ hbase-prefix-tree/ hbase-shell/ LICENSE.txt README.txt CHANGES.txt hbase-assembly/ hbase-examples/ hbase-hadoop-compat/ hbase-protocol/ hbase-testing-util/ NOTICE.txt src/ conf/ hbase-client/hbase-hadoop1-compat/ hbase-it/ hbase-server/ hbase-thrift/ pom.xml target/ where as directory structure of hbase-0.94.14 # ls ../hbase-0.94.14 bin/ CHANGES.txt conf/ docs/ hbase-0.94.14.jar hbase-0.94.14-tests.jar hbase-webapps/ lib/ LICENSE.txt NOTICE.txt pom.xml README.txt sbin/ security/ src/ we downloaded hbase 0.98 and hbase 0.94 from http://www.webhostingjams.com/mirror/apache/hbase/hbase-0.98.2/hbase-0.98.2-src.tar.gzand http://www.webhostingjams.com/mirror/apache/hbase/stable/hbase-0.94.19.tar.gz respectively . This was done in 0.95 release to shift from traditional single src/ directory to a multi-module layout as it makes more sense and easy to manage. More discussion about it in this jira https://issues.apache.org/jira/browse/HBASE-4336 2) we are trying to build an rpm out of hbase 0.98 version , where can i get the spec file or srpm for the same if available ? Does this work? Bigtop manages it now IIRC. https://github.com/apache/bigtop/blob/master/bigtop-packages/src/rpm/hbase/SPECS/hbase.spec Thanks oc.tsdb -- Bharath Vissapragada http://www.cloudera.com
HBase 0.98 rpm
Hi , 1) Where can we find hbase.098 tar with hbase 0.98 directory structure similar to hbase-0.94.19 ? i,e .. Directory structure of hbase 0.98 # ls bin/ dev-support/ hbase-common/ hbase-hadoop2-compat/ hbase-prefix-tree/ hbase-shell/ LICENSE.txt README.txt CHANGES.txt hbase-assembly/ hbase-examples/ hbase-hadoop-compat/ hbase-protocol/ hbase-testing-util/ NOTICE.txt src/ conf/ hbase-client/hbase-hadoop1-compat/ hbase-it/ hbase-server/ hbase-thrift/ pom.xml target/ where as directory structure of hbase-0.94.14 # ls ../hbase-0.94.14 bin/ CHANGES.txt conf/ docs/ hbase-0.94.14.jar hbase-0.94.14-tests.jar hbase-webapps/ lib/ LICENSE.txt NOTICE.txt pom.xml README.txt sbin/ security/ src/ we downloaded hbase 0.98 and hbase 0.94 from http://www.webhostingjams.com/mirror/apache/hbase/hbase-0.98.2/hbase-0.98.2-src.tar.gzand http://www.webhostingjams.com/mirror/apache/hbase/stable/hbase-0.94.19.tar.gz respectively . 2) we are trying to build an rpm out of hbase 0.98 version , where can i get the spec file or srpm for the same if available ? Thanks oc.tsdb
How to remove duplicate data in HBase?
Hi all, We want to know if there is any option to remove duplicate data in Hbase based on column family dynamically? Thanks, OC
directly export hbase snapshots to local fs
Hi, We are using HBase 0.94.14 and have only one cluster with 1NN and 4 DNs. We are trying to export snapshot directly to local system(e.g local fs path: /tmp/hbase_backup) as specified below.It is just exporting/copying snapshots (.hbase-snapshot) but not actual data(.archive).Why the below command is not copying actual data? hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot hbase_tbl_snapshot_name -copy-to file:///tmp/hbase_backup -mappers 16; It is always first need to export to HDFS and then copy to local file system using hadoop get command? Thanks in advance. -OC
Re: HBase 0.94.10 snapshot FileNotFoundException exception
verify snapshot with: hbase org.apache.hadoop.hbase.snapshot.SnapshotInfo -snapshot snapshotName - from a shell on Cluster-1: clone_snapshot 'snapshotName', 'tableName' 4 and 5 I've no idea what you mean Matteo On Mon, Dec 9, 2013 at 11:25 AM, oc tsdb oc.t...@gmail.com wrote: Hi, we are taking tsdb data backup using HBase snapshots. We tested the scenario as mentioned below: 1. Created snapshots and exported all snapshots successfully. Exported data contains archive+hbase-snapshot 2. Started HBase from scratch (removed all Hbase tabled and created again) 3. Successfully imported all snapshots created in step 1. 4. Again Created snapshots on imported data (step 3) and exported all snapshots. Exported data contains only hbase-snapshot but not archive.Why? 5. Now tried to import snapshots created in step 4 but we are getting below exception. Exception in thread main java.io.FileNotFoundException: Unable to open link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs:// tsdb.test.com:54310/tsdb_full_backup_2013-12-09_00.24.03/tsdb/1a3e8367302757a653efffbbde3b462b/t/tsdb=1a3e8367302757a653efffbbde3b462b-e265daa7b19d4911a5196c6f56a4162e , hdfs:// tsdb.test.com:54310/tsdb_full_backup_2013-12-09_00.24.03/.tmp/tsdb/1a3e8367302757a653efffbbde3b462b/t/tsdb=1a3e8367302757a653efffbbde3b462b-e265daa7b19d4911a5196c6f56a4162e , hdfs:// tsdb.test.com:54310/tsdb_full_backup_2013-12-09_00.24.03/.archive/tsdb/1a3e8367302757a653efffbbde3b462b/t/tsdb=1a3e8367302757a653efffbbde3b462b-e265daa7b19d4911a5196c6f56a4162e ] at org.apache.hadoop.hbase.io.FileLink.getFileStatus(FileLink.java:376) at org.apache.hadoop.hbase.snapshot.ExportSnapshot$1.storeFile(ExportSnapshot.java:390) at org.apache.hadoop.hbase.util.FSVisitor.visitRegionStoreFiles(FSVisitor.java:115) at org.apache.hadoop.hbase.util.FSVisitor.visitTableStoreFiles(FSVisitor.java:81) at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitTableStoreFiles(SnapshotReferenceUtil.java:116) at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitReferencedFiles(SnapshotReferenceUtil.java:101) at org.apache.hadoop.hbase.snapshot.ExportSnapshot.getSnapshotFiles(ExportSnapshot.java:385) at org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:633) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:705) at org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:709) Exception in thread main java.io.FileNotFoundException: Unable to open link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs:// tsdb.test.com:54310/tsdb_full_backup_2013-12-09_00.24.03/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/tsdb-uid=f9e5e554f111dc0679dfc8069b282ff7-e76015449a5247b988d15c64c273d43c , hdfs:// tsdb.test.com:54310/tsdb_full_backup_2013-12-09_00.24.03/.tmp/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/tsdb-uid=f9e5e554f111dc0679dfc8069b282ff7-e76015449a5247b988d15c64c273d43c , hdfs:// tsdb.test.com:54310/tsdb_full_backup_2013-12-09_00.24.03/.archive/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/tsdb-uid=f9e5e554f111dc0679dfc8069b282ff7-e76015449a5247b988d15c64c273d43c ] at org.apache.hadoop.hbase.io.FileLink.getFileStatus(FileLink.java:376) at org.apache.hadoop.hbase.snapshot.ExportSnapshot$1.storeFile(ExportSnapshot.java:390) at org.apache.hadoop.hbase.util.FSVisitor.visitRegionStoreFiles(FSVisitor.java:115) at org.apache.hadoop.hbase.util.FSVisitor.visitTableStoreFiles(FSVisitor.java:81) at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitTableStoreFiles(SnapshotReferenceUtil.java:116) at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitReferencedFiles(SnapshotReferenceUtil.java:101) at org.apache.hadoop.hbase.snapshot.ExportSnapshot.getSnapshotFiles(ExportSnapshot.java:385) at org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:633) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:705) at org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:709) Can someone help us to understand the reason to fail snapshot import. Is this issue similar to HBASE-8455? Thanks -OC
Re: HBase 0.94.10 snapshot FileNotFoundException exception
Thanks for confirming the issue.So, to overcome this issue we need to move from 0.94.10 to either 0.94.12 or above. Thanks -OC On Tue, Dec 10, 2013 at 4:09 PM, Matteo Bertozzi theo.berto...@gmail.comwrote: Hi, thank you for the follow up. The problem is related to a bug in the name resolution of a clone (that in your case the is the restore of a snapshot of a restored table). This problem was fixed as part of HBASE-8760, which should be integrated in 0.94.12 Matteo On Tue, Dec 10, 2013 at 10:00 AM, oc tsdb oc.t...@gmail.com wrote: Hi, Thanks for your help. More details are added below.Please let us know if any additional logs required. We have only one cluster (cluster-1) with 1 NN and with 4 DNs and HBase version is 0.94.10 We are using hbase shell to Create, Export snapshots. In short, what we are trying to do is: -Create snapshots for all HBase tables on cluster-1 and exporte to local file system. Here exported data contains archive+hbase-snapshot -Remove everything from Cluster-1 and importe all exported snapshots from local file system to same cluster-1. Working fine without any issues. -Again,after restoring snapshots, create snapshots for all HBase tables on cluster-1 and exporte to local file system Exported data contains only hbase-snapshot but not archive directory. As we don't have archive directory, snapshot restore is failing with FileNotFoundException. Basically snapshot export on restored snapshot data is not working in our case. Detailed Steps are given below: 1.Created snapshots for all HBase tables on cluster-1 snapshot 'tsdb', 'tsdb_snap_backup' snapshot 'tsdb-meta', 'tsdb-meta_snap_backup' snapshot 'tsdb-tree', 'tsdb-tree_snap_backup' snapshot 'tsdb-uid', 'tsdb-uid_snap_backup' 2.Exported above snapshots to same cluster(cluster-1) under hdfs root (/tsdb_full_backup_2013-12-09_23.32.19) hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot tsdb_snap_backup -copy-to /tsdb_full_backup_2013-12-09_23.32.19 -mappers 16; hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot tsdb-meta_snap_backup -copy-to /tsdb_full_backup_2013-12-09_23.32.19 -mappers 16; hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot tsdb-tree_snap_backup -copy-to /tsdb_full_backup_2013-12-09_23.32.19 -mappers 16; hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot tsdb-uid_snap_backup -copy-to /tsdb_full_backup_2013-12-09_23.32.19 -mappers 16; 3. Copied /tsdb_full_backup_2013-12-09_23.32.19 to local file system /opt/airwave/lib/hadoop/bin/hadoop fs -get /tsdb_full_backup_2013-12-09_23.32.19 /tmp/ We could see archive+hbase-snapshot directories copied to local file system /tmp/tsdb_full_backup_2013-12-09_23.32.19 4. Formatted cluster-1 Remove everything from Cluster-1.Now its fresh cluster. 5. copied /tmp/tsdb_full_backup_2013-12-09_23.32.19/ to hdfs path (root dir) /opt/airwave/lib/hadoop/bin/hadoop fs -put /tmp/tsdb_full_backup_2013-12-09_23.32.19 / 6.Import snapshots from hdfs path /tsdb_full_backup_2013-12-09_23.32.19 hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -D hbase.rootdir=hdfs:/tsdb_full_backup_2013-12-09_23.32.19 -snapshot tsdb_snap_backup -copy-to hdfs:/hbase hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -D hbase.rootdir=hdfs:/tsdb_full_backup_2013-12-09_23.32.19 -snapshot tsdb-meta_snap_backup -copy-to hdfs:/hbase hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -D hbase.rootdir=hdfs:/tsdb_full_backup_2013-12-09_23.32.19 -snapshot tsdb-tree_snap_backup -copy-to hdfs:/hbase hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -D hbase.rootdir=hdfs:/tsdb_full_backup_2013-12-09_23.32.19 -snapshot tsdb-uid_snap_backup -copy-to hdfs:/hbase 7.disable all HBase tables disable 'tsdb' disable'tsdb-meta' disable 'tsdb-tree' disable 'tsdb-uid' 8.restore all HBase tables from snapshot restore_snapshot 'tsdb' restore_snapshot 'tsdb-meta' restore_snapshot 'tsdb-tree' restore_snapshot 'tsdb-uid' 9.enable all HBase tables enable 'tsdb' enable'tsdb-meta' enable 'tsdb-tree' enable 'tsdb-uid' 10.Deleted all snapshots 11.Verified HBase data is imported properly or not. data is imported properly.No issues found. 12.Now again repeat step 1 to step 11 on imported data. In this case hbase-snapshot directory only exported to local file system (eg : /tmp/tsdb_full_backup_2013-12-09_23.42.19) But we dont see archive directory in local file system (eg : /tmp/tsdb_full_backup_2013-12-09_23.42.19) why? Thanks -OC On Mon, Dec 9, 2013 at 5:08 PM, Matteo Bertozzi theo.berto...@gmail.com wrote: can you clarify your steps, maybe adding the commands that you used? Which hbase version are you using? (HBASE-8455 doesn't seems to be related from the log) I assume: 1
HBase 0.94.10 snapshot FileNotFoundException exception
Hi, we are taking tsdb data backup using HBase snapshots. We tested the scenario as mentioned below: 1. Created snapshots and exported all snapshots successfully. Exported data contains archive+hbase-snapshot 2. Started HBase from scratch (removed all Hbase tabled and created again) 3. Successfully imported all snapshots created in step 1. 4. Again Created snapshots on imported data (step 3) and exported all snapshots. Exported data contains only hbase-snapshot but not archive.Why? 5. Now tried to import snapshots created in step 4 but we are getting below exception. Exception in thread main java.io.FileNotFoundException: Unable to open link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs:// tsdb.test.com:54310/tsdb_full_backup_2013-12-09_00.24.03/tsdb/1a3e8367302757a653efffbbde3b462b/t/tsdb=1a3e8367302757a653efffbbde3b462b-e265daa7b19d4911a5196c6f56a4162e, hdfs:// tsdb.test.com:54310/tsdb_full_backup_2013-12-09_00.24.03/.tmp/tsdb/1a3e8367302757a653efffbbde3b462b/t/tsdb=1a3e8367302757a653efffbbde3b462b-e265daa7b19d4911a5196c6f56a4162e, hdfs:// tsdb.test.com:54310/tsdb_full_backup_2013-12-09_00.24.03/.archive/tsdb/1a3e8367302757a653efffbbde3b462b/t/tsdb=1a3e8367302757a653efffbbde3b462b-e265daa7b19d4911a5196c6f56a4162e ] at org.apache.hadoop.hbase.io.FileLink.getFileStatus(FileLink.java:376) at org.apache.hadoop.hbase.snapshot.ExportSnapshot$1.storeFile(ExportSnapshot.java:390) at org.apache.hadoop.hbase.util.FSVisitor.visitRegionStoreFiles(FSVisitor.java:115) at org.apache.hadoop.hbase.util.FSVisitor.visitTableStoreFiles(FSVisitor.java:81) at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitTableStoreFiles(SnapshotReferenceUtil.java:116) at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitReferencedFiles(SnapshotReferenceUtil.java:101) at org.apache.hadoop.hbase.snapshot.ExportSnapshot.getSnapshotFiles(ExportSnapshot.java:385) at org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:633) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:705) at org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:709) Exception in thread main java.io.FileNotFoundException: Unable to open link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs:// tsdb.test.com:54310/tsdb_full_backup_2013-12-09_00.24.03/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/tsdb-uid=f9e5e554f111dc0679dfc8069b282ff7-e76015449a5247b988d15c64c273d43c, hdfs:// tsdb.test.com:54310/tsdb_full_backup_2013-12-09_00.24.03/.tmp/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/tsdb-uid=f9e5e554f111dc0679dfc8069b282ff7-e76015449a5247b988d15c64c273d43c, hdfs:// tsdb.test.com:54310/tsdb_full_backup_2013-12-09_00.24.03/.archive/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/tsdb-uid=f9e5e554f111dc0679dfc8069b282ff7-e76015449a5247b988d15c64c273d43c ] at org.apache.hadoop.hbase.io.FileLink.getFileStatus(FileLink.java:376) at org.apache.hadoop.hbase.snapshot.ExportSnapshot$1.storeFile(ExportSnapshot.java:390) at org.apache.hadoop.hbase.util.FSVisitor.visitRegionStoreFiles(FSVisitor.java:115) at org.apache.hadoop.hbase.util.FSVisitor.visitTableStoreFiles(FSVisitor.java:81) at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitTableStoreFiles(SnapshotReferenceUtil.java:116) at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitReferencedFiles(SnapshotReferenceUtil.java:101) at org.apache.hadoop.hbase.snapshot.ExportSnapshot.getSnapshotFiles(ExportSnapshot.java:385) at org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:633) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:705) at org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:709) Can someone help us to understand the reason to fail snapshot import. Is this issue similar to HBASE-8455? Thanks -OC
HBase snapshot export - mappers
Hi, While exporting HBase snapshots we need to specify number of mappers to use as mentioned below.To get better performance how many mappers can be used and please let us know based on which parameters we need to decide on number of mappers to use.I am new to HBase. Please refer if there are good link on this. $ bin/hbase class org.apache.hadoop.hbase.snapshot.tool.ExportSnapshot -snapshot MySnapshot -copy-to hdfs:///srv2:8082/hbase -mappers 16 Thanks -OC
Re: HBase ExportSnapshot
I think that error is because of missing .archive files? We only see this issue when we export snapshot to local file system (file:///tmp/hbase_backup) but not when we export snapshot to hdfs. When we export snapshot to hdfs we are getting all the data.Its only not working when we export snapshot data to local file system directly? On Tue, Dec 3, 2013 at 5:39 PM, Ted Yu yuzhih...@gmail.com wrote: Please check what happened to the HFileLink mentioned in the exception below - this would be the cause for snapshot export failure: Exception in thread main java.io.FileNotFoundException: Unable to open link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs:// site.com:54310/data_full_backup_2013-12-02_21.49.20/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/ed071cd010534856adc4be997498d645 , hdfs:// site.com:54310/data_full_backup_2013-12-02_21.49.20/.tmp/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/ed071cd010534856adc4be997498d645 , hdfs:// site.com:54310/data_full_backup_2013-12-02_21.49.20/.archive/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/ed071cd010534856adc4be997498d645 ] at org.apache.hadoop.hbase.io.FileLink.getFileStatus(FileLink.java:376) On Mon, Dec 2, 2013 at 10:01 PM, oc tsdb oc.t...@gmail.com wrote: here is snapshot export logs. mastre log: === 2013-12-02 21:54:30,840 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing because balanced cluster; servers=1 regions=1 average=1.0 mostloaded=1 leastloaded=1 2013-12-02 21:54:30,841 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing because balanced cluster; servers=1 regions=1 average=1.0 mostloaded=1 leastloaded=1 2013-12-02 21:54:30,841 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing because balanced cluster; servers=1 regions=1 average=1.0 mostloaded=1 leastloaded=1 2013-12-02 21:54:30,841 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing because balanced cluster; servers=1 regions=1 average=1.0 mostloaded=1 leastloaded=1 snapshot export console log: = 2013-12-02 21:54:30,841 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing because balanced cluster; servers=1 regions=1 average=1.0 mostloaded=1 leastloaded=1 2013-12-02 21:54:30,841 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing because balanced cluster; servers=1 regions=1 average=1.0 mostloaded=1 leastloaded=1 at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitReferencedFiles(SnapshotReferenceUtil.java:101) at org.apache.hadoop.hbase.snapshot.ExportSnapshot.getSnapshotFiles(ExportSnapshot.java:385) at org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:633) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:705) at org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:709) 13/12/02 21:54:24 INFO util.FSVisitor: No families under region directory:hdfs:// site.com:54310/data_full_backup_2013-12-02_21.49.20/.hbase-snapshot/tsdb-meta_snap_backup/f06335933b32019c4369f95001d996fb 13/12/02 21:54:24 INFO util.FSVisitor: No logs under directory:hdfs:// site.com:54310/data_full_backup_2013-12-02_21.49.20/.hbase-snapshot/tsdb-meta_snap_backup/.logs 13/12/02 21:54:24 WARN snapshot.ExportSnapshot: There are 0 store file to be copied. There may be no data in the table. 13/12/02 21:54:25 INFO util.FSVisitor: No families under region directory:hdfs:// site.com:54310/data_full_backup_2013-12-02_21.49.20/.hbase-snapshot/tsdb-tree_snap_backup/c40c34c4312ccb3302fbaf62caa91b9c 13/12/02 21:54:25 INFO util.FSVisitor: No logs under directory:hdfs:// site.com:54310/data_full_backup_2013-12-02_21.49.20/.hbase-snapshot/tsdb-tree_snap_backup/.logs 13/12/02 21:54:25 WARN snapshot.ExportSnapshot: There are 0 store file to be copied. There may be no data in the table. Exception in thread main java.io.FileNotFoundException: Unable to open link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs:// site.com:54310/data_full_backup_2013-12-02_21.49.20/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/ed071cd010534856adc4be997498d645 , hdfs:// site.com:54310/data_full_backup_2013-12-02_21.49.20/.tmp/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/ed071cd010534856adc4be997498d645 , hdfs:// site.com:54310/data_full_backup_2013-12-02_21.49.20/.archive/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/ed071cd010534856adc4be997498d645 ] at org.apache.hadoop.hbase.io.FileLink.getFileStatus(FileLink.java:376) at org.apache.hadoop.hbase.snapshot.ExportSnapshot$1.storeFile(ExportSnapshot.java:390
Incremental backup of hbase with export not working
Hi , In order to take incremental backup using export of hbase , we followed http://hbase.apache.org/book/ops_mgt.html#import Few things that need clarification are : 1. what does version mean? is is the same version number which we give during creation of hbase table ? 2. what if we don't specify the version and just specify start and end time stamps ? kindly provide us an example how to take incremental hbase backup using exportin an interval. We did some experiments with version and start time combination and results are as follows 1. we created a table with version=1 and tested the import CLI using the same version (version =1 ) and start-end times .even though the data is present between start and end intervals , we didn't get any data. 2. without specifying the version , we got all the data irrespective of the start and end times Kindly clarify us how to specify the version , time stamp range to match our requirements . Thanks, Oc.tsdb
Re: HBase ExportSnapshot
We see same logs for both options 013-12-02 09:47:41,311 INFO org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler: Running FLUSH table snapshot tsdb_snap_backup C_M_SNAPSHOT_TABLE on table tsdb 2013-12-02 09:47:41,312 INFO org.apache.hadoop.hbase.util.FSUtils: FileSystem doesn't support getDefaultReplication 2013-12-02 09:47:41,312 INFO org.apache.hadoop.hbase.util.FSUtils: FileSystem doesn't support getDefaultBlockSize 2013-12-02 09:47:41,337 INFO org.apache.hadoop.hbase.procedure.Procedure: Starting procedure 'tsdb_snap_backup' 2013-12-02 09:47:41,724 INFO org.apache.hadoop.hbase.procedure.Procedure: Procedure 'tsdb_snap_backup' execution completed 2013-12-02 09:47:41,724 INFO org.apache.hadoop.hbase.procedure.ZKProcedureUtil: Clearing all znodes for procedure tsdb_snap_backupincluding nodes /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2013-12-02 09:47:41,730 INFO org.apache.hadoop.hbase.master.snapshot.EnabledTableSnapshotHandler: Done waiting - snapshot for tsdb_snap_backup finished! It seems we can't export complete snapshot data directly to local file system using 'ExportSnapshot' command. If we want to copy to outside of cluster first we need to export it to hdfs and then use hadoop get command to copy to local file system. Is this correct? What is the difference between below two commands? hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot hbase_tbl_snapshot_name -copy-to file:///tmp/hbase_backup -mappers 16; hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot hbase_tbl_snapshot_name -copy-to hdfs:/hbase_backup -mappers 16; Thanks -OC On Mon, Dec 2, 2013 at 10:56 PM, Ted Yu yuzhih...@gmail.com wrote: Can you pastebin master log during operation #2 ? There have been at least two fixes since 0.94.10, listed below. It would be nice if you can verify this behavior using 0.94.14 Cheers r1515967 | mbertozzi | 2013-08-20 13:49:38 -0700 (Tue, 20 Aug 2013) | 1 line HBASE-8760 possible loss of data in snapshot taken after region split r1507792 | mbertozzi | 2013-07-28 05:17:39 -0700 (Sun, 28 Jul 2013) | 1 line HBASE-9060 ExportSnapshot job fails if target path contains percentage character (Jerry He) On Mon, Dec 2, 2013 at 9:19 AM, oc tsdb oc.t...@gmail.com wrote: Hi, We have cluster with 4 data nodes and HBase version is 0.94.10. We have created snapshot for all hbase tables and trying to export snapshot in two ways. option 1.Export snapshot into same cluster hdfs hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot hbase_tbl_snapshot_name -copy-to *hdfs:/hbase_backup *-mappers 16; Here we are getting full data ( .archive + .hbase-snapshot) exported to hdfs:/hbase_backup option 2.Export snapshot to local filesystem command : hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot hbase_tbl_snapshot_name -copy-to *file:///tmp/hbase_backup* -mappers 16; But with option 2 we only getting .hbase-snapshot exported to local dir (/tmp/hbase_backup) but .archive files are not exported.It is expected behavior or something wrong in option 2. Thanks OC
Re: HBase ExportSnapshot
: FileSystem outputFs = FileSystem.get(outputRoot.toUri(), conf); For 'file:///tmp/hbase_backup' argument, outputFs would be an instance of org.apache.hadoop.fs.LocalFileSystem. Cheers On Mon, Dec 2, 2013 at 9:06 PM, oc tsdb oc.t...@gmail.com wrote: We see same logs for both options 013-12-02 09:47:41,311 INFO org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler: Running FLUSH table snapshot tsdb_snap_backup C_M_SNAPSHOT_TABLE on table tsdb 2013-12-02 09:47:41,312 INFO org.apache.hadoop.hbase.util.FSUtils: FileSystem doesn't support getDefaultReplication 2013-12-02 09:47:41,312 INFO org.apache.hadoop.hbase.util.FSUtils: FileSystem doesn't support getDefaultBlockSize 2013-12-02 09:47:41,337 INFO org.apache.hadoop.hbase.procedure.Procedure: Starting procedure 'tsdb_snap_backup' 2013-12-02 09:47:41,724 INFO org.apache.hadoop.hbase.procedure.Procedure: Procedure 'tsdb_snap_backup' execution completed 2013-12-02 09:47:41,724 INFO org.apache.hadoop.hbase.procedure.ZKProcedureUtil: Clearing all znodes for procedure tsdb_snap_backupincluding nodes /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2013-12-02 09:47:41,730 INFO org.apache.hadoop.hbase.master.snapshot.EnabledTableSnapshotHandler: Done waiting - snapshot for tsdb_snap_backup finished! It seems we can't export complete snapshot data directly to local file system using 'ExportSnapshot' command. If we want to copy to outside of cluster first we need to export it to hdfs and then use hadoop get command to copy to local file system. Is this correct? What is the difference between below two commands? hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot hbase_tbl_snapshot_name -copy-to file:///tmp/hbase_backup -mappers 16; hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot hbase_tbl_snapshot_name -copy-to hdfs:/hbase_backup -mappers 16; Thanks -OC On Mon, Dec 2, 2013 at 10:56 PM, Ted Yu yuzhih...@gmail.com wrote: Can you pastebin master log during operation #2 ? There have been at least two fixes since 0.94.10, listed below. It would be nice if you can verify this behavior using 0.94.14 Cheers r1515967 | mbertozzi | 2013-08-20 13:49:38 -0700 (Tue, 20 Aug 2013) | 1 line HBASE-8760 possible loss of data in snapshot taken after region split r1507792 | mbertozzi | 2013-07-28 05:17:39 -0700 (Sun, 28 Jul 2013) | 1 line HBASE-9060 ExportSnapshot job fails if target path contains percentage character (Jerry He) On Mon, Dec 2, 2013 at 9:19 AM, oc tsdb oc.t...@gmail.com wrote: Hi, We have cluster with 4 data nodes and HBase version is 0.94.10. We have created snapshot for all hbase tables and trying to export snapshot in two ways. option 1.Export snapshot into same cluster hdfs hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot hbase_tbl_snapshot_name -copy-to *hdfs:/hbase_backup *-mappers 16; Here we are getting full data ( .archive + .hbase-snapshot) exported to hdfs:/hbase_backup option 2.Export snapshot to local filesystem command : hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot hbase_tbl_snapshot_name -copy-to *file:///tmp/hbase_backup* -mappers 16; But with option 2 we only getting .hbase-snapshot exported to local dir (/tmp/hbase_backup) but .archive files are not exported.It is expected behavior or something wrong in option 2. Thanks OC
Re: snapshotexport to outside of cluster
Hi, Can you clarify what 'without copying to local file system' means ? We want to take HBase tables backup.So we periodically take snapshots and export data to backup server which is not in cluster. Currently we are achieving this as mentioned below. 1.Create snapshots from HBase command line lets say we have created snapshot for a table called 'hbase_tabel' and its snapshot name is 'hbase_tabel_snapshot' 2. Then export snapshot data to remote backup server which is not in cluster.We are achieving this in two steps. a. Export snapshot to local file system(sample path /tmp/hbase_backup) from hdfs file system as mentioned below hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot hbase_tabel_snapshot -copy-to file:///tmp/hbase_backup - mappers 16 Here we should have enough disc space in local file system to export snapshot from hdfs. b.Transfer(scp) exported snapshot from local file system (/tmp/hbase_backup) to backup server which is not hadoop/HBase running server. That means first we have to export snapshot to local system from hdfs and then transfer ( scp ) to remote server.Instead of doing in two steps can we directly export snapshot to remote server(which is not in cluster) like how it is exporting to another cluster. . Thanks OC On Sat, Nov 30, 2013 at 11:11 AM, Ted Yu yuzhih...@gmail.com wrote: HBASE-4618 is not related to snapshots. Lars did some work in HBASE-5509, a sub-task of HBASE-4618. FYI On Fri, Nov 29, 2013 at 9:31 PM, Yu Li car...@gmail.com wrote: I guess this might be a use case of HBASE-7912https://issues.apache.org/jira/browse/HBASE-7912or HBASE-4618 https://issues.apache.org/jira/browse/HBASE-4618? Maybe some backup/restore tools taking use of snapshot would be nice to have? On 30 November 2013 12:58, Ted Yu yuzhih...@gmail.com wrote: The remote server is in different cluster, right ? Looking at ExportSnapshot#ExportMapper#copyFile(), it directly copies data to outputPath which is in the other cluster. Can you clarify what 'without copying to local file system' means ? Cheers On Fri, Nov 29, 2013 at 8:49 PM, oc tsdb oc.t...@gmail.com wrote: Hi, thanks for your replay. We are able to export snapshot to another cluster But this is not what we are looking for. We want to export snapshot to remote server without copying to local file system. Thanks OC On Fri, Nov 29, 2013 at 8:52 PM, Ted Yu yuzhih...@gmail.com wrote: Here is a sample command exporting to another cluster: hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot gs_1383068263 -copy-to hdfs://experiment-1d:8020/hbase -mappers 6 On Fri, Nov 29, 2013 at 4:02 AM, oc tsdb oc.t...@gmail.com wrote: Hi, can someone please let us know if we can export snapshots data directly to outside of cluster without copying to local. Please refer us if there are any blog with example. Below command is copying data to local file system under /tmp. hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot mytable -copy-to file:///tmp/hbase_backup -mappers 16 Without copying to local system, we want to export snapshot(eg scp) data to a server which is not part of any cluster but its reachable from cluster. Thanks OC -- Best Regards, Li Yu
snapshotexport to outside of cluster
Hi, can someone please let us know if we can export snapshots data directly to outside of cluster without copying to local. Please refer us if there are any blog with example. Below command is copying data to local file system under /tmp. hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot mytable -copy-to file:///tmp/hbase_backup -mappers 16 Without copying to local system, we want to export snapshot(eg scp) data to a server which is not part of any cluster but its reachable from cluster. Thanks OC
Re: snapshotexport to outside of cluster
Hi, thanks for your replay. We are able to export snapshot to another cluster But this is not what we are looking for. We want to export snapshot to remote server without copying to local file system. Thanks OC On Fri, Nov 29, 2013 at 8:52 PM, Ted Yu yuzhih...@gmail.com wrote: Here is a sample command exporting to another cluster: hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot gs_1383068263 -copy-to hdfs://experiment-1d:8020/hbase -mappers 6 On Fri, Nov 29, 2013 at 4:02 AM, oc tsdb oc.t...@gmail.com wrote: Hi, can someone please let us know if we can export snapshots data directly to outside of cluster without copying to local. Please refer us if there are any blog with example. Below command is copying data to local file system under /tmp. hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot mytable -copy-to file:///tmp/hbase_backup -mappers 16 Without copying to local system, we want to export snapshot(eg scp) data to a server which is not part of any cluster but its reachable from cluster. Thanks OC
HBase snapshots
Hi, We are using HBase 0.94.10 version. We could able to take snapshots for all hbase tables and export into another cluster successfully. To take HBase table snapshots we followed procedure given at below apache link. http://hbase.apache.org/book/ops.snapshots.html Purpose of taking snapshots is to take cluster backup. Can someone please help us on below queries. 1.Is there a way to take incremental snapshots on HBase tables? 2.Can we export snapshots to outside of HDFS(e.g backup server)? 3.Is it possible to take a snapshot for all tables in one command? 4.Does HBase support to take automatic snapshots based on configured timeout? Please also let us know if there is any better way to take HBase tables backup. Thanks in advance. -OC
Re: HBase snapshots
Hi Matteo, Thanks a lot for your quick and detailed explanation. -OC On Tue, Nov 19, 2013 at 10:43 PM, Matteo Bertozzi theo.berto...@gmail.comwrote: 1.Is there a way to take incremental snapshots on HBase tables? Due to the nature of hbase, when hfiles gets compacted your table will not share files with the snapshot causing the next snapshot to don't share hfiles (or share a few) with the previous snapshot. The other approach will be using logs to have an incremental snapshots but logs contains different tables, so you may end up with more space used, and for sure more time to restore since you must replay the logs. 2.Can we export snapshots to outside of HDFS(e.g backup server)? the ExportSnapshot tool takes a destination path as argument, so as long as the destination filesystem is supported by the FileSystem class you can. (e.g. you can export with -copy-to=file:///mnt/backup-mount-point/hbase) 3.Is it possible to take a snapshot for all tables in one command? No, but you can do a simple script that does a for each table in admin.listTable(), admin.snapshot(table, snapshotName) 4.Does HBase support to take automatic snapshots based on configured timeout? No, hbase doesn't do job scheduling. you may use a cron job to execute a shell command, echo snapshot 'tableName', 'snapshotName' | hbase shell Please also let us know if there is any better way to take HBase tables backup. There are different solutions that you can use: Replication, CopyTable, HDFS files copy + CopyTable and maybe others it depends on what you're looking for. There is an article here that explains how to use CopyTable as backup tool: http://blog.cloudera.com/blog/2012/06/online-hbase-backups-with-copytable-2/
is jdk required to run hbase or jre alone is sufficient
HI , I would like to know if JRE alone is sufficient to run HBASE services or JDK is required ? we are planning to install latest stable version of hbase Thanks, Oc.tsdb
How to recover data from hadoop/hbase cluster
Hi, I am running a hadoop/hbase cluster with 6 nodes (1 name node, 5 data nodes). Two days back two of the machines went down due to some kernel crash. These machines were rebooted after some time. Meanwhile the cluster became unresponsive. Today, to create disk space and let other application services continue in cluster, we had to wipe out hadoop data directory on one of the data nodes, thinking that we should still be able to recover cluster as the replication count was 3. Could you please point me to any procedure to recover data in my cluster. Thanks in advance. Regards, OC.
getting splitmanager debug logs continuously
Hi, I am running a cluster with 6 nodes; Two of 6 nodes in my cluster went down (due to other application failure) and came back after some time (had to do a power reboot). When these nodes are back I use to get WARN org.apache.hadoop.DFSClient: Failed to connect to , add to deadnodes and continue. Now these messages are stopped and getting continuous debug message as follows. 2013-08-08 12:57:36,628 DEBUG org.apache.hadoop.hbase. master.SplitLogManager: total tasks = 14 unassigned = 14 2013-08-08 12:57:37,628 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 14 unassigned = 14 2013-08-08 12:57:37,629 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired /hbase/splitlog/hdfs%3A%2F%2Fmb-1.corp.oc.com%3A54310%2Fhbase%2F.logs% 2Fmb-3.corp.oc.com%2C60020%2C1375466447768-splitting%2Fmb-3.corp.oc.com%252C60020%252C1375466447768.1375631802971 ver = 0 2013-08-08 12:57:37,629 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired /hbase/splitlog/hdfs%3A%2F%2Fmb-1.corp.oc.com%3A54310%2Fhbase%2F.logs% 2Fmb-6.corp.oc.com%2C60020%2C1375466460755-splitting%2Fmb-6.corp.oc.com%252C60020%252C1375466460755.1375623787557 ver = 0 2013-08-08 12:57:37,629 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired /hbase/splitlog/hdfs%3A%2F%2Fmb-1.corp.oc.com%3A54310%2Fhbase%2F.logs% 2Fmb-6.corp.oc.com%2C60020%2C1375466460755-splitting%2Fmb-6.corp.oc.com%252C60020%252C1375466460755.1375619231059 ver = 3 2013-08-08 12:57:37,629 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired /hbase/splitlog/hdfs%3A%2F%2Fmb-1.corp.oc.com%3A54310%2Fhbase%2F.logs% 2Fmb-2.corp.oc.com%2C60020%2C1375466479427-splitting%2Fmb-2.corp.oc.com%252C60020%252C1375466479427.1375639017535 ver = 0 2013-08-08 12:57:37,629 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired /hbase/splitlog/hdfs%3A%2F%2Fmb-1.corp.oc.com%3A54310%2Fhbase%2F.logs% 2Fmb-6.corp.oc.com%2C60020%2C1375466460755-splitting%2Fmb-6.corp.oc.com%252C60020%252C1375466460755.1375623021175 ver = 0 2013-08-08 12:57:37,629 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired /hbase/splitlog/hdfs%3A%2F%2Fmb-1.corp.oc.com%3A54310%2Fhbase%2F.logs% 2Fmb-3.corp.oc.com%2C60020%2C1375466447768-splitting%2Fmb-3.corp.oc.com%252C60020%252C1375466447768.1375630425141 ver = 0 2013-08-08 12:57:37,629 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: resubmitting unassigned task(s) after timeout 2013-08-08 12:57:37,629 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired /hbase/splitlog/hdfs%3A%2F%2Fmb-1.corp.oc.com%3A54310%2Fhbase%2F.logs% 2Fmb-6.corp.oc.com%2C60020%2C1375466460755-splitting%2Fmb-6.corp.oc.com%252C60020%252C1375466460755.1375620714514 ver = 3 2013-08-08 12:57:37,630 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired /hbase/splitlog/hdfs%3A%2F%2Fmb-1.corp.oc.com%3A54310%2Fhbase%2F.logs% 2Fmb-6.corp.oc.com%2C60020%2C1375924525310-splitting%2Fmb-6.corp.oc.com%252C60020%252C1375924525310.1375924529658 ver = 0 2013-08-08 12:57:37,630 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired /hbase/splitlog/hdfs%3A%2F%2Fmb-1.corp.oc.com%3A54310%2Fhbase%2F.logs% 2Fmb-4.corp.oc.com%2C60020%2C1375466551673-splitting%2Fmb-4.corp.oc.com%252C60020%252C1375466551673.1375641592581 ver = 0 2013-08-08 12:57:37,630 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired /hbase/splitlog/hdfs%3A%2F%2Fmb-1.corp.oc.com%3A54310%2Fhbase%2F.logs% 2Fmb-5.corp.oc.com%2C60020%2C1375924528073-splitting%2Fmb-5.corp.oc.com%252C60020%252C1375924528073.1375924532442 ver = 0 2013-08-08 12:57:37,630 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired /hbase/splitlog/hdfs%3A%2F%2Fmb-1.corp.oc.com%3A54310%2Fhbase%2F.logs% 2Fmb-6.corp.oc.com%2C60020%2C1375466460755-splitting%2Fmb-6.corp.oc.com%252C60020%252C1375466460755.1375622290167 ver = 3 2013-08-08 12:57:37,630 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired /hbase/splitlog/hdfs%3A%2F%2Fmb-1.corp.oc.com%3A54310%2Fhbase%2F.logs% 2Fmb-5.corp.oc.com%2C60020%2C1375466463385-splitting%2Fmb-5.corp.oc.com%252C60020%252C1375466463385.1375638183425 ver = 0 2013-08-08 12:57:37,630 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired /hbase/splitlog/hdfs%3A%2F%2Fmb-1.corp.oc.com%3A54310%2Fhbase%2F.logs% 2Fmb-5.corp.oc.com%2C60020%2C1375466463385-splitting%2Fmb-5.corp.oc.com%252C60020%252C1375466463385.1375639599559 ver = 0 2013-08-08 12:57:37,630 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired /hbase/splitlog/hdfs%3A%2F%2Fmb-1.corp.oc.com%3A54310%2Fhbase%2F.logs% 2Fmb-5.corp.oc.com%2C60020%2C1375466463385-splitting%2Fmb-5.corp.oc.com%252C60020%252C1375466463385.1375641710787 ver = 3 2013-08-08 12:57:37,633 INFO org.apache.hadoop.hbase.master.SplitLogManager: task /hbase/splitlog/RESCAN006975 entered state done mb-1.corp.oc.com
Re: getting splitmanager debug logs continuously
all the nodes are running but master does not run region-server; master was limited to run nameNode, quorum, and HMaster functionality. you mean to run region server on Master node as well? On Thu, Aug 8, 2013 at 2:48 PM, Jimmy Xiang jxi...@cloudera.com wrote: Can you start the master as well (besides region servers)? On Thu, Aug 8, 2013 at 2:41 PM, oc tsdb oc.t...@gmail.com wrote: I am using hbase-0.92 Region server was not running on any of the nodes. Restarted the cluster. It started region server on all nodes except HMaster but still unresponsive. processes running on master are TSDMain HMaster SecondaryNameNode NameNode JobTracker HQuorumPeer processes running on all other nodes are DataNode TaskTracker RegionServer TSDMain This time, I see the error messages in the attached log. Could you please suggest if I can recover/restore the data and get the cluster up. Thanks Regards, VSR On Thu, Aug 8, 2013 at 1:40 PM, Ted Yu yuzhih...@gmail.com wrote: Can you tell us the version of HBase you're using ? Do you find something in region server logs on the 4 remaining nodes ? Cheers On Thu, Aug 8, 2013 at 1:36 PM, oc tsdb oc.t...@gmail.com wrote: Hi, I am running a cluster with 6 nodes; Two of 6 nodes in my cluster went down (due to other application failure) and came back after some time (had to do a power reboot). When these nodes are back I use to get WARN org.apache.hadoop.DFSClient: Failed to connect to , add to deadnodes and continue. Now these messages are stopped and getting continuous debug message as follows. 2013-08-08 12:57:36,628 DEBUG org.apache.hadoop.hbase. master.SplitLogManager: total tasks = 14 unassigned = 14 2013-08-08 12:57:37,628 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 14 unassigned = 14 2013-08-08 12:57:37,629 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired /hbase/splitlog/hdfs%3A%2F%2Fmb-1.corp.oc.com %3A54310%2Fhbase%2F.logs% 2Fmb-3.corp.oc.com%2C60020%2C1375466447768-splitting% 2Fmb-3.corp.oc.com %252C60020%252C1375466447768.1375631802971 ver = 0 2013-08-08 12:57:37,629 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired /hbase/splitlog/hdfs%3A%2F%2Fmb-1.corp.oc.com %3A54310%2Fhbase%2F.logs% 2Fmb-6.corp.oc.com%2C60020%2C1375466460755-splitting% 2Fmb-6.corp.oc.com %252C60020%252C1375466460755.1375623787557 ver = 0 2013-08-08 12:57:37,629 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired /hbase/splitlog/hdfs%3A%2F%2Fmb-1.corp.oc.com %3A54310%2Fhbase%2F.logs% 2Fmb-6.corp.oc.com%2C60020%2C1375466460755-splitting% 2Fmb-6.corp.oc.com %252C60020%252C1375466460755.1375619231059 ver = 3 2013-08-08 12:57:37,629 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired /hbase/splitlog/hdfs%3A%2F%2Fmb-1.corp.oc.com %3A54310%2Fhbase%2F.logs% 2Fmb-2.corp.oc.com%2C60020%2C1375466479427-splitting% 2Fmb-2.corp.oc.com %252C60020%252C1375466479427.1375639017535 ver = 0 2013-08-08 12:57:37,629 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired /hbase/splitlog/hdfs%3A%2F%2Fmb-1.corp.oc.com %3A54310%2Fhbase%2F.logs% 2Fmb-6.corp.oc.com%2C60020%2C1375466460755-splitting% 2Fmb-6.corp.oc.com %252C60020%252C1375466460755.1375623021175 ver = 0 2013-08-08 12:57:37,629 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired /hbase/splitlog/hdfs%3A%2F%2Fmb-1.corp.oc.com %3A54310%2Fhbase%2F.logs% 2Fmb-3.corp.oc.com%2C60020%2C1375466447768-splitting% 2Fmb-3.corp.oc.com %252C60020%252C1375466447768.1375630425141 ver = 0 2013-08-08 12:57:37,629 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: resubmitting unassigned task(s) after timeout 2013-08-08 12:57:37,629 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired /hbase/splitlog/hdfs%3A%2F%2Fmb-1.corp.oc.com %3A54310%2Fhbase%2F.logs% 2Fmb-6.corp.oc.com%2C60020%2C1375466460755-splitting% 2Fmb-6.corp.oc.com %252C60020%252C1375466460755.1375620714514 ver = 3 2013-08-08 12:57:37,630 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired /hbase/splitlog/hdfs%3A%2F%2Fmb-1.corp.oc.com %3A54310%2Fhbase%2F.logs% 2Fmb-6.corp.oc.com%2C60020%2C1375924525310-splitting% 2Fmb-6.corp.oc.com %252C60020%252C1375924525310.1375924529658 ver = 0 2013-08-08 12:57:37,630 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired /hbase/splitlog/hdfs%3A%2F%2Fmb-1.corp.oc.com %3A54310%2Fhbase%2F.logs% 2Fmb-4.corp.oc.com%2C60020%2C1375466551673-splitting% 2Fmb-4.corp.oc.com %252C60020%252C1375466551673.1375641592581 ver = 0 2013-08-08 12:57:37,630 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired /hbase
