Re: how to reduce hbase-regionserver connection retries to zookeeper
Hi,
Thanks for your response.
We are using Hbase 0.98.2.
You mean we need to add the following two properties in hbase-site.xml?
zookeeper.recovery.retry = 1
zookeeper.recovery.retry.intervalmill=1000
Thanks
oc.tsdb
On Tue, Jul 29, 2014 at 8:09 PM, Ted Yu wrote:
> You can adjust values for the following config parameters (values below are
> only for illustration):
>
> config.setInt("zookeeper.recovery.retry", 1);
> config.setInt("zookeeper.recovery.retry.intervalmill", 1000);
>
> What hbase release are you using ?
>
> Cheers
>
>
> On Tue, Jul 29, 2014 at 3:48 AM, oc tsdb wrote:
>
> > Hi,
> >
> > if the namenode (or zookeeper) is down for some reason,
> > hbase-regionserver(on datanode) keep retrying to connect to zookeeper
> > continuously for long time.
> >
> > We tried by setting "hbase.client.retries.number" to 1 in hbase-site.xml,
> > But no effect.
> >
> > Can someone please let us know, hbase configuration to stop
> > hbase-regionserver in 5 sec, in case, if zookeeper is not running.
> >
> > Thanks
> > oc.tsdb
> >
>
how to reduce hbase-regionserver connection retries to zookeeper
Hi, if the namenode (or zookeeper) is down for some reason, hbase-regionserver(on datanode) keep retrying to connect to zookeeper continuously for long time. We tried by setting "hbase.client.retries.number" to 1 in hbase-site.xml, But no effect. Can someone please let us know, hbase configuration to stop hbase-regionserver in 5 sec, in case, if zookeeper is not running. Thanks oc.tsdb
restoring old hbase backup in new hbase
Hi, We are planning to move our cluster from hbase 0.94.14 to 0.98.2. Our query is - If we have backup taken(using snapshots) on hbase 0.94.14, can we restore the same backup on newer version of hbase 0.98.2? Thanks oc.tsdb
incremental cluster backup using snapshots
Hi, Does new hbase version(>0.99) supports incremental backup using snapshots? If not supported in current releases, is it planned to add in future releases? can we export snapshots to local file system directly? Thanks oc.tsdb
Re: HBase 0.98 rpm
Hi, That means first we should setup bigtop work space and then build rpm? Can you please point me, if there are any links describing in details about building rpms from scratch? Thanks oc.tsdb On Wed, Jun 4, 2014 at 4:52 AM, Ted Yu wrote: > This is what I did: > clone bigtop master branch > under root of bigtop workspace, issue the following command: > > make hbase-rpm > > You should see this at the tail of output: > > + exit 0 > cp -r /homes/hortonzy/bigtop/build/hbase//rpm/RPMS/*/* > /homes/hortonzy/bigtop/output/hbase/ > touch /homes/hortonzy/bigtop/build/hbase//.rpm > > You can find rpm's under output/hbase/ > > Cheers > > > On Mon, Jun 2, 2014 at 10:52 PM, oc tsdb wrote: > > > Hi, > > > > To build rpm, we downloaded HBase 0.98.2 src from apache download link( > > > http://mirror.sdunix.com/apache/hbase/hbase-0.98.2/hbase-0.98.2-src.tar.gz > > ) > > and other files (including specs) from bigtop ( > > > > > https://github.com/apache/bigtop/tree/master/bigtop-packages/src/common/hbase > > ) > > > > But while building rpm we are getting the following error. > > *hbase/bigtop.bom: No such file or directory* > > > > i see following line in hbase/do-component-build > > *. `dirname $0`/bigtop.bom* > > > > Please let us know when do we get this error? and how to resolve. > > > > > > Thanks > > oc.tsdb > > > > > > > > On Tue, May 27, 2014 at 3:07 PM, Bharath Vissapragada < > > bhara...@cloudera.com > > > wrote: > > > > > On Tue, May 27, 2014 at 2:38 PM, oc tsdb wrote: > > > > > > > Hi , > > > > > > > > 1) Where can we find hbase.098 tar > > > > with hbase 0.98 directory structure similar to hbase-0.94.19 ? > > > > i,e .. > > > > > > > > Directory structure of hbase 0.98 > > > > # ls > > > > bin/ dev-support/ hbase-common/ hbase-hadoop2-compat/ > > > > hbase-prefix-tree/ hbase-shell/ LICENSE.txt README.txt > > > > CHANGES.txt hbase-assembly/ hbase-examples/ > hbase-hadoop-compat/ > > > > hbase-protocol/ hbase-testing-util/ NOTICE.txt src/ > > > > conf/ hbase-client/hbase-hadoop1-compat/ hbase-it/ > > > > hbase-server/ hbase-thrift/ pom.xml target/ > > > > > > > > where as directory structure of hbase-0.94.14 > > > > > > > > # ls ../hbase-0.94.14 > > > > bin/ CHANGES.txt conf/ docs/ hbase-0.94.14.jar > > > hbase-0.94.14-tests.jar > > > > hbase-webapps/ lib/ LICENSE.txt NOTICE.txt pom.xml README.txt > > > > sbin/ security/ > > > > src/ > > > > > > > > we downloaded hbase 0.98 and hbase 0.94 from > > > > > > > > > > > > > > http://www.webhostingjams.com/mirror/apache/hbase/hbase-0.98.2/hbase-0.98.2-src.tar.gzand > > > > > > > > > > > > > > > > > > http://www.webhostingjams.com/mirror/apache/hbase/stable/hbase-0.94.19.tar.gz > > > > > > > > respectively . > > > > > > > > > > This was done in 0.95 release to shift from traditional single src/ > > > directory to a multi-module layout as it makes more sense and easy to > > > manage. More discussion about it in this jira > > > https://issues.apache.org/jira/browse/HBASE-4336 > > > > > > > > > > > > > > > > > > 2) we are trying to build an rpm out of hbase 0.98 version , where > can > > i > > > > get the spec file or srpm for the same if available ? > > > > > > > > > > Does this work? Bigtop manages it now IIRC. > > > > > > > > > > > > https://github.com/apache/bigtop/blob/master/bigtop-packages/src/rpm/hbase/SPECS/hbase.spec > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > oc.tsdb > > > > > > > > > > > > > > > > -- > > > Bharath Vissapragada > > > <http://www.cloudera.com> > > > > > >
Re: HBase 0.98 rpm
Hi, To build rpm, we downloaded HBase 0.98.2 src from apache download link( http://mirror.sdunix.com/apache/hbase/hbase-0.98.2/hbase-0.98.2-src.tar.gz) and other files (including specs) from bigtop ( https://github.com/apache/bigtop/tree/master/bigtop-packages/src/common/hbase ) But while building rpm we are getting the following error. *hbase/bigtop.bom: No such file or directory* i see following line in hbase/do-component-build *. `dirname $0`/bigtop.bom* Please let us know when do we get this error? and how to resolve. Thanks oc.tsdb On Tue, May 27, 2014 at 3:07 PM, Bharath Vissapragada wrote: > On Tue, May 27, 2014 at 2:38 PM, oc tsdb wrote: > > > Hi , > > > > 1) Where can we find hbase.098 tar > > with hbase 0.98 directory structure similar to hbase-0.94.19 ? > > i,e .. > > > > Directory structure of hbase 0.98 > > # ls > > bin/ dev-support/ hbase-common/ hbase-hadoop2-compat/ > > hbase-prefix-tree/ hbase-shell/ LICENSE.txt README.txt > > CHANGES.txt hbase-assembly/ hbase-examples/ hbase-hadoop-compat/ > > hbase-protocol/ hbase-testing-util/ NOTICE.txt src/ > > conf/ hbase-client/hbase-hadoop1-compat/ hbase-it/ > > hbase-server/ hbase-thrift/ pom.xml target/ > > > > where as directory structure of hbase-0.94.14 > > > > # ls ../hbase-0.94.14 > > bin/ CHANGES.txt conf/ docs/ hbase-0.94.14.jar > hbase-0.94.14-tests.jar > > hbase-webapps/ lib/ LICENSE.txt NOTICE.txt pom.xml README.txt > > sbin/ security/ > > src/ > > > > we downloaded hbase 0.98 and hbase 0.94 from > > > > > http://www.webhostingjams.com/mirror/apache/hbase/hbase-0.98.2/hbase-0.98.2-src.tar.gzand > > > > > > > http://www.webhostingjams.com/mirror/apache/hbase/stable/hbase-0.94.19.tar.gz > > > > respectively . > > > > This was done in 0.95 release to shift from traditional single src/ > directory to a multi-module layout as it makes more sense and easy to > manage. More discussion about it in this jira > https://issues.apache.org/jira/browse/HBASE-4336 > > > > > > > > 2) we are trying to build an rpm out of hbase 0.98 version , where can i > > get the spec file or srpm for the same if available ? > > > > Does this work? Bigtop manages it now IIRC. > > > https://github.com/apache/bigtop/blob/master/bigtop-packages/src/rpm/hbase/SPECS/hbase.spec > > > > > > > > > Thanks > > oc.tsdb > > > > > > -- > Bharath Vissapragada > <http://www.cloudera.com> >
HBase 0.98 rpm
Hi , 1) Where can we find hbase.098 tar with hbase 0.98 directory structure similar to hbase-0.94.19 ? i,e .. Directory structure of hbase 0.98 # ls bin/ dev-support/ hbase-common/ hbase-hadoop2-compat/ hbase-prefix-tree/ hbase-shell/ LICENSE.txt README.txt CHANGES.txt hbase-assembly/ hbase-examples/ hbase-hadoop-compat/ hbase-protocol/ hbase-testing-util/ NOTICE.txt src/ conf/ hbase-client/hbase-hadoop1-compat/ hbase-it/ hbase-server/ hbase-thrift/ pom.xml target/ where as directory structure of hbase-0.94.14 # ls ../hbase-0.94.14 bin/ CHANGES.txt conf/ docs/ hbase-0.94.14.jar hbase-0.94.14-tests.jar hbase-webapps/ lib/ LICENSE.txt NOTICE.txt pom.xml README.txt sbin/ security/ src/ we downloaded hbase 0.98 and hbase 0.94 from http://www.webhostingjams.com/mirror/apache/hbase/hbase-0.98.2/hbase-0.98.2-src.tar.gzand http://www.webhostingjams.com/mirror/apache/hbase/stable/hbase-0.94.19.tar.gz respectively . 2) we are trying to build an rpm out of hbase 0.98 version , where can i get the spec file or srpm for the same if available ? Thanks oc.tsdb
How to remove duplicate data in HBase?
Hi all, We want to know if there is any option to remove duplicate data in Hbase based on column family dynamically? Thanks, OC
directly export hbase snapshots to local fs
Hi, We are using HBase 0.94.14 and have only one cluster with 1NN and 4 DNs. We are trying to export snapshot directly to local system(e.g local fs path: /tmp/hbase_backup) as specified below.It is just exporting/copying snapshots (.hbase-snapshot) but not actual data(.archive).Why the below command is not copying actual data? hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot hbase_tbl_snapshot_name -copy-to file:///tmp/hbase_backup -mappers 16; It is always first need to export to HDFS and then copy to local file system using hadoop get command? Thanks in advance. -OC
Re: HBase 0.94.10 snapshot FileNotFoundException exception
Thanks for confirming the issue.So, to overcome this issue we need to move from 0.94.10 to either 0.94.12 or above. Thanks -OC On Tue, Dec 10, 2013 at 4:09 PM, Matteo Bertozzi wrote: > Hi, thank you for the follow up. > > The problem is related to a bug in the name resolution of a "clone" (that > in your case the is the restore of a snapshot of a restored table). > This problem was fixed as part of HBASE-8760, which should be integrated in > 0.94.12 > > > Matteo > > > > On Tue, Dec 10, 2013 at 10:00 AM, oc tsdb wrote: > > > Hi, > > > > Thanks for your help. > > > > More details are added below.Please let us know if any additional logs > > required. > > > > We have only one cluster (cluster-1) with 1 NN and with 4 DNs and HBase > > version is 0.94.10 > > > > We are using hbase shell to Create, Export snapshots. > > > > In short, what we are trying to do is: > > > > -Create snapshots for all HBase tables on cluster-1 and exporte to local > > file system. > >Here exported data contains archive+hbase-snapshot > > -Remove everything from Cluster-1 and importe all exported snapshots from > > local file system to same cluster-1. > > Working fine without any issues. > > -Again,after restoring snapshots, create snapshots for all HBase tables > on > > cluster-1 and exporte to local file system > > Exported data contains only hbase-snapshot but not archive directory. > > As we don't have archive directory, snapshot restore is failing with > > FileNotFoundException. > > > > Basically snapshot export on restored snapshot data is not working in our > > case. > > > > Detailed Steps are given below: > > > > 1.Created snapshots for all HBase tables on cluster-1 > > > > snapshot 'tsdb', 'tsdb_snap_backup' > > snapshot 'tsdb-meta', 'tsdb-meta_snap_backup' > > snapshot 'tsdb-tree', 'tsdb-tree_snap_backup' > > snapshot 'tsdb-uid', 'tsdb-uid_snap_backup' > > > > 2.Exported above snapshots to same cluster(cluster-1) under hdfs root > > (/tsdb_full_backup_2013-12-09_23.32.19) > > > > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot > > tsdb_snap_backup -copy-to /tsdb_full_backup_2013-12-09_23.32.19 -mappers > > 16; > > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot > > tsdb-meta_snap_backup -copy-to /tsdb_full_backup_2013-12-09_23.32.19 > > -mappers 16; > > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot > > tsdb-tree_snap_backup -copy-to /tsdb_full_backup_2013-12-09_23.32.19 > > -mappers 16; > > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot > > tsdb-uid_snap_backup -copy-to /tsdb_full_backup_2013-12-09_23.32.19 > > -mappers 16; > > > > > > 3. Copied /tsdb_full_backup_2013-12-09_23.32.19 to local file system > > > > /opt/airwave/lib/hadoop/bin/hadoop fs -get > > /tsdb_full_backup_2013-12-09_23.32.19 /tmp/ > > > > We could see archive+hbase-snapshot directories copied to local file > system > > /tmp/tsdb_full_backup_2013-12-09_23.32.19 > > > > 4. Formatted cluster-1 > > Remove everything from Cluster-1.Now its fresh cluster. > > > > 5. copied /tmp/tsdb_full_backup_2013-12-09_23.32.19/ to hdfs path (root > > dir) > > > > /opt/airwave/lib/hadoop/bin/hadoop fs -put > > /tmp/tsdb_full_backup_2013-12-09_23.32.19 / > > > > 6.Import snapshots from hdfs path /tsdb_full_backup_2013-12-09_23.32.19 > > > > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -D > > hbase.rootdir=hdfs:/tsdb_full_backup_2013-12-09_23.32.19 -snapshot > > tsdb_snap_backup -copy-to hdfs:/hbase > > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -D > > hbase.rootdir=hdfs:/tsdb_full_backup_2013-12-09_23.32.19 -snapshot > > tsdb-meta_snap_backup -copy-to hdfs:/hbase > > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -D > > hbase.rootdir=hdfs:/tsdb_full_backup_2013-12-09_23.32.19 -snapshot > > tsdb-tree_snap_backup -copy-to hdfs:/hbase > > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -D > > hbase.rootdir=hdfs:/tsdb_full_backup_2013-12-09_23.32.19 -snapshot > > tsdb-uid_snap_backup -copy-to hdfs:/hbase > > > > 7.disable all HBase tables > > disable 'tsdb' > > disable'tsdb-meta' > > disable 'tsdb-tree' > > disable 'tsdb-uid' > > > > 8.restore all HBase tables from snapshot > > > > > > restore_snapshot '
Re: HBase 0.94.10 snapshot FileNotFoundException exception
.. drop > 'tableName' .. delete_snapshot 'snapshotName' > 3) Import snapshot created in Step 1 means > - from a shell in Cluster-2 export snapshot to Cluster-1: hbase > org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot snapshotName ... > - from a shell on Cluster-1 verify snapshot with: hbase > org.apache.hadoop.hbase.snapshot.SnapshotInfo -snapshot snapshotName > - from a shell on Cluster-1: clone_snapshot 'snapshotName', 'tableName' > 4 and 5 I've no idea what you mean > > Matteo > > > > On Mon, Dec 9, 2013 at 11:25 AM, oc tsdb wrote: > > > Hi, > > > > we are taking tsdb data backup using HBase snapshots. > > > > We tested the scenario as mentioned below: > > > > 1. Created snapshots and exported all snapshots successfully. > > Exported data contains archive+hbase-snapshot > > 2. Started HBase from scratch (removed all Hbase tabled and created > again) > > 3. Successfully imported all snapshots created in step 1. > > 4. Again Created snapshots on imported data (step 3) and exported all > > snapshots. > > Exported data contains only hbase-snapshot but not archive.Why? > > 5. Now tried to import snapshots created in step 4 but we are getting > below > > exception. > > > > Exception in thread "main" java.io.FileNotFoundException: Unable to open > > link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs:// > > > > > tsdb.test.com:54310/tsdb_full_backup_2013-12-09_00.24.03/tsdb/1a3e8367302757a653efffbbde3b462b/t/tsdb=1a3e8367302757a653efffbbde3b462b-e265daa7b19d4911a5196c6f56a4162e > > , > > hdfs:// > > > > > tsdb.test.com:54310/tsdb_full_backup_2013-12-09_00.24.03/.tmp/tsdb/1a3e8367302757a653efffbbde3b462b/t/tsdb=1a3e8367302757a653efffbbde3b462b-e265daa7b19d4911a5196c6f56a4162e > > , > > hdfs:// > > > > > tsdb.test.com:54310/tsdb_full_backup_2013-12-09_00.24.03/.archive/tsdb/1a3e8367302757a653efffbbde3b462b/t/tsdb=1a3e8367302757a653efffbbde3b462b-e265daa7b19d4911a5196c6f56a4162e > > ] > > at org.apache.hadoop.hbase.io.FileLink.getFileStatus(FileLink.java:376) > > at > > > > > org.apache.hadoop.hbase.snapshot.ExportSnapshot$1.storeFile(ExportSnapshot.java:390) > > at > > > > > org.apache.hadoop.hbase.util.FSVisitor.visitRegionStoreFiles(FSVisitor.java:115) > > at > > > > > org.apache.hadoop.hbase.util.FSVisitor.visitTableStoreFiles(FSVisitor.java:81) > > at > > > > > org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitTableStoreFiles(SnapshotReferenceUtil.java:116) > > at > > > > > org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitReferencedFiles(SnapshotReferenceUtil.java:101) > > at > > > > > org.apache.hadoop.hbase.snapshot.ExportSnapshot.getSnapshotFiles(ExportSnapshot.java:385) > > at > > > > > org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:633) > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) > > at > > > > > org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:705) > > at > > > > > org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:709) > > > > > > Exception in thread "main" java.io.FileNotFoundException: Unable to open > > link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs:// > > > > > tsdb.test.com:54310/tsdb_full_backup_2013-12-09_00.24.03/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/tsdb-uid=f9e5e554f111dc0679dfc8069b282ff7-e76015449a5247b988d15c64c273d43c > > , > > hdfs:// > > > > > tsdb.test.com:54310/tsdb_full_backup_2013-12-09_00.24.03/.tmp/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/tsdb-uid=f9e5e554f111dc0679dfc8069b282ff7-e76015449a5247b988d15c64c273d43c > > , > > hdfs:// > > > > > tsdb.test.com:54310/tsdb_full_backup_2013-12-09_00.24.03/.archive/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/tsdb-uid=f9e5e554f111dc0679dfc8069b282ff7-e76015449a5247b988d15c64c273d43c > > ] > > at org.apache.hadoop.hbase.io.FileLink.getFileStatus(FileLink.java:376) > > at > > > > > org.apache.hadoop.hbase.snapshot.ExportSnapshot$1.storeFile(ExportSnapshot.java:390) > > at > > > > > org.apache.hadoop.hbase.util.FSVisitor.visitRegionStoreFiles(FSVisitor.java:115) > > at > > > > > org.apache.hadoop.hbase.util.FSVisitor.visitTableStoreFiles(FSVisitor.java:81) > > at > > > > > org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitTableStoreFiles(SnapshotReferenceUtil.java:116) > > at > > > > > org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitReferencedFiles(SnapshotReferenceUtil.java:101) > > at > > > > > org.apache.hadoop.hbase.snapshot.ExportSnapshot.getSnapshotFiles(ExportSnapshot.java:385) > > at > > > > > org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:633) > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) > > at > > > > > org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:705) > > at > > > > > org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:709) > > > > > > Can someone help us to understand the reason to fail snapshot import. > > Is this issue similar to HBASE-8455? > > > > Thanks > > -OC > > >
HBase 0.94.10 snapshot FileNotFoundException exception
Hi, we are taking tsdb data backup using HBase snapshots. We tested the scenario as mentioned below: 1. Created snapshots and exported all snapshots successfully. Exported data contains archive+hbase-snapshot 2. Started HBase from scratch (removed all Hbase tabled and created again) 3. Successfully imported all snapshots created in step 1. 4. Again Created snapshots on imported data (step 3) and exported all snapshots. Exported data contains only hbase-snapshot but not archive.Why? 5. Now tried to import snapshots created in step 4 but we are getting below exception. Exception in thread "main" java.io.FileNotFoundException: Unable to open link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs:// tsdb.test.com:54310/tsdb_full_backup_2013-12-09_00.24.03/tsdb/1a3e8367302757a653efffbbde3b462b/t/tsdb=1a3e8367302757a653efffbbde3b462b-e265daa7b19d4911a5196c6f56a4162e, hdfs:// tsdb.test.com:54310/tsdb_full_backup_2013-12-09_00.24.03/.tmp/tsdb/1a3e8367302757a653efffbbde3b462b/t/tsdb=1a3e8367302757a653efffbbde3b462b-e265daa7b19d4911a5196c6f56a4162e, hdfs:// tsdb.test.com:54310/tsdb_full_backup_2013-12-09_00.24.03/.archive/tsdb/1a3e8367302757a653efffbbde3b462b/t/tsdb=1a3e8367302757a653efffbbde3b462b-e265daa7b19d4911a5196c6f56a4162e ] at org.apache.hadoop.hbase.io.FileLink.getFileStatus(FileLink.java:376) at org.apache.hadoop.hbase.snapshot.ExportSnapshot$1.storeFile(ExportSnapshot.java:390) at org.apache.hadoop.hbase.util.FSVisitor.visitRegionStoreFiles(FSVisitor.java:115) at org.apache.hadoop.hbase.util.FSVisitor.visitTableStoreFiles(FSVisitor.java:81) at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitTableStoreFiles(SnapshotReferenceUtil.java:116) at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitReferencedFiles(SnapshotReferenceUtil.java:101) at org.apache.hadoop.hbase.snapshot.ExportSnapshot.getSnapshotFiles(ExportSnapshot.java:385) at org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:633) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:705) at org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:709) Exception in thread "main" java.io.FileNotFoundException: Unable to open link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs:// tsdb.test.com:54310/tsdb_full_backup_2013-12-09_00.24.03/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/tsdb-uid=f9e5e554f111dc0679dfc8069b282ff7-e76015449a5247b988d15c64c273d43c, hdfs:// tsdb.test.com:54310/tsdb_full_backup_2013-12-09_00.24.03/.tmp/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/tsdb-uid=f9e5e554f111dc0679dfc8069b282ff7-e76015449a5247b988d15c64c273d43c, hdfs:// tsdb.test.com:54310/tsdb_full_backup_2013-12-09_00.24.03/.archive/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/tsdb-uid=f9e5e554f111dc0679dfc8069b282ff7-e76015449a5247b988d15c64c273d43c ] at org.apache.hadoop.hbase.io.FileLink.getFileStatus(FileLink.java:376) at org.apache.hadoop.hbase.snapshot.ExportSnapshot$1.storeFile(ExportSnapshot.java:390) at org.apache.hadoop.hbase.util.FSVisitor.visitRegionStoreFiles(FSVisitor.java:115) at org.apache.hadoop.hbase.util.FSVisitor.visitTableStoreFiles(FSVisitor.java:81) at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitTableStoreFiles(SnapshotReferenceUtil.java:116) at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitReferencedFiles(SnapshotReferenceUtil.java:101) at org.apache.hadoop.hbase.snapshot.ExportSnapshot.getSnapshotFiles(ExportSnapshot.java:385) at org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:633) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:705) at org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:709) Can someone help us to understand the reason to fail snapshot import. Is this issue similar to HBASE-8455? Thanks -OC
Re: HBase snapshot export - mappers
Hi, That means if i have cluster with 5 data nodes,i can have mappers upto 5. more mappers means more throughput. Am I correct? Thanks -OC On Thu, Dec 5, 2013 at 7:31 PM, Matteo Bertozzi wrote: > to make it simple, the number of mappers is the number of "machines" that > you want to use. > each machine will get N files to copy based on: total files to export / > nmappers > > Matteo > > > > On Thu, Dec 5, 2013 at 12:58 PM, oc tsdb wrote: > > > Hi, > > > > > > While exporting HBase snapshots we need to specify number of mappers to > use > > as mentioned below.To get better performance how many mappers can be used > > and please let us know based on which parameters we need to decide on > > number of mappers to use.I am new to HBase. > > > > Please refer if there are good link on this. > > > > $ bin/hbase class org.apache.hadoop.hbase.snapshot.tool.ExportSnapshot > > -snapshot MySnapshot -copy-to hdfs:///srv2:8082/hbase -mappers 16 > > > > > > Thanks > > -OC > > >
HBase snapshot export - mappers
Hi, While exporting HBase snapshots we need to specify number of mappers to use as mentioned below.To get better performance how many mappers can be used and please let us know based on which parameters we need to decide on number of mappers to use.I am new to HBase. Please refer if there are good link on this. $ bin/hbase class org.apache.hadoop.hbase.snapshot.tool.ExportSnapshot -snapshot MySnapshot -copy-to hdfs:///srv2:8082/hbase -mappers 16 Thanks -OC
Re: HBase ExportSnapshot
I think that error is because of missing .archive files? We only see this issue when we export snapshot to local file system (file:///tmp/hbase_backup) but not when we export snapshot to hdfs. When we export snapshot to hdfs we are getting all the data.Its only not working when we export snapshot data to local file system directly? On Tue, Dec 3, 2013 at 5:39 PM, Ted Yu wrote: > Please check what happened to the HFileLink mentioned in the exception > below - this would be the cause for snapshot export failure: > > Exception in thread "main" java.io.FileNotFoundException: Unable to open > link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs:// > > site.com:54310/data_full_backup_2013-12-02_21.49.20/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/ed071cd010534856adc4be997498d645 > , > hdfs:// > > site.com:54310/data_full_backup_2013-12-02_21.49.20/.tmp/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/ed071cd010534856adc4be997498d645 > , > hdfs:// > > site.com:54310/data_full_backup_2013-12-02_21.49.20/.archive/tsdb-uid/f9e5e554f111dc0679dfc8069b282ff7/id/ed071cd010534856adc4be997498d645 > ] > at > org.apache.hadoop.hbase.io.FileLink.getFileStatus(FileLink.java:376) > > > On Mon, Dec 2, 2013 at 10:01 PM, oc tsdb wrote: > > > here is snapshot export logs. > > > > mastre log: > > === > > 2013-12-02 21:54:30,840 INFO org.apache.hadoop.hbase.master.LoadBalancer: > > Skipping load balancing because balanced cluster; servers=1 regions=1 > > average=1.0 mostloaded=1 leastloaded=1 > > 2013-12-02 21:54:30,841 INFO org.apache.hadoop.hbase.master.LoadBalancer: > > Skipping load balancing because balanced cluster; servers=1 regions=1 > > average=1.0 mostloaded=1 leastloaded=1 > > 2013-12-02 21:54:30,841 INFO org.apache.hadoop.hbase.master.LoadBalancer: > > Skipping load balancing because balanced cluster; servers=1 regions=1 > > average=1.0 mostloaded=1 leastloaded=1 > > 2013-12-02 21:54:30,841 INFO org.apache.hadoop.hbase.master.LoadBalancer: > > Skipping load balancing because balanced cluster; servers=1 regions=1 > > average=1.0 mostloaded=1 leastloaded=1 > > > > snapshot export console log: > > = > > > > 2013-12-02 21:54:30,841 INFO org.apache.hadoop.hbase.master.LoadBalancer: > > Skipping load balancing because balanced cluster; servers=1 regions=1 > > average=1.0 mostloaded=1 leastloaded=1 > > 2013-12-02 21:54:30,841 INFO org.apache.hadoop.hbase.master.LoadBalancer: > > Skipping load balancing because balanced cluster; servers=1 regions=1 > > average=1.0 mostloaded=1 leastloaded=1 > > > > at > > > > > org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitReferencedFiles(SnapshotReferenceUtil.java:101) > > at > > > > > org.apache.hadoop.hbase.snapshot.ExportSnapshot.getSnapshotFiles(ExportSnapshot.java:385) > > at > > > > > org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:633) > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) > > at > > > > > org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:705) > > at > > > > > org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:709) > > 13/12/02 21:54:24 INFO util.FSVisitor: No families under region > > directory:hdfs:// > > > > > site.com:54310/data_full_backup_2013-12-02_21.49.20/.hbase-snapshot/tsdb-meta_snap_backup/f06335933b32019c4369f95001d996fb > > 13/12/02 21:54:24 INFO util.FSVisitor: No logs under directory:hdfs:// > > > > > site.com:54310/data_full_backup_2013-12-02_21.49.20/.hbase-snapshot/tsdb-meta_snap_backup/.logs > > 13/12/02 21:54:24 WARN snapshot.ExportSnapshot: There are 0 store file to > > be copied. There may be no data in the table. > > 13/12/02 21:54:25 INFO util.FSVisitor: No families under region > > directory:hdfs:// > > > > > site.com:54310/data_full_backup_2013-12-02_21.49.20/.hbase-snapshot/tsdb-tree_snap_backup/c40c34c4312ccb3302fbaf62caa91b9c > > 13/12/02 21:54:25 INFO util.FSVisitor: No logs under directory:hdfs:// > > > > > site.com:54310/data_full_backup_2013-12-02_21.49.20/.hbase-snapshot/tsdb-tree_snap_backup/.logs > > 13/12/02 21:54:25 WARN snapshot.ExportSnapshot: There are 0 store file to > > be copied. There may be no data in the table. > > Exception in thread "main" java.io.FileNotFoundException: Unable to open > > link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs:// > > > > > site.com:54310/data_full_backup_2013-12-02_21.49.20/tsdb-uid/f
Re: HBase ExportSnapshot
the code that determines output FileSystem: > > FileSystem outputFs = FileSystem.get(outputRoot.toUri(), conf); > > For 'file:///tmp/hbase_backup' argument, outputFs would be an instance of > org.apache.hadoop.fs.LocalFileSystem. > > Cheers > > > On Mon, Dec 2, 2013 at 9:06 PM, oc tsdb wrote: > > > We see same logs for both options > > > > 013-12-02 09:47:41,311 INFO > > org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler: Running > FLUSH > > table snapshot tsdb_snap_backup C_M_SNAPSHOT_TABLE on table tsdb > > 2013-12-02 09:47:41,312 INFO org.apache.hadoop.hbase.util.FSUtils: > > FileSystem doesn't support getDefaultReplication > > 2013-12-02 09:47:41,312 INFO org.apache.hadoop.hbase.util.FSUtils: > > FileSystem doesn't support getDefaultBlockSize > > 2013-12-02 09:47:41,337 INFO org.apache.hadoop.hbase.procedure.Procedure: > > Starting procedure 'tsdb_snap_backup' > > 2013-12-02 09:47:41,724 INFO org.apache.hadoop.hbase.procedure.Procedure: > > Procedure 'tsdb_snap_backup' execution completed > > 2013-12-02 09:47:41,724 INFO > > org.apache.hadoop.hbase.procedure.ZKProcedureUtil: Clearing all znodes > for > > procedure tsdb_snap_backupincluding nodes /hbase/online-snapshot/acquired > > /hbase/online-snapshot/reached /hbase/online-snapshot/abort > > 2013-12-02 09:47:41,730 INFO > > org.apache.hadoop.hbase.master.snapshot.EnabledTableSnapshotHandler: Done > > waiting - snapshot for tsdb_snap_backup finished! > > > > It seems we can't export complete snapshot data directly to local file > > system using 'ExportSnapshot' command. > > If we want to copy to outside of cluster first we need to export it to > hdfs > > and then use hadoop get command to copy to local file system. > > Is this correct? > > > > What is the difference between below two commands? > > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot > > hbase_tbl_snapshot_name -copy-to file:///tmp/hbase_backup -mappers 16; > > > > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot > > hbase_tbl_snapshot_name -copy-to hdfs:/hbase_backup -mappers 16; > > > > Thanks > > -OC > > > > > > > > On Mon, Dec 2, 2013 at 10:56 PM, Ted Yu wrote: > > > > > Can you pastebin master log during operation #2 ? > > > > > > There have been at least two fixes since 0.94.10, listed below. > > > It would be nice if you can verify this behavior using 0.94.14 > > > > > > Cheers > > > > > > r1515967 | mbertozzi | 2013-08-20 13:49:38 -0700 (Tue, 20 Aug 2013) | 1 > > > line > > > > > > HBASE-8760 possible loss of data in snapshot taken after region split > > > > > > > r1507792 | mbertozzi | 2013-07-28 05:17:39 -0700 (Sun, 28 Jul 2013) | 1 > > > line > > > > > > HBASE-9060 ExportSnapshot job fails if target path contains percentage > > > character (Jerry He) > > > > > > > > > On Mon, Dec 2, 2013 at 9:19 AM, oc tsdb wrote: > > > > > > > Hi, > > > > > > > > We have cluster with 4 data nodes and HBase version is 0.94.10. > > > > > > > > We have created snapshot for all hbase tables and trying to export > > > snapshot > > > > in two ways. > > > > > > > > option 1.Export snapshot into same cluster hdfs > > > > > > > > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot > > > > hbase_tbl_snapshot_name -copy-to *hdfs:/hbase_backup *-mappers 16; > > > > > > > > Here we are getting full data ( .archive + .hbase-snapshot) exported > to > > > > hdfs:/hbase_backup > > > > > > > > option 2.Export snapshot to local filesystem > > > > command : > > > > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot > > > > hbase_tbl_snapshot_name -copy-to *file:///tmp/hbase_backup* > -mappers > > > 16; > > > > > > > > But with option 2 we only getting .hbase-snapshot exported to local > dir > > > > (/tmp/hbase_backup) but .archive files are not exported.It is > expected > > > > behavior or something wrong in option 2. > > > > > > > > Thanks > > > > OC > > > > > > > > > >
Re: HBase ExportSnapshot
We see same logs for both options 013-12-02 09:47:41,311 INFO org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler: Running FLUSH table snapshot tsdb_snap_backup C_M_SNAPSHOT_TABLE on table tsdb 2013-12-02 09:47:41,312 INFO org.apache.hadoop.hbase.util.FSUtils: FileSystem doesn't support getDefaultReplication 2013-12-02 09:47:41,312 INFO org.apache.hadoop.hbase.util.FSUtils: FileSystem doesn't support getDefaultBlockSize 2013-12-02 09:47:41,337 INFO org.apache.hadoop.hbase.procedure.Procedure: Starting procedure 'tsdb_snap_backup' 2013-12-02 09:47:41,724 INFO org.apache.hadoop.hbase.procedure.Procedure: Procedure 'tsdb_snap_backup' execution completed 2013-12-02 09:47:41,724 INFO org.apache.hadoop.hbase.procedure.ZKProcedureUtil: Clearing all znodes for procedure tsdb_snap_backupincluding nodes /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2013-12-02 09:47:41,730 INFO org.apache.hadoop.hbase.master.snapshot.EnabledTableSnapshotHandler: Done waiting - snapshot for tsdb_snap_backup finished! It seems we can't export complete snapshot data directly to local file system using 'ExportSnapshot' command. If we want to copy to outside of cluster first we need to export it to hdfs and then use hadoop get command to copy to local file system. Is this correct? What is the difference between below two commands? hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot hbase_tbl_snapshot_name -copy-to file:///tmp/hbase_backup -mappers 16; hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot hbase_tbl_snapshot_name -copy-to hdfs:/hbase_backup -mappers 16; Thanks -OC On Mon, Dec 2, 2013 at 10:56 PM, Ted Yu wrote: > Can you pastebin master log during operation #2 ? > > There have been at least two fixes since 0.94.10, listed below. > It would be nice if you can verify this behavior using 0.94.14 > > Cheers > > r1515967 | mbertozzi | 2013-08-20 13:49:38 -0700 (Tue, 20 Aug 2013) | 1 > line > > HBASE-8760 possible loss of data in snapshot taken after region split > > r1507792 | mbertozzi | 2013-07-28 05:17:39 -0700 (Sun, 28 Jul 2013) | 1 > line > > HBASE-9060 ExportSnapshot job fails if target path contains percentage > character (Jerry He) > > > On Mon, Dec 2, 2013 at 9:19 AM, oc tsdb wrote: > > > Hi, > > > > We have cluster with 4 data nodes and HBase version is 0.94.10. > > > > We have created snapshot for all hbase tables and trying to export > snapshot > > in two ways. > > > > option 1.Export snapshot into same cluster hdfs > > > > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot > > hbase_tbl_snapshot_name -copy-to *hdfs:/hbase_backup *-mappers 16; > > > > Here we are getting full data ( .archive + .hbase-snapshot) exported to > > hdfs:/hbase_backup > > > > option 2.Export snapshot to local filesystem > > command : > > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot > > hbase_tbl_snapshot_name -copy-to *file:///tmp/hbase_backup* -mappers > 16; > > > > But with option 2 we only getting .hbase-snapshot exported to local dir > > (/tmp/hbase_backup) but .archive files are not exported.It is expected > > behavior or something wrong in option 2. > > > > Thanks > > OC > > >
HBase ExportSnapshot
Hi, We have cluster with 4 data nodes and HBase version is 0.94.10. We have created snapshot for all hbase tables and trying to export snapshot in two ways. option 1.Export snapshot into same cluster hdfs hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot hbase_tbl_snapshot_name -copy-to *hdfs:/hbase_backup *-mappers 16; Here we are getting full data ( .archive + .hbase-snapshot) exported to hdfs:/hbase_backup option 2.Export snapshot to local filesystem command : hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot hbase_tbl_snapshot_name -copy-to *file:///tmp/hbase_backup* -mappers 16; But with option 2 we only getting .hbase-snapshot exported to local dir (/tmp/hbase_backup) but .archive files are not exported.It is expected behavior or something wrong in option 2. Thanks OC
Incremental backup of hbase with export not working
Hi , In order to take incremental backup using export of hbase , we followed http://hbase.apache.org/book/ops_mgt.html#import Few things that need clarification are : 1. what does version mean? is is the same version number which we give during creation of hbase table ? 2. what if we don't specify the version and just specify start and end time stamps ? kindly provide us an example how to take incremental hbase backup using exportin an interval. We did some experiments with version and start time combination and results are as follows 1. we created a table with version=>1 and tested the import CLI using the same version (version =1 ) and start-end times .even though the data is present between start and end intervals , we didn't get any data. 2. without specifying the version , we got all the data irrespective of the start and end times Kindly clarify us how to specify the version , time stamp range to match our requirements . Thanks, Oc.tsdb
Re: snapshotexport to outside of cluster
We want to use that exported data in case of any disaster happens. Thanks -OC On Sun, Dec 1, 2013 at 2:25 PM, Ted Yu wrote: > Since the remote server doesn't have hdfs, what would the backup be used > for ? > > Cheers > > On Nov 30, 2013, at 10:34 PM, oc tsdb wrote: > > > Hi, > > > >>> Can you clarify what 'without copying to local file system' means ? > > > > We want to take HBase tables backup.So we periodically take snapshots and > > export data to backup server which is not in cluster. > > > > Currently we are achieving this as mentioned below. > > > > 1.Create snapshots from HBase command line > > lets say we have created snapshot for a table called 'hbase_tabel' and > > its snapshot name is 'hbase_tabel_snapshot' > > 2. Then export snapshot data to remote backup server which is not in > > cluster.We are achieving this in two steps. > > a. Export snapshot to local file system(sample path /tmp/hbase_backup) > > from hdfs file system as mentioned below > > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot > > hbase_tabel_snapshot -copy-to file:///tmp/hbase_backup - > > mappers 16 > > Here we should have enough disc space in local file system to export > > snapshot from hdfs. > >b.Transfer(scp) exported snapshot from local file system > > (/tmp/hbase_backup) > > to backup server which is not hadoop/HBase running > > server. > > > > That means first we have to export snapshot to local system from hdfs > and > > then transfer ( scp ) to remote server.Instead of doing in two steps can > we > > directly export snapshot to remote server(which is not in cluster) like > how > > it is exporting to another cluster. > > . > > Thanks > > OC > > > > > > > > > > > > On Sat, Nov 30, 2013 at 11:11 AM, Ted Yu wrote: > > > >> HBASE-4618 is not related to snapshots. > >> Lars did some work in HBASE-5509, a sub-task of HBASE-4618. > >> > >> FYI > >> > >> > >> On Fri, Nov 29, 2013 at 9:31 PM, Yu Li wrote: > >> > >>> I guess this might be a use case of > >>> HBASE-7912<https://issues.apache.org/jira/browse/HBASE-7912>or > >>> HBASE-4618 <https://issues.apache.org/jira/browse/HBASE-4618>? Maybe > >> some > >>> backup/restore tools taking use of snapshot would be nice to have? > >>> > >>> > >>> On 30 November 2013 12:58, Ted Yu wrote: > >>> > >>>> The remote server is in different cluster, right ? > >>>> > >>>> Looking at ExportSnapshot#ExportMapper#copyFile(), it directly copies > >>> data > >>>> to outputPath which is in the other cluster. > >>>> > >>>> Can you clarify what 'without copying to local file system' means ? > >>>> > >>>> Cheers > >>>> > >>>> > >>>> On Fri, Nov 29, 2013 at 8:49 PM, oc tsdb wrote: > >>>> > >>>>> Hi, > >>>>> thanks for your replay. > >>>>> > >>>>> We are able to export snapshot to another cluster But this is not > >> what > >>> we > >>>>> are looking for. > >>>>> We want to export snapshot to remote server without copying to local > >>> file > >>>>> system. > >>>>> > >>>>> Thanks > >>>>> OC > >>>>> > >>>>> On Fri, Nov 29, 2013 at 8:52 PM, Ted Yu wrote: > >>>>> > >>>>>> Here is a sample command exporting to another cluster: > >>>>>> > >>>>>> hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot > >>>>>> gs_1383068263 -copy-to hdfs://experiment-1d:8020/hbase -mappers 6 > >>>>>> > >>>>>> > >>>>>> On Fri, Nov 29, 2013 at 4:02 AM, oc tsdb > >> wrote: > >>>>>> > >>>>>>> Hi, > >>>>>>> > >>>>>>> can someone please let us know if we can export snapshots data > >>>> directly > >>>>>> to > >>>>>>> outside of cluster without copying to local. > >>>>>>> > >>>>>>> Please refer us if there are any blog with example. > >>>>>>> > >>>>>>> Below command is copying data to local file system under /tmp. > >>>>>>> hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot > >>>> mytable > >>>>>>> -copy-to file:///tmp/hbase_backup -mappers 16 > >>>>>>> > >>>>>>> Without copying to local system, we want to export snapshot(eg > >> scp) > >>>>> data > >>>>>> to > >>>>>>> a server which is not part of any cluster but its reachable from > >>>>> cluster. > >>>>>>> > >>>>>>> Thanks > >>>>>>> OC > >>> > >>> > >>> > >>> -- > >>> Best Regards, > >>> Li Yu > >> >
Re: snapshotexport to outside of cluster
Hi, >>Can you clarify what 'without copying to local file system' means ? We want to take HBase tables backup.So we periodically take snapshots and export data to backup server which is not in cluster. Currently we are achieving this as mentioned below. 1.Create snapshots from HBase command line lets say we have created snapshot for a table called 'hbase_tabel' and its snapshot name is 'hbase_tabel_snapshot' 2. Then export snapshot data to remote backup server which is not in cluster.We are achieving this in two steps. a. Export snapshot to local file system(sample path /tmp/hbase_backup) from hdfs file system as mentioned below hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot hbase_tabel_snapshot -copy-to file:///tmp/hbase_backup - mappers 16 Here we should have enough disc space in local file system to export snapshot from hdfs. b.Transfer(scp) exported snapshot from local file system (/tmp/hbase_backup) to backup server which is not hadoop/HBase running server. That means first we have to export snapshot to local system from hdfs and then transfer ( scp ) to remote server.Instead of doing in two steps can we directly export snapshot to remote server(which is not in cluster) like how it is exporting to another cluster. . Thanks OC On Sat, Nov 30, 2013 at 11:11 AM, Ted Yu wrote: > HBASE-4618 is not related to snapshots. > Lars did some work in HBASE-5509, a sub-task of HBASE-4618. > > FYI > > > On Fri, Nov 29, 2013 at 9:31 PM, Yu Li wrote: > > > I guess this might be a use case of > > HBASE-7912<https://issues.apache.org/jira/browse/HBASE-7912>or > > HBASE-4618 <https://issues.apache.org/jira/browse/HBASE-4618>? Maybe > some > > backup/restore tools taking use of snapshot would be nice to have? > > > > > > On 30 November 2013 12:58, Ted Yu wrote: > > > > > The remote server is in different cluster, right ? > > > > > > Looking at ExportSnapshot#ExportMapper#copyFile(), it directly copies > > data > > > to outputPath which is in the other cluster. > > > > > > Can you clarify what 'without copying to local file system' means ? > > > > > > Cheers > > > > > > > > > On Fri, Nov 29, 2013 at 8:49 PM, oc tsdb wrote: > > > > > > > Hi, > > > > thanks for your replay. > > > > > > > > We are able to export snapshot to another cluster But this is not > what > > we > > > > are looking for. > > > > We want to export snapshot to remote server without copying to local > > file > > > > system. > > > > > > > > Thanks > > > > OC > > > > > > > > On Fri, Nov 29, 2013 at 8:52 PM, Ted Yu wrote: > > > > > > > > > Here is a sample command exporting to another cluster: > > > > > > > > > > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot > > > > > gs_1383068263 -copy-to hdfs://experiment-1d:8020/hbase -mappers 6 > > > > > > > > > > > > > > > On Fri, Nov 29, 2013 at 4:02 AM, oc tsdb > wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > can someone please let us know if we can export snapshots data > > > directly > > > > > to > > > > > > outside of cluster without copying to local. > > > > > > > > > > > > Please refer us if there are any blog with example. > > > > > > > > > > > > Below command is copying data to local file system under /tmp. > > > > > > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot > > > mytable > > > > > > -copy-to file:///tmp/hbase_backup -mappers 16 > > > > > > > > > > > > Without copying to local system, we want to export snapshot(eg > scp) > > > > data > > > > > to > > > > > > a server which is not part of any cluster but its reachable from > > > > cluster. > > > > > > > > > > > > Thanks > > > > > > OC > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > Best Regards, > > Li Yu > > >
Re: snapshotexport to outside of cluster
Hi, thanks for your replay. We are able to export snapshot to another cluster But this is not what we are looking for. We want to export snapshot to remote server without copying to local file system. Thanks OC On Fri, Nov 29, 2013 at 8:52 PM, Ted Yu wrote: > Here is a sample command exporting to another cluster: > > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot > gs_1383068263 -copy-to hdfs://experiment-1d:8020/hbase -mappers 6 > > > On Fri, Nov 29, 2013 at 4:02 AM, oc tsdb wrote: > > > Hi, > > > > can someone please let us know if we can export snapshots data directly > to > > outside of cluster without copying to local. > > > > Please refer us if there are any blog with example. > > > > Below command is copying data to local file system under /tmp. > > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot mytable > > -copy-to file:///tmp/hbase_backup -mappers 16 > > > > Without copying to local system, we want to export snapshot(eg scp) data > to > > a server which is not part of any cluster but its reachable from cluster. > > > > Thanks > > OC > > >
snapshotexport to outside of cluster
Hi, can someone please let us know if we can export snapshots data directly to outside of cluster without copying to local. Please refer us if there are any blog with example. Below command is copying data to local file system under /tmp. hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot mytable -copy-to file:///tmp/hbase_backup -mappers 16 Without copying to local system, we want to export snapshot(eg scp) data to a server which is not part of any cluster but its reachable from cluster. Thanks OC
Re: HBase snapshots
Hi Matteo, Thanks a lot for your quick and detailed explanation. -OC On Tue, Nov 19, 2013 at 10:43 PM, Matteo Bertozzi wrote: > > 1.Is there a way to take incremental snapshots on HBase tables? > > > > Due to the nature of hbase, when hfiles gets compacted your table will not > share files with the snapshot causing the next snapshot to don't share > hfiles (or share a few) with the previous snapshot. The other approach will > be using logs to have an incremental snapshots but logs contains different > tables, so you may end up with more space used, and for sure more time to > restore since you must replay the logs. > > > > 2.Can we export snapshots to outside of HDFS(e.g backup server)? > > > > the ExportSnapshot tool takes a destination path as argument, so as long as > the destination filesystem is supported by the FileSystem class you can. > (e.g. you can export with -copy-to=file:///mnt/backup-mount-point/hbase) > > > > 3.Is it possible to take a snapshot for all tables in one command? > > > > No, but you can do a simple script that does a for each table in > admin.listTable(), admin.snapshot(table, snapshotName) > > > > 4.Does HBase support to take automatic snapshots based on configured > > timeout? > > > > No, hbase doesn't do job scheduling. > you may use a cron job to execute a shell command, "echo snapshot > 'tableName', 'snapshotName' | hbase shell" > > > > > > Please also let us know if there is any better way to take HBase tables > > backup. > > > > There are different solutions that you can use: Replication, CopyTable, > HDFS files copy + CopyTable and maybe others it depends on what you're > looking for. > There is an article here that explains how to use CopyTable as backup > tool: > > http://blog.cloudera.com/blog/2012/06/online-hbase-backups-with-copytable-2/ >
HBase snapshots
Hi, We are using HBase 0.94.10 version. We could able to take snapshots for all hbase tables and export into another cluster successfully. To take HBase table snapshots we followed procedure given at below apache link. http://hbase.apache.org/book/ops.snapshots.html Purpose of taking snapshots is to take cluster backup. Can someone please help us on below queries. 1.Is there a way to take incremental snapshots on HBase tables? 2.Can we export snapshots to outside of HDFS(e.g backup server)? 3.Is it possible to take a snapshot for all tables in one command? 4.Does HBase support to take automatic snapshots based on configured timeout? Please also let us know if there is any better way to take HBase tables backup. Thanks in advance. -OC
is jdk required to run hbase or jre alone is sufficient
HI , I would like to know if JRE alone is sufficient to run HBASE services or JDK is required ? we are planning to install latest stable version of hbase Thanks, Oc.tsdb
How to recover data from hadoop/hbase cluster
Hi, I am running a hadoop/hbase cluster with 6 nodes (1 name node, 5 data nodes). Two days back two of the machines went down due to some kernel crash. These machines were rebooted after some time. Meanwhile the cluster became unresponsive. Today, to create disk space and let other application services continue in cluster, we had to wipe out hadoop data directory on one of the data nodes, thinking that we should still be able to recover cluster as the replication count was 3. Could you please point me to any procedure to recover data in my cluster. Thanks in advance. Regards, OC.
Re: getting splitmanager debug logs continuously
all the nodes are running but master does not run region-server; master was limited to run nameNode, quorum, and HMaster functionality. you mean to run region server on Master node as well? On Thu, Aug 8, 2013 at 2:48 PM, Jimmy Xiang wrote: > Can you start the master as well (besides region servers)? > > > On Thu, Aug 8, 2013 at 2:41 PM, oc tsdb wrote: > > > I am using hbase-0.92 > > > > Region server was not running on any of the nodes. > > > > Restarted the cluster. It started region server on all nodes except > > HMaster but still unresponsive. > > > > processes running on master are > > TSDMain > > HMaster > > SecondaryNameNode > > NameNode > > JobTracker > > HQuorumPeer > > > > processes running on all other nodes are > > DataNode > > TaskTracker > > RegionServer > > TSDMain > > > > This time, I see the error messages in the attached log. > > > > Could you please suggest if I can recover/restore the data and get the > > cluster up. > > > > Thanks & Regards, > > VSR > > > > > > > > On Thu, Aug 8, 2013 at 1:40 PM, Ted Yu wrote: > > > >> Can you tell us the version of HBase you're using ? > >> > >> Do you find something in region server logs on the 4 remaining nodes ? > >> > >> Cheers > >> > >> On Thu, Aug 8, 2013 at 1:36 PM, oc tsdb wrote: > >> > >> > Hi, > >> > > >> > I am running a cluster with 6 nodes; > >> > Two of 6 nodes in my cluster went down (due to other application > >> failure) > >> > and came back after some time (had to do a power reboot). > >> > When these nodes are back I use to get "WARN > >> org.apache.hadoop.DFSClient: > >> > Failed to connect to , add to deadnodes and continue". > >> > Now these messages are stopped and getting continuous debug message as > >> > follows. > >> > > >> > 2013-08-08 12:57:36,628 DEBUG org.apache.hadoop.hbase. > >> > master.SplitLogManager: total tasks = 14 unassigned = 14 > >> > 2013-08-08 12:57:37,628 DEBUG > >> > org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 14 > >> unassigned > >> > = 14 > >> > 2013-08-08 12:57:37,629 DEBUG > >> > org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired > >> > /hbase/splitlog/hdfs%3A%2F%2Fmb-1.corp.oc.com > %3A54310%2Fhbase%2F.logs% > >> > 2Fmb-3.corp.oc.com%2C60020%2C1375466447768-splitting% > 2Fmb-3.corp.oc.com > >> > %252C60020%252C1375466447768.1375631802971 > >> > ver = 0 > >> > 2013-08-08 12:57:37,629 DEBUG > >> > org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired > >> > /hbase/splitlog/hdfs%3A%2F%2Fmb-1.corp.oc.com > %3A54310%2Fhbase%2F.logs% > >> > 2Fmb-6.corp.oc.com%2C60020%2C1375466460755-splitting% > 2Fmb-6.corp.oc.com > >> > %252C60020%252C1375466460755.1375623787557 > >> > ver = 0 > >> > 2013-08-08 12:57:37,629 DEBUG > >> > org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired > >> > /hbase/splitlog/hdfs%3A%2F%2Fmb-1.corp.oc.com > %3A54310%2Fhbase%2F.logs% > >> > 2Fmb-6.corp.oc.com%2C60020%2C1375466460755-splitting% > 2Fmb-6.corp.oc.com > >> > %252C60020%252C1375466460755.1375619231059 > >> > ver = 3 > >> > 2013-08-08 12:57:37,629 DEBUG > >> > org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired > >> > /hbase/splitlog/hdfs%3A%2F%2Fmb-1.corp.oc.com > %3A54310%2Fhbase%2F.logs% > >> > 2Fmb-2.corp.oc.com%2C60020%2C1375466479427-splitting% > 2Fmb-2.corp.oc.com > >> > %252C60020%252C1375466479427.1375639017535 > >> > ver = 0 > >> > 2013-08-08 12:57:37,629 DEBUG > >> > org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired > >> > /hbase/splitlog/hdfs%3A%2F%2Fmb-1.corp.oc.com > %3A54310%2Fhbase%2F.logs% > >> > 2Fmb-6.corp.oc.com%2C60020%2C1375466460755-splitting% > 2Fmb-6.corp.oc.com > >> > %252C60020%252C1375466460755.1375623021175 > >> > ver = 0 > >> > 2013-08-08 12:57:37,629 DEBUG > >> > org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired > >> > /hbase/splitlog/hdfs%3A%2F%2Fmb-1.corp.oc.com > %3A54310%2Fhbase%2F.logs% > >> > 2Fmb-3.corp.oc.com%2C60020%2C1375466447768-splitting% > 2Fmb-3.corp.oc.com > >> > %252C60020%252C1375466447768.137563
getting splitmanager debug logs continuously
Hi, I am running a cluster with 6 nodes; Two of 6 nodes in my cluster went down (due to other application failure) and came back after some time (had to do a power reboot). When these nodes are back I use to get "WARN org.apache.hadoop.DFSClient: Failed to connect to , add to deadnodes and continue". Now these messages are stopped and getting continuous debug message as follows. 2013-08-08 12:57:36,628 DEBUG org.apache.hadoop.hbase. master.SplitLogManager: total tasks = 14 unassigned = 14 2013-08-08 12:57:37,628 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 14 unassigned = 14 2013-08-08 12:57:37,629 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired /hbase/splitlog/hdfs%3A%2F%2Fmb-1.corp.oc.com%3A54310%2Fhbase%2F.logs% 2Fmb-3.corp.oc.com%2C60020%2C1375466447768-splitting%2Fmb-3.corp.oc.com%252C60020%252C1375466447768.1375631802971 ver = 0 2013-08-08 12:57:37,629 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired /hbase/splitlog/hdfs%3A%2F%2Fmb-1.corp.oc.com%3A54310%2Fhbase%2F.logs% 2Fmb-6.corp.oc.com%2C60020%2C1375466460755-splitting%2Fmb-6.corp.oc.com%252C60020%252C1375466460755.1375623787557 ver = 0 2013-08-08 12:57:37,629 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired /hbase/splitlog/hdfs%3A%2F%2Fmb-1.corp.oc.com%3A54310%2Fhbase%2F.logs% 2Fmb-6.corp.oc.com%2C60020%2C1375466460755-splitting%2Fmb-6.corp.oc.com%252C60020%252C1375466460755.1375619231059 ver = 3 2013-08-08 12:57:37,629 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired /hbase/splitlog/hdfs%3A%2F%2Fmb-1.corp.oc.com%3A54310%2Fhbase%2F.logs% 2Fmb-2.corp.oc.com%2C60020%2C1375466479427-splitting%2Fmb-2.corp.oc.com%252C60020%252C1375466479427.1375639017535 ver = 0 2013-08-08 12:57:37,629 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired /hbase/splitlog/hdfs%3A%2F%2Fmb-1.corp.oc.com%3A54310%2Fhbase%2F.logs% 2Fmb-6.corp.oc.com%2C60020%2C1375466460755-splitting%2Fmb-6.corp.oc.com%252C60020%252C1375466460755.1375623021175 ver = 0 2013-08-08 12:57:37,629 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired /hbase/splitlog/hdfs%3A%2F%2Fmb-1.corp.oc.com%3A54310%2Fhbase%2F.logs% 2Fmb-3.corp.oc.com%2C60020%2C1375466447768-splitting%2Fmb-3.corp.oc.com%252C60020%252C1375466447768.1375630425141 ver = 0 2013-08-08 12:57:37,629 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: resubmitting unassigned task(s) after timeout 2013-08-08 12:57:37,629 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired /hbase/splitlog/hdfs%3A%2F%2Fmb-1.corp.oc.com%3A54310%2Fhbase%2F.logs% 2Fmb-6.corp.oc.com%2C60020%2C1375466460755-splitting%2Fmb-6.corp.oc.com%252C60020%252C1375466460755.1375620714514 ver = 3 2013-08-08 12:57:37,630 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired /hbase/splitlog/hdfs%3A%2F%2Fmb-1.corp.oc.com%3A54310%2Fhbase%2F.logs% 2Fmb-6.corp.oc.com%2C60020%2C1375924525310-splitting%2Fmb-6.corp.oc.com%252C60020%252C1375924525310.1375924529658 ver = 0 2013-08-08 12:57:37,630 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired /hbase/splitlog/hdfs%3A%2F%2Fmb-1.corp.oc.com%3A54310%2Fhbase%2F.logs% 2Fmb-4.corp.oc.com%2C60020%2C1375466551673-splitting%2Fmb-4.corp.oc.com%252C60020%252C1375466551673.1375641592581 ver = 0 2013-08-08 12:57:37,630 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired /hbase/splitlog/hdfs%3A%2F%2Fmb-1.corp.oc.com%3A54310%2Fhbase%2F.logs% 2Fmb-5.corp.oc.com%2C60020%2C1375924528073-splitting%2Fmb-5.corp.oc.com%252C60020%252C1375924528073.1375924532442 ver = 0 2013-08-08 12:57:37,630 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired /hbase/splitlog/hdfs%3A%2F%2Fmb-1.corp.oc.com%3A54310%2Fhbase%2F.logs% 2Fmb-6.corp.oc.com%2C60020%2C1375466460755-splitting%2Fmb-6.corp.oc.com%252C60020%252C1375466460755.1375622290167 ver = 3 2013-08-08 12:57:37,630 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired /hbase/splitlog/hdfs%3A%2F%2Fmb-1.corp.oc.com%3A54310%2Fhbase%2F.logs% 2Fmb-5.corp.oc.com%2C60020%2C1375466463385-splitting%2Fmb-5.corp.oc.com%252C60020%252C1375466463385.1375638183425 ver = 0 2013-08-08 12:57:37,630 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired /hbase/splitlog/hdfs%3A%2F%2Fmb-1.corp.oc.com%3A54310%2Fhbase%2F.logs% 2Fmb-5.corp.oc.com%2C60020%2C1375466463385-splitting%2Fmb-5.corp.oc.com%252C60020%252C1375466463385.1375639599559 ver = 0 2013-08-08 12:57:37,630 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired /hbase/splitlog/hdfs%3A%2F%2Fmb-1.corp.oc.com%3A54310%2Fhbase%2F.logs% 2Fmb-5.corp.oc.com%2C60020%2C1375466463385-splitting%2Fmb-5.corp.oc.com%252C60020%252C1375466463385.1375641710787 ver = 3 2013-08-08 12:57:37,633 INFO org.apache.hadoop.hbase.master.SplitLogManager: task /hbase/splitlog/RESCAN006975 entered state done mb-1.corp.oc.com ,6,13759245
