Re: HBase Backup and Restore
Backport is possible. We have backported to 2.1.x and using it. Waiting for some PRs to merge so that I can put some effort in getting the backport available for active 2.x releases. On Tue, Jul 12, 2022, 10:42 PM anup ahire wrote: > Thanks Mallikarjun. Will it be possible to backport this to 2.x ? trying to > understand if there are any hard dependencies on 3.0.0. > > On Tue, Jul 12, 2022 at 10:01 AM Mallikarjun > wrote: > > > 3.0.0 should be the first release to include Backup and Restore. So far > > there are no releases. > > > > On Tue, Jul 12, 2022, 10:26 PM anup ahire wrote: > > > > > Hello, > > > > > > I am trying to understand, why is > > > https://github.com/apache/hbase/tree/master/hbase-backup part of > master > > > but > > > getting left out from release tags ? > > > > > > Also, What is the minimum required version of the HBase to use Backup > and > > > Restore ? I am on 2.2.6 > > > > > > > > > Thanks, > > > Anup > > > > > >
Re: HBase Backup and Restore
3.0.0 should be the first release to include Backup and Restore. So far there are no releases. On Tue, Jul 12, 2022, 10:26 PM anup ahire wrote: > Hello, > > I am trying to understand, why is > https://github.com/apache/hbase/tree/master/hbase-backup part of master > but > getting left out from release tags ? > > Also, What is the minimum required version of the HBase to use Backup and > Restore ? I am on 2.2.6 > > > Thanks, > Anup >
Re: Hbase export snapshot
The problem is that it seems like hfile TTL cleaner. Which can be configured with --> *hbase.master.hfilecleaner.ttl* On ExportSnapshot to hbase filesystem, temporary files are placed under */archive*, which is cleaned up by ttl cleaner on frequent intervals (I think 15 minutes is default) if there are no references for these files (which won't be created until snapshot is completed). *Fix*: Increase the above configuration value to how long export snapshot takes and then restart hmaster. Later on you can revert back. --- Mallikarjun On Wed, Feb 9, 2022 at 7:15 PM Hamado Dene wrote: > Hi hbase community, > I'm trying to make an export snapshot of a table that has large hfiles, > however.With smaller tables I have not had problems, but bigger tables I > can't export and I get the exception: > rror: java.io.FileNotFoundException: File does not exist: > /hbase/archive/data/default/mn1_7482_hevents/d37341ab3adad67e2c911edd6d5e6de7/d/27f6d74f99654685b5518a8db1c1496a > (inode 604969) Holder > DFSClient_attempt_1643276298721_0182_m_00_0_1307333633_1 does not have > any open files. at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2674) > at > org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.analyzeFileState(FSDirWriteFileOp.java:521) > at > org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.validateAddBlock(FSDirWriteFileOp.java:161) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2555) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:829) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:510) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:447) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989) at > org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:850) at > org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:793) at > java.security.AccessController.doPrivileged(Native Method) at > javax.security.auth.Subject.doAs(Subject.java:422) at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2489) at > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at > org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) > at > org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) > at > org.apache.hadoop.hdfs.DataStreamer.locateFollowingBlock(DataStreamer.java:1842) > at > org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1638) > at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:704) Caused > by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): > File does not exist: > /hbase/archive/data/default/mn1_7482_hevents/d37341ab3adad67e2c911edd6d5e6de7/d/27f6d74f99654685b5518a8db1c1496a > (inode 604969) Holder > DFSClient_attempt_1643276298721_0182_m_00_0_1307333633_1 does not have > any open files. at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2674) > at > org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.analyzeFileState(FSDirWriteFileOp.java:521) > at > org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.validateAddBlock(FSDirWriteFileOp.java:161) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2555) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:829) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:510) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:447) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989) at > org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:850) at > org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:793) at > java.security.AccessController.doPrivileged(Native Method
Re: Sync two cluster
You can do this several ways. I recommend you can do following way 1. Stop and remove existing replication. 2. Setup replication again and disable it temporarily 3. Export snapshot to secondary cluster. 4. Delete the table there. And restore from export snapshot 5. Enable replication. 6. You can check from hamster ui replication tab for status This is recommended because it uses least resources. Disk and network are the major resources used. And they can be controlled via bandwidth parameter of export snapshot Alternatively 1. You can disable replication 2. Run Hash table, sync table utility 3. Enable replication This is resource intensive if the difference is huge. Because it will do in hbase layer and scan whole table and ship batch of rows at a time. On Sun, Jan 23, 2022, 11:22 PM Hamado Dene wrote: > hi Hbase community, In our production environment we have a main cluster > and one used as a replica in a second datacenter. We have found that the > disk space used on the primary is much greater than on the replica. So we > think the replica is far behind the primary. Is there a way to synchronize > the two clusters without impacting the primary cluster too much? Still > keeping replication on? > Hbase version: 2.2.Hadoop version: 2.8.5 > Thanks > Hamado Dene
Re: My Hbase replication is not working
Thanks Duo I will patch this and verify for the issue I mentioned above. On Sun, Dec 12, 2021, 8:06 PM 张铎(Duo Zhang) wrote: > We have fixed several replication related issues which may cause data loss, > for example, this one > > https://issues.apache.org/jira/browse/HBASE-26482 > > For serial replication, if we miss some wal files, it usually causes > replication to be stuck... > > Mallikarjun 于2021年12月12日周日 18:19写道： > > > Sync table is to be run manually when you think there can be > > inconsistencies between the 2 clusters only for specific time period. > > > > As soon as you disable serial replication, it should start replicating > from > > the time it was stuck. You can build dashboards from jmx metrics > generated > > from hmaster to know about these and setup alerts as well. > > > > > > > > On Sun, Dec 12, 2021, 3:33 PM Hamado Dene > > wrote: > > > > > Ok perfect.How often should this sync run? I guess in this case you > have > > > to automate it somehow, correct? > > > Since I will have to disable serial mode, do I first have to align > tables > > > manually or the moment I disable serial mode, the regionservers will > > start > > > replicating from where they were blocked? > > > > > > > > > Il domenica 12 dicembre 2021, 10:55:05 CET, Mallikarjun < > > > mallik.v.ar...@gmail.com> ha scritto: > > > > > > https://hbase.apache.org/book.html#hashtable.synctable > > > > > > To copy the difference between tables for a specific time period. > > > > > > On Sun, Dec 12, 2021, 3:12 PM Hamado Dene > > > > wrote: > > > > > > > Interesting, thank you very much for the info. I'll try to disable > > serial > > > > replication.As for "sync table utility" what do you mean?I am new to > > > Hbase, > > > > I am not yet familiar with all Hbase tools. > > > > > > > > > > > > > > > >Il domenica 12 dicembre 2021, 10:15:01 CET, Mallikarjun < > > > > mallik.v.ar...@gmail.com> ha scritto: > > > > > > > > We have faced issues with serial replication when one of the region > > > server > > > > of either cluster goes into hardware failure, typically memory from > my > > > > understanding. I could not spend enough time to reproduce reliably to > > > > identify the root cause. So I don't know why it is caused. > > > > > > > > Issue could be your serial replication has got into deadlock mode > among > > > the > > > > region servers. Who are not able to make any progress because older > > > > sequence ID is not replicated and older sequence ID is not in front > of > > > the > > > > line to be able to replicate itself. > > > > > > > > Quick fix: disable serial replication temporarily so that out of > > ordering > > > > is allowed to unblock the replication. Can result into some > > > inconsistencies > > > > between clusters which can be fixed using sync table utility since > your > > > > setup is active passive > > > > > > > > Another fix: delete barriers for each regions in hbase:meta. Same > > > > consequence as above. > > > > > > > > > > > > On Sun, Dec 12, 2021, 2:24 PM Hamado Dene > > > > > > > wrote: > > > > > > > > > I'm using hbase 2.2.6 with hadoop 2.8.5.Yes, My replication serial > is > > > > > enabled.This is my peer configuration > > > > > > > > > > > > > > > > > > > > | > > > > > | Peer Id | Cluster Key | Endpoint | State | IsSerial | Bandwidth | > > > > > ReplicateAll | Namespaces | Exclude Namespaces | Table Cfs | > Exclude > > > > Table > > > > > Cfs | > > > > > | replicav1 | acv-db10-hn,acv-db11-hn,acv-db12-hn:2181:/hbase | | > > > > ENABLED > > > > > | true | UNLIMITED | true > > > > > > > > > > | > > > > > > > > > >Il domenica 12 dicembre 2021, 09:39:44 CET, Mallikarjun < > > > > > mallik.v.ar...@gmail.com> ha scritto: > > > > > > > > > > Which version of hbase are you using? Is your replication serial > > > > enabled? > > > > > > > > > > --- > > > > > Mallikarjun > > > > >
Re: My Hbase replication is not working
Sync table is to be run manually when you think there can be inconsistencies between the 2 clusters only for specific time period. As soon as you disable serial replication, it should start replicating from the time it was stuck. You can build dashboards from jmx metrics generated from hmaster to know about these and setup alerts as well. On Sun, Dec 12, 2021, 3:33 PM Hamado Dene wrote: > Ok perfect.How often should this sync run? I guess in this case you have > to automate it somehow, correct? > Since I will have to disable serial mode, do I first have to align tables > manually or the moment I disable serial mode, the regionservers will start > replicating from where they were blocked? > > > Il domenica 12 dicembre 2021, 10:55:05 CET, Mallikarjun < > mallik.v.ar...@gmail.com> ha scritto: > > https://hbase.apache.org/book.html#hashtable.synctable > > To copy the difference between tables for a specific time period. > > On Sun, Dec 12, 2021, 3:12 PM Hamado Dene > wrote: > > > Interesting, thank you very much for the info. I'll try to disable serial > > replication.As for "sync table utility" what do you mean?I am new to > Hbase, > > I am not yet familiar with all Hbase tools. > > > > > > > >Il domenica 12 dicembre 2021, 10:15:01 CET, Mallikarjun < > > mallik.v.ar...@gmail.com> ha scritto: > > > > We have faced issues with serial replication when one of the region > server > > of either cluster goes into hardware failure, typically memory from my > > understanding. I could not spend enough time to reproduce reliably to > > identify the root cause. So I don't know why it is caused. > > > > Issue could be your serial replication has got into deadlock mode among > the > > region servers. Who are not able to make any progress because older > > sequence ID is not replicated and older sequence ID is not in front of > the > > line to be able to replicate itself. > > > > Quick fix: disable serial replication temporarily so that out of ordering > > is allowed to unblock the replication. Can result into some > inconsistencies > > between clusters which can be fixed using sync table utility since your > > setup is active passive > > > > Another fix: delete barriers for each regions in hbase:meta. Same > > consequence as above. > > > > > > On Sun, Dec 12, 2021, 2:24 PM Hamado Dene > > wrote: > > > > > I'm using hbase 2.2.6 with hadoop 2.8.5.Yes, My replication serial is > > > enabled.This is my peer configuration > > > > > > > > > > > > | > > > | Peer Id | Cluster Key | Endpoint | State | IsSerial | Bandwidth | > > > ReplicateAll | Namespaces | Exclude Namespaces | Table Cfs | Exclude > > Table > > > Cfs | > > > | replicav1 | acv-db10-hn,acv-db11-hn,acv-db12-hn:2181:/hbase | | > > ENABLED > > > | true | UNLIMITED | true > > > > > > | > > > > > >Il domenica 12 dicembre 2021, 09:39:44 CET, Mallikarjun < > > > mallik.v.ar...@gmail.com> ha scritto: > > > > > > Which version of hbase are you using? Is your replication serial > > enabled? > > > > > > --- > > > Mallikarjun > > > > > > > > > On Sun, Dec 12, 2021 at 1:54 PM Hamado Dene > > > > > > wrote: > > > > > > > Hi Hbase community, > > > > > > > > On our production installation we have two hbase clusters in two > > > different > > > > datacenters.The primary datacenter replicates the data to the > secondary > > > > datacenter.When we create the tables, we first create on the > secondary > > > > datacenter and then on the primary and then we set replication scope > > to 1 > > > > on the primary.The peer pointing to quorum zk of the secondary > cluster > > is > > > > configured on the primary. > > > > Initially, replication worked fine and data was replicated.We have > > > > recently noticed that some tables are empty in the secondary > > datacenter. > > > So > > > > most likely the data is no longer replicated. I'm seeing lines like > > this > > > in > > > > the logs: > > > > > > > > > > > > Recovered source for cluster/machine(s) replicav1: Total replicated > > > edits: > > > > 0, current progress:walGroup [db11%2C16020%2C1637849866921]: > currently > > > > replicating from: > > > > > > > > > > hdfs://ro
Re: My Hbase replication is not working
https://hbase.apache.org/book.html#hashtable.synctable To copy the difference between tables for a specific time period. On Sun, Dec 12, 2021, 3:12 PM Hamado Dene wrote: > Interesting, thank you very much for the info. I'll try to disable serial > replication.As for "sync table utility" what do you mean?I am new to Hbase, > I am not yet familiar with all Hbase tools. > > > > Il domenica 12 dicembre 2021, 10:15:01 CET, Mallikarjun < > mallik.v.ar...@gmail.com> ha scritto: > > We have faced issues with serial replication when one of the region server > of either cluster goes into hardware failure, typically memory from my > understanding. I could not spend enough time to reproduce reliably to > identify the root cause. So I don't know why it is caused. > > Issue could be your serial replication has got into deadlock mode among the > region servers. Who are not able to make any progress because older > sequence ID is not replicated and older sequence ID is not in front of the > line to be able to replicate itself. > > Quick fix: disable serial replication temporarily so that out of ordering > is allowed to unblock the replication. Can result into some inconsistencies > between clusters which can be fixed using sync table utility since your > setup is active passive > > Another fix: delete barriers for each regions in hbase:meta. Same > consequence as above. > > > On Sun, Dec 12, 2021, 2:24 PM Hamado Dene > wrote: > > > I'm using hbase 2.2.6 with hadoop 2.8.5.Yes, My replication serial is > > enabled.This is my peer configuration > > > > > > > > | > > | Peer Id | Cluster Key | Endpoint | State | IsSerial | Bandwidth | > > ReplicateAll | Namespaces | Exclude Namespaces | Table Cfs | Exclude > Table > > Cfs | > > | replicav1 | acv-db10-hn,acv-db11-hn,acv-db12-hn:2181:/hbase | | > ENABLED > > | true | UNLIMITED | true > > > > | > > > >Il domenica 12 dicembre 2021, 09:39:44 CET, Mallikarjun < > > mallik.v.ar...@gmail.com> ha scritto: > > > > Which version of hbase are you using? Is your replication serial > enabled? > > > > --- > > Mallikarjun > > > > > > On Sun, Dec 12, 2021 at 1:54 PM Hamado Dene > > > wrote: > > > > > Hi Hbase community, > > > > > > On our production installation we have two hbase clusters in two > > different > > > datacenters.The primary datacenter replicates the data to the secondary > > > datacenter.When we create the tables, we first create on the secondary > > > datacenter and then on the primary and then we set replication scope > to 1 > > > on the primary.The peer pointing to quorum zk of the secondary cluster > is > > > configured on the primary. > > > Initially, replication worked fine and data was replicated.We have > > > recently noticed that some tables are empty in the secondary > datacenter. > > So > > > most likely the data is no longer replicated. I'm seeing lines like > this > > in > > > the logs: > > > > > > > > > Recovered source for cluster/machine(s) replicav1: Total replicated > > edits: > > > 0, current progress:walGroup [db11%2C16020%2C1637849866921]: currently > > > replicating from: > > > > > > hdfs://rozzanohadoopcluster/hbase/oldWALs/db11-hd%2C16020%2C1637849866921.1637849874263 > > > at position: -1 > > > Recovered source for cluster/machine(s) replicav1: Total replicated > > edits: > > > 0, current progress:walGroup [db09%2C16020%2C1637589840862]: currently > > > replicating from: > > > > > > hdfs://rozzanohadoopcluster/hbase/oldWALs/db09-hd%2C16020%2C1637589840862.1637589846870 > > > at position: -1 > > > Recovered source for cluster/machine(s) replicav1: Total replicated > > edits: > > > 0, current progress:walGroup [db13%2C16020%2C1635424806449]: currently > > > replicating from: > > > > > > hdfs://rozzanohadoopcluster/hbase/oldWALs/db13%2C16020%2C1635424806449.1635424812985 > > > at position: -1 > > > > > > > > > > > > 2021-12-12 09:13:47,148 INFO [rzv-db09-hd:16020Replication Statistics > > #0] > > > regionserver.Replication: ormal source for cluster replicav1: Total > > > replicated edits: 0, current progress:walGroup > > > [db09%2C16020%2C1638791923537]: currently replicating from: > > > > > > hdfs://rozzanohadoopcluster/hbase/WALs/rzv-db09-hd.rozzano.diennea.lan,16020,1638791923537/rzv-db09-hd.rozzano.diennea.lan%2C16020%2C1638791
Re: My Hbase replication is not working
We have faced issues with serial replication when one of the region server of either cluster goes into hardware failure, typically memory from my understanding. I could not spend enough time to reproduce reliably to identify the root cause. So I don't know why it is caused. Issue could be your serial replication has got into deadlock mode among the region servers. Who are not able to make any progress because older sequence ID is not replicated and older sequence ID is not in front of the line to be able to replicate itself. Quick fix: disable serial replication temporarily so that out of ordering is allowed to unblock the replication. Can result into some inconsistencies between clusters which can be fixed using sync table utility since your setup is active passive Another fix: delete barriers for each regions in hbase:meta. Same consequence as above. On Sun, Dec 12, 2021, 2:24 PM Hamado Dene wrote: > I'm using hbase 2.2.6 with hadoop 2.8.5.Yes, My replication serial is > enabled.This is my peer configuration > > > > | > | Peer Id | Cluster Key | Endpoint | State | IsSerial | Bandwidth | > ReplicateAll | Namespaces | Exclude Namespaces | Table Cfs | Exclude Table > Cfs | > | replicav1 | acv-db10-hn,acv-db11-hn,acv-db12-hn:2181:/hbase | | ENABLED > | true | UNLIMITED | true > > | > > Il domenica 12 dicembre 2021, 09:39:44 CET, Mallikarjun < > mallik.v.ar...@gmail.com> ha scritto: > > Which version of hbase are you using? Is your replication serial enabled? > > --- > Mallikarjun > > > On Sun, Dec 12, 2021 at 1:54 PM Hamado Dene > wrote: > > > Hi Hbase community, > > > > On our production installation we have two hbase clusters in two > different > > datacenters.The primary datacenter replicates the data to the secondary > > datacenter.When we create the tables, we first create on the secondary > > datacenter and then on the primary and then we set replication scope to 1 > > on the primary.The peer pointing to quorum zk of the secondary cluster is > > configured on the primary. > > Initially, replication worked fine and data was replicated.We have > > recently noticed that some tables are empty in the secondary datacenter. > So > > most likely the data is no longer replicated. I'm seeing lines like this > in > > the logs: > > > > > > Recovered source for cluster/machine(s) replicav1: Total replicated > edits: > > 0, current progress:walGroup [db11%2C16020%2C1637849866921]: currently > > replicating from: > > > hdfs://rozzanohadoopcluster/hbase/oldWALs/db11-hd%2C16020%2C1637849866921.1637849874263 > > at position: -1 > > Recovered source for cluster/machine(s) replicav1: Total replicated > edits: > > 0, current progress:walGroup [db09%2C16020%2C1637589840862]: currently > > replicating from: > > > hdfs://rozzanohadoopcluster/hbase/oldWALs/db09-hd%2C16020%2C1637589840862.1637589846870 > > at position: -1 > > Recovered source for cluster/machine(s) replicav1: Total replicated > edits: > > 0, current progress:walGroup [db13%2C16020%2C1635424806449]: currently > > replicating from: > > > hdfs://rozzanohadoopcluster/hbase/oldWALs/db13%2C16020%2C1635424806449.1635424812985 > > at position: -1 > > > > > > > > 2021-12-12 09:13:47,148 INFO [rzv-db09-hd:16020Replication Statistics > #0] > > regionserver.Replication: ormal source for cluster replicav1: Total > > replicated edits: 0, current progress:walGroup > > [db09%2C16020%2C1638791923537]: currently replicating from: > > > hdfs://rozzanohadoopcluster/hbase/WALs/rzv-db09-hd.rozzano.diennea.lan,16020,1638791923537/rzv-db09-hd.rozzano.diennea.lan%2C16020%2C1638791923537.1638791930213 > > at position: -1 > > Recovered source for cluster/machine(s) replicav1: Total replicated > edits: > > 0, current progress:walGroup [db09%2C16020%2C1634401671527]: currently > > replicating from: > > > hdfs://rozzanohadoopcluster/hbase/oldWALs/rzv-db09-hd.rozzano.diennea.lan%2C16020%2C1634401671527.1634401679218 > > at position: -1 > > Recovered source for cluster/machine(s) replicav1: Total replicated > edits: > > 0, current progress:walGroup [db10%2C16020%2C163758587]: currently > > replicating from: > > > hdfs://rozzanohadoopcluster/hbase/oldWALs/rzv-db10-hd.rozzano.diennea.lan%2C16020%2C163758587.1637585906625 > > at position: -1 > > > > > > > > 2021-12-12 08:24:58,561 WARN [regionserver/rzv-db12-hd:16020.logRoller] > > regionserver.ReplicationSource: WAL group db12%2C16020%2C1638790692057 > > queue size: 187 exceeds value of replication.source.log.queue.warn: 2 > > Do you have any info on what could be the problem? > > > > Thanks >
Re: My Hbase replication is not working
Which version of hbase are you using? Is your replication serial enabled? --- Mallikarjun On Sun, Dec 12, 2021 at 1:54 PM Hamado Dene wrote: > Hi Hbase community, > > On our production installation we have two hbase clusters in two different > datacenters.The primary datacenter replicates the data to the secondary > datacenter.When we create the tables, we first create on the secondary > datacenter and then on the primary and then we set replication scope to 1 > on the primary.The peer pointing to quorum zk of the secondary cluster is > configured on the primary. > Initially, replication worked fine and data was replicated.We have > recently noticed that some tables are empty in the secondary datacenter. So > most likely the data is no longer replicated. I'm seeing lines like this in > the logs: > > > Recovered source for cluster/machine(s) replicav1: Total replicated edits: > 0, current progress:walGroup [db11%2C16020%2C1637849866921]: currently > replicating from: > hdfs://rozzanohadoopcluster/hbase/oldWALs/db11-hd%2C16020%2C1637849866921.1637849874263 > at position: -1 > Recovered source for cluster/machine(s) replicav1: Total replicated edits: > 0, current progress:walGroup [db09%2C16020%2C1637589840862]: currently > replicating from: > hdfs://rozzanohadoopcluster/hbase/oldWALs/db09-hd%2C16020%2C1637589840862.1637589846870 > at position: -1 > Recovered source for cluster/machine(s) replicav1: Total replicated edits: > 0, current progress:walGroup [db13%2C16020%2C1635424806449]: currently > replicating from: > hdfs://rozzanohadoopcluster/hbase/oldWALs/db13%2C16020%2C1635424806449.1635424812985 > at position: -1 > > > > 2021-12-12 09:13:47,148 INFO [rzv-db09-hd:16020Replication Statistics #0] > regionserver.Replication: ormal source for cluster replicav1: Total > replicated edits: 0, current progress:walGroup > [db09%2C16020%2C1638791923537]: currently replicating from: > hdfs://rozzanohadoopcluster/hbase/WALs/rzv-db09-hd.rozzano.diennea.lan,16020,1638791923537/rzv-db09-hd.rozzano.diennea.lan%2C16020%2C1638791923537.1638791930213 > at position: -1 > Recovered source for cluster/machine(s) replicav1: Total replicated edits: > 0, current progress:walGroup [db09%2C16020%2C1634401671527]: currently > replicating from: > hdfs://rozzanohadoopcluster/hbase/oldWALs/rzv-db09-hd.rozzano.diennea.lan%2C16020%2C1634401671527.1634401679218 > at position: -1 > Recovered source for cluster/machine(s) replicav1: Total replicated edits: > 0, current progress:walGroup [db10%2C16020%2C163758587]: currently > replicating from: > hdfs://rozzanohadoopcluster/hbase/oldWALs/rzv-db10-hd.rozzano.diennea.lan%2C16020%2C163758587.1637585906625 > at position: -1 > > > > 2021-12-12 08:24:58,561 WARN [regionserver/rzv-db12-hd:16020.logRoller] > regionserver.ReplicationSource: WAL group db12%2C16020%2C1638790692057 > queue size: 187 exceeds value of replication.source.log.queue.warn: 2 > Do you have any info on what could be the problem? > > Thanks
Re: Number of Regions with small Tables
Do you have any configuration for Region Normalizer ( https://hbase.apache.org/book.html#normalizer) or something? Balancer does not split or merge regions. AFAIK, split policy controlled by `hbase.regionserver.region.split.policy` does the splitting and there is nothing similar for merges. --- Mallikarjun On Mon, Jul 12, 2021 at 2:48 PM Christian Pfarr wrote: > Hello @all, > > i´ve a quesion regarding controlling the number of regions on small tables > in HBase. > But first i have to give you some hints about our Usecase. > > We´ve built a lambda architecture with HDFS (Batch), HBase(Speed) and > Drill as Serving Layer where we are combining Parquet Files from HDFS with > HBase Rows that are newer then the most recent Row in HDFS. > The HBase table is filled in realtime via Nifi, while it is cleaned up > every Batch (nightly) so that Drill can put the most workload on HDFS. > Unfortunately the hbase table is very small and because of this, we have > only one region and because of that, drill cannot parallelize the query, > which leads to long query times. > > If i pre-split the hbase table everything is fine, until the balancer > comes and merges the small regions. So after a few hours everything is slow > again :-/ > > So... my question is now, whats the best way to handle these parallization > issue. > I thought about setting hbase.hregion.max.filesize to a very small > number, for example HDFS Blocksize = 128 MB but i´m not shure if this leads > to new problems. > > What do you think? Is there a better way to handle this? > > Regards, > z0ltrix > > >
Re: HotSpot detection/mitigation worker?
Agree. It solves the problem in terms of randomizing distribution, but does not increase cardinality. On Tue, May 18, 2021 at 6:29 PM Bryan Beaudreault wrote: > Hi Mallikarjun, thanks for the response. > > I agree that it is hard to fully mitigate a bad rowkey design. We do make > pretty heavy use of hash prefixes, and we don't really have many examples > of the common problem you describe where the "latest" data is in 1-2 > regions. Our distribution issues instead come from the fact that we have > customers both big and small living in the same table -- a small customer > might have only a few rows and might only use the product enough to access > those rows a few times a day. A large cluster might have thousands or > millions of rows, and be constantly using the product and accessing those > rows. > > While we do use hash prefixes, developers need to choose what to put in > that hash -- for example if the rowkey is customerId + emailAddress + > somethingElse, ideally the hash could at the very least be > hash(customerId + emailAddress). This would result in a relatively well > distributed schema, assuming the cardinality of somethingElse wasn't too > big. But often developers might need to scan data for a given customer, so > the hash might just be hash(customerId). For a large customer with many > email addresses, this presents a problem. In these cases we do try to lean > on small index tables + multi gets where possible, but that's not always > possible and doesn't fully solve the potential for hotspotting. > > We have about 1000 tables across 40 clusters and over 5000 regionservers. > This sort of scenario plays out over and over, where large customers end up > causing certain regions to get more traffic than others. One might hope > that these "heavy" regions might also be bigger in size, if read and write > volume were relatively proportional. If that were true, the HFile > size-based normalizer might be enough. For us though, that is not always > the case. My thought is to identify these read-heavy regions and split them > so we can better distribute the load with the balancer. > > Perhaps this problem is very specific to our use-case, which is why this > problem has not been more generally solved by a load-based normalizer. > > On Mon, May 17, 2021 at 10:09 PM Mallikarjun > wrote: > > > I think, no matter how good a balancer cost function be written, it > cannot > > cover for a not so optimal row key design. Say for example, you have 10 > > regionservers, 100 regions and your application is heavy on the latest > data > > which is mostly 1 or 2 regions, how many ever splits and/or merges it > > becomes very hard to balance the load among the regionservers. > > > > Here is how we have solved this problem among our clients. Which might > not > > work for existing clients, but can be a thought for new clients. > > > > Every request with a row key goes through the enrichment process, which > > prefixes with a hash (from say murmurhash) based on the client requested > > distribution (this stays throughout the lifetime of that table for that > > client). Also We wrote a hbase client abstraction to take care of this > in a > > seamless manager for our clients. > > > > Example: Actual row key --> *0QUPHSBTLGM*, and client requested a 3 digit > > prefix based on table region range (000 - 999), would translate to > > *115-0QUPHSBTLGM* with murmurhash > > > > --- > > Mallikarjun > > > > > > On Tue, May 18, 2021 at 1:33 AM Bryan Beaudreault > > wrote: > > > > > Hey all, > > > > > > We run a bunch of big hbase clusters that get used by hundreds of > product > > > teams for a variety of real-time workloads. We are a B2B company, so > most > > > data has a customerId somewhere in the rowkey. As the team that owns > the > > > hbase infrastructure, we try to help product teams properly design > > schemas > > > to avoid hotspotting, but inevitably it happens. It may not necessarily > > > just be hotspotting, but for example request volume may not be evenly > > > distributed across all regions of a table. > > > > > > This hotspotting/distribution issue makes it hard for the balancer to > > keep > > > the cluster balanced from a load perspective -- sure, all RS have the > > same > > > number of regions, but those regions are not all created equal from a > > load > > > perspective. This results in cases where one RS might be consistently > at > > > 70% cpu, another might be at 30%, and all the rest are in a band > > > in-between. &g
Re: HotSpot detection/mitigation worker?
I think, no matter how good a balancer cost function be written, it cannot cover for a not so optimal row key design. Say for example, you have 10 regionservers, 100 regions and your application is heavy on the latest data which is mostly 1 or 2 regions, how many ever splits and/or merges it becomes very hard to balance the load among the regionservers. Here is how we have solved this problem among our clients. Which might not work for existing clients, but can be a thought for new clients. Every request with a row key goes through the enrichment process, which prefixes with a hash (from say murmurhash) based on the client requested distribution (this stays throughout the lifetime of that table for that client). Also We wrote a hbase client abstraction to take care of this in a seamless manager for our clients. Example: Actual row key --> *0QUPHSBTLGM*, and client requested a 3 digit prefix based on table region range (000 - 999), would translate to *115-0QUPHSBTLGM* with murmurhash --- Mallikarjun On Tue, May 18, 2021 at 1:33 AM Bryan Beaudreault wrote: > Hey all, > > We run a bunch of big hbase clusters that get used by hundreds of product > teams for a variety of real-time workloads. We are a B2B company, so most > data has a customerId somewhere in the rowkey. As the team that owns the > hbase infrastructure, we try to help product teams properly design schemas > to avoid hotspotting, but inevitably it happens. It may not necessarily > just be hotspotting, but for example request volume may not be evenly > distributed across all regions of a table. > > This hotspotting/distribution issue makes it hard for the balancer to keep > the cluster balanced from a load perspective -- sure, all RS have the same > number of regions, but those regions are not all created equal from a load > perspective. This results in cases where one RS might be consistently at > 70% cpu, another might be at 30%, and all the rest are in a band > in-between. > > We already have a normalizer job which works similarly to the > SimpleRegionNormalizer -- keeping regions approximately the same size from > a data size perspective. I'm considering updating our normalizer to also > take into account region load. > > My general plan is to follow a similar strategy to the balancer -- keep a > configurable number of RegionLoad objects in memory per-region, and extract > averages for readRequestsCount from those. If a region's average load is > > some threshold relative to other regions in the same table, split it. If > it's < some threshold relative to other regions in the same table, merge > it. > > I'm writing because I'm wondering if anyone else has had this problem and > if there exists prior art here. Is there a reason HBase does not provide a > configurable load-based normalizer (beyond typical OSS reasons -- no one > contributed it yet)? > > Thanks! >
Re: [SURVEY] The current usage of favor node balancer across the community
Inline reply On Tue, Apr 27, 2021 at 1:03 AM Stack wrote: > On Mon, Apr 26, 2021 at 12:30 PM Stack wrote: > > > On Mon, Apr 26, 2021 at 8:10 AM Mallikarjun > > wrote: > > > >> We use FavoredStochasticBalancer, which by description says the same > thing > >> as FavoredNodeLoadBalancer. Ignoring that fact, problem appears to be > >> > >> > > > > Other concerns: > > > > * Hard-coded triplet of nodes that will inevitably rot as machines come > > and go (Are there tools for remediation?) > It doesn't really rot, if you think it with balancer responsible to assigning regions 1. On every region assigned to a particular regionserver, the balancer would have to reassign this triplet and hence there is no scope of rot (Same logic applied to WAL as well). (On compaction hdfs blocks will be pulled back if any spill over) 2. We used hostnames only (so, come and go is not going to be new nodes but same hostnames) Couple of outstanding problems though. 1. We couldn't increase replication factor to > 3. Which was fine so far for our use cases. But we have had thoughts around fixing them. 2. Balancer doesn't understand favored nodes construct, perfect balanced fn among the rsgroup datanodes isn't possible, but with some variance like 10-20% difference is expected > > * A workaround for a facility that belongs in the NN > Probably, you can argue both ways. Hbase is the owner of data and hbase has the authority to dictate where a particular region replica sits. Benefits like data locality will be mostly around 1, rack awareness is more aligned to this strategy and so on. Moreover, HDFS has data pinning for clients to make use of it. Isn't it? > > * Opaque in operation > We haven't looked around wrapping these operations around metrics, so that it is no longer opaque and reasons mentioned in the above point. > > * My understanding was that the feature was never finished; in > particular > > the balancer wasn't properly wired- up (Happy to be incorrect here). > > > > > One more concern was that the feature was dead/unused. You seem to refute > this notion of mine. > S > We have been using this for more than a year with hbase 2.1 in highly critical workloads for our company. And several years with hbase 1.2 as well with backporting rsgroup from master at that time. (2017-18 ish) And it has been very smooth operationally in hbase 2.1 > > > > > > > >> Going a step back. > >> > >> Did we ever consider giving a thought towards truely multi-tenant hbase? > >> > > > > Always. > > > > > >> Where each rsgroup has a group of datanodes and namespace tables data > >> created under that particular rsgroup would sit on those datanodes only? > >> We > >> have attempted to do that and have largely been very successful running > >> clusters of hundreds of terabytes with hundreds of > >> regionservers(datanodes) > >> per cluster. > >> > >> > > So isolation of load by node? (I believe this is where the rsgroup > feature > > came from originally; the desire for a deploy like you describe above. > > IIUC, its what Thiru and crew run). > > > > > > > >> 1. We use a modified version of RSGroupBasedFavoredNodeLoadBalancer > >> contributed by Thiruvel Thirumoolan --> > >> https://issues.apache.org/jira/browse/HBASE-15533 > >> > >> On each balance operation, while the region is moved around (or while > >> creating table), favored nodes are assigned based on the rsgroup that > >> region is pinned to. And hence data is pinned to those datanodes only > >> (Pinning favored nodes is best effort from the hdfs side, but there are > >> only a few exception scenarios where data will be spilled over and they > >> recover after a major compaction). > >> > >> > > Sounds like you have studied this deploy in operation. Write it up? Blog > > post on hbase.apache.org? > > > Definitely will write up. > > > > > >> 2. We have introduced several balancer cost functions to restore things > to > >> normalcy (multi tenancy with fn pinning) such as when a node is dead, or > >> when fn's are imbalanced within the same rsgroup, etc. > >> > >> 3. We had diverse workloads under the same cluster and WAL isolation > >> became > >> a requirement and we went ahead with similar philosophy mentioned in > line > >> 1. Where WAL's are created with FN pinning so that they are tied to > >> datanodes belonging to the same rsgroup. Some discussion seems to have > >> happen
Re: [SURVEY] The current usage of favor node balancer across the community
We use FavoredStochasticBalancer, which by description says the same thing as FavoredNodeLoadBalancer. Ignoring that fact, problem appears to be favor node balancer is a problem, as it stores the favor node information > in hbase:meta. > Going a step back. Did we ever consider giving a thought towards truely multi-tenant hbase? Where each rsgroup has a group of datanodes and namespace tables data created under that particular rsgroup would sit on those datanodes only? We have attempted to do that and have largely been very successful running clusters of hundreds of terabytes with hundreds of regionservers(datanodes) per cluster. 1. We use a modified version of RSGroupBasedFavoredNodeLoadBalancer contributed by Thiruvel Thirumoolan --> https://issues.apache.org/jira/browse/HBASE-15533 On each balance operation, while the region is moved around (or while creating table), favored nodes are assigned based on the rsgroup that region is pinned to. And hence data is pinned to those datanodes only (Pinning favored nodes is best effort from the hdfs side, but there are only a few exception scenarios where data will be spilled over and they recover after a major compaction). 2. We have introduced several balancer cost functions to restore things to normalcy (multi tenancy with fn pinning) such as when a node is dead, or when fn's are imbalanced within the same rsgroup, etc. 3. We had diverse workloads under the same cluster and WAL isolation became a requirement and we went ahead with similar philosophy mentioned in line 1. Where WAL's are created with FN pinning so that they are tied to datanodes belonging to the same rsgroup. Some discussion seems to have happened here --> https://issues.apache.org/jira/browse/HBASE-21641 There are several other enhancements we have worked on with respect to rsgroup aware export snapshot, rsaware regionmover, rsaware cluster replication, etc. For above use cases, we would be needing fn information on hbase:meta. If the use case seems to be a fit for how we would want hbase to be taken forward as one of the supported use cases, willing to contribute our changes back to the community. (I was anyway planning to initiate this discussion) To strengthen the above use case. Here is what one of our multi tenant cluster looks like RSGroups(Tenants): 21 (With tenant isolation) Regionservers: 275 Regions Hosted: 6k Tables Hosted: 87 Capacity: 250 TB (100TB used) --- Mallikarjun On Mon, Apr 26, 2021 at 9:15 AM 张铎(Duo Zhang) wrote: > As you all know, we always want to reduce the size of the hbase-server > module. This time we want to separate the balancer related code to another > sub module. > > The design doc: > > https://docs.google.com/document/d/1T7WSgcQBJTtbJIjqi8sZYLxD2Z7JbIHx4TJaKKdkBbE/edit# > > You can see the bottom of the design doc, favor node balancer is a problem, > as it stores the favor node information in hbase:meta. Stack mentioned that > the feature is already dead, maybe we could just purge it from our code > base. > > So here we want to know if there are still some users in the community who > still use favor node balancer. Please share your experience and whether you > still want to use it. > > Thanks. >
Re: HBase Operating System compatabiliity
Doesn't java philosophy of `*write once and run anywhere*` apply to hbase/hadoop? We have deployed our production setup on Debian 7/8/9 at different points of time. --- Mallikarjun On Tue, Apr 20, 2021 at 10:05 AM Debraj Manna wrote: > Hi > > Can someone let me know if there is a compatibility matrix for different > HBase and OS Versions? Which all OS versions and flavors hbase is tested > on? > > I could not find anything in hbase doc <http://hbase.apache.org/book.html > >. > Hadoop hardware/software requirements > < > https://hadoop.apache.org/docs/r3.0.0/hadoop-project-dist/hadoop-common/Compatibility.html#:~:text=Currently%20GNU%2FLinux%20and%20Microsoft,Apple%20MacOSX%2C%20Solaris%2C%20etc > .> > also do not say anything explicitly. > > Thanks, >
Re: Hbase Migrations
*Simple procedure:*
1. Stop writing in one cluster.
2. Export snapshot as described below and Restore on other cluster for each
of the tables
3. Start writing to another cluster.
*Not so Simple Procedure (Zero Downtime)*
*If your source cluster > 2.1 (has Serial replication)*
You can do the following in sequence
1. Export snapshot to the destination cluster. Which starts at time say *t1*
and it takes say an hour for example.
hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
> $snapshot_name -copy-to hdfs://$dest_hdfs_ip:8020/hbase -bandwidth
> $bandwidth_in_MB
2. Setup up a Peer to destination cluster at say time *t2* which replicates
the table(s) to be migrated (works only if your source cluster has serial
replication > hbase 2.1) (make sure replication scope is 1 on cf you want
to copy over)
add_peer 'cluster_dest', CLUSTER_KEY =>
> "zk1,zk2,zk3:2182:/hbase", TABLE_CFS => { "table1" => ["cf1", "cf2"] },
> STATE => "ENABLED", SERIAL => true, REPLICATE_ALL => false
>
3. Make sure cluster replication is propagating data and then disable
cluster replication temporarily. This will start piling up the data in
oldWAL's and logs are kept in zk replication queues
disable_peer 'cluster_dest'
>
4. Copy the diff data from time *t1 *to present time using Copy Table
which starts at time say *t3 ( > t2) *and it takes say 10 minutes for
example
hbase org.apache.hadoop.hbase.mapreduce.CopyTable --starttime=
> $copy_starttime --endtime=$copy_endtime --peer.adr=$dest_zk_addr '
> $full_table_name'
5. Enable cluster replication back to ensure data is replicated from time
*t2*
enable_peer 'cluster_dest'
6. Stop writes to old cluster by marking tables read only
alter '$full_table_name',{METHOD=>'table_att', READONLY=>true}
>
7. (optional) Verify if data is the same between 2 clusters for the window
of migration. (Note: This comes with caveats. If you have data being
updated for the same rows, or ttl expiring data some row differences are
expected.)
On Source cluster
> hbase org.apache.hadoop.hbase.mapreduce.HashTable --starttime=
> $verify_starttime --endtime=$verify_endtime '$full_table_name'
> $hash_table_full_path
On Destination Cluster
> hbase org.apache.hadoop.hbase.mapreduce.SyncTable --sourcezkcluster=
> $src_zk_addr *--dryrun=true* hdfs://$src_hdfs_ip:8020$hash_table_full_path
> '$full_table_name' '$full_table_name'
*If your source cluster < 2.1 (No serial replication)*
you will have some write downtime. Can skip step 3 and 5. and step 4 can be
repeated as many times as you want until the downtime window is very small.
Otherwise largely the same.
---
Mallikarjun
On Thu, Apr 15, 2021 at 7:23 PM Sajid Mohammed
wrote:
> Hello All
>
> Any one know simple procedure to migrate all Hbase Tables from one cluster
> to another in one go ?
>
>
> Thanks
> Sajid.
>
