It makes sense to turn on async wal replication when hbase.meta.replicas.use = true. Let me run couple rounds of itbll with hbase.region.replica.replication.catalog.enabled (lastest 2.4 and 2.5.0 candidates) to get more confidence before proposing turn on async wal replication for meta.
Thanks, Huaxiang On 2022/03/26 04:03:15 Andrew Purtell wrote: > Just to be clear when I say "it seems pointless to have meta replicas which > do not actually receive updates (by default)", what I should have said is > 'timely updates', because a long delay in updating meta might as well be a > missed update. > > On Fri, Mar 25, 2022 at 9:01 PM Andrew Purtell <[email protected]> wrote: > > > > "Async WAL replication for META is added as a new feature in 2.4.0. It > > is still under active development. Use with caution. Set > > hbase.region.replica.replication.catalog.enabled to enable async WAL > > Replication for META region replicas. It is off by default." > > > > Do we still need this warning? > > > > Should hbase.region.replica.replication.catalog.enabled have a default of > > 'true' (enabled) if hbase.meta.replicas.use = true ? Otherwise, it seems > > pointless to have meta replicas which do not actually receive updates (by > > default). > > > > > > On Fri, Mar 25, 2022 at 10:51 AM Huaxiang Sun <[email protected]> > > wrote: > > > >> Hi Andor, > >> > >> I get what you are saying. The HFile refreshing is the old way for > >> replica regions to refresh hfiles periodically, default is 5 minutes. In > >> this itbll case, we need to have the wal replication enabled for meta > >> replica. Please check out, > >> > >> https://hbase.apache.org/book.html#_async_wal_replication_for_meta_table_as_of_hbase_2_4_0. > >> Basically, you need to set > >> "hbase.region.replica.replication.catalog.enabled" to true in the > >> configuration and rerun itbll. Otherwise, all meta changes at the primary > >> meta region wont be updated at the replica meta regions and it will result > >> in itbll failures. > >> > >> Hope this helps, > >> > >> Huaxiang > >> > >> > >> On 2022/03/25 13:46:42 Andor Molnar wrote: > >> > Hi Huaxiang, > >> > > >> > We use 2.4.6 for the tests. > >> > > >> > I run itbll with the following command: > >> > > >> > hbase org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList > >> generator 15 1000000 /tmp/hbase-itbll > >> > > >> > for the generator step and essentially jobs have failed. We can see the > >> meta request are spanning out to replicas, but writes start failing after > >> this due to the stale cache which is not getting updated. > >> > > >> > Would you please tell me more about ‘hfile refresh’ and how to > >> configure it? > >> > > >> > Thanks, > >> > Andor > >> > > >> > > >> > > >> > > >> > > On 2022. Mar 24., at 17:43, Huaxiang Sun <[email protected]> > >> wrote: > >> > > > >> > > Hi Andor, > >> > > > >> > > Which 2.4 release do you test in your lab? We use this feature at > >> production cluster with 2.4.5. > >> > > At server side, we use hfile refresh instead of wal replication. I > >> used to run itbll for each release with this feature enabled. How did you > >> find the errors, did itbll fail? > >> > > > >> > > Regards, > >> > > Huaxiang > >> > > >> > > >> > > > > > > -- > > Best regards, > > Andrew > > > > Unrest, ignorance distilled, nihilistic imbeciles - > > It's what we’ve earned > > Welcome, apocalypse, what’s taken you so long? > > Bring us the fitting end that we’ve been counting on > > - A23, Welcome, Apocalypse > > > > > -- > Best regards, > Andrew > > Unrest, ignorance distilled, nihilistic imbeciles - > It's what we’ve earned > Welcome, apocalypse, what’s taken you so long? > Bring us the fitting end that we’ve been counting on > - A23, Welcome, Apocalypse >
