Sorry for replying late. Yeah, at server side, since region move is very rare in our production cluster, we did not enable the async wal replication. It is only being tested in itbll cluster.
On 2022/03/31 13:35:01 Andor Molnar wrote: > One more question Huaxiang which makes me confused: > > You mentioned that "At server side, we use hfile refresh instead of wal > replication.” which means that async wal replication for meta is not tested > by you. > > Is that correct? > > Andor > > > > > > On 2022. Mar 30., at 19:22, Andor Molnar <an...@apache.org> wrote: > > > > Hi Huaxiang, > > > > Given that you already use this feature in production with no problems, I > > created a short patch to remove the “be careful” warnings from the HBase > > documentation. > > > > PTAL if you agree. > > > > https://github.com/apache/hbase/pull/4301 > > > > Thanks, > > Andor > > > > > > > > > >> On 2022. Mar 29., at 18:52, Huaxiang Sun <huaxiang...@apache.org> wrote: > >> > >> This is great, thanks for the testing results! > >> > >> Huaxiang > >> > >> On 2022/03/29 13:29:48 Andor Molnar wrote: > >>> Works! > >>> > >>> I enabled async wal replication with the suggested option and ITBLL ran > >>> successfully. > >>> > >>> generator step: > >>> hbase org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList generator > >>> 15 1000000 /tmp/hbase-itbll > >>> > >>> verification step: > >>> hbase org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList verify > >>> /tmp/hbase-itbll-verify 15 > >>> > >>> Both succeeded. I also confirmed that meta replicas are written and read > >>> by the clients, so must be in load balance mode. > >>> > >>> Thanks for the help! > >>> > >>> Andor > >>> > >>> > >>> > >>>> On 2022. Mar 27., at 6:59, Huaxiang Sun <huaxiang...@apache.org> wrote: > >>>> > >>>> It makes sense to turn on async wal replication when > >>>> hbase.meta.replicas.use = true. Let me run couple rounds of itbll with > >>>> hbase.region.replica.replication.catalog.enabled (lastest 2.4 and 2.5.0 > >>>> candidates) to get more confidence before proposing turn on async wal > >>>> replication for meta. > >>>> > >>>> Thanks, > >>>> Huaxiang > >>>> > >>>> On 2022/03/26 04:03:15 Andrew Purtell wrote: > >>>>> Just to be clear when I say "it seems pointless to have meta replicas > >>>>> which > >>>>> do not actually receive updates (by default)", what I should have said > >>>>> is > >>>>> 'timely updates', because a long delay in updating meta might as well > >>>>> be a > >>>>> missed update. > >>>>> > >>>>> On Fri, Mar 25, 2022 at 9:01 PM Andrew Purtell <apurt...@apache.org> > >>>>> wrote: > >>>>> > >>>>>>> "Async WAL replication for META is added as a new feature in 2.4.0. It > >>>>>> is still under active development. Use with caution. Set > >>>>>> hbase.region.replica.replication.catalog.enabled to enable async WAL > >>>>>> Replication for META region replicas. It is off by default." > >>>>>> > >>>>>> Do we still need this warning? > >>>>>> > >>>>>> Should hbase.region.replica.replication.catalog.enabled have a default > >>>>>> of > >>>>>> 'true' (enabled) if hbase.meta.replicas.use = true ? Otherwise, it > >>>>>> seems > >>>>>> pointless to have meta replicas which do not actually receive updates > >>>>>> (by > >>>>>> default). > >>>>>> > >>>>>> > >>>>>> On Fri, Mar 25, 2022 at 10:51 AM Huaxiang Sun <huaxiang...@apache.org> > >>>>>> wrote: > >>>>>> > >>>>>>> Hi Andor, > >>>>>>> > >>>>>>> I get what you are saying. The HFile refreshing is the old way for > >>>>>>> replica regions to refresh hfiles periodically, default is 5 minutes. > >>>>>>> In > >>>>>>> this itbll case, we need to have the wal replication enabled for meta > >>>>>>> replica. Please check out, > >>>>>>> > >>>>>>> https://hbase.apache.org/book.html#_async_wal_replication_for_meta_table_as_of_hbase_2_4_0. > >>>>>>> Basically, you need to set > >>>>>>> "hbase.region.replica.replication.catalog.enabled" to true in the > >>>>>>> configuration and rerun itbll. Otherwise, all meta changes at the > >>>>>>> primary > >>>>>>> meta region wont be updated at the replica meta regions and it will > >>>>>>> result > >>>>>>> in itbll failures. > >>>>>>> > >>>>>>> Hope this helps, > >>>>>>> > >>>>>>> Huaxiang > >>>>>>> > >>>>>>> > >>>>>>> On 2022/03/25 13:46:42 Andor Molnar wrote: > >>>>>>>> Hi Huaxiang, > >>>>>>>> > >>>>>>>> We use 2.4.6 for the tests. > >>>>>>>> > >>>>>>>> I run itbll with the following command: > >>>>>>>> > >>>>>>>> hbase org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList > >>>>>>> generator 15 1000000 /tmp/hbase-itbll > >>>>>>>> > >>>>>>>> for the generator step and essentially jobs have failed. We can see > >>>>>>>> the > >>>>>>> meta request are spanning out to replicas, but writes start failing > >>>>>>> after > >>>>>>> this due to the stale cache which is not getting updated. > >>>>>>>> > >>>>>>>> Would you please tell me more about ‘hfile refresh’ and how to > >>>>>>> configure it? > >>>>>>>> > >>>>>>>> Thanks, > >>>>>>>> Andor > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>>> On 2022. Mar 24., at 17:43, Huaxiang Sun <huaxiang...@apache.org> > >>>>>>> wrote: > >>>>>>>>> > >>>>>>>>> Hi Andor, > >>>>>>>>> > >>>>>>>>> Which 2.4 release do you test in your lab? We use this feature at > >>>>>>> production cluster with 2.4.5. > >>>>>>>>> At server side, we use hfile refresh instead of wal replication. I > >>>>>>> used to run itbll for each release with this feature enabled. How did > >>>>>>> you > >>>>>>> find the errors, did itbll fail? > >>>>>>>>> > >>>>>>>>> Regards, > >>>>>>>>> Huaxiang > >>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>>> -- > >>>>>> Best regards, > >>>>>> Andrew > >>>>>> > >>>>>> Unrest, ignorance distilled, nihilistic imbeciles - > >>>>>> It's what we’ve earned > >>>>>> Welcome, apocalypse, what’s taken you so long? > >>>>>> Bring us the fitting end that we’ve been counting on > >>>>>> - A23, Welcome, Apocalypse > >>>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> Best regards, > >>>>> Andrew > >>>>> > >>>>> Unrest, ignorance distilled, nihilistic imbeciles - > >>>>> It's what we’ve earned > >>>>> Welcome, apocalypse, what’s taken you so long? > >>>>> Bring us the fitting end that we’ve been counting on > >>>>> - A23, Welcome, Apocalypse > >>>>> > >>> > >>> > > > >
