Prathyusha created HBASE-30493:
----------------------------------
Summary: SplitWALProcedure creates SplitWALRemoteProcedures in a
loop when WAL archival fails without any retry strategy
Key: HBASE-30493
URL: https://issues.apache.org/jira/browse/HBASE-30493
Project: HBase
Issue Type: Bug
Reporter: Prathyusha
SplitWALProcedure creates millions of SplitWALRemoteProcedures in a loop when
WAL archival fails without any retry strategy.
On our new cluster due to a directory file limit issue at HDFS, WAL archival
failed and we saw
~11 Million SplitWALRemoteProcedures created in a loop
Storming the Procedure store
blocked all SCPs, bringing the cluster down
MasterData/WALs reached around ~22.5K, which on crash resulted in 6-7 hours of
startup time
HMaster memory had to be increased to accommodate the loading of procedure
store at startup (this is cause all the SplitWALRemoteProcedures are unfinished
childs of SCPs).
Proposed fix
Instead of creating SplitWALRemoteProcedures in a loop, we attach a retry
strategy [here
|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/SplitWALProcedure.java#L96]
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
