Wei-Chiu Chuang created HDDS-10651:
--------------------------------------
Summary: HBase WAL spitting fails due to lease recovery
Key: HDDS-10651
URL: https://issues.apache.org/jira/browse/HDDS-10651
Project: Apache Ozone
Issue Type: Sub-task
Reporter: Wei-Chiu Chuang
HBase RS log:
{noformat}
2024-04-04 21:33:03,337 INFO org.apache.hadoop.hbase.wal.WALSplitter: Splitting
ofs://ozone1710266930/vol1/bucket1/hbase/WALs/ccycloud-2.weichiu-src.root.comops.site,22101,1712266104607-spli
tting/ccycloud-2.weichiu-src.root.comops.site%2C22101%2C1712266104607.ccycloud-2.weichiu-src.root.comops.site%2C22101%2C1712266104607.regiongroup-0.1712266120442,
size=83 (83bytes)
2024-04-04 21:33:03,337 INFO org.apache.hadoop.hbase.wal.WALSplitter: Splitting
ofs://ozone1710266930/vol1/bucket1/hbase/WALs/ccycloud-2.weichiu-src.root.comops.site,22101,1712266104607-spli
tting/ccycloud-2.weichiu-src.root.comops.site%2C22101%2C1712266104607.ccycloud-2.weichiu-src.root.comops.site%2C22101%2C1712266104607.regiongroup-0.1712266382475,
size=83 (83bytes)
2024-04-04 21:33:03,340 INFO org.apache.hadoop.hbase.util.RecoverLeaseFSUtils:
Recover lease on dfs file
ofs://ozone1710266930/vol1/bucket1/hbase/WALs/ccycloud-2.weichiu-src.root.comops.site
,22101,1712266104607-splitting/ccycloud-2.weichiu-src.root.comops.site%2C22101%2C1712266104607.ccycloud-2.weichiu-src.root.comops.site%2C22101%2C1712266104607.regiongroup-0.1712266120442
2024-04-04 21:33:03,340 INFO org.apache.hadoop.hbase.util.RecoverLeaseFSUtils:
Recover lease on dfs file
ofs://ozone1710266930/vol1/bucket1/hbase/WALs/ccycloud-2.weichiu-src.root.comops.site
,22101,1712266104607-splitting/ccycloud-2.weichiu-src.root.comops.site%2C22101%2C1712266104607.ccycloud-2.weichiu-src.root.comops.site%2C22101%2C1712266104607.regiongroup-0.1712266382475
2024-04-04 21:33:03,361 WARN org.apache.hadoop.hbase.util.RecoverLeaseFSUtils:
attempt=0 on
file=ofs://ozone1710266930/vol1/bucket1/hbase/WALs/ccycloud-2.weichiu-src.root.comops.site,22101,1
712266104607-splitting/ccycloud-2.weichiu-src.root.comops.site%2C22101%2C1712266104607.ccycloud-2.weichiu-src.root.comops.site%2C22101%2C1712266104607.regiongroup-0.1712266382475
after 19ms
KEY_UNDER_LEASE_SOFT_LIMIT_PERIOD
org.apache.hadoop.ozone.om.exceptions.OMException: Open Key
hbase/WALs/ccycloud-2.weichiu-src.root.comops.site,22101,1712266104607-splitting/ccycloud-2.weic
hiu-src.root.comops.site%2C22101%2C1712266104607.ccycloud-2.weichiu-src.root.comops.site%2C22101%2C1712266104607.regiongroup-0.1712266382475
updated recently and is inside soft limit period
at
org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:756)
at
org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.recoverLease(OzoneManagerProtocolClientSideTranslatorPB.java:2560)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.hdds.tracing.TraceAllMethod.invoke(TraceAllMethod.java:76)
at com.sun.proxy.$Proxy27.recoverLease(Unknown Source)
at
org.apache.hadoop.fs.ozone.BasicRootedOzoneClientAdapterImpl.recoverFilePrepare(BasicRootedOzoneClientAdapterImpl.java:1384)
at
org.apache.hadoop.fs.ozone.RootedOzoneFileSystem.recoverLease(RootedOzoneFileSystem.java:154)
at
org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:183)
at
org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:115)
at
org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:70)
at
org.apache.hadoop.hbase.wal.WALSplitter.getReader(WALSplitter.java:455)
at
org.apache.hadoop.hbase.wal.WALSplitter.splitWAL(WALSplitter.java:306)
at
org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:201)
at
org.apache.hadoop.hbase.regionserver.SplitLogWorker.splitLog(SplitLogWorker.java:108)
at
org.apache.hadoop.hbase.regionserver.SplitWALCallable.call(SplitWALCallable.java:100)
at
org.apache.hadoop.hbase.regionserver.SplitWALCallable.call(SplitWALCallable.java:46)
at
org.apache.hadoop.hbase.regionserver.handler.RSProcedureHandler.process(RSProcedureHandler.java:49)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:100)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{noformat}
The Hbase code:
https://github.com/apache/hbase/blob/e48c4485db8d9255510ee1cc9cf465e14de637d7/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java#L457
note: HBase RS wal spliting opens an input stream to the WAL file. It
forcefully invokes lease recovery regardless of the state of file.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
