[
https://issues.apache.org/jira/browse/HDDS-8620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pratyush Bhatt resolved HDDS-8620.
----------------------------------
Resolution: Fixed
> [Hbase-Ozone] Cannot create file, as parent directory doesn't exist
> -------------------------------------------------------------------
>
> Key: HDDS-8620
> URL: https://issues.apache.org/jira/browse/HDDS-8620
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Manager
> Affects Versions: 1.4.0
> Reporter: Pratyush Bhatt
> Priority: Major
>
> Seeing errors like below in Hbase master logs:
> {noformat}
> DIRECTORY_NOT_FOUND org.apache.hadoop.ozone.om.exceptions.OMException: Cannot
> create file{noformat}
> Master also aborted after this:
> {noformat}
> 2023-05-13 15:18:38,441 ERROR org.apache.hadoop.hbase.master.HMaster: *****
> ABORTING master ozn-lease4-5.ozn-lease4.root.hwx.site,22001,1683980247749:
> IOE in log roller *****{noformat}
> Related logs:
> {noformat}
> 2023-05-13 15:18:38,019 WARN
> org.apache.hadoop.hbase.master.MasterRpcServices:
> ozn-lease4-6.ozn-lease4.root.hwx.site,22101,1683980245792 reported a fatal
> error:
> ***** ABORTING region server
> ozn-lease4-6.ozn-lease4.root.hwx.site,22101,1683980245792: Unrecoverable
> exception while closing hbase:meta,,1.1588230740 *****
> Cause:
> DIRECTORY_NOT_FOUND org.apache.hadoop.ozone.om.exceptions.OMException: Cannot
> create file :
> hbase/WALs/ozn-lease4-6.ozn-lease4.root.hwx.site,22101,1683980245792/ozn-lease4-6.ozn-lease4.root.hwx.site%2C22101%2C1683980245792.meta.1683987819776.meta
> as parent directory doesn't exist
> at
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:711)
> at
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.updateKey(OzoneManagerProtocolClientSideTranslatorPB.java:802)
> at
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.hsyncKey(OzoneManagerProtocolClientSideTranslatorPB.java:754)
> at
> org.apache.hadoop.ozone.client.io.BlockOutputStreamEntryPool.hsyncKey(BlockOutputStreamEntryPool.java:362)
> at
> org.apache.hadoop.ozone.client.io.KeyOutputStream.hsync(KeyOutputStream.java:472)
> at
> org.apache.hadoop.ozone.client.io.OzoneOutputStream.hsync(OzoneOutputStream.java:76)
> at
> org.apache.hadoop.fs.ozone.OzoneFSOutputStream.hsync(OzoneFSOutputStream.java:70)
> at
> org.apache.hadoop.fs.ozone.OzoneFSOutputStream.hflush(OzoneFSOutputStream.java:65)
> at
> org.apache.hadoop.fs.FSDataOutputStream.hflush(FSDataOutputStream.java:136)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.sync(ProtobufLogWriter.java:89)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:669)2023-05-13
> 15:18:38,289 INFO org.apache.hadoop.hbase.master.RegionServerTracker:
> RegionServer ephemeral node deleted, processing expiration
> [ozn-lease4-6.ozn-lease4.root.hwx.site,22101,1683980245792]
> 2023-05-13 15:18:38,295 INFO org.apache.hadoop.hbase.master.ServerManager:
> Processing expiration of
> ozn-lease4-6.ozn-lease4.root.hwx.site,22101,1683980245792 on
> ozn-lease4-5.ozn-lease4.root.hwx.site,22001,1683980247749
> 2023-05-13 15:18:38,325 WARN
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL: sync failed
> DIRECTORY_NOT_FOUND org.apache.hadoop.ozone.om.exceptions.OMException: Cannot
> create file :
> hbase/MasterData/WALs/ozn-lease4-5.ozn-lease4.root.hwx.site,22001,1683980247749/ozn-lease4-5.ozn-lease4.root.hwx.site%2C22001%2C1683980247749.1683991068651
> as parent directory doesn't exist
> at
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:711)
> at
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.updateKey(OzoneManagerProtocolClientSideTranslatorPB.java:802)
> at
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.hsyncKey(OzoneManagerProtocolClientSideTranslatorPB.java:754)
> at
> org.apache.hadoop.ozone.client.io.BlockOutputStreamEntryPool.hsyncKey(BlockOutputStreamEntryPool.java:362)
> at
> org.apache.hadoop.ozone.client.io.KeyOutputStream.hsync(KeyOutputStream.java:472)
> at
> org.apache.hadoop.ozone.client.io.OzoneOutputStream.hsync(OzoneOutputStream.java:76)
> at
> org.apache.hadoop.fs.ozone.OzoneFSOutputStream.hsync(OzoneFSOutputStream.java:70)
> at
> org.apache.hadoop.fs.ozone.OzoneFSOutputStream.hflush(OzoneFSOutputStream.java:65)
> at
> org.apache.hadoop.fs.FSDataOutputStream.hflush(FSDataOutputStream.java:136)
> at
> org.apache.hadoop.hbase.io.asyncfs.WrapperAsyncFSOutput.flush0(WrapperAsyncFSOutput.java:92)
> at
> org.apache.hadoop.hbase.io.asyncfs.WrapperAsyncFSOutput.lambda$flush$0(WrapperAsyncFSOutput.java:113)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2023-05-13 15:18:38,378 WARN
> org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter: Init
> output failed,
> path=ofs://ozone1/vol1/bucket1/hbase/MasterData/WALs/ozn-lease4-5.ozn-lease4.root.hwx.site,22001,1683980247749/ozn-lease4-5.ozn-lease4.root.hwx.site%2C22001%2C1683980247749.1683991118335
> DIRECTORY_NOT_FOUND org.apache.hadoop.ozone.om.exceptions.OMException: Cannot
> create file :
> hbase/MasterData/WALs/ozn-lease4-5.ozn-lease4.root.hwx.site,22001,1683980247749/ozn-lease4-5.ozn-lease4.root.hwx.site%2C22001%2C1683980247749.1683991118335
> as one of parent directory is not created
> at
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:711)
> at
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.createFile(OzoneManagerProtocolClientSideTranslatorPB.java:2006)
> at
> org.apache.hadoop.ozone.client.rpc.RpcClient.createFile(RpcClient.java:2011)
> at
> org.apache.hadoop.ozone.client.OzoneBucket.createFile(OzoneBucket.java:821)
> at
> org.apache.hadoop.fs.ozone.BasicRootedOzoneClientAdapterImpl.createFile(BasicRootedOzoneClientAdapterImpl.java:384)
> at
> org.apache.hadoop.fs.ozone.BasicRootedOzoneFileSystem.createOutputStream(BasicRootedOzoneFileSystem.java:275)
> at
> org.apache.hadoop.fs.ozone.BasicRootedOzoneFileSystem.createNonRecursive(BasicRootedOzoneFileSystem.java:261)
> at
> org.apache.hadoop.fs.FileSystem.createNonRecursive(FileSystem.java:1382)
> at
> org.apache.hadoop.fs.FileSystem.createNonRecursive(FileSystem.java:1360)
> at
> org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:63)
> at
> org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:190)
> at
> org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:160)
> at
> org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:116)
> at
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:726)
> at
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:129)
> at
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:890)
> at
> org.apache.hadoop.hbase.wal.AbstractWALRoller$RollController.rollWal(AbstractWALRoller.java:306)
> at
> org.apache.hadoop.hbase.wal.AbstractWALRoller.run(AbstractWALRoller.java:211)
> 2023-05-13 15:18:38,382 ERROR org.apache.hadoop.hbase.wal.AbstractWALRoller:
> Roll wal failed and waiting timeout, will not retry
> DIRECTORY_NOT_FOUND org.apache.hadoop.ozone.om.exceptions.OMException: Cannot
> create file :
> hbase/MasterData/WALs/ozn-lease4-5.ozn-lease4.root.hwx.site,22001,1683980247749/ozn-lease4-5.ozn-lease4.root.hwx.site%2C22001%2C1683980247749.1683991118335
> as one of parent directory is not created
> at
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:711)
> at
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.createFile(OzoneManagerProtocolClientSideTranslatorPB.java:2006)
> at
> org.apache.hadoop.ozone.client.rpc.RpcClient.createFile(RpcClient.java:2011)
> at
> org.apache.hadoop.ozone.client.OzoneBucket.createFile(OzoneBucket.java:821)
> at
> org.apache.hadoop.fs.ozone.BasicRootedOzoneClientAdapterImpl.createFile(BasicRootedOzoneClientAdapterImpl.java:384)
> at
> org.apache.hadoop.fs.ozone.BasicRootedOzoneFileSystem.createOutputStream(BasicRootedOzoneFileSystem.java:275)
> at
> org.apache.hadoop.fs.ozone.BasicRootedOzoneFileSystem.createNonRecursive(BasicRootedOzoneFileSystem.java:261)
> at
> org.apache.hadoop.fs.FileSystem.createNonRecursive(FileSystem.java:1382)
> at
> org.apache.hadoop.fs.FileSystem.createNonRecursive(FileSystem.java:1360)
> at
> org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:63)
> at
> org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:190)
> at
> org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:160)
> at
> org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:116)
> at
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:726)
> at
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:129)
> at
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:890)
> at
> org.apache.hadoop.hbase.wal.AbstractWALRoller$RollController.rollWal(AbstractWALRoller.java:306)
> at
> org.apache.hadoop.hbase.wal.AbstractWALRoller.run(AbstractWALRoller.java:211)
> 2023-05-13 15:18:38,406 WARN
> org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter: Failed to
> write trailer, non-fatal, continuing...
> DIRECTORY_NOT_FOUND org.apache.hadoop.ozone.om.exceptions.OMException: Cannot
> create file :
> hbase/MasterData/WALs/ozn-lease4-5.ozn-lease4.root.hwx.site,22001,1683980247749/ozn-lease4-5.ozn-lease4.root.hwx.site%2C22001%2C1683980247749.1683991068651
> as parent directory doesn't exist
> at
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:711)
> at
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.updateKey(OzoneManagerProtocolClientSideTranslatorPB.java:802)
> at
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.hsyncKey(OzoneManagerProtocolClientSideTranslatorPB.java:754)
> at
> org.apache.hadoop.ozone.client.io.BlockOutputStreamEntryPool.hsyncKey(BlockOutputStreamEntryPool.java:362)
> at
> org.apache.hadoop.ozone.client.io.KeyOutputStream.hsync(KeyOutputStream.java:472)
> at
> org.apache.hadoop.ozone.client.io.OzoneOutputStream.hsync(OzoneOutputStream.java:76)
> at
> org.apache.hadoop.fs.ozone.OzoneFSOutputStream.hsync(OzoneFSOutputStream.java:70)
> at
> org.apache.hadoop.fs.ozone.OzoneFSOutputStream.hflush(OzoneFSOutputStream.java:65)
> at
> org.apache.hadoop.fs.FSDataOutputStream.hflush(FSDataOutputStream.java:136)
> at
> org.apache.hadoop.hbase.io.asyncfs.WrapperAsyncFSOutput.flush0(WrapperAsyncFSOutput.java:92)
> at
> org.apache.hadoop.hbase.io.asyncfs.WrapperAsyncFSOutput.lambda$flush$0(WrapperAsyncFSOutput.java:113)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2023-05-13 15:18:38,428 WARN
> org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter: normal close
> failed, try recover
> DIRECTORY_NOT_FOUND org.apache.hadoop.ozone.om.exceptions.OMException: Cannot
> create file :
> hbase/MasterData/WALs/ozn-lease4-5.ozn-lease4.root.hwx.site,22001,1683980247749/ozn-lease4-5.ozn-lease4.root.hwx.site%2C22001%2C1683980247749.1683991068651
> as parent directory doesn't exist
> at
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:711)
> at
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.updateKey(OzoneManagerProtocolClientSideTranslatorPB.java:802)
> at
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.commitKey(OzoneManagerProtocolClientSideTranslatorPB.java:760)
> at
> org.apache.hadoop.ozone.client.io.BlockOutputStreamEntryPool.commitKey(BlockOutputStreamEntryPool.java:341)
> at
> org.apache.hadoop.ozone.client.io.KeyOutputStream.close(KeyOutputStream.java:559)
> at
> org.apache.hadoop.ozone.client.io.OzoneOutputStream.close(OzoneOutputStream.java:68)
> at
> org.apache.hadoop.fs.ozone.OzoneFSOutputStream.close(OzoneFSOutputStream.java:60)
> at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:77)
> at
> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)
> at
> org.apache.hadoop.hbase.io.asyncfs.WrapperAsyncFSOutput.close(WrapperAsyncFSOutput.java:127)
> at
> org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:169)
> at
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:753)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2023-05-13 15:18:38,441 ERROR org.apache.hadoop.hbase.master.HMaster: Master
> server abort: loaded coprocessors are:
> [org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor]
> 2023-05-13 15:18:38,441 ERROR org.apache.hadoop.hbase.master.HMaster: *****
> ABORTING master ozn-lease4-5.ozn-lease4.root.hwx.site,22001,1683980247749:
> IOE in log roller *****{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
