Sammi Chen created HDDS-15150:
---------------------------------
Summary: Datanode scanncer should not mark container as UNHEALTHY
when FD exhausted
Key: HDDS-15150
URL: https://issues.apache.org/jira/browse/HDDS-15150
Project: Apache Ozone
Issue Type: Bug
Reporter: Sammi Chen
Metadata scanner
{code:java}
2026-04-20 22:01:43,978 ERROR
[ContainerMetadataScanner]-org.apache.hadoop.ozone.container.ozoneimpl.BackgroundContainerMetadataScanner:
Corruption detected in container [3980819]. Marking it UNHEALTHY.
java.io.FileNotFoundException:
/data6/hadoop-ozone/datanode/data/hdds/CID-637fe7c5-f40b-4e49-98b3-52154bd669e2/current/containerDir95/3980819/metadata/3980819.container
(Too many open files)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at
org.apache.hadoop.ozone.container.common.impl.ContainerDataYaml.readContainerFile(ContainerDataYaml.java:132)
at
org.apache.hadoop.ozone.container.keyvalue.KeyValueContainerCheck.loadContainerData(KeyValueContainerCheck.java:452)
at
org.apache.hadoop.ozone.container.keyvalue.KeyValueContainerCheck.fastCheck(KeyValueContainerCheck.java:118)
at
org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.scanMetaData(KeyValueContainer.java:959)
at
org.apache.hadoop.ozone.container.ozoneimpl.BackgroundContainerMetadataScanner.scanContainer(BackgroundContainerMetadataScanner.java:78)
at
org.apache.hadoop.ozone.container.ozoneimpl.AbstractBackgroundContainerScanner.scanContainers(AbstractBackgroundContainerScanner.java:98)
at
org.apache.hadoop.ozone.container.ozoneimpl.AbstractBackgroundContainerScanner.runIteration(AbstractBackgroundContainerScanner.java:73)
at
org.apache.hadoop.ozone.container.ozoneimpl.AbstractBackgroundContainerScanner.run(AbstractBackgroundContainerScanner.java:56)
{code}
data scanner
{code:java}
2026-04-20 22:01:43,982 ERROR
[ContainerDataScanner(/data12/hadoop-ozone/datanode/data/hdds)]-org.apache.hadoop.ozone.container.ozoneimpl.BackgroundContainerDataScanner:
Corruption detected in container [16326340]. Marking it UNHEALTHY.
java.nio.file.FileSystemException:
/data12/hadoop-ozone/datanode/data/hdds/CID-637fe7c5-f40b-4e49-98b3-52154bd669e2/current/containerDir143/16326340/chunks/115816904944438982.block:
Too many open files
at
sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at
sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)
at java.nio.channels.FileChannel.open(FileChannel.java:287)
at
org.apache.hadoop.ozone.container.keyvalue.KeyValueContainerCheck.verifyChecksum(KeyValueContainerCheck.java:390)
at
org.apache.hadoop.ozone.container.keyvalue.KeyValueContainerCheck.scanBlock(KeyValueContainerCheck.java:367)
at
org.apache.hadoop.ozone.container.keyvalue.KeyValueContainerCheck.scanData(KeyValueContainerCheck.java:261)
at
org.apache.hadoop.ozone.container.keyvalue.KeyValueContainerCheck.fullCheck(KeyValueContainerCheck.java:159)
at
org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.scanData(KeyValueContainer.java:989)
at
org.apache.hadoop.ozone.container.ozoneimpl.BackgroundContainerDataScanner.scanContainer(BackgroundContainerDataScanner.java:90)
at
org.apache.hadoop.ozone.container.ozoneimpl.AbstractBackgroundContainerScanner.scanContainers(AbstractBackgroundContainerScanner.java:98)
at
org.apache.hadoop.ozone.container.ozoneimpl.AbstractBackgroundContainerScanner.runIteration(AbstractBackgroundContainerScanner.java:73)
at
org.apache.hadoop.ozone.container.ozoneimpl.AbstractBackgroundContainerScanner.run(AbstractBackgroundContainerScanner.java:56)
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
