[jira] [Resolved] (HDFS-8564) BlockPoolSlice.checkDirs() will trigger excessive IO while traversing all sub-directories under finalizedDir
[
https://issues.apache.org/jira/browse/HDFS-8564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Esteban Gutierrez resolved HDFS-8564.
-
Resolution: Duplicate
> BlockPoolSlice.checkDirs() will trigger excessive IO while traversing all
> sub-directories under finalizedDir
>
>
> Key: HDFS-8564
> URL: https://issues.apache.org/jira/browse/HDFS-8564
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.0.0
>Reporter: Esteban Gutierrez
>Assignee: Esteban Gutierrez
>Priority: Critical
>
> DataNodes continuously call checkDiskErrorAsync() for multiple operations in
> the DN in order to verify if a volume hasn't experienced any failure. When
> DN.startCheckDiskErrorThread() is invoked we need to traverse all configured
> data volumes on a DN to see which volumes need to be removed (see
> FsVolumeList.checkDir(s)) however that means that for each directory on
> BlockPoolSlice we need to call DiskChecker.checkDirs() which will
> recursively will look into the rbw, tmp and finalized directories:
> {code}
> void checkDirs() throws DiskErrorException {
> DiskChecker.checkDirs(finalizedDir);
> DiskChecker.checkDir(tmpDir);
> DiskChecker.checkDir(rbwDir);
> }
> {code}
> Unfortunately after HDFS-6482, the subdirectory structure is created with the
> following algorithm:
> {code}
> public static File idToBlockDir(File root, long blockId) {
> int d1 = (int)((blockId >> 16) & 0xff);
> int d2 = (int)((blockId >> 8) & 0xff);
> String path = DataStorage.BLOCK_SUBDIR_PREFIX + d1 + SEP +
> DataStorage.BLOCK_SUBDIR_PREFIX + d2;
> return new File(root, path);
> }
> {code}
> Which leaves each data volume with 64K directories (256 directories x 256
> subdirectories) A side effect of this is that if the dentries haven't been
> cached by the OS, then the DN needs to recursively scan up to 64k directories
> x the number of configured data volumes (x number of files) impacting IO for
> other operations while DiskChecker.checkDirs(finalizedDir) is running.
> There are few possibilities to address this problem:
> 1. Do not scan at all finalizedDir
> 2. Limit to one level the number of sub directories to scan recursively. (256)
> 3. Remove a subdirectory immediately it doesn't have any block under it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8564) BlockPoolSlice.checkDirs() will trigger excessive IO while traversing all sub-directories under finalizedDir
Esteban Gutierrez created HDFS-8564:
---
Summary: BlockPoolSlice.checkDirs() will trigger excessive IO
while traversing all sub-directories under finalizedDir
Key: HDFS-8564
URL: https://issues.apache.org/jira/browse/HDFS-8564
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode, HDFS
Affects Versions: 3.0.0
Reporter: Esteban Gutierrez
Priority: Critical
DataNodes continuously call checkDiskErrorAsync() for multiple operations in
the DN in order to verify if a volume hasn't experienced any failure. When
DN.startCheckDiskErrorThread() is invoked we need to traverse all configured
data volumes on a DN to see which volumes need to be removed (see
FsVolumeList.checkDir(s)) however that means that for each directory on
BlockPoolSlice we need to call DiskChecker.checkDirs() which will recursively
will look into the rbw, tmp and finalized directories:
{code}
void checkDirs() throws DiskErrorException {
DiskChecker.checkDirs(finalizedDir);
DiskChecker.checkDir(tmpDir);
DiskChecker.checkDir(rbwDir);
}
{code}
Unfortunately after HDFS-6482, the subdirectory structure is created with the
following algorithm:
{code}
public static File idToBlockDir(File root, long blockId) {
int d1 = (int)((blockId 16) 0xff);
int d2 = (int)((blockId 8) 0xff);
String path = DataStorage.BLOCK_SUBDIR_PREFIX + d1 + SEP +
DataStorage.BLOCK_SUBDIR_PREFIX + d2;
return new File(root, path);
}
{code}
Which leaves each data volume with 64K directories (256 directories x 256
subdirectories) A side effect of this is that if the dentries haven't been
cached by the OS, then the DN needs to recursively scan up to 64k directories x
the number of configured data volumes (x number of files) impacting IO for
other operations while DiskChecker.checkDirs(finalizedDir) is running.
There are few possibilities to address this problem:
1. Do not scan at all finalizedDir
2. Limit to one level the number of sub directories to scan recursively. (256)
3. Remove a subdirectory immediately it doesn't have any block under it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8359) Normalization of timeouts in InputStream and OutputStream
Esteban Gutierrez created HDFS-8359: --- Summary: Normalization of timeouts in InputStream and OutputStream Key: HDFS-8359 URL: https://issues.apache.org/jira/browse/HDFS-8359 Project: Hadoop HDFS Issue Type: Bug Components: datanode, hdfs-client Reporter: Esteban Gutierrez This is a follow up from HDFS-8311. As noticed by [~yzhangal] there are many other places where we need to provide a timeout in the InputStream and OutputStream (perhaps in lesser extent in OS). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-8311) DataStreamer.transfer() should timeout the socket InputStream.
Esteban Gutierrez created HDFS-8311:
---
Summary: DataStreamer.transfer() should timeout the socket
InputStream.
Key: HDFS-8311
URL: https://issues.apache.org/jira/browse/HDFS-8311
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs-client
Reporter: Esteban Gutierrez
While validating some HA failure modes we found that HDFS clients can take a
long time to recover or sometimes don't recover at all since we don't setup the
socket timeout in the InputStream:
{code}
private void transfer () { ...
...
OutputStream unbufOut = NetUtils.getOutputStream(sock, writeTimeout);
InputStream unbufIn = NetUtils.getInputStream(sock);
...
}
{code}
The InputStream should have its own timeout in the same way as the OutputStream.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
