[
https://issues.apache.org/jira/browse/GOBBLIN-1619?focusedWorklogId=758834&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-758834
]
ASF GitHub Bot logged work on GOBBLIN-1619:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 19/Apr/22 22:02
Start Date: 19/Apr/22 22:02
Worklog Time Spent: 10m
Work Description: hanghangliu commented on PR #3477:
URL: https://github.com/apache/gobblin/pull/3477#issuecomment-1103212963
LGTM
Issue Time Tracking
-------------------
Worklog Id: (was: 758834)
Time Spent: 3h 10m (was: 3h)
> WriterUtils.mkdirsWithRecursivePermission contains race condition and puts
> unnecessary load on filesystem
> ---------------------------------------------------------------------------------------------------------
>
> Key: GOBBLIN-1619
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1619
> Project: Apache Gobblin
> Issue Type: Bug
> Reporter: Matthew Ho
> Priority: Minor
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> The current implementation recursively calls fs.mkdirs has the following
> issues:
> * *Race condition for creating parent directories, causing FileNotFound
> exception even when the file exists on file system*
> * {*}HDFS fs.mkdirs atomically creates missing parent directories. Thus, the
> recursive approach is making unnecessary calls.{*}{*}{*}
> HDFS, which the current FileSystem interface is built upon, guarantees the
> parents will be created. So all FileSystem class implementations should also
> follow this behavior.
>
> *Note the
> [FileSystem|https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/fs/FileSystem.html]
> abstract class documentation says the following:*
> The behaviour of the filesystem is [specified in the Hadoop documentation.
> |https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/filesystem/filesystem.html]However,
> the normative specification of the behavior of this class is actually HDFS:
> {color:#de350b}if HDFS does not behave the way these Javadocs or the
> specification in the Hadoop documentations define, assume that the
> documentation is incorrect{color}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
