[
https://issues.apache.org/jira/browse/GOBBLIN-1619?focusedWorklogId=753863&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-753863
]
ASF GitHub Bot logged work on GOBBLIN-1619:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 07/Apr/22 07:03
Start Date: 07/Apr/22 07:03
Worklog Time Spent: 10m
Work Description: homatthew commented on code in PR #3477:
URL: https://github.com/apache/gobblin/pull/3477#discussion_r844783234
##########
gobblin-utility/src/main/java/org/apache/gobblin/util/WriterUtils.java:
##########
@@ -307,10 +303,20 @@ public static void
mkdirsWithRecursivePermissionWithRetry(final FileSystem fs, f
throw new IOException("Path " + path + "does not exist however it
should. Giving up..."+ e);
}
}
+ }
+
+ private static void gobblinMkDirs(final FileSystem fs, final Path path,
FsPermission perm) throws IOException {
+ Set<Path> parentsThatDidntExistBefore = new HashSet<>();
+ for (Path p = path.getParent(); p != null && !fs.exists(p); p =
p.getParent()) {
+ parentsThatDidntExistBefore.add(p);
+ }
+
+ if (!FileSystem.mkdirs(fs, path, perm)) {
+ throw new IOException(String.format("Unable to mkdir %s with permission
%s", path, perm));
+ }
- // Double check permission, since fs.mkdirs() may not guarantee to set the
permission correctly
Review Comment:
In the latest iteration, I make sure to check for existence based on the
retry policy before setting the permission.
This is a minor edge case for eventually consistent systems, such as S3,
where there could be mutate after creation consistency issues. When there is no
retry policy enabled, there will be no performance hit and it won't check for
existence before setting the permission. (this is the case all the pipelines
internally verify using `go/code data.publisher.retry.enabled` or checking any
containers for log line `org.apache.gobblin.publisher.BaseDataPublisher -
Retry disabled for publish.`)
This is the ideal behavior for HDFS.
Issue Time Tracking
-------------------
Worklog Id: (was: 753863)
Time Spent: 3h (was: 2h 50m)
> WriterUtils.mkdirsWithRecursivePermission contains race condition and puts
> unnecessary load on filesystem
> ---------------------------------------------------------------------------------------------------------
>
> Key: GOBBLIN-1619
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1619
> Project: Apache Gobblin
> Issue Type: Bug
> Reporter: Matthew Ho
> Priority: Minor
> Time Spent: 3h
> Remaining Estimate: 0h
>
> The current implementation recursively calls fs.mkdirs has the following
> issues:
> * *Race condition for creating parent directories, causing FileNotFound
> exception even when the file exists on file system*
> * {*}HDFS fs.mkdirs atomically creates missing parent directories. Thus, the
> recursive approach is making unnecessary calls.{*}{*}{*}
> HDFS, which the current FileSystem interface is built upon, guarantees the
> parents will be created. So all FileSystem class implementations should also
> follow this behavior.
>
> *Note the
> [FileSystem|https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/fs/FileSystem.html]
> abstract class documentation says the following:*
> The behaviour of the filesystem is [specified in the Hadoop documentation.
> |https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/filesystem/filesystem.html]However,
> the normative specification of the behavior of this class is actually HDFS:
> {color:#de350b}if HDFS does not behave the way these Javadocs or the
> specification in the Hadoop documentations define, assume that the
> documentation is incorrect{color}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
