[
https://issues.apache.org/jira/browse/MAPREDUCE-7101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16502212#comment-16502212
]
Vinod Kumar Vavilapalli commented on MAPREDUCE-7101:
----------------------------------------------------
There are two options
# Have FS specific scan algorithms.
# Have both algos for all FSes - i.e. instead of completely removing the
modification-time check, have it (helps HDFS), but augment it with a fall-back
more relaxed monitoring which just scans all users' directories every so often
(helps those cloud FSes)
bq. Not sure how bad it could impact performance.
In long running large mutli-tenant cluster scenarios, this is bad. This is
true for all HDFS deployments, and may be some deployments in the cloud. For
cloud deployments that are per user/tenant, it's okay there.
> Revisit behavior of JHS scan file behavior
> ------------------------------------------
>
> Key: MAPREDUCE-7101
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7101
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Wangda Tan
> Priority: Critical
>
> Currently, the JHS scan directory if the modification of *directory* changed:
> {code}
> public synchronized void scanIfNeeded(FileStatus fs) {
> long newModTime = fs.getModificationTime();
> if (modTime != newModTime) {
> <... omitted some logics ...>
> // reset scanTime before scanning happens
> scanTime = System.currentTimeMillis();
> Path p = fs.getPath();
> try {
> scanIntermediateDirectory(p);
> {code}
> This logic relies on an assumption that, the directory's modification time
> will be updated if a file got placed under the directory.
> However, the semantic of directory's modification time is not consistent in
> different FS implementations. For example, MAPREDUCE-6680 fixed some issues
> of truncated modification time. And HADOOP-12837 mentioned on S3, the
> directory's modification time is always 0.
> I think we need to revisit behavior of this logic to make it to more robustly
> work on different file systems.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]