[
https://issues.apache.org/jira/browse/SPARK-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14518721#comment-14518721
]
Zhang, Liye commented on SPARK-7189:
------------------------------------
Hi [~vanzin], I think using timestamp is not that precise. This method is very
similar with the way using modification time. There will always be situations
that several operations finished within very short time (say less than 1
millisecond or even shorter). So timestamp and modification time can not be
trusted.
The target is to get the status change of the files, including contents change
(write operation) and permission change (rename operation). `Inotify` can get
the change but it's not available in HDFS before version 2.7. One way to tell
the change is to set one flag after each operation and reset the flag after
reloading the file. But this will make the code really ugly, a bad option.
> History server will always reload the same file even when no log file is
> updated
> --------------------------------------------------------------------------------
>
> Key: SPARK-7189
> URL: https://issues.apache.org/jira/browse/SPARK-7189
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.3.0
> Reporter: Zhang, Liye
> Priority: Minor
>
> History server will check every log file with it's modification time. It will
> reload the file if the file's modification time is later or equal to the
> latest modification time it remembered. So it will reload the same file(s)
> periodically if the file(s) with the latest modification time even if there
> is nothing change. This is not necessary.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
