[
https://issues.apache.org/jira/browse/SPARK-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14519761#comment-14519761
]
Marcelo Vanzin commented on SPARK-7189:
---------------------------------------
bq. There will always be situations that several operations finished within
very short time
That's true. But if we want to go with the assumption that these files are only
even appended to, which is the case for the writer (EventLoggingListener), we
can check {{(timestamp, file size)}} to detect modifications. Or event just
file size, for that matter, and just use timestamp for ordering.
> History server will always reload the same file even when no log file is
> updated
> --------------------------------------------------------------------------------
>
> Key: SPARK-7189
> URL: https://issues.apache.org/jira/browse/SPARK-7189
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.3.0
> Reporter: Zhang, Liye
> Priority: Minor
>
> History server will check every log file with it's modification time. It will
> reload the file if the file's modification time is later or equal to the
> latest modification time it remembered. So it will reload the same file(s)
> periodically if the file(s) with the latest modification time even if there
> is nothing change. This is not necessary.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
