[
https://issues.apache.org/jira/browse/SPARK-6270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15215996#comment-15215996
]
Steve Loughran commented on SPARK-6270:
---------------------------------------
There's actually some security implications too: the shared history server
means shared-writeable data dir and no privacy in who can see your histories.
And it has its own scaleability issues with long-logs.
Security wise, users are free to launch their own history server instances; the
existing one could be enhanced by adding in the yarn module the ability to
launch one. Alternatively, use kerberos all the way through, so the caller's
credentials are used to read the data in the FS, or at least to restrict access
to all but job summaries.
Scale-wise streaming jobs create logs too big to read in again. One short term
(i.e. ugly) workaround would be for the logger to be configurable with a log
limit: once a threshold of events/jobs had been saved, it would stop logging
more. But that essentially hard-codes in assumptions about the scalability of
the reader, which may not hold in future.
> Standalone Master hangs when streaming job completes and event logging is
> enabled
> ---------------------------------------------------------------------------------
>
> Key: SPARK-6270
> URL: https://issues.apache.org/jira/browse/SPARK-6270
> Project: Spark
> Issue Type: Bug
> Components: Deploy, Streaming
> Affects Versions: 1.2.0, 1.2.1, 1.3.0, 1.5.1
> Reporter: Tathagata Das
> Priority: Critical
>
> If the event logging is enabled, the Spark Standalone Master tries to
> recreate the web UI of a completed Spark application from its event logs.
> However if this event log is huge (e.g. for a Spark Streaming application),
> then the master hangs in its attempt to read and recreate the web ui. This
> hang causes the whole standalone cluster to be unusable.
> Workaround is to disable the event logging.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
