[
https://issues.apache.org/jira/browse/SPARK-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14190863#comment-14190863
]
Marcelo Vanzin commented on SPARK-1537:
---------------------------------------
bq. ...security (coming 2.6), high availability, scalability, better client
libs and so on...
That's exactly my point about the ATS not being production-level quality yet.
The current plans I'm aware of would require changes in the ATS API. Since
Spark does not support the ATS at the moment, I'd rather have it support the
new-and-secure-and-scalable-and-available API than the current one. Otherwise
you'll get into the mess of having to conditionally compile code for both APIs,
or implement part of those features into your own client code (something I've
done in my proof-of-concept but I'd really like to avoid, because it's really
just trying to work around limitations in the current ATS design).
So, short version of what I'm trying to say: yes, you can build something that
talks to the current ATS. But given that it currently has shortcomings, and the
fix for those will, as far as I know, affect the client API, I don't see the
point in trying to push that integration at this moment when Spark already has
a working solution for job history, just so that you'll ship code that will be
immediately deprecated by the new ATS...
> Add integration with Yarn's Application Timeline Server
> -------------------------------------------------------
>
> Key: SPARK-1537
> URL: https://issues.apache.org/jira/browse/SPARK-1537
> Project: Spark
> Issue Type: New Feature
> Components: YARN
> Reporter: Marcelo Vanzin
> Assignee: Marcelo Vanzin
>
> It would be nice to have Spark integrate with Yarn's Application Timeline
> Server (see YARN-321, YARN-1530). This would allow users running Spark on
> Yarn to have a single place to go for all their history needs, and avoid
> having to manage a separate service (Spark's built-in server).
> At the moment, there's a working version of the ATS in the Hadoop 2.4 branch,
> although there is still some ongoing work. But the basics are there, and I
> wouldn't expect them to change (much) at this point.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
