[
https://issues.apache.org/jira/browse/SPARK-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14192491#comment-14192491
]
Zhijie Shen commented on SPARK-1537:
------------------------------------
bq. BTW, if you want a list of things I think are important for Spark, here are
some quick ones:
Thanks for sharing the details, which are more helpful to clean up the puzzles
than some big but vague statement. Let me go through the aforementioned Jiras:
* YARN-2521: I'd like to keep it open for some further client improvement, such
as local timeline data caching, while YARN-2673 already made the client retry
when the server temporally doesn't respond. Please note that "I think it's
pretty critical when you can't upload your data because the server is down" is
*no longer true* after YARN-2673. On the other side, At the point of view of
the API, it should keep stable.
* YARN-2423: This is proposed to improve the Java libs by adding GET APIs. They
are used to query data, NOT to put data. We do this to help the use case that
the developers write Java code to implement the UI to analyze the timeline
data. Framework integration mainly deals with PUT APIs, and the Java client
libs are already there. Take one step back, apart from the client libs, the
RESTful APIs are always there, which is programming language neutral, and
useful to non-Java developers.
* YARN-2444: It's may be a bug or an improper use case. According to the
exception, the user doesn't pass the authorization for some reason. It is
reported for 2.5, and is probably no longer valid after we fixed a bunch of
security issues for 2.6. We need to do more validation for this issue before a
conclusion. Anyway, it's obviously an internal issue happening in secure mode
only, which should not the API CHANGES.
bq. I understand it doesn't affect the client API and we can still have the
code in,
It seems that we have the agreement that the current timeline service offering
is not blocking the Spark integration work.
> Add integration with Yarn's Application Timeline Server
> -------------------------------------------------------
>
> Key: SPARK-1537
> URL: https://issues.apache.org/jira/browse/SPARK-1537
> Project: Spark
> Issue Type: New Feature
> Components: YARN
> Reporter: Marcelo Vanzin
> Assignee: Marcelo Vanzin
>
> It would be nice to have Spark integrate with Yarn's Application Timeline
> Server (see YARN-321, YARN-1530). This would allow users running Spark on
> Yarn to have a single place to go for all their history needs, and avoid
> having to manage a separate service (Spark's built-in server).
> At the moment, there's a working version of the ATS in the Hadoop 2.4 branch,
> although there is still some ongoing work. But the basics are there, and I
> wouldn't expect them to change (much) at this point.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
