[
https://issues.apache.org/jira/browse/SPARK-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14192561#comment-14192561
]
Marcelo Vanzin commented on SPARK-1537:
---------------------------------------
bq. It's again a vague statement.
I don't know what is vague about wanting to read the data you write.
bq. Can you share your design detail
I already did way better than that, way earlier in this bug: I shared the
actual code. For this particular question, here it is:
https://github.com/vanzin/spark/blob/yarn-timeline/yarn/timeline/src/main/scala/org/apache/spark/deploy/yarn/timeline/YarnTimelineProvider.scala
See how it reads data from the ATS? It feeds it into the Spark history server,
where the data can be visualized. It's using Yarn internal APIs, which is
generally bad practice.
bq. If you don't agree on it, please post your investigation on YARN-2444,
YARN folks will help you on this issue.
I posted the error and the code to reproduce it. I don't know what else do you
expect from me. If you think it's an authorization issue, test it with 2.6 and
close the bug if you believe it's fixed.
bq. No matter the integration with timeline service, Spark on YARN is picking
Hadoop versions now. It doesn't make sense to ask for a feature by using an
early version that hasn't it.
I'm not sure I really understood what you're trying to say here. Yes, we have
to pick versions. We need a version that supports the features we need. Even if
the API in 2.5 didn't change in 2.6, it seems to have bugs that prevent my
current code from working, so there is no point in trying to integrate with 2.5
as far as I'm concerned. And as far as I know, 2.6 hasn't been released yet.
(BTW, my code used to work with 2.4.)
> Add integration with Yarn's Application Timeline Server
> -------------------------------------------------------
>
> Key: SPARK-1537
> URL: https://issues.apache.org/jira/browse/SPARK-1537
> Project: Spark
> Issue Type: New Feature
> Components: YARN
> Reporter: Marcelo Vanzin
> Assignee: Marcelo Vanzin
>
> It would be nice to have Spark integrate with Yarn's Application Timeline
> Server (see YARN-321, YARN-1530). This would allow users running Spark on
> Yarn to have a single place to go for all their history needs, and avoid
> having to manage a separate service (Spark's built-in server).
> At the moment, there's a working version of the ATS in the Hadoop 2.4 branch,
> although there is still some ongoing work. But the basics are there, and I
> wouldn't expect them to change (much) at this point.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
