[ https://issues.apache.org/jira/browse/SPARK-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14190946#comment-14190946 ]
Zhijie Shen commented on SPARK-1537: ------------------------------------ bq. That's exactly my point about the ATS not being production-level quality yet. The current plans I'm aware of would require changes in the ATS API. Not to mention the definition of production ready (which differs from community to community, such as Tez and MapReduce), I'm curious about the required API changes of the timeline server. Please elaborate the *changes* in case I've missed some discussion. On the other side, according to my understanding of the timeline server, the ongoing and the future improvement is: 1) Security is coming with Hadoop 2.6, which doesn't affect the usage of the existing APIs in a insecure mode. AFAIK, Spark is working with Hadoop 2.3(4). It should be okay to ride on the timeline server in insecure mode. Whenever upgrading to Hadoop 2.6, you just need to turn on the security switch. 2) Timeline availability and scalability is going to be a server side improvement, but doesn't affect user-faced API. In the scope of YARN, we have already successfully enhance RM with the HA feature while making it transparent to the user. I'm not aware of the major blocker that prevents the timeline server to achieve the same goal. 3) For the client libs, we're trying to help to users to utilize the timeline service more easily (e.g., YARN-2517, YARN-2673), which are either transparent or additions. As I've mentioned before, we're careful about any proposed changes that will break the incompatibility. I'm commenting on this Jira to share more insights about the timeline server to Spark folks in case the folks interested in this YARN offer. It's up to Spark folks to decide whether they want to make use of it or when they make use of it. > Add integration with Yarn's Application Timeline Server > ------------------------------------------------------- > > Key: SPARK-1537 > URL: https://issues.apache.org/jira/browse/SPARK-1537 > Project: Spark > Issue Type: New Feature > Components: YARN > Reporter: Marcelo Vanzin > Assignee: Marcelo Vanzin > > It would be nice to have Spark integrate with Yarn's Application Timeline > Server (see YARN-321, YARN-1530). This would allow users running Spark on > Yarn to have a single place to go for all their history needs, and avoid > having to manage a separate service (Spark's built-in server). > At the moment, there's a working version of the ATS in the Hadoop 2.4 branch, > although there is still some ongoing work. But the basics are there, and I > wouldn't expect them to change (much) at this point. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org