Hey Kevin and Ron, So is the main shortcoming of the launcher library the inability to get an app ID back from YARN? Or are there other issues here that fundamentally regress things for you.
It seems like adding a way to get back the appID would be a reasonable addition to the launcher. - Patrick On Tue, May 12, 2015 at 12:51 PM, Marcelo Vanzin <van...@cloudera.com> wrote: > On Tue, May 12, 2015 at 11:34 AM, Kevin Markey <kevin.mar...@oracle.com> > wrote: > >> I understand that SparkLauncher was supposed to address these issues, but >> it really doesn't. Yarn already provides indirection and an arm's length >> transaction for starting Spark on a cluster. The launcher introduces yet >> another layer of indirection and dissociates the Yarn Client from the >> application that launches it. >> > > Well, not fully. The launcher was supposed to solve "how to launch a Spark > app programatically", but in the first version nothing was added to > actually gather information about the running app. It's also limited in the > way it works because of Spark's limitations (one context per JVM, etc). > > Still, adding things like this is something that is definitely in the scope > for the launcher library; information such as app id can be useful for the > code launching the app, not just in yarn mode. We just have to find a clean > way to provide that information to the caller. > > >> I am still reading the newest code, and we are still researching options >> to move forward. If there are alternatives, we'd like to know. >> >> > Super hacky, but if you launch Spark as a child process you could parse the > stderr and get the app ID. > > -- > Marcelo --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
