[ https://issues.apache.org/jira/browse/SPARK-1807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14057511#comment-14057511 ]
jay vyas commented on SPARK-1807: --------------------------------- Question: Are you saying the {{SPARK_EXECUTOR_URI}} should be a URI that points to a script? Or that we can replace the URI with a simple path to an executable script. Proposed Alternative: Take this extension to its logical extreme, and actually deprecate the SPARK_EXECUTOR_URI parameter, and instead have a parameter {{SPARK_EXECUTOR_CLASS}}, which is invoked with any number of args, one of which could be SPARK_EXECUTOR_URI. Something like this: {noformat} val c = Class.forName(conf.get("SPARK_EXECUTOR_CLASS")) val m = c.getMethod("startSpark",classOf[SparkExecutor]) m.invoke(null,conf.get("SPARK_EXECUTOR_URI")) {noformat} The advantage of using a class, rather than a shell script, for this - is that we are gauranteed off the bat that the class is providing a contract - to start spark - rather than just a one off script which could do any number of things. Also it means that various classes for this can be maintained and tested over time as first class "spark environment" providers. > Modify SPARK_EXECUTOR_URI to allow for script execution in Mesos. > ----------------------------------------------------------------- > > Key: SPARK-1807 > URL: https://issues.apache.org/jira/browse/SPARK-1807 > Project: Spark > Issue Type: Improvement > Components: Mesos > Affects Versions: 0.9.0 > Reporter: Timothy St. Clair > > Modify Mesos Scheduler integration to allow SPARK_EXECUTOR_URI to be an > executable script. This allows admins to launch spark in any fashion they > desire, vs. just tarball fetching + implied context. -- This message was sent by Atlassian JIRA (v6.2#6252)