Hi Arun,
You are correct Spark is the first framework, and in my mind,
frameworks should be treated as plugins. Also, we need to consider that not
all frameworks will run under the JVM.
Last, each framework has two modules, a runner (used by both the executor
and the leader) and runtime, to be used by the actions themselves
I would suggest the following structure to start with:
frameworks
|-> spark
|-> runner
|-> runtime
As for the shell scripts, I will leave that for @Nadav, but please have a
look at PR #17 containing the CLI that will replace the scripts as of
0.2.1-incubating.
Cheers,
Yaniv
On Sat, May 26, 2018 at 5:16 PM, Arun Manivannan <a...@arunma.com> wrote:
> Gentlemen,
>
> I am looking into Amaterasu-24 and would like to run the intended changes
> by you before I make them.
>
> Refactor Spark out of Amaterasu executor to it's own project
> <https://issues.apache.org/jira/projects/AMATERASU/
> issues/AMATERASU-24?filter=allopenissues>
>
> I understand Spark is just the first of many frameworks that has been lined
> up for support by Amaterasu.
>
> These are the intended changes :
>
> 1. Create a new module called "runners" and have the Spark runners under
> executor pulled into this project
> (org.apache.executor.execution.actions.runners.spark). We could call it
> "frameworks" if "runners" is not a great name for this.
> 2. Will also pull away the Spark dependencies from the Executor to the
> respective sub-sub-projects (at the moment, just Spark).
> 3. Since the result of the framework modules would be different bundles,
> the pattern that I am considering to name the bundle is - "runner-spark".
> So, it would be "runners:runner-spark" in gradle.
> 4. On the shell scripts (miniconda and load-spark-env") and the "-cp"
> passed as commands for the ActionsExecutorLauncher, I could pull them as a
> separate properties of Spark (inside the runner), so that the Application
> master can use it.
>
> Is it okay if I rename the Miniconda install file to miniconda-install
> using the "wget -O". The reason why this change is proposed is to avoid
> hardcoding the conda version inside the code and possibly pull it away into
> amaterasu.properties file. (The changes are in the ama-start shell scripts
> and a couple of places inside the code).
>
> Please let me know if this would work.
>
> Cheers,
> Arun
>
--
Yaniv Rodenski
+61 477 778 405
ya...@shinto.io
