Gentlemen, I am looking into Amaterasu-24 and would like to run the intended changes by you before I make them.
Refactor Spark out of Amaterasu executor to it's own project <https://issues.apache.org/jira/projects/AMATERASU/issues/AMATERASU-24?filter=allopenissues> I understand Spark is just the first of many frameworks that has been lined up for support by Amaterasu. These are the intended changes : 1. Create a new module called "runners" and have the Spark runners under executor pulled into this project (org.apache.executor.execution.actions.runners.spark). We could call it "frameworks" if "runners" is not a great name for this. 2. Will also pull away the Spark dependencies from the Executor to the respective sub-sub-projects (at the moment, just Spark). 3. Since the result of the framework modules would be different bundles, the pattern that I am considering to name the bundle is - "runner-spark". So, it would be "runners:runner-spark" in gradle. 4. On the shell scripts (miniconda and load-spark-env") and the "-cp" passed as commands for the ActionsExecutorLauncher, I could pull them as a separate properties of Spark (inside the runner), so that the Application master can use it. Is it okay if I rename the Miniconda install file to miniconda-install using the "wget -O". The reason why this change is proposed is to avoid hardcoding the conda version inside the code and possibly pull it away into amaterasu.properties file. (The changes are in the ama-start shell scripts and a couple of places inside the code). Please let me know if this would work. Cheers, Arun
