[jira] [Created] (AMATERASU-27) ama CLI doesn't take into account amaterasu.properties changes in YARN
Nadav Har Tzvi created AMATERASU-27:
---
Summary: ama CLI doesn't take into account amaterasu.properties
changes in YARN
Key: AMATERASU-27
URL: https://issues.apache.org/jira/browse/AMATERASU-27
Project: AMATERASU
Issue Type: Bug
Affects Versions: 0.2.1-incubating
Environment: any hadoop cluster
Reporter: Nadav Har Tzvi
Assignee: Nadav Har Tzvi
Fix For: 0.2.1-incubating
To reproduce:
# On a hadoop cluster
# Setup Amaterasu
# Run a job
# Run ama setup again and change something
# Run a job. The changed setting will not be taken into account.
How to fix:
We need an indication that amaterasu.properties has changed, it can be any
mechanism
(boolean flag, keep record of last 2 file hashes, etc)
When we execute {{ama run}} then the CLI should check whether or not there is a
new version of amaterasu.properties. If there is a new version, upload it to
HDFS.
Existing workarounds:
executing {{ama run}} with {{--force-bin}} will completely remove the existing
Amaterasu HDFS assets and will upload everything again. While it is not amazing
and consumes tons of time (has to upload the Spark client again), it works.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AMATERASU-27) ama CLI doesn't take into account amaterasu.properties changes
[
https://issues.apache.org/jira/browse/AMATERASU-27?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nadav Har Tzvi updated AMATERASU-27:
Summary: ama CLI doesn't take into account amaterasu.properties changes
(was: ama CLI doesn't take into account amaterasu.properties changes in YARN)
> ama CLI doesn't take into account amaterasu.properties changes
> --
>
> Key: AMATERASU-27
> URL: https://issues.apache.org/jira/browse/AMATERASU-27
> Project: AMATERASU
> Issue Type: Bug
>Affects Versions: 0.2.1-incubating
> Environment: any hadoop cluster
>Reporter: Nadav Har Tzvi
>Assignee: Nadav Har Tzvi
>Priority: Major
> Labels: cli, yarn
> Fix For: 0.2.1-incubating
>
>
> To reproduce:
> # On a hadoop cluster
> # Setup Amaterasu
> # Run a job
> # Run ama setup again and change something
> # Run a job. The changed setting will not be taken into account.
> How to fix:
> We need an indication that amaterasu.properties has changed, it can be any
> mechanism
> (boolean flag, keep record of last 2 file hashes, etc)
> When we execute {{ama run}} then the CLI should check whether or not there is
> a new version of amaterasu.properties. If there is a new version, upload it
> to HDFS.
>
> Existing workarounds:
> executing {{ama run}} with {{--force-bin}} will completely remove the
> existing Amaterasu HDFS assets and will upload everything again. While it is
> not amazing and consumes tons of time (has to upload the Spark client again),
> it works.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AMATERASU-3) Scaffolding based on maki.yml
[ https://issues.apache.org/jira/browse/AMATERASU-3?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nadav Har Tzvi resolved AMATERASU-3. Resolution: Implemented Fix Version/s: 0.2.1-incubating > Scaffolding based on maki.yml > - > > Key: AMATERASU-3 > URL: https://issues.apache.org/jira/browse/AMATERASU-3 > Project: AMATERASU > Issue Type: Sub-task >Reporter: Nadav Har Tzvi >Assignee: Nadav Har Tzvi >Priority: Major > Labels: CLI > Fix For: 0.2.1-incubating > > Original Estimate: 24h > Remaining Estimate: 24h > > When the user invokes "ama init -m " > An amaterasu repository will be created/updated with the src files as > specified in the maki. > e.g.: > Given a maki.yml file: > job-name: amaterasu-test # Replace this with your job's name > flow: > - name: start # Name of this step > runner: > group: spark # Currently supporting spark only, but expect more > here in the future! > type: scala # scala, sql, r, python > file: file.scala # Source code for the step > exports: > odd: parquet > - name: step2 > runner: > group: spark > type: scala > file: file2.scala > Then an Amaterasu job repository will be created with src/file.scala and > src/file2.scala -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AMATERASU-2) Initialization of an Amaterasu job repository via the CLI
[ https://issues.apache.org/jira/browse/AMATERASU-2?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nadav Har Tzvi resolved AMATERASU-2. Resolution: Implemented Fix Version/s: 0.2.1-incubating > Initialization of an Amaterasu job repository via the CLI > - > > Key: AMATERASU-2 > URL: https://issues.apache.org/jira/browse/AMATERASU-2 > Project: AMATERASU > Issue Type: Sub-task >Reporter: Nadav Har Tzvi >Assignee: Nadav Har Tzvi >Priority: Major > Labels: CLI > Fix For: 0.2.1-incubating > > Original Estimate: 96h > Remaining Estimate: 96h > > The user will invoke "ama init" to create an Amaterasu job repository. > There are 2 invocation modes: > # "ama init" creates a repository at CWD > # "ama init " creates a repository at > The created repository will consist of: > # A git repository > # maki file > # env + env/default directories, where env/default contains a spark.yml and > job.yml > # empty src directory (maybe we would like to include one scala example and > one python example) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AMATERASU-4) Run an Amaterasu pipeline
[ https://issues.apache.org/jira/browse/AMATERASU-4?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nadav Har Tzvi resolved AMATERASU-4. Resolution: Implemented Fix Version/s: 0.2.1-incubating > Run an Amaterasu pipeline > - > > Key: AMATERASU-4 > URL: https://issues.apache.org/jira/browse/AMATERASU-4 > Project: AMATERASU > Issue Type: Sub-task >Reporter: Nadav Har Tzvi >Assignee: Nadav Har Tzvi >Priority: Major > Fix For: 0.2.1-incubating > > Original Estimate: 168h > Remaining Estimate: 168h > > The user will invoke "ama run" > "ama run" will take in the following parameters (based on ama-start.sh): > -r, --repo = > -b, --branch = , the default is "master" > -e, --env = , this should correspond to a path under /env > directory, e.g. /env/default, /env/test, etc. The default value is "default" > -n, --name = > -i, --job-id = TBD > -r, --report = > Invocation will start Amaterasu on demand. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
