[jira] [Created] (AMATERASU-84) Create local keystore implementation for CryptoUtil

[Yaniv Rodenski (JIRA)]

Yaniv Rodenski created AMATERASU-84:
---

 Summary: Create local keystore implementation for CryptoUtil
 Key: AMATERASU-84
 URL: https://issues.apache.org/jira/browse/AMATERASU-84
 Project: AMATERASU
  Issue Type: Sub-task
Reporter: Yaniv Rodenski






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AMATERASU-83) Support key configuration on configuration objects

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-83?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski resolved AMATERASU-83.
-
Resolution: Fixed

> Support key configuration on configuration objects
> --
>
> Key: AMATERASU-83
> URL: https://issues.apache.org/jira/browse/AMATERASU-83
> Project: AMATERASU
>  Issue Type: Sub-task
>    Reporter: Yaniv Rodenski
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AMATERASU-80) Encrypt field values with encryption keys

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-80?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski updated AMATERASU-80:

Summary: Encrypt field values with encryption keys  (was: Encript field 
values with encryption keys)

> Encrypt field values with encryption keys
> -
>
> Key: AMATERASU-80
> URL: https://issues.apache.org/jira/browse/AMATERASU-80
> Project: AMATERASU
>  Issue Type: Improvement
>    Reporter: Yaniv Rodenski
>Priority: Major
>
> Add support for env-level objects to have their values encrypted by an 
> encryption key.
> in the following example job.yaml all the configuration items are expected to 
> be encrypted by an aws kms key:
> name: test 
> master: local[*] i
> nputRootPath: file:///amaterasu/input 
> outputRootPath: file:///amaterasu/output 
> workingDir: file:///tmp/amaterasu/work_dir 
> configuration:
>     *key:*
>     *type: aws-kms*
>     *id: 40be3055-e2da-460f-aa4a-5e7a648e4d0a*
>     breweryDBApiKey: 7f70ba8064256a93ae81ea95172ed6e6



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[DISCUSS] podling report - Amaterasu

[Yaniv Rodenski]

Hi All,

I propose the following report to be submitted.
Amaterasu


Apache Amaterasu is a framework providing continuous deployment for Big
Data pipelines.

Amaterasu has been incubating since 2017-09-07.
Three most important unfinished issues to address before graduating:


   1.
   2.  Grow up user and contributor communities
   3. Prepare documentation

Are there any issues that the IPMC or ASF Board need to be aware of?

N/A
How has the community developed since the last report?

Since the last report we have been progressing with some use cases, but so
far no one the can be made public.
How has the project developed since the last report?

Since the last report 7 PRs ware merged with 53 commits. In addition there
are 4 PRs with 19 commits so far waiting fixes based on a review.

In addition there are only 3 more pending tasks for the next release that
is planned in two weeks. This version will also contain the much awaited
documentation (one of our main milestones).

Also work on the release following it (0.3.0) have began with 2 of the
pending PRs which are to the 0.3.0 working branch.
How would you assess the podling's maturity?


Please feel free to add your own commentary.

   -  Initial setup
   -  Working towards first release
   -  Community building
   -  Nearing graduation
   -  Other:

Date of last release:


2018-07-18
When were the last committers or PPMC members elected?

n/a however a vote thread has been restarted in the private@ mailing list
to vote in two committers.
Have your mentors been helpful and responsive?


n/a
Signed-off-by:


   -  (amaterasu) Jean-Baptiste Onofré Comments:
   -  (amaterasu) Olivier Lamy Comments:
   -  (amaterasu) Davor Bonaci Comments:

IPMC/Shepherd notes:

[jira] [Closed] (AMATERASU-53) Support action level dependencies

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-53?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski closed AMATERASU-53.
---
Resolution: Fixed

> Support action level dependencies 
> --
>
> Key: AMATERASU-53
> URL: https://issues.apache.org/jira/browse/AMATERASU-53
> Project: AMATERASU
>  Issue Type: Task
>    Reporter: Yaniv Rodenski
>Assignee: Kirupa
>Priority: Major
> Fix For: 0.2.1-incubating
>
>
> Add support for dependencies files to be stored per action.
> This should be stored under *src/\{action_name}/deps/* and be deployed with 
> the job level dependencies.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (AMATERASU-22) Improve the way yarn containers are allocated to consider specific task requirments

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-22?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski closed AMATERASU-22.
---
Resolution: Fixed

> Improve the way yarn containers are allocated to consider specific task 
> requirments
> ---
>
> Key: AMATERASU-22
> URL: https://issues.apache.org/jira/browse/AMATERASU-22
> Project: AMATERASU
>  Issue Type: Improvement
>Reporter: Eyal Ben Ivri
>Assignee: Eyal Ben Ivri
>Priority: Major
> Fix For: 0.2.1-incubating
>
>
> Currently, YARN containers are created without consideration for specific 
> requirements of a task, just the job configuration.
> As a pipeline developer, I would like to be able to override the 
> specifications (v-cores or memory) of a specific task inside a flow. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AMATERASU-83) Support key configuration on configuration objects

[Yaniv Rodenski (JIRA)]

Yaniv Rodenski created AMATERASU-83:
---

 Summary: Support key configuration on configuration objects
 Key: AMATERASU-83
 URL: https://issues.apache.org/jira/browse/AMATERASU-83
 Project: AMATERASU
  Issue Type: Sub-task
Reporter: Yaniv Rodenski






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AMATERASU-82) Create AWS KMS implementation for CryptoUtil

[Yaniv Rodenski (JIRA)]

Yaniv Rodenski created AMATERASU-82:
---

 Summary: Create AWS KMS implementation for CryptoUtil
 Key: AMATERASU-82
 URL: https://issues.apache.org/jira/browse/AMATERASU-82
 Project: AMATERASU
  Issue Type: Sub-task
Reporter: Yaniv Rodenski






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AMATERASU-81) CryptoUtil

[Yaniv Rodenski (JIRA)]

Yaniv Rodenski created AMATERASU-81:
---

 Summary: CryptoUtil
 Key: AMATERASU-81
 URL: https://issues.apache.org/jira/browse/AMATERASU-81
 Project: AMATERASU
  Issue Type: Sub-task
Reporter: Yaniv Rodenski


Create a utility to encrypt/decrypt configuration values, the CryptoUtil.

The util should should support different KMSs implementations



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AMATERASU-80) Encript field values with encryption keys

[Yaniv Rodenski (JIRA)]

Yaniv Rodenski created AMATERASU-80:
---

 Summary: Encript field values with encryption keys
 Key: AMATERASU-80
 URL: https://issues.apache.org/jira/browse/AMATERASU-80
 Project: AMATERASU
  Issue Type: Improvement
Reporter: Yaniv Rodenski


Add support for env-level objects to have their values encrypted by an 
encryption key.

in the following example job.yaml all the configuration items are expected to 
be encrypted by an aws kms key:

name: test 
master: local[*] i
nputRootPath: file:///amaterasu/input 
outputRootPath: file:///amaterasu/output 
workingDir: file:///tmp/amaterasu/work_dir 
configuration:
    *key:*
    *type: aws-kms*
    *id: 40be3055-e2da-460f-aa4a-5e7a648e4d0a*
    breweryDBApiKey: 7f70ba8064256a93ae81ea95172ed6e6



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Podling Report Reminder - July 2019

[Yaniv Rodenski]

Hi Davor,

That seems wrong to me. According to GitHub, just in the last month we had
8 PRs 5 of which are already merged and 3 are waiting for fixes based on
review. In the last 3 months I believe we are getting closer to 100
commits, I'll get the exact count for the report soon.

Also, this is from our GitHub pulse for the last month - Excluding merges, *3
authors* have pushed *22 commits* to master and *22 commits* to all
branches. On master, *130 files* have changed and there have been *1,567*
*additions* and *5,829* *deletions*
<https://github.com/apache/incubator-amaterasu/compare/master@%7B1558659799%7D...master>.


As for the release, we are very close to finishing our next release that
includes another major milestone, the documentation. out of 38 issues, 33
are done and two are in progress. I believe this is a matter of a couple of
weeks before we can start voting on it.

In addition 2 PRs ware opened for the next version.

Once we have the documentation out (we are hoping for a public use-case by
than as well) I think we can start test the community growth, but that my
opinion, happy to have that discussion.

Cheers,
Yaniv

On Sun, Jun 23, 2019 at 7:45 AM Davor Bonaci  wrote:

> Hey everybody --
> what's the current view on the state of the project?
>
> I see a few commits in the last 3 months (9 commits, 3 contributors), ~no
> mailing list activity, ~no community growth since the start of the project,
> no releases in a year, and no plans of one.
>
> Davor
>
> On Fri, Jun 21, 2019 at 7:30 PM  wrote:
>
> > Dear podling,
> >
> > This email was sent by an automated system on behalf of the Apache
> > Incubator PMC. It is an initial reminder to give you plenty of time to
> > prepare your quarterly board report.
> >
> > The board meeting is scheduled for Wed, 17 July 2019, 10:30 am PDT.
> > The report for your podling will form a part of the Incubator PMC
> > report. The Incubator PMC requires your report to be submitted 2 weeks
> > before the board meeting, to allow sufficient time for review and
> > submission (Wed, July 03).
> >
> > Please submit your report with sufficient time to allow the Incubator
> > PMC, and subsequently board members to review and digest. Again, the
> > very latest you should submit your report is 2 weeks prior to the board
> > meeting.
> >
> > Candidate names should not be made public before people are actually
> > elected, so please do not include the names of potential committers or
> > PPMC members in your report.
> >
> > Thanks,
> >
> > The Apache Incubator PMC
> >
> > Submitting your Report
> >
> > --
> >
> > Your report should contain the following:
> >
> > *   Your project name
> > *   A brief description of your project, which assumes no knowledge of
> > the project or necessarily of its field
> > *   A list of the three most important issues to address in the move
> > towards graduation.
> > *   Any issues that the Incubator PMC or ASF Board might wish/need to be
> > aware of
> > *   How has the community developed since the last report
> > *   How has the project developed since the last report.
> > *   How does the podling rate their own maturity.
> >
> > This should be appended to the Incubator Wiki page at:
> >
> > https://cwiki.apache.org/confluence/display/INCUBATOR/July2019
> >
> > Note: This is manually populated. You may need to wait a little before
> > this page is created from a template.
> >
> > Note: The format of the report has changed to use markdown.
> >
> > Mentors
> > ---
> >
> > Mentors should review reports for their project(s) and sign them off on
> > the Incubator wiki page. Signing off reports shows that you are
> > following the project - projects that are not signed may raise alarms
> > for the Incubator PMC.
> >
> > Incubator PMC
> >
>


-- 
Yaniv Rodenski

+61 477 778 405
ya...@shinto.io

[jira] [Resolved] (AMATERASU-58) Convert Leader Common to Kotlin

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-58?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski resolved AMATERASU-58.
-
Resolution: Fixed

> Convert Leader Common to Kotlin
> ---
>
> Key: AMATERASU-58
> URL: https://issues.apache.org/jira/browse/AMATERASU-58
> Project: AMATERASU
>  Issue Type: Sub-task
>Reporter: Guy Peleg
>Priority: Major
>
> Lets rewrite the package from scratch in Kotlin, without touching the scala 
> code first.
> points for discussion,
> Immutability, Statelessness other good design 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AMATERASU-60) Remove old leader-common

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-60?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski resolved AMATERASU-60.
-
Resolution: Fixed

> Remove old leader-common
> 
>
> Key: AMATERASU-60
> URL: https://issues.apache.org/jira/browse/AMATERASU-60
> Project: AMATERASU
>  Issue Type: Sub-task
>Reporter: Guy Peleg
>Priority: Major
>
> remove the old code, run tests and regression



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AMATERASU-59) Convert Leader to Kotlin

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-59?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski resolved AMATERASU-59.
-
Resolution: Fixed

> Convert Leader to Kotlin
> 
>
> Key: AMATERASU-59
> URL: https://issues.apache.org/jira/browse/AMATERASU-59
> Project: AMATERASU
>  Issue Type: Sub-task
>Reporter: Guy Peleg
>Priority: Major
>
> let's start converting leader code to Kotlin by using the new Leader-Common



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AMATERASU-61) Remove old Leader

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-61?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski resolved AMATERASU-61.
-
Resolution: Fixed

> Remove old Leader
> -
>
> Key: AMATERASU-61
> URL: https://issues.apache.org/jira/browse/AMATERASU-61
> Project: AMATERASU
>  Issue Type: Sub-task
>Reporter: Guy Peleg
>Priority: Major
>
> Remove old Leader run reggression



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AMATERASU-36) Refactoring - leader: extract projects per resource manager + common

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-36?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski resolved AMATERASU-36.
-
Resolution: Fixed

> Refactoring - leader: extract projects per resource manager + common
> 
>
> Key: AMATERASU-36
> URL: https://issues.apache.org/jira/browse/AMATERASU-36
> Project: AMATERASU
>  Issue Type: Improvement
>    Reporter: Yaniv Rodenski
>    Assignee: Yaniv Rodenski
>Priority: Major
> Fix For: 0.2.1-incubating
>
>
> To simplify deployments and the Amaterasu codebase, we will create 
> differently projects/binaries per resource manager, for example:
> leader
>    |>leader.common 
>    |>leader.mesos
>    |>leader.YARN
>    |>leader.K8s
> ...



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (AMATERASU-36) Refactoring - leader: extract projects per resource manager + common

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-36?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski reassigned AMATERASU-36:
---

Assignee: Yaniv Rodenski  (was: Guy Peleg)

> Refactoring - leader: extract projects per resource manager + common
> 
>
> Key: AMATERASU-36
> URL: https://issues.apache.org/jira/browse/AMATERASU-36
> Project: AMATERASU
>  Issue Type: Improvement
>    Reporter: Yaniv Rodenski
>    Assignee: Yaniv Rodenski
>Priority: Major
> Fix For: 0.2.1-incubating
>
>
> To simplify deployments and the Amaterasu codebase, we will create 
> differently projects/binaries per resource manager, for example:
> leader
>    |>leader.common 
>    |>leader.mesos
>    |>leader.YARN
>    |>leader.K8s
> ...



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AMATERASU-30) Job level spark memory requirements aren't respected in mesos

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-30?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski resolved AMATERASU-30.
-
Resolution: Fixed

> Job level spark memory requirements aren't respected in mesos
> -
>
> Key: AMATERASU-30
> URL: https://issues.apache.org/jira/browse/AMATERASU-30
> Project: AMATERASU
>  Issue Type: Bug
>Affects Versions: 0.2.0-incubating
>Reporter: Nadav Har Tzvi
>    Assignee: Yaniv Rodenski
>Priority: Major
>  Labels: Spark, scheduler
> Fix For: 0.2.1-incubating
>
>
> When trying to override the default spark memory requirements using job.yml, 
> the entry is completely ignored.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AMATERASU-79) Remove deprecated spark runner

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-79?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski resolved AMATERASU-79.
-
Resolution: Fixed

> Remove deprecated spark runner
> --
>
> Key: AMATERASU-79
> URL: https://issues.apache.org/jira/browse/AMATERASU-79
> Project: AMATERASU
>  Issue Type: Bug
>    Reporter: Yaniv Rodenski
>    Assignee: Yaniv Rodenski
>Priority: Major
> Fix For: 0.2.1-incubating
>
>
> The spark-runner module is no longer in use



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AMATERASU-46) PySpark - Move the persistence functionality from the runner implementation into ama_context

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-46?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski resolved AMATERASU-46.
-
Resolution: Fixed

> PySpark - Move the persistence functionality from the runner implementation 
> into ama_context 
> -
>
> Key: AMATERASU-46
> URL: https://issues.apache.org/jira/browse/AMATERASU-46
> Project: AMATERASU
>  Issue Type: Task
>    Reporter: Yaniv Rodenski
>Assignee: Nadav Har Tzvi
>Priority: Major
> Fix For: 0.2.1-incubating
>
>
> Persistence of datasets should be done explicitly via ama_context.persist 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AMATERASU-71) Add Python tests to Travis CI

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-71?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski resolved AMATERASU-71.
-
Resolution: Fixed

> Add Python tests to Travis CI
> -
>
> Key: AMATERASU-71
> URL: https://issues.apache.org/jira/browse/AMATERASU-71
> Project: AMATERASU
>  Issue Type: Task
>Affects Versions: 0.2.1-incubating
>Reporter: Nadav Har Tzvi
>Priority: Major
>  Labels: travis-ci
> Fix For: 0.2.1-incubating
>
>
> We need to add the upcoming python tests to be automatically invoked by 
> travis.
> The first feature that will need this is [AMATERASU-45]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AMATERASU-46) PySpark - Move the persistence functionality from the runner implementation into ama_context

[Yaniv Rodenski (JIRA)]

[ 
https://issues.apache.org/jira/browse/AMATERASU-46?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16856183#comment-16856183
 ] 

Yaniv Rodenski commented on AMATERASU-46:
-

Is this still relevant? I think this was already merged.

> PySpark - Move the persistence functionality from the runner implementation 
> into ama_context 
> -
>
> Key: AMATERASU-46
> URL: https://issues.apache.org/jira/browse/AMATERASU-46
> Project: AMATERASU
>  Issue Type: Task
>    Reporter: Yaniv Rodenski
>Assignee: Nadav Har Tzvi
>Priority: Major
> Fix For: 0.2.1-incubating
>
>
> Persistence of datasets should be done explicitly via ama_context.persist 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AMATERASU-52) Implement AmaContext.datastores

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-52?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski resolved AMATERASU-52.
-
Resolution: Fixed

> Implement AmaContext.datastores
> ---
>
> Key: AMATERASU-52
> URL: https://issues.apache.org/jira/browse/AMATERASU-52
> Project: AMATERASU
>  Issue Type: Task
>    Reporter: Yaniv Rodenski
>Assignee: Arun Manivannan
>Priority: Major
> Fix For: 0.2.1-incubating
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> AmaContext.datastores should contain the data from datastores.yaml



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AMATERASU-79) Remove deprecated spark runner

[Yaniv Rodenski (JIRA)]

Yaniv Rodenski created AMATERASU-79:
---

 Summary: Remove deprecated spark runner
 Key: AMATERASU-79
 URL: https://issues.apache.org/jira/browse/AMATERASU-79
 Project: AMATERASU
  Issue Type: Bug
Reporter: Yaniv Rodenski
Assignee: Yaniv Rodenski
 Fix For: 0.2.1-incubating


The spark-runner module is no longer in use



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AMATERASU-78) ama build - crates Docker images from a job definistion

[Yaniv Rodenski (JIRA)]

Yaniv Rodenski created AMATERASU-78:
---

 Summary: ama build - crates Docker images from a job definistion
 Key: AMATERASU-78
 URL: https://issues.apache.org/jira/browse/AMATERASU-78
 Project: AMATERASU
  Issue Type: Sub-task
Reporter: Yaniv Rodenski
 Fix For: 0.3.0-incubating


the cli command ama build should create Docker images for each action and push 
them to a provided docker registry.

parameters:

 - target: the registry's url



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AMATERASU-77) ama-cli

[Yaniv Rodenski (JIRA)]

Yaniv Rodenski created AMATERASU-77:
---

 Summary: ama-cli
 Key: AMATERASU-77
 URL: https://issues.apache.org/jira/browse/AMATERASU-77
 Project: AMATERASU
  Issue Type: Improvement
Reporter: Yaniv Rodenski
 Fix For: 0.3.0-incubating


create a cli project for amaterasu



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AMATERASU-76) add support for username and password for private repos

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-76?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski resolved AMATERASU-76.
-
Resolution: Fixed

> add support for username and password for private repos
> ---
>
> Key: AMATERASU-76
> URL: https://issues.apache.org/jira/browse/AMATERASU-76
> Project: AMATERASU
>  Issue Type: Improvement
>    Reporter: Yaniv Rodenski
>    Assignee: Yaniv Rodenski
>Priority: Major
> Fix For: 0.2.1-incubating
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> add optional --user-name and --password parameters to ama-start-yarn and 
> ama-start-mesos to be used while authenticating to private repos.
> for example:
> ./ama-start-yarn.sh 
> --repo=https://github.com/roadan/amaterasu-beers-job-example.git 
> --branch=master --env=test --report=code --username=DarkHelmet 
> --password=12345



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (AMATERASU-32) Investigate why Amaterasu requires minimum of 2G memory to run on DC/OS

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-32?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski closed AMATERASU-32.
---
Resolution: Invalid

> Investigate why Amaterasu requires minimum of 2G memory to run on DC/OS
> ---
>
> Key: AMATERASU-32
> URL: https://issues.apache.org/jira/browse/AMATERASU-32
> Project: AMATERASU
>  Issue Type: Task
>Affects Versions: 0.2.1-incubating
>Reporter: Nadav Har Tzvi
>Priority: Major
>  Labels: mesos
> Fix For: 0.2.1-incubating
>
>
> This is even weirder than the problem we have in EMR. In DC/OS we can't do 
> anything without requesting 2G of memory from Mesos, and that's for the 
> job-samples.
> Why on standalone deployment of Mesos we need 1G of memory and on DC/OS it is 
> 2G?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AMATERASU-69) support URL values for action files

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-69?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski resolved AMATERASU-69.
-
Resolution: Fixed

> support URL values for action files
> ---
>
> Key: AMATERASU-69
> URL: https://issues.apache.org/jira/browse/AMATERASU-69
> Project: AMATERASU
>  Issue Type: Improvement
>    Reporter: Yaniv Rodenski
>    Assignee: Yaniv Rodenski
>Priority: Major
> Fix For: 0.2.1-incubating
>
>
> This is an improvement request, to allow pyspark (but not only) source files 
> to be located on a remote server.
> Example:
> ---
> job-name: pyspark-pipeline
> flow:
>   - name: start
>     runner:
>       group: spark
>       type: pyspark
>     file: [https://webserver/files/action1.py]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (AMATERASU-31) Investigate why PySpark actions on EMR require minimum of 2G memory

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-31?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski closed AMATERASU-31.
---
Resolution: Invalid

> Investigate why PySpark actions on EMR require minimum of 2G memory
> ---
>
> Key: AMATERASU-31
> URL: https://issues.apache.org/jira/browse/AMATERASU-31
> Project: AMATERASU
>  Issue Type: Task
>Affects Versions: 0.2.0-incubating
>Reporter: Nadav Har Tzvi
>Assignee: Eyal Ben Ivri
>Priority: Major
> Fix For: 0.2.1-incubating
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (AMATERASU-76) add support for username and password for private repos

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-76?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski reassigned AMATERASU-76:
---

Assignee: Yaniv Rodenski

> add support for username and password for private repos
> ---
>
> Key: AMATERASU-76
> URL: https://issues.apache.org/jira/browse/AMATERASU-76
> Project: AMATERASU
>  Issue Type: Improvement
>    Reporter: Yaniv Rodenski
>    Assignee: Yaniv Rodenski
>Priority: Major
> Fix For: 0.2.1-incubating
>
>
> add optional --user-name and --password parameters to ama-start-yarn and 
> ama-start-mesos to be used while authenticating to private repos.
> for example:
> ./ama-start-yarn.sh 
> --repo=https://github.com/roadan/amaterasu-beers-job-example.git 
> --branch=master --env=test --report=code --username=DarkHelmet 
> --password=12345



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AMATERASU-76) add support for username and password for private repos

[Yaniv Rodenski (JIRA)]

Yaniv Rodenski created AMATERASU-76:
---

 Summary: add support for username and password for private repos
 Key: AMATERASU-76
 URL: https://issues.apache.org/jira/browse/AMATERASU-76
 Project: AMATERASU
  Issue Type: Improvement
Reporter: Yaniv Rodenski
 Fix For: 0.2.1-incubating


add optional --user-name and --password parameters to ama-start-yarn and 
ama-start-mesos to be used while authenticating to private repos.

for example:

./ama-start-yarn.sh 
--repo=https://github.com/roadan/amaterasu-beers-job-example.git 
--branch=master --env=test --report=code --username=DarkHelmet --password=12345



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (AMATERASU-30) Job level spark memory requirements aren't respected in mesos

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-30?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski reassigned AMATERASU-30:
---

Assignee: Yaniv Rodenski

> Job level spark memory requirements aren't respected in mesos
> -
>
> Key: AMATERASU-30
> URL: https://issues.apache.org/jira/browse/AMATERASU-30
> Project: AMATERASU
>  Issue Type: Bug
>Affects Versions: 0.2.0-incubating
>Reporter: Nadav Har Tzvi
>    Assignee: Yaniv Rodenski
>Priority: Major
>  Labels: Spark, scheduler
> Fix For: 0.2.1-incubating
>
>
> When trying to override the default spark memory requirements using job.yml, 
> the entry is completely ignored.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AMATERASU-64) Create runtime.yaml for YARN

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-64?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski resolved AMATERASU-64.
-
Resolution: Fixed

> Create runtime.yaml for YARN
> 
>
> Key: AMATERASU-64
> URL: https://issues.apache.org/jira/browse/AMATERASU-64
> Project: AMATERASU
>  Issue Type: Sub-task
>Affects Versions: 0.2.1-incubating
>    Reporter: Yaniv Rodenski
>    Assignee: Yaniv Rodenski
>Priority: Major
> Fix For: 0.2.1-incubating
>
>
> Each container should be created with a runtime.yaml containing the jobId and 
> action name



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AMATERASU-62) Add the use of the ConfigManager to create env.yml on YARN

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-62?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski resolved AMATERASU-62.
-
Resolution: Fixed

> Add the use of the ConfigManager to create env.yml on YARN
> --
>
> Key: AMATERASU-62
> URL: https://issues.apache.org/jira/browse/AMATERASU-62
> Project: AMATERASU
>  Issue Type: Sub-task
>    Reporter: Yaniv Rodenski
>    Assignee: Yaniv Rodenski
>Priority: Major
> Fix For: 0.2.1-incubating
>
>
> use the ConfigManager to create an env.yml file in each container as 
> implemented in the Mesos JobScheduler lines 161-163



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AMATERASU-63) Create datasets.yaml for YARN

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-63?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski resolved AMATERASU-63.
-
Resolution: Fixed

> Create datasets.yaml for YARN
> -
>
> Key: AMATERASU-63
> URL: https://issues.apache.org/jira/browse/AMATERASU-63
> Project: AMATERASU
>  Issue Type: Sub-task
>    Reporter: Yaniv Rodenski
>    Assignee: Yaniv Rodenski
>Priority: Major
> Fix For: 0.2.1-incubating
>
>
> Create a datasets.yaml file in each container. the datasets.yaml is currently 
> created out of the exports section in the yaml, to be replaced by 
> AMATERASU-52 later



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[Discuss] April report draft

[Yaniv Rodenski]

Hi All


First of all, a big congrats to Nadav and wife or the birth of their first
child two days ago. While it is major progress, it did not make it to this
months report :)


Please see below the suggested report:


Amaterasu


Apache Amaterasu is a framework providing configuration management and

deployment for Big Data Pipelines.


It provides the following capabilities:

Continuous integration tools to package pipelines and run tests.

A repository to store those packaged applications: the applications

repository.

A repository to store the pipelines, and engine configuration (for

instance, the location of the Spark master, etc.): per environment - the

configuration repository.

A dashboard to monitor the pipelines.

A DSL and integration hooks allowing third parties to easily integrate.


Amaterasu has been incubating since 2017-09.


Three most important issues to address in the move towards graduation:


  1. Grow up user and contributor communities

  2. Prepare documentation


Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be aware

of?


  There is still concern regarding the progress of the project. The
community tries to address this by making the project more accessible. The
first major step is releasing the documentation and simplifying the install
process. In addition, we are working closely with some of the newer
contributors to mentor them and help them be more productive.


How has the community developed since the last report?


  Since the last report, we have been focusing on the next release with
strong focus building the documentation which we hope will help with
adoption. We are still working with a couple of organization on their use
case, hopefully, those could become a public use cases soon.


How has the project developed since the last report?


  * 11 pull requests by 4 contributors have been opened since the last
report. Out of those 2 have been closed, 2 are awaiting review/fixes and
the rest have been merged

  * Documentation has been progressing and is due for the upcoming release.


Date of the last release:


  12 July 2018


When were the last committers or PMC members elected?


  N/A


Have your mentors been helpful and responsive or are things falling through
the cracks? In the latter case, please list any open issues that need to be
addressed

.


  N/A


Signed-off-by:


  [X](amaterasu) Jean-Baptiste Onofré

 Comments:

  [X](amaterasu) Olivier Lamy

 Comments:

  [X](amaterasu) Davor Bonaci

 Comments:



IPMC/Shepherd notes:

[jira] [Assigned] (AMATERASU-69) support URL values for action files

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-69?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski reassigned AMATERASU-69:
---

Assignee: Yaniv Rodenski

> support URL values for action files
> ---
>
> Key: AMATERASU-69
> URL: https://issues.apache.org/jira/browse/AMATERASU-69
> Project: AMATERASU
>  Issue Type: Improvement
>    Reporter: Yaniv Rodenski
>    Assignee: Yaniv Rodenski
>Priority: Major
> Fix For: 0.2.1-incubating
>
>
> This is an improvement request, to allow pyspark (but not only) source files 
> to be located on a remote server.
> Example:
> ---
> job-name: pyspark-pipeline
> flow:
>   - name: start
>     runner:
>       group: spark
>       type: pyspark
>     file: [https://webserver/files/action1.py]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (AMATERASU-69) support URL values for action files

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-69?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski reassigned AMATERASU-69:
---

Assignee: (was: Yaniv Rodenski)

> support URL values for action files
> ---
>
> Key: AMATERASU-69
> URL: https://issues.apache.org/jira/browse/AMATERASU-69
> Project: AMATERASU
>  Issue Type: Improvement
>    Reporter: Yaniv Rodenski
>Priority: Major
> Fix For: 0.2.1-incubating
>
>
> This is an improvement request, to allow pyspark (but not only) source files 
> to be located on a remote server.
> Example:
> ---
> job-name: pyspark-pipeline
> flow:
>   - name: start
>     runner:
>       group: spark
>       type: pyspark
>     file: [https://webserver/files/action1.py]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Podling Report Reminder - April 2019

[Yaniv Rodenski]

Hi Davor,

Nadav volunteered to work on this month's report, but he is currently in
the delivery room (might even be out by now) with his wife, so I'm taking
over, I'll send a report later today/tomorrow.

In general, we have been progressing with the next release, and one of the
biggest things we have progressed with is the documentation, which should
help to get new projects onboard. In addition, we are progressing with one
of the projects I've mentioned in the previous report, adding some features
based on their needs, and hopefully, we can report on that publically soon
as well.

As I wrote above, I'll send the detailed draft soon.
Yaniv

On Sun, Mar 31, 2019 at 4:17 PM Davor Bonaci  wrote:

> Hi guys --
> What do we think about the project status these days?
>
> Davor
>
> On Fri, Mar 29, 2019 at 4:07 PM  wrote:
>
> > Dear podling,
> >
> > This email was sent by an automated system on behalf of the Apache
> > Incubator PMC. It is an initial reminder to give you plenty of time to
> > prepare your quarterly board report.
> >
> > The board meeting is scheduled for Wed, 17 April 2019, 10:30 am PDT.
> > The report for your podling will form a part of the Incubator PMC
> > report. The Incubator PMC requires your report to be submitted 2 weeks
> > before the board meeting, to allow sufficient time for review and
> > submission (Wed, April 03).
> >
> > Please submit your report with sufficient time to allow the Incubator
> > PMC, and subsequently board members to review and digest. Again, the
> > very latest you should submit your report is 2 weeks prior to the board
> > meeting.
> >
> > Candidate names should not be made public before people are actually
> > elected, so please do not include the names of potential committers or
> > PPMC members in your report.
> >
> > Thanks,
> >
> > The Apache Incubator PMC
> >
> > Submitting your Report
> >
> > --
> >
> > Your report should contain the following:
> >
> > *   Your project name
> > *   A brief description of your project, which assumes no knowledge of
> > the project or necessarily of its field
> > *   A list of the three most important issues to address in the move
> > towards graduation.
> > *   Any issues that the Incubator PMC or ASF Board might wish/need to be
> > aware of
> > *   How has the community developed since the last report
> > *   How has the project developed since the last report.
> > *   How does the podling rate their own maturity.
> >
> > This should be appended to the Incubator Wiki page at:
> >
> > https://wiki.apache.org/incubator/April2019
> >
> > Note: This is manually populated. You may need to wait a little before
> > this page is created from a template.
> >
> > Mentors
> > ---
> >
> > Mentors should review reports for their project(s) and sign them off on
> > the Incubator wiki page. Signing off reports shows that you are
> > following the project - projects that are not signed may raise alarms
> > for the Incubator PMC.
> >
> > Incubator PMC
> >
>


-- 
Yaniv Rodenski

+61 477 778 405
ya...@shinto.io

[jira] [Assigned] (AMATERASU-63) Create datasets.yaml for YARN

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-63?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski reassigned AMATERASU-63:
---

Assignee: Yaniv Rodenski  (was: Eyal Ben Ivri)

> Create datasets.yaml for YARN
> -
>
> Key: AMATERASU-63
> URL: https://issues.apache.org/jira/browse/AMATERASU-63
> Project: AMATERASU
>  Issue Type: Sub-task
>    Reporter: Yaniv Rodenski
>    Assignee: Yaniv Rodenski
>Priority: Major
> Fix For: 0.2.1-incubating
>
>
> Create a datasets.yaml file in each container. the datasets.yaml is currently 
> created out of the exports section in the yaml, to be replaced by 
> AMATERASU-52 later



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (AMATERASU-64) Create runtime.yaml for YARN

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-64?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski reassigned AMATERASU-64:
---

Assignee: Yaniv Rodenski  (was: Eyal Ben Ivri)

> Create runtime.yaml for YARN
> 
>
> Key: AMATERASU-64
> URL: https://issues.apache.org/jira/browse/AMATERASU-64
> Project: AMATERASU
>  Issue Type: Sub-task
>Affects Versions: 0.2.1-incubating
>    Reporter: Yaniv Rodenski
>    Assignee: Yaniv Rodenski
>Priority: Major
> Fix For: 0.2.1-incubating
>
>
> Each container should be created with a runtime.yaml containing the jobId and 
> action name



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [Discuss] datasets input file in user's repository

[Yaniv Rodenski]

Hi Nadav,

I think datasets should be per environment, (for example, it is very common
to use different databases for dev/test/prod), so I think that datasets as
configurations in Amaterasu should sit under env).

Cheers,
Yaniv

On Tue, Mar 19, 2019 at 5:13 AM Nadav Har Tzvi 
wrote:

> Hi,
>
> Just wanna open this up for discussion as it seems we somehow skipped this
> point.
> Basically, by now we pretty much have the new datasets APIs in place in the
> Python SDK and in implementing frameworks. (amaterasu-pyspark,
> amaterasu-pandas, amaterasu-python)
> The only question left is regarding the way we get the datasets
> definitions.
> Currently, we still look up the datasets definitions in the maki file,
> under the action's exports.
> Do we intend to keep it that way? I assume not as I think that every action
> needs access to all defined datasets.
> In that case, how will the user submit datasets configuration? Is it
> another file next to the maki.yaml? Is it a file that resides in the
> environment, e.g. next to the env.yaml? Is it not even a file on its own
> but a part of the env.yaml?
> Ideas, anyone?
>
> Let's discuss this please!
>
> Cheers,
> Nadav
>


-- 
Yaniv Rodenski

+61 477 778 405
ya...@shinto.io

[jira] [Assigned] (AMATERASU-75) Change ClusterConfig to load using Konf

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-75?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski reassigned AMATERASU-75:
---

Assignee: Yaniv Rodenski

> Change ClusterConfig to load using Konf
> ---
>
> Key: AMATERASU-75
> URL: https://issues.apache.org/jira/browse/AMATERASU-75
> Project: AMATERASU
>  Issue Type: Improvement
>    Reporter: Yaniv Rodenski
>    Assignee: Yaniv Rodenski
>Priority: Major
>
> Current implementation is both too verbose and also causes issues in 
> non-scala code



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AMATERASU-75) Change ClusterConfig to load using Konf

[Yaniv Rodenski (JIRA)]

Yaniv Rodenski created AMATERASU-75:
---

 Summary: Change ClusterConfig to load using Konf
 Key: AMATERASU-75
 URL: https://issues.apache.org/jira/browse/AMATERASU-75
 Project: AMATERASU
  Issue Type: Improvement
Reporter: Yaniv Rodenski


Current implementation is both too verbose and also causes issues in non-scala 
code



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (AMATERASU-62) Add the use of the ConfigManager to create env.yml on YARN

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-62?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski reassigned AMATERASU-62:
---

Assignee: Yaniv Rodenski  (was: Eyal Ben Ivri)

> Add the use of the ConfigManager to create env.yml on YARN
> --
>
> Key: AMATERASU-62
> URL: https://issues.apache.org/jira/browse/AMATERASU-62
> Project: AMATERASU
>  Issue Type: Sub-task
>    Reporter: Yaniv Rodenski
>    Assignee: Yaniv Rodenski
>Priority: Major
> Fix For: 0.2.1-incubating
>
>
> use the ConfigManager to create an env.yml file in each container as 
> implemented in the Mesos JobScheduler lines 161-163



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AMATERASU-25) Create documentation with ReadTheDocs

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-25?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski resolved AMATERASU-25.
-
Resolution: Delivered

> Create documentation with ReadTheDocs
> -
>
> Key: AMATERASU-25
> URL: https://issues.apache.org/jira/browse/AMATERASU-25
> Project: AMATERASU
>  Issue Type: Task
>Affects Versions: 0.2.1-incubating
>Reporter: Nadav Har Tzvi
>    Assignee: Yaniv Rodenski
>Priority: Major
>  Labels: docuentation
> Fix For: 0.2.1-incubating
>
>
> We need to start filling in documentation for Apache Amaterasu.
> We will use readthedocs for this purpose.
> We need to set up a /docs directory with rst files.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AMATERASU-73) Long running leader

[Yaniv Rodenski (JIRA)]

Yaniv Rodenski created AMATERASU-73:
---

 Summary: Long running leader
 Key: AMATERASU-73
 URL: https://issues.apache.org/jira/browse/AMATERASU-73
 Project: AMATERASU
  Issue Type: Improvement
Reporter: Yaniv Rodenski


The Amaterasu leader should become a long-running service, supporting multiple 
deployments at a time.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AMATERASU-72) Add ActiveMQ implementation for Mesos

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-72?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski updated AMATERASU-72:

Fix Version/s: 0.3.0-incubating

> Add ActiveMQ implementation for Mesos
> -
>
> Key: AMATERASU-72
> URL: https://issues.apache.org/jira/browse/AMATERASU-72
> Project: AMATERASU
>  Issue Type: Sub-task
>    Reporter: Yaniv Rodenski
>Priority: Major
> Fix For: 0.3.0-incubating
>
>
> ActiveMQ should be available in Mesos environments, to be used as a log store



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AMATERASU-72) Add ActiveMQ implementation for Mesos

[Yaniv Rodenski (JIRA)]

Yaniv Rodenski created AMATERASU-72:
---

 Summary: Add ActiveMQ implementation for Mesos
 Key: AMATERASU-72
 URL: https://issues.apache.org/jira/browse/AMATERASU-72
 Project: AMATERASU
  Issue Type: Sub-task
Reporter: Yaniv Rodenski


ActiveMQ should be available in Mesos environments, to be used as a log store



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (AMATERASU-27) ama CLI doesn't take into account amaterasu.properties changes

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-27?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski closed AMATERASU-27.
---
Resolution: Invalid

> ama CLI doesn't take into account amaterasu.properties changes
> --
>
> Key: AMATERASU-27
> URL: https://issues.apache.org/jira/browse/AMATERASU-27
> Project: AMATERASU
>  Issue Type: Bug
>Affects Versions: 0.2.1-incubating
> Environment: any hadoop cluster
>Reporter: Nadav Har Tzvi
>Assignee: Nadav Har Tzvi
>Priority: Major
>  Labels: cli, yarn
> Fix For: 0.2.1-incubating
>
>
> To reproduce:
>  # On a hadoop cluster
>  # Setup Amaterasu
>  # Run a job
>  # Run ama setup again and change something
>  # Run a job. The changed setting will not be taken into account.
> How to fix:
> We need an indication that amaterasu.properties has changed, it can be any 
> mechanism
> (boolean flag, keep record of last 2 file hashes, etc)
> When we execute {{ama run}} then the CLI should check whether or not there is 
> a new version of amaterasu.properties. If there is a new version, upload it 
> to HDFS.
>  
> Existing workarounds:
> executing {{ama run}} with {{--force-bin}} will completely remove the 
> existing Amaterasu HDFS assets and will upload everything again. While it is 
> not amazing and consumes tons of time (has to upload the Spark client again), 
> it works.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (AMATERASU-69) support URL values for action files

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-69?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski reassigned AMATERASU-69:
---

Assignee: Yaniv Rodenski

> support URL values for action files
> ---
>
> Key: AMATERASU-69
> URL: https://issues.apache.org/jira/browse/AMATERASU-69
> Project: AMATERASU
>  Issue Type: Improvement
>    Reporter: Yaniv Rodenski
>    Assignee: Yaniv Rodenski
>Priority: Major
> Fix For: 0.2.1-incubating
>
>
> This is an improvement request, to allow pyspark (but not only) source files 
> to be located on a remote server.
> Example:
> ---
> job-name: pyspark-pipeline
> flow:
>   - name: start
>     runner:
>       group: spark
>       type: pyspark
>     file: [https://webserver/files/action1.py]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (AMATERASU-48) Create a spark-shell based setup provider for spark scala

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-48?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski closed AMATERASU-48.
---
Resolution: Invalid

We will deprecate the scala shell soon

> Create a spark-shell based setup provider for spark scala
> -
>
> Key: AMATERASU-48
> URL: https://issues.apache.org/jira/browse/AMATERASU-48
> Project: AMATERASU
>  Issue Type: Task
>    Reporter: Yaniv Rodenski
>    Assignee: Yaniv Rodenski
>Priority: Major
> Fix For: 0.2.1-incubating
>
>
> As a first step towards the new execution model, a scala-submit group will be 
> created using the spark-shell command line to execute a spark script as a 
> whole (not line-by-line)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AMATERASU-8) Support for Jar/External actions

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-8?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski resolved AMATERASU-8.

Resolution: Fixed

> Support for Jar/External actions
> 
>
> Key: AMATERASU-8
> URL: https://issues.apache.org/jira/browse/AMATERASU-8
> Project: AMATERASU
>  Issue Type: New Feature
>Affects Versions: 0.2.1-incubating
>    Reporter: Yaniv Rodenski
>    Assignee: Yaniv Rodenski
>Priority: Major
> Fix For: 0.2.1-incubating
>
>
> As a pipeline developer, I can use a pre-built jar or other packaged formats 
> for my actions definitions.
> Note: This need to be implemented in the framework's runner (i.e. Spark etc.) 
> as well as in the JobManager 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AMATERASU-70) Support s3 urls in actions source files

[Yaniv Rodenski (JIRA)]

Yaniv Rodenski created AMATERASU-70:
---

 Summary: Support s3 urls in actions source files
 Key: AMATERASU-70
 URL: https://issues.apache.org/jira/browse/AMATERASU-70
 Project: AMATERASU
  Issue Type: Sub-task
Reporter: Yaniv Rodenski


Extend the ability to have the file attribute to support s3 urls.

Example:

---
job-name: pyspark-pipeline
flow:
  - name: start
    runner:
      group: spark
      type: pyspark
    file: *s3://mybyucket/files/action1.py* 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AMATERASU-69) support URL values for action files

[Yaniv Rodenski (JIRA)]

Yaniv Rodenski created AMATERASU-69:
---

 Summary: support URL values for action files
 Key: AMATERASU-69
 URL: https://issues.apache.org/jira/browse/AMATERASU-69
 Project: AMATERASU
  Issue Type: Improvement
Reporter: Yaniv Rodenski
 Fix For: 0.2.1-incubating


This is an improvement request, to allow pyspark (but not only) source files to 
be located on a remote server.

Example:

---
job-name: pyspark-pipeline
flow:
  - name: start
    runner:
      group: spark
      type: pyspark
    file: [https://webserver/files/action1.py]

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AMATERASU-68) Support test pipelines

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-68?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski updated AMATERASU-68:

Fix Version/s: 0.3.0-incubating

> Support test pipelines
> --
>
> Key: AMATERASU-68
> URL: https://issues.apache.org/jira/browse/AMATERASU-68
> Project: AMATERASU
>  Issue Type: Bug
>    Reporter: Yaniv Rodenski
>Priority: Major
> Fix For: 0.3.0-incubating
>
>
> Test pipelines are additional YAML definitions that run to validate 
> integration points in the pipeline:
> repo/
>    |- env
>    |- src
>    |- *test*
> To run test cases, use the --test switch for ama-start



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AMATERASU-68) Support test pipelines

[Yaniv Rodenski (JIRA)]

Yaniv Rodenski created AMATERASU-68:
---

 Summary: Support test pipelines
 Key: AMATERASU-68
 URL: https://issues.apache.org/jira/browse/AMATERASU-68
 Project: AMATERASU
  Issue Type: Bug
Reporter: Yaniv Rodenski


Test pipelines are additional YAML definitions that run to validate integration 
points in the pipeline:

repo/
   |- env
   |- src
   |- *test*

To run test cases, use the --test switch for ama-start



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AMATERASU-12) Spark framework - support for spark streaming

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-12?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski updated AMATERASU-12:

Affects Version/s: (was: 0.3.0-incubating)

> Spark framework - support for spark streaming
> -
>
> Key: AMATERASU-12
> URL: https://issues.apache.org/jira/browse/AMATERASU-12
> Project: AMATERASU
>  Issue Type: New Feature
>    Reporter: Yaniv Rodenski
>Priority: Major
> Fix For: 0.4.0-incubating
>
>
> As a pipeline developer, I can use Amaterasu to deploy long-running spark 
> streaming jobs



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AMATERASU-12) Spark framework - support for spark streaming

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-12?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski updated AMATERASU-12:

Fix Version/s: (was: 0.3.0-incubating)
   0.4.0-incubating

> Spark framework - support for spark streaming
> -
>
> Key: AMATERASU-12
> URL: https://issues.apache.org/jira/browse/AMATERASU-12
> Project: AMATERASU
>  Issue Type: New Feature
>Affects Versions: 0.3.0-incubating
>    Reporter: Yaniv Rodenski
>Priority: Major
> Fix For: 0.4.0-incubating
>
>
> As a pipeline developer, I can use Amaterasu to deploy long-running spark 
> streaming jobs



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AMATERASU-65) Log centralisation

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-65?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski updated AMATERASU-65:

Issue Type: New Feature  (was: Bug)

> Log centralisation
> --
>
> Key: AMATERASU-65
> URL: https://issues.apache.org/jira/browse/AMATERASU-65
> Project: AMATERASU
>  Issue Type: New Feature
>    Reporter: Yaniv Rodenski
>Assignee: Eyal Ben Ivri
>Priority: Major
> Fix For: 0.3.0-incubating
>
>
> Logs from all actions and the leader should aggregate and be stored in 
> configurable locations such as:
>  - The Amaterasu Monitoring UI internal datastore
>  - Time series databases
>  - Storage locations (such as S3/HDFS/etc)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AMATERASU-67) rename the flow directive to seq

[Yaniv Rodenski (JIRA)]

Yaniv Rodenski created AMATERASU-67:
---

 Summary: rename the flow directive to seq
 Key: AMATERASU-67
 URL: https://issues.apache.org/jira/browse/AMATERASU-67
 Project: AMATERASU
  Issue Type: Sub-task
Reporter: Yaniv Rodenski
Assignee: Yaniv Rodenski






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AMATERASU-10) Action level configuration

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-10?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski updated AMATERASU-10:

Fix Version/s: (was: 0.3.0-incubating)
   0.2.1-incubating

> Action level configuration
> --
>
> Key: AMATERASU-10
> URL: https://issues.apache.org/jira/browse/AMATERASU-10
> Project: AMATERASU
>  Issue Type: New Feature
>Affects Versions: 0.3.0-incubating
>    Reporter: Yaniv Rodenski
>Priority: Major
> Fix For: 0.2.1-incubating
>
>
> As a pipeline developer I can define configuration elements for a specific 
> action.
> Need to think how clashing configuration will be handled, for the first 
> version we can go with letting pipeline configuration override any clashing 
> configuration in the action.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AMATERASU-66) add batch directive to the maki DSL

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-66?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski updated AMATERASU-66:

Issue Type: New Feature  (was: Task)

> add batch directive to the maki DSL
> ---
>
> Key: AMATERASU-66
> URL: https://issues.apache.org/jira/browse/AMATERASU-66
> Project: AMATERASU
>  Issue Type: New Feature
>    Reporter: Yaniv Rodenski
>    Assignee: Yaniv Rodenski
>Priority: Major
> Fix For: 0.3.0-incubating
>
>
> The batch directive allows actions and flows defined under it to run in 
> parallel
> Considering the following definition:
> ---
> job-name: amaterasu-test 
> batch: 
>   - name: action1 
>     runner: 
>       group: spark 
>       type: jar 
>     artifact: ...
>      - name: action2 
>        runner: 
>          group: spark 
>          type: jar 
>       artifact: ...
> ...
> action1 and action2 will be queued immediately for execution (the queueing 
> order will be determined by the order of the actions in the YAML).
> *Batch and sequences:*
> a batch element can contain a seq element and also the other way round which 
> will lead to the following:
>  - When a sec element is queued during a batch, it will execute all of its 
> internal actions in order.
>  - When a batch element is queued during a sequential flow, all of its 
> internal actions and sequences must complete before the next action is 
> executed 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AMATERASU-66) add batch directive to the maki DSL

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-66?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski updated AMATERASU-66:

Issue Type: Task  (was: New Feature)

> add batch directive to the maki DSL
> ---
>
> Key: AMATERASU-66
> URL: https://issues.apache.org/jira/browse/AMATERASU-66
> Project: AMATERASU
>  Issue Type: Task
>    Reporter: Yaniv Rodenski
>    Assignee: Yaniv Rodenski
>Priority: Major
> Fix For: 0.3.0-incubating
>
>
> The batch directive allows actions and flows defined under it to run in 
> parallel
> Considering the following definition:
> ---
> job-name: amaterasu-test 
> batch: 
>   - name: action1 
>     runner: 
>       group: spark 
>       type: jar 
>     artifact: ...
>      - name: action2 
>        runner: 
>          group: spark 
>          type: jar 
>       artifact: ...
> ...
> action1 and action2 will be queued immediately for execution (the queueing 
> order will be determined by the order of the actions in the YAML).
> *Batch and sequences:*
> a batch element can contain a seq element and also the other way round which 
> will lead to the following:
>  - When a sec element is queued during a batch, it will execute all of its 
> internal actions in order.
>  - When a batch element is queued during a sequential flow, all of its 
> internal actions and sequences must complete before the next action is 
> executed 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AMATERASU-66) add batch directive to the maki DSL

[Yaniv Rodenski (JIRA)]

Yaniv Rodenski created AMATERASU-66:
---

 Summary: add batch directive to the maki DSL
 Key: AMATERASU-66
 URL: https://issues.apache.org/jira/browse/AMATERASU-66
 Project: AMATERASU
  Issue Type: Bug
Reporter: Yaniv Rodenski
Assignee: Yaniv Rodenski
 Fix For: 0.3.0-incubating


The batch directive allows actions and flows defined under it to run in parallel

Considering the following definition:

---

job-name: amaterasu-test 
batch: 
  - name: action1 
    runner: 
      group: spark 
      type: jar 
    artifact: ...
     - name: action2 
       runner: 
         group: spark 
         type: jar 
      artifact: ...
...

action1 and action2 will be queued immediately for execution (the queueing 
order will be determined by the order of the actions in the YAML).

*Batch and sequences:*

a batch element can contain a seq element and also the other way round which 
will lead to the following:

 - When a sec element is queued during a batch, it will execute all of its 
internal actions in order.
 - When a batch element is queued during a sequential flow, all of its internal 
actions and sequences must complete before the next action is executed 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AMATERASU-66) add batch directive to the maki DSL

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-66?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski updated AMATERASU-66:

Issue Type: New Feature  (was: Bug)

> add batch directive to the maki DSL
> ---
>
> Key: AMATERASU-66
> URL: https://issues.apache.org/jira/browse/AMATERASU-66
> Project: AMATERASU
>  Issue Type: New Feature
>    Reporter: Yaniv Rodenski
>    Assignee: Yaniv Rodenski
>Priority: Major
> Fix For: 0.3.0-incubating
>
>
> The batch directive allows actions and flows defined under it to run in 
> parallel
> Considering the following definition:
> ---
> job-name: amaterasu-test 
> batch: 
>   - name: action1 
>     runner: 
>       group: spark 
>       type: jar 
>     artifact: ...
>      - name: action2 
>        runner: 
>          group: spark 
>          type: jar 
>       artifact: ...
> ...
> action1 and action2 will be queued immediately for execution (the queueing 
> order will be determined by the order of the actions in the YAML).
> *Batch and sequences:*
> a batch element can contain a seq element and also the other way round which 
> will lead to the following:
>  - When a sec element is queued during a batch, it will execute all of its 
> internal actions in order.
>  - When a batch element is queued during a sequential flow, all of its 
> internal actions and sequences must complete before the next action is 
> executed 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AMATERASU-65) Log centralisation

[Yaniv Rodenski (JIRA)]

Yaniv Rodenski created AMATERASU-65:
---

 Summary: Log centralisation
 Key: AMATERASU-65
 URL: https://issues.apache.org/jira/browse/AMATERASU-65
 Project: AMATERASU
  Issue Type: Bug
Reporter: Yaniv Rodenski
Assignee: Eyal Ben Ivri
 Fix For: 0.3.0-incubating


Logs from all actions and the leader should aggregate and be stored in 
configurable locations such as:

 - The Amaterasu Monitoring UI internal datastore
 - Time series databases
 - Storage locations (such as S3/HDFS/etc)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AMATERASU-7) Scheduled pipelines/actions

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-7?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski updated AMATERASU-7:
---
Affects Version/s: (was: 0.3.0-incubating)
   0.4.0-incubating

> Scheduled pipelines/actions
> ---
>
> Key: AMATERASU-7
> URL: https://issues.apache.org/jira/browse/AMATERASU-7
> Project: AMATERASU
>  Issue Type: New Feature
>Affects Versions: 0.4.0-incubating
>    Reporter: Yaniv Rodenski
>    Assignee: Yaniv Rodenski
>Priority: Major
> Fix For: 0.3.0-incubating
>
>
> As a pipeline developer, I can define a schedule in the maki.yml for my 
> pipeline to run. I can also define a schedule for an action in a long-running 
> pipeline
> job-name:amaterasu-test
> type: long-running
> flow:
>- name: action1
>  type: long-running
>  runner:
>  group: spark
>  type: scala   
>  file: src/file.scala
>  exports:
>  oddData: parquet
>- name: action2
>  type: scheduled
>  schedule: 10 * * * *
>  runner:
>  group: spark
>  type: python
>  file: src/file2.scala
>  imports: 
>  oddData: parquet
>  file: file2.py



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AMATERASU-54) Use Docker for Mesos containerization

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-54?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski updated AMATERASU-54:

Fix Version/s: (was: 0.4.0-incubating)
   0.3.0-incubating

> Use Docker for Mesos containerization
> -
>
> Key: AMATERASU-54
> URL: https://issues.apache.org/jira/browse/AMATERASU-54
> Project: AMATERASU
>  Issue Type: Task
>    Reporter: Yaniv Rodenski
>Assignee: Yariv Triffon
>Priority: Major
> Fix For: 0.3.0-incubating
>
>
> The Amaterasu Mesos framework should launch Docker containers rather than 
> vanilla Mesos containers.
> Note that resources should be added using Docker and not the Mesos addUri 
> method.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (AMATERASU-10) Action level configuration

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-10?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski closed AMATERASU-10.
---
Resolution: Duplicate

> Action level configuration
> --
>
> Key: AMATERASU-10
> URL: https://issues.apache.org/jira/browse/AMATERASU-10
> Project: AMATERASU
>  Issue Type: New Feature
>Affects Versions: 0.3.0-incubating
>    Reporter: Yaniv Rodenski
>Priority: Major
> Fix For: 0.3.0-incubating
>
>
> As a pipeline developer I can define configuration elements for a specific 
> action.
> Need to think how clashing configuration will be handled, for the first 
> version we can go with letting pipeline configuration override any clashing 
> configuration in the action.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AMATERASU-9) Nested pipelines

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-9?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski updated AMATERASU-9:
---
Fix Version/s: (was: 0.3.0-incubating)
   0.4.0-incubating

> Nested pipelines
> 
>
> Key: AMATERASU-9
> URL: https://issues.apache.org/jira/browse/AMATERASU-9
> Project: AMATERASU
>  Issue Type: Improvement
>Affects Versions: 0.4.0-incubating
>    Reporter: Yaniv Rodenski
>Priority: Major
> Fix For: 0.4.0-incubating
>
>
> As a pipeline developer, I can use existing pipelines in my maki.yml.
> Need to think how clashing configuration will be handled, for the first 
> version we can go with letting configuration settings in the calling pipeline 
> will override any existing configuration in embedded pipelines 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AMATERASU-9) Nested pipelines

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-9?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski updated AMATERASU-9:
---
Affects Version/s: (was: 0.3.0-incubating)
   0.4.0-incubating

> Nested pipelines
> 
>
> Key: AMATERASU-9
> URL: https://issues.apache.org/jira/browse/AMATERASU-9
> Project: AMATERASU
>  Issue Type: Improvement
>Affects Versions: 0.4.0-incubating
>    Reporter: Yaniv Rodenski
>Priority: Major
> Fix For: 0.3.0-incubating
>
>
> As a pipeline developer, I can use existing pipelines in my maki.yml.
> Need to think how clashing configuration will be handled, for the first 
> version we can go with letting configuration settings in the calling pipeline 
> will override any existing configuration in embedded pipelines 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AMATERASU-7) Scheduled pipelines/actions

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-7?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski updated AMATERASU-7:
---
Fix Version/s: (was: 0.3.0-incubating)
   0.4.0-incubating

> Scheduled pipelines/actions
> ---
>
> Key: AMATERASU-7
> URL: https://issues.apache.org/jira/browse/AMATERASU-7
> Project: AMATERASU
>  Issue Type: New Feature
>Affects Versions: 0.4.0-incubating
>    Reporter: Yaniv Rodenski
>    Assignee: Yaniv Rodenski
>Priority: Major
> Fix For: 0.4.0-incubating
>
>
> As a pipeline developer, I can define a schedule in the maki.yml for my 
> pipeline to run. I can also define a schedule for an action in a long-running 
> pipeline
> job-name:amaterasu-test
> type: long-running
> flow:
>- name: action1
>  type: long-running
>  runner:
>  group: spark
>  type: scala   
>  file: src/file.scala
>  exports:
>  oddData: parquet
>- name: action2
>  type: scheduled
>  schedule: 10 * * * *
>  runner:
>  group: spark
>  type: python
>  file: src/file2.scala
>  imports: 
>  oddData: parquet
>  file: file2.py



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AMATERASU-6) Long running actions/pipelines

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-6?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski updated AMATERASU-6:
---
Affects Version/s: (was: 0.3.0-incubating)
   0.4.0-incubating

> Long running actions/pipelines 
> ---
>
> Key: AMATERASU-6
> URL: https://issues.apache.org/jira/browse/AMATERASU-6
> Project: AMATERASU
>  Issue Type: New Feature
>Affects Versions: 0.4.0-incubating
>    Reporter: Yaniv Rodenski
>Assignee: Eyal Ben Ivri
>Priority: Major
> Fix For: 0.3.0-incubating
>
>
>  As a pipeline developer, I can define pipelines and actions to be 
> long-running, for workloads such as stream processing etc. 
> Example maki.yml
> job-name:amaterasu-test
> type: long-running
> flow:
>- name: action1
>  type: long-running
>  runner:
>  group: spark
>  type: scala   
>  file: src/file.scala
>  exports:
>  oddData: parquet



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AMATERASU-6) Long running actions/pipelines

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-6?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski updated AMATERASU-6:
---
Fix Version/s: (was: 0.3.0-incubating)
   0.4.0-incubating

> Long running actions/pipelines 
> ---
>
> Key: AMATERASU-6
> URL: https://issues.apache.org/jira/browse/AMATERASU-6
> Project: AMATERASU
>  Issue Type: New Feature
>Affects Versions: 0.4.0-incubating
>    Reporter: Yaniv Rodenski
>Assignee: Eyal Ben Ivri
>Priority: Major
> Fix For: 0.4.0-incubating
>
>
>  As a pipeline developer, I can define pipelines and actions to be 
> long-running, for workloads such as stream processing etc. 
> Example maki.yml
> job-name:amaterasu-test
> type: long-running
> flow:
>- name: action1
>  type: long-running
>  runner:
>  group: spark
>  type: scala   
>  file: src/file.scala
>  exports:
>  oddData: parquet



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AMATERASU-54) Use Docker for Mesos containerization

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-54?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski updated AMATERASU-54:

Fix Version/s: (was: 0.2.1-incubating)
   0.2.2-incubating

> Use Docker for Mesos containerization
> -
>
> Key: AMATERASU-54
> URL: https://issues.apache.org/jira/browse/AMATERASU-54
> Project: AMATERASU
>  Issue Type: Task
>    Reporter: Yaniv Rodenski
>Assignee: Yariv Triffon
>Priority: Major
> Fix For: 0.2.2-incubating
>
>
> The Amaterasu Mesos framework should launch Docker containers rather than 
> vanilla Mesos containers.
> Note that resources should be added using Docker and not the Mesos addUri 
> method.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AMATERASU-64) Create runtime.yaml for YARN

[Yaniv Rodenski (JIRA)]

Yaniv Rodenski created AMATERASU-64:
---

 Summary: Create runtime.yaml for YARN
 Key: AMATERASU-64
 URL: https://issues.apache.org/jira/browse/AMATERASU-64
 Project: AMATERASU
  Issue Type: Sub-task
Affects Versions: 0.2.1-incubating
Reporter: Yaniv Rodenski
Assignee: Eyal Ben Ivri
 Fix For: 0.2.1-incubating


Each container should be created with a runtime.yaml containing the jobId and 
action name



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AMATERASU-63) Create datasets.yaml for YARN

[Yaniv Rodenski (JIRA)]

Yaniv Rodenski created AMATERASU-63:
---

 Summary: Create datasets.yaml for YARN
 Key: AMATERASU-63
 URL: https://issues.apache.org/jira/browse/AMATERASU-63
 Project: AMATERASU
  Issue Type: Sub-task
Reporter: Yaniv Rodenski
Assignee: Eyal Ben Ivri
 Fix For: 0.2.1-incubating


Create a datasets.yaml file in each container. the datasets.yaml is currently 
created out of the exports section in the yaml, to be replaced by AMATERASU-52 
later



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AMATERASU-62) Add the use of the ConfigManager to create env.yml on YARN

[Yaniv Rodenski (JIRA)]

Yaniv Rodenski created AMATERASU-62:
---

 Summary: Add the use of the ConfigManager to create env.yml on YARN
 Key: AMATERASU-62
 URL: https://issues.apache.org/jira/browse/AMATERASU-62
 Project: AMATERASU
  Issue Type: Sub-task
Reporter: Yaniv Rodenski
Assignee: Eyal Ben Ivri
 Fix For: 0.2.1-incubating


use the ConfigManager to create an env.yml file in each container as 
implemented in the Mesos JobScheduler lines 161-163



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (AMATERASU-8) Support for Jar/External actions

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-8?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski reassigned AMATERASU-8:
--

Assignee: Yaniv Rodenski  (was: Shad Amez)

> Support for Jar/External actions
> 
>
> Key: AMATERASU-8
> URL: https://issues.apache.org/jira/browse/AMATERASU-8
> Project: AMATERASU
>  Issue Type: New Feature
>Affects Versions: 0.2.1-incubating
>    Reporter: Yaniv Rodenski
>    Assignee: Yaniv Rodenski
>Priority: Major
> Fix For: 0.2.1-incubating
>
>
> As a pipeline developer, I can use a pre-built jar or other packaged formats 
> for my actions definitions.
> Note: This need to be implemented in the framework's runner (i.e. Spark etc.) 
> as well as in the JobManager 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AMATERASU-19) Setup travis-ci release pipeline

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-19?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski resolved AMATERASU-19.
-
Resolution: Fixed

> Setup travis-ci release pipeline
> 
>
> Key: AMATERASU-19
> URL: https://issues.apache.org/jira/browse/AMATERASU-19
> Project: AMATERASU
>  Issue Type: Task
>    Reporter: Yaniv Rodenski
>Assignee: Guy Peleg
>Priority: Major
>
> Based on the release policy as described in the mailing list



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [jira] [Created] (AMATERASU-52) Implement AmaContext.datastores

[Yaniv Rodenski]

Hi Arun,

I've added my comments to the PR, but good call, I agree @Nadav Har Tzvi
 should at least review as you both need to
maintain compatible APIs.

Cheers,
Yaniv

On Sun, Jan 13, 2019 at 10:21 PM Arun Manivannan  wrote:

> Hi Guy, Yaniv and Nadiv,
>
> This PR <https://github.com/apache/incubator-amaterasu/pull/39> just
> captures part of the issue - the datasets.yaml, ConfigManager and the
> testcases. The Integration with the AmaContext is yet to be done but I
> would like to get your thoughts on the implementation.
>
> Guy - Would it be okay if you could help throw some light on the syntax and
> the idiomatic part of Kotlin itself. Newbie here.
>
> Cheers,
> Arun
>
> On Fri, Oct 12, 2018 at 7:15 PM Yaniv Rodenski (JIRA) 
> wrote:
>
> > Yaniv Rodenski created AMATERASU-52:
> > ---
> >
> >  Summary: Implement AmaContext.datastores
> >  Key: AMATERASU-52
> >  URL: https://issues.apache.org/jira/browse/AMATERASU-52
> >      Project: AMATERASU
> >   Issue Type: Task
> > Reporter: Yaniv Rodenski
> > Assignee: Arun Manivannan
> >  Fix For: 0.2.1-incubating
> >
> >
> > AmaContext.datastores should contain the data from datastores.yaml
> >
> >
> >
> > --
> > This message was sent by Atlassian JIRA
> > (v7.6.3#76005)
> >
>


-- 
Yaniv Rodenski

+61 477 778 405
ya...@shinto.io

[jira] [Assigned] (AMATERASU-25) Create documentation with ReadTheDocs

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-25?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski reassigned AMATERASU-25:
---

Assignee: Yaniv Rodenski  (was: Nadav Har Tzvi)

> Create documentation with ReadTheDocs
> -
>
> Key: AMATERASU-25
> URL: https://issues.apache.org/jira/browse/AMATERASU-25
> Project: AMATERASU
>  Issue Type: Task
>Affects Versions: 0.2.1-incubating
>Reporter: Nadav Har Tzvi
>    Assignee: Yaniv Rodenski
>Priority: Major
>  Labels: docuentation
> Fix For: 0.2.1-incubating
>
>
> We need to start filling in documentation for Apache Amaterasu.
> We will use readthedocs for this purpose.
> We need to set up a /docs directory with rst files.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] podling report

[Yaniv Rodenski]

Hi Davor,

The report was made in a rush and a bit from memory as I am traveling. I
will rectify, please see below:

> Two new contributors have contributed code that have been merged.
This was from my memory, @Yariv Triffon  did start
committing (he is working on AMATERASU-54), but also, he helped me and
pushed to by branch during this iteration (@Yariv, correct me if I'm
wrong). It does seem that those changes did not make it at the end (or I
might have redone them manually) but I will remove him from the report.

> In addition, we are actively looking for more use cases and organizations
> to use Amaterasu.
This is done by having direct discussions with different organizations, in
fact, most of my efforts in the last months have been around trying to find
use cases for Amaterasu, and I have been spending a lot of time approaching
organizations (with some help from @Kirupagaran Devarajan
 and @ey...@apache.org ).

I think this is key to the success of the project no less than the number
of commits, but it takes time, especially for a small project.

> * 5 pull requests have been opened
You are correct, as I said this report was done in a rush  I will fix this
in the report.

I understand that you are not trying to be difficult, and if you and the
other mentors think we should retire we can discuss that, but also, I
wanted to know if maybe there are things you and the rest of the mentors
think you can do to help us pass this rough patch and get the project to
the next level (and I'm not speaking about graduation yet).

I will also add a comment in the report regarding this discussion.

Cheers,
Yaniv

On Sun, Jan 6, 2019 at 8:39 AM Davor Bonaci  wrote:

> The report doesn't have my sign-off in the current form.
>
>
> > Two new contributors have contributed code that have been merged.
>
>
> Can you please clarify who are those? From what I can tell [1], the code
> activity includes past contributors Yaniv (11 commits, 1.5k lines) and
> anonymousGiraffe (50 lines). The (only) new contributor seems to be ebarten
> at 40 lines.
>
>
> > In addition, we are actively looking for more use cases and organizations
> > to use Amaterasu.
> >
>
> Can you please clarify what exactly has been done here? Links and evidence
> would be super helpful.
>
>
> > * 5 pull requests have been opened
>
>
> I can see four (#34 - #37). #33 was opened in early August. Which is the
> fifth one?
>
> Firstly, I don't think it matters whether there are 1 or 2 new
> contributors, 4 or 5 pull requests, but it does bother me the accuracy of
> the report. Please be accurate when reporting statistics and possibly quote
> the source if there's any doubt about them, otherwise you easily lose all
> credibility and appear disrespectful to your audience. Please.
>
> Secondly, I think it is safe to assume that everybody understands that the
> project is dormant and not making any progress towards graduation. Should
> the report at least mention it? Even after being explicitly asked by
> Justin?
>
> Please note that I'm not trying to be difficult here, but things cannot
> continue this way.
>
> Davor
>
> [1]
>
> https://github.com/apache/incubator-amaterasu/graphs/contributors?from=2018-10-05&to=2019-01-05&type=c
>


-- 
Yaniv Rodenski

+61 477 778 405
ya...@shinto.io

Re: Podling Report Reminder - January 2019

[Yaniv Rodenski]

Hi All,

We did not have further discussions regarding this. We did have progress
from everyone who committed to continuing and contributing to the project,
as well as identify and approach more contributors which had some success.

On one hand, I do understand the concern regarding the progress being made,
but there is a commitment by the contributors as well as some progress.

I guess maybe other than that, the only thing I can think of maybe looks at
some milestones or maybe quantifying what would be a success/failure for
the project?

Also, maybe the mentors do have ideas on how to make progress on this
front? in your experience, ware there any tactics that worked in such
scenarios?

Cheers,
Yaniv

On Mon, Dec 31, 2018 at 10:13 AM Davor Bonaci  wrote:

> Any comments?
>
> On Fri, Dec 21, 2018 at 5:48 PM Justin Mclean  wrote:
>
> > Hi,
> >
> >  In your last report it was mentioned that:
> > "A concern was raised by Davor Bonaci (mentor) that the community is too
> > small and not active enough and that Amaterasu should consider retiring.
> > currently, the community voted on trying to address those issues and
> > reconsider in a few months."
> >
> > Has any progress / decision been made? I was not able to find any further
> > discussion about this other than the initial thread. Please include in
> your
> > report how the PPMC is addressing this issue.
> >
> > Thanks,
> > Justin
> >
>


-- 
Yaniv Rodenski

+61 477 778 405
ya...@shinto.io

[DISCUSS] podling report

[Yaniv Rodenski]

Hi All,

I propose the following report to be submitted.

Amaterasu


Apache Amaterasu is a framework providing configuration management and

deployment for Big Data Pipelines.


It provides the following capabilities:


Continuous integration tools to package pipelines and run tests.

A repository to store those packaged applications: the applications

repository.

A repository to store the pipelines, and engine configuration (for

instance, the location of the Spark master, etc.): per environment - the

configuration repository.

A dashboard to monitor the pipelines.

A DSL and integration hooks allowing third parties to easily integrate.


Amaterasu has been incubating since 2017-09.


Three most important issues to address in the move towards graduation:


  1. Grow up user and contributor communities

  2. Prepare documentation


Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be

aware of?


How has the community developed since the last report?


  Two new contributors have contributed code that have been merged. In
addition, we are actively looking for more use cases and organizations to
use Amaterasu.


How has the project developed since the last report?


  * 5 pull requests have been opened since the last report and 4 have been
merged

  * Since last report 9 more issues have been created and 4 out of them
have been assigned


Date of the last release:


  12 July 2018


When were the last committers or PMC members elected?


  N/A


Have your mentors been helpful and responsive or are things falling through
the cracks? In the latter case, please list any open issues that need to be
addressed.



 N/A


Signed-off-by:


  [](amaterasu) Jean-Baptiste Onofré

  [](amaterasu) Olivier Lamy

  [](amaterasu) Davor Bonaci

-- 
Yaniv Rodenski

[jira] [Commented] (AMATERASU-36) Refactoring - leader: extract projects per resource manager + common

[Yaniv Rodenski (JIRA)]

[ 
https://issues.apache.org/jira/browse/AMATERASU-36?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16706613#comment-16706613
 ] 

Yaniv Rodenski commented on AMATERASU-36:
-

Need to add sub-tasks including ones for Kotlin refatoring 

> Refactoring - leader: extract projects per resource manager + common
> 
>
> Key: AMATERASU-36
> URL: https://issues.apache.org/jira/browse/AMATERASU-36
> Project: AMATERASU
>  Issue Type: Improvement
>Reporter: Yaniv Rodenski
>Assignee: Guy Peleg
>Priority: Major
> Fix For: 0.2.1-incubating
>
>
> To simplify deployments and the Amaterasu codebase, we will create 
> differently projects/binaries per resource manager, for example:
> leader
>    |>leader.common 
>    |>leader.mesos
>    |>leader.YARN
>    |>leader.K8s
> ...



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (AMATERASU-36) Refactoring - leader: extract projects per resource manager + common

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-36?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski reassigned AMATERASU-36:
---

Assignee: Guy Peleg  (was: Yaniv Rodenski)

> Refactoring - leader: extract projects per resource manager + common
> 
>
> Key: AMATERASU-36
> URL: https://issues.apache.org/jira/browse/AMATERASU-36
> Project: AMATERASU
>  Issue Type: Improvement
>    Reporter: Yaniv Rodenski
>Assignee: Guy Peleg
>Priority: Major
> Fix For: 0.2.1-incubating
>
>
> To simplify deployments and the Amaterasu codebase, we will create 
> differently projects/binaries per resource manager, for example:
> leader
>    |>leader.common 
>    |>leader.mesos
>    |>leader.YARN
>    |>leader.K8s
> ...



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AMATERASU-49) Generate env.yml per container

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-49?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski resolved AMATERASU-49.
-
Resolution: Fixed

> Generate env.yml per container
> --
>
> Key: AMATERASU-49
> URL: https://issues.apache.org/jira/browse/AMATERASU-49
> Project: AMATERASU
>  Issue Type: Task
>    Reporter: Yaniv Rodenski
>    Assignee: Yaniv Rodenski
>Priority: Major
> Fix For: 0.2.1-incubating
>
>
> in the new execution model, each framework runtime component will read it's 
> configuration from an env.yml file located within the container.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AMATERASU-51) Add support for configuring the path for action level environment path

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-51?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski resolved AMATERASU-51.
-
Resolution: Fixed

> Add support for configuring the path for action level environment path
> --
>
> Key: AMATERASU-51
> URL: https://issues.apache.org/jira/browse/AMATERASU-51
> Project: AMATERASU
>  Issue Type: Improvement
>    Reporter: Yaniv Rodenski
>Assignee: Eran Bartenstein
>Priority: Major
> Fix For: 0.2.1-incubating
>
>
> Need to add support in the JobPartser, ActionData and JobManager for an 
> action level config element as follows:
> flow:
>   - name: start
>        *config: src/start/\{env}/spark-start.yml*
>        runner:
>           group: spark
>           type: scala
>       file: src/main/\{action_name}/start.scala
>       exports:
>           odd: parquet
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AMATERASU-53) Support action level dependencies

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-53?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski updated AMATERASU-53:

Description: 
Add support for dependencies files to be stored per action.

This should be stored under *src/\{action_name}/deps/* and be deployed with the 
job level dependencies.

  was:
Add support for dependencies files to be stored per action.

This should be stored under src/\{action_name}/deps/ and be deployed with the 
job level dependencies.


> Support action level dependencies 
> --
>
> Key: AMATERASU-53
> URL: https://issues.apache.org/jira/browse/AMATERASU-53
> Project: AMATERASU
>  Issue Type: Task
>    Reporter: Yaniv Rodenski
>Assignee: Kirupa
>Priority: Major
> Fix For: 0.2.1-incubating
>
>
> Add support for dependencies files to be stored per action.
> This should be stored under *src/\{action_name}/deps/* and be deployed with 
> the job level dependencies.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AMATERASU-56) Create a Kotlin logger implementation

[Yaniv Rodenski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMATERASU-56?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yaniv Rodenski resolved AMATERASU-56.
-
Resolution: Fixed

> Create a Kotlin logger implementation
> -
>
> Key: AMATERASU-56
> URL: https://issues.apache.org/jira/browse/AMATERASU-56
> Project: AMATERASU
>  Issue Type: Sub-task
>    Reporter: Yaniv Rodenski
>Assignee: Eran Bartenstein
>Priority: Major
> Fix For: 0.2.1-incubating
>
>
> During the refactoring for AMATERASU-51 the need to create a Kotlin 
> implementation for the logger for the newly refactored Kotlin classes



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AMATERASU-56) Create a Kotlin logger implementation

[Yaniv Rodenski (JIRA)]

Yaniv Rodenski created AMATERASU-56:
---

 Summary: Create a Kotlin logger implementation
 Key: AMATERASU-56
 URL: https://issues.apache.org/jira/browse/AMATERASU-56
 Project: AMATERASU
  Issue Type: Sub-task
Reporter: Yaniv Rodenski
Assignee: Eran Bartenstein
 Fix For: 0.2.1-incubating


During the refactoring for AMATERASU-51 the need to create a Kotlin 
implementation for the logger for the newly refactored Kotlin classes



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AMATERASU-55) Retire org.reflections.Reflections

[Yaniv Rodenski (JIRA)]

Yaniv Rodenski created AMATERASU-55:
---

 Summary: Retire org.reflections.Reflections
 Key: AMATERASU-55
 URL: https://issues.apache.org/jira/browse/AMATERASU-55
 Project: AMATERASU
  Issue Type: Task
Reporter: Yaniv Rodenski
Assignee: Guy Peleg






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] Dependencies resolution and action level dependencies

[Yaniv Rodenski]

Excellent,

I've added:
AMATERASU-53 - Support action level dependencies
<https://issues.apache.org/jira/browse/AMATERASU-53>
AMATERASU-54 - Use Docker for Mesos containerization
<https://issues.apache.org/jira/browse/AMATERASU-54>
I suggest we review after AMATERASU-54 how to approach the YARN
implementation.

Cheers,
Yaniv

On Tue, Oct 23, 2018 at 9:51 PM Yariv Triffon  wrote:

> Hi Yaniv,
> i'm good to grab the task of moving Mesos to use Docker containers.
>
> Cheers,
> Yariv
>
> On Tue, Oct 23, 2018 at 5:13 PM Yaniv Rodenski  wrote:
>
> > Thanks, Kirupa,
> >
> > I'll create the JIRA tasks shortly and assign that one to you.
> >
> >
> >
> > On Tue, Oct 23, 2018 at 5:09 PM Kirupa Devarajan <
> kirupagara...@gmail.com>
> > wrote:
> >
> > > Hi Yaniv,
> > >
> > > I am happy to pick up the following task
> > >
> > > 1. Add to the JobManager the functionality to read action level
> > > dependencies
> > >
> > > Regards,
> > > Kirupa
> > >
> > > On Tue., 23 Oct. 2018, 11:04 am Yaniv Rodenski, 
> wrote:
> > >
> > > > Hi Nadav,
> > > >
> > > > It does make sense, in fact, we actually have action level resources
> > > > already, however they are limited to the configuration files for the
> > > > container.
> > > > I also think that we need to revision the way we set up those.
> > Correctly
> > > we
> > > > use YARN/Mesos to copy dependencies to the containers. With YARN 3.0
> I
> > > > think it makes sense to move to use Docker as the way to manage
> > resources
> > > > in the containers.
> > > > This should also have performance benefits + will make life easier (I
> > > hope)
> > > > when we start working on K8s.
> > > >
> > > > To do this, I think we need to add the following tasks:
> > > > 1. Add to the JobManager the functionality to read action level
> > > > dependencies
> > > > 2. Move from Mesos/YARN containers to Docker (probably at least two
> > > tasks)
> > > >
> > > > I'll add them to JIRA asap, for version 0.2.1-incubating if everyone
> is
> > > OK
> > > > with it.
> > > >
> > > > On Sat, Oct 20, 2018 at 6:43 PM Nadav Har Tzvi <
> nadavhart...@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Hey everyone,
> > > > >
> > > > > Yaniv and I were just discussing how to resolve dependencies in the
> > new
> > > > > frameworks architecture and integrate the dependencies with the
> > > concrete
> > > > > cluster resource manager (Mesos/YARN)
> > > > > We rolled with the idea of each runner (or base runner) performing
> > the
> > > > > dependencies resolution on its own.
> > > > > So for example, the Spark Scala runner would resolve the required
> > JARs
> > > > and
> > > > > do whatever it needs to do with them (e.g. spark-submit --jars
> > > --packages
> > > > > --repositories, etc).
> > > > > The base Python provider will resolve dependencies and dynamically
> > > > generate
> > > > > a requirement.txt file that will deployed to the executor.
> > > > > The handling of the requirements.txt file differs between different
> > > > > concrete Python runners. For example, a regular Python runner would
> > > > simply
> > > > > run pip install, while the pyspark runner would need to rearrange
> the
> > > > > dependencies in a way that would be acceptable by spark-submit (
> > > > >
> > > > >
> > > >
> > >
> >
> https://bytes.grubhub.com/managing-dependencies-and-artifacts-in-pyspark-7641aa89ddb7
> > > > > sounds like a decent idea, comment if you have a better idea
> please)
> > > > >
> > > > > So far I hope it makes sense.
> > > > >
> > > > > The next item I want to discuss is as follows:
> > > > > In the new architecture, we do hierarchical runtime environment
> > > > resolution,
> > > > > starting at the top job level and drilling down to the action
> level,
> > > > > outputting one unified environment configuration file that is
> > deployed
> > > to
> > > > > the executor.
> > > > > I suggest d

[jira] [Created] (AMATERASU-54) Use Docker for Mesos containerization

[Yaniv Rodenski (JIRA)]

Yaniv Rodenski created AMATERASU-54:
---

 Summary: Use Docker for Mesos containerization
 Key: AMATERASU-54
 URL: https://issues.apache.org/jira/browse/AMATERASU-54
 Project: AMATERASU
  Issue Type: Task
Reporter: Yaniv Rodenski
Assignee: Yariv Triffon
 Fix For: 0.2.1-incubating


The Amaterasu Mesos framework should launch Docker containers rather than 
vanilla Mesos containers.

Note that resources should be added using Docker and not the Mesos addUri 
method.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AMATERASU-53) Support action level dependencies

[Yaniv Rodenski (JIRA)]

Yaniv Rodenski created AMATERASU-53:
---

 Summary: Support action level dependencies 
 Key: AMATERASU-53
 URL: https://issues.apache.org/jira/browse/AMATERASU-53
 Project: AMATERASU
  Issue Type: Task
Reporter: Yaniv Rodenski
Assignee: Kirupa
 Fix For: 0.2.1-incubating


Add support for dependencies files to be stored per action.

This should be stored under src/\{action_name}/deps/ and be deployed with the 
job level dependencies.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] Dependencies resolution and action level dependencies

[Yaniv Rodenski]

Thanks, Kirupa,

I'll create the JIRA tasks shortly and assign that one to you.



On Tue, Oct 23, 2018 at 5:09 PM Kirupa Devarajan 
wrote:

> Hi Yaniv,
>
> I am happy to pick up the following task
>
> 1. Add to the JobManager the functionality to read action level
> dependencies
>
> Regards,
> Kirupa
>
> On Tue., 23 Oct. 2018, 11:04 am Yaniv Rodenski,  wrote:
>
> > Hi Nadav,
> >
> > It does make sense, in fact, we actually have action level resources
> > already, however they are limited to the configuration files for the
> > container.
> > I also think that we need to revision the way we set up those. Correctly
> we
> > use YARN/Mesos to copy dependencies to the containers. With YARN 3.0 I
> > think it makes sense to move to use Docker as the way to manage resources
> > in the containers.
> > This should also have performance benefits + will make life easier (I
> hope)
> > when we start working on K8s.
> >
> > To do this, I think we need to add the following tasks:
> > 1. Add to the JobManager the functionality to read action level
> > dependencies
> > 2. Move from Mesos/YARN containers to Docker (probably at least two
> tasks)
> >
> > I'll add them to JIRA asap, for version 0.2.1-incubating if everyone is
> OK
> > with it.
> >
> > On Sat, Oct 20, 2018 at 6:43 PM Nadav Har Tzvi 
> > wrote:
> >
> > > Hey everyone,
> > >
> > > Yaniv and I were just discussing how to resolve dependencies in the new
> > > frameworks architecture and integrate the dependencies with the
> concrete
> > > cluster resource manager (Mesos/YARN)
> > > We rolled with the idea of each runner (or base runner) performing the
> > > dependencies resolution on its own.
> > > So for example, the Spark Scala runner would resolve the required JARs
> > and
> > > do whatever it needs to do with them (e.g. spark-submit --jars
> --packages
> > > --repositories, etc).
> > > The base Python provider will resolve dependencies and dynamically
> > generate
> > > a requirement.txt file that will deployed to the executor.
> > > The handling of the requirements.txt file differs between different
> > > concrete Python runners. For example, a regular Python runner would
> > simply
> > > run pip install, while the pyspark runner would need to rearrange the
> > > dependencies in a way that would be acceptable by spark-submit (
> > >
> > >
> >
> https://bytes.grubhub.com/managing-dependencies-and-artifacts-in-pyspark-7641aa89ddb7
> > > sounds like a decent idea, comment if you have a better idea please)
> > >
> > > So far I hope it makes sense.
> > >
> > > The next item I want to discuss is as follows:
> > > In the new architecture, we do hierarchical runtime environment
> > resolution,
> > > starting at the top job level and drilling down to the action level,
> > > outputting one unified environment configuration file that is deployed
> to
> > > the executor.
> > > I suggest doing the same with dependencies.
> > > Currently, we only have job level dependencies. I suggest that we
> provide
> > > action level dependencies and resolve them in exactly the same manner
> as
> > we
> > > resolve the environment.
> > > There should be quite a few benefits for this approach:
> > >
> > >1. It will give the option to have different versions of the same
> > >package in different actions. This is especially important if you
> have
> > > 2+
> > >pipeline developers working independently, this would reduce the
> > >integration costs by letting each action be more self-contained.
> > >2. It should lower the startup time per action. The more
> dependencies
> > >you have, the longer it takes to resolve and install them. Actions
> > will
> > > no
> > >longer get any unnecessary dependencies.
> > >
> > >
> > > What do you think? Does it make sense?
> > >
> > > Cheers,
> > > Nadav
> > >
> >
> >
> > --
> > Yaniv Rodenski
> >
> > +61 477 778 405
> > ya...@shinto.io
> >
>


-- 
Yaniv Rodenski

+61 477 778 405
ya...@shinto.io

Re: [DISCUSS] Dependencies resolution and action level dependencies

[Yaniv Rodenski]

Hi Nadav,

It does make sense, in fact, we actually have action level resources
already, however they are limited to the configuration files for the
container.
I also think that we need to revision the way we set up those. Correctly we
use YARN/Mesos to copy dependencies to the containers. With YARN 3.0 I
think it makes sense to move to use Docker as the way to manage resources
in the containers.
This should also have performance benefits + will make life easier (I hope)
when we start working on K8s.

To do this, I think we need to add the following tasks:
1. Add to the JobManager the functionality to read action level dependencies
2. Move from Mesos/YARN containers to Docker (probably at least two tasks)

I'll add them to JIRA asap, for version 0.2.1-incubating if everyone is OK
with it.

On Sat, Oct 20, 2018 at 6:43 PM Nadav Har Tzvi 
wrote:

> Hey everyone,
>
> Yaniv and I were just discussing how to resolve dependencies in the new
> frameworks architecture and integrate the dependencies with the concrete
> cluster resource manager (Mesos/YARN)
> We rolled with the idea of each runner (or base runner) performing the
> dependencies resolution on its own.
> So for example, the Spark Scala runner would resolve the required JARs and
> do whatever it needs to do with them (e.g. spark-submit --jars --packages
> --repositories, etc).
> The base Python provider will resolve dependencies and dynamically generate
> a requirement.txt file that will deployed to the executor.
> The handling of the requirements.txt file differs between different
> concrete Python runners. For example, a regular Python runner would simply
> run pip install, while the pyspark runner would need to rearrange the
> dependencies in a way that would be acceptable by spark-submit (
>
> https://bytes.grubhub.com/managing-dependencies-and-artifacts-in-pyspark-7641aa89ddb7
> sounds like a decent idea, comment if you have a better idea please)
>
> So far I hope it makes sense.
>
> The next item I want to discuss is as follows:
> In the new architecture, we do hierarchical runtime environment resolution,
> starting at the top job level and drilling down to the action level,
> outputting one unified environment configuration file that is deployed to
> the executor.
> I suggest doing the same with dependencies.
> Currently, we only have job level dependencies. I suggest that we provide
> action level dependencies and resolve them in exactly the same manner as we
> resolve the environment.
> There should be quite a few benefits for this approach:
>
>1. It will give the option to have different versions of the same
>package in different actions. This is especially important if you have
> 2+
>pipeline developers working independently, this would reduce the
>integration costs by letting each action be more self-contained.
>2. It should lower the startup time per action. The more dependencies
>you have, the longer it takes to resolve and install them. Actions will
> no
>    longer get any unnecessary dependencies.
>
>
> What do you think? Does it make sense?
>
> Cheers,
> Nadav
>


-- 
Yaniv Rodenski

+61 477 778 405
ya...@shinto.io