Re: [DISCUSS] podling report
+1 On 6. July 2018 at 18:50:37, Davor Bonaci (da...@apache.org) wrote: +1 "have been released" --> "have been built and voted upon" On Fri, Jul 6, 2018 at 12:51 AM, Yaniv Rodenski wrote: > Hi All, > > Sorry for doing this late again, but I propose the following report to be > submitted: > > " > Apache Amaterasu is a framework providing configuration management and > deployment for Big Data Pipelines. > > It provides the following capabilities: > > Continuous integration tools to package pipelines and run tests. > A repository to store those packaged applications: the applications > repository. > A repository to store the pipelines, and engine configuration (for > instance, the location of the Spark master, etc.): per environment - the > configuration repository. > A dashboard to monitor the pipelines. > A DSL and integration hooks allowing third parties to easily integrate. > > Amaterasu has been incubating since 2017-09. > > Three most important issues to address in the move towards graduation: > > 1. Prepare the first release > 2. Grow up user and contributor communities > 3. Prepare documentation > > Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be > aware of? > > None > > How has the community developed since the last report? > > * Two conference talks have been delivered (PyCon il and SDP) > * Initial documentation has been created, targeted for Amaterasu's next > release > > How has the project developed since the last report? > > * since the last report 4 release candidates have been released, at the > time of this report the last RC is being voted on in the general@incubator > mailing list > * Two additional contributors started contributing to the code base > * One more organization we are aware of have started a POC with Amaterasu > > Date of the last release: > > N/A > > When were the last committers or PMC members elected? > > N/A > " > > If there are no objections I will update the wiki. > > Cheers, > Yaniv >
Re: [VOTE] Release Apache Amaterasu (incubating) 0.2.0 (rc4)
+1 On 24. June 2018 at 12:00:12, Arun Manivannan (a...@arunma.com) wrote: Clean built. Testcases run fine. Ran it on HDP and HDP 2.6.5.0-292 with minor change on the leader AM. Works perfect. +1 Cheers, Arun On Sun, Jun 24, 2018 at 5:10 PM guy peleg wrote: > Downloaded the source code and the artifact, everything looks good on my > side. > +1 > > Guy > > On Sun, Jun 24, 2018, 19:09 Kirupa Devarajan > wrote: > > > gradle build and assemble successful on the branch > > version-0.2.0-incubating-rc4 > > < > > > https://github.com/apache/incubator-amaterasu/tree/version-0.2.0-incubating-rc4 > > > > > > . > > > > +1 > > > > Regards, > > Kirupa > > > > > > On Sun, Jun 24, 2018 at 6:57 PM, Nadav Har Tzvi > > wrote: > > > > > Tested on standalone Mesos, EMR and HDP. > > > Spark-Scala and PySpark both work on each of the environments. > > > Thus, I vote +1 > > > > > > Cheers, > > > Nadav > > > > > > > > > > > > On Sat, 23 Jun 2018 at 15:35, Yaniv Rodenski wrote: > > > > > > > Hi everyone, > > > > > > > > After cancelling the vote in the general@ list we've fixed the > > > following: > > > > * Headers added to all non-code files where applicable as remarked by > > > Davor > > > > * Sources now match the released version + no keys are present > > > > * The gradle-wrapper.jar was removed and instructions ware added to > the > > > > readme.md file on how to add it. > > > > * Missing licenses found by Justin Mclean during the general@ vote > > ware > > > > added when applicable. > > > > > > > > So, hoping for the best, please review and vote on the release > > candidate > > > #4 > > > > for the version 0.2.0-incubating, as follows > > > > > > > > [ ] +1, Approve the release > > > > [ ] -1, Do not approve the release (please provide specific comments) > > > > > > > > The complete staging area is available for your review, which > includes: > > > > > > > > * JIRA release notes [1], > > > > * the official Apache source release to be deployed to > dist.apache.org > > > > [2], > > > > which is signed with the key with fingerprint [3], > > > > * source code tag "version-0.2.0-incubating-rc4" [4], > > > > * Java artifacts were built with Gradle 3.1 and OpenJDK/Oracle JDK > > > > 1.8.0_151 > > > > > > > > The vote will be open for at least 72 hours. It is adopted by > majority > > > > approval, with at least 3 PMC affirmative votes. > > > > > > > > Thanks, > > > > Yaniv > > > > > > > > [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?p > > > > rojectId=12321521&version=12342793 > > > > [2] > > https://dist.apache.org/repos/dist/dev/incubator/amaterasu/0.2.0rc4/ > > > > [3] https://dist.apache.org/repos/dist/dev/incubator/amaterasu/KEYS > > > > [4] https://github.com/apache/incubator-amaterasu/tags > > > > > > > > Thanks everyone > > > > Yaniv > > > > > > > > > >
[jira] [Updated] (AMATERASU-22) Improve the way yarn containers are allocated to consider specific task requirments
[ https://issues.apache.org/jira/browse/AMATERASU-22?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eyal Ben Ivri updated AMATERASU-22: --- Description: Currently, YARN containers are created without consideration for specific requirements of a task, just the job configuration. As a pipeline developer, I would like to be able to override the specifications (v-cores or memory) of a specific task inside a flow. was: Currently, YARN containers are created without consideration for specific requirements of a task, just the job configuration. As a Flow developer, i would like to be able to override the specifications (vcores or memory) of a specific task inside a flow. > Improve the way yarn containers are allocated to consider specific task > requirments > --- > > Key: AMATERASU-22 > URL: https://issues.apache.org/jira/browse/AMATERASU-22 > Project: AMATERASU > Issue Type: Improvement > Reporter: Eyal Ben Ivri > Assignee: Eyal Ben Ivri >Priority: Major > Fix For: 0.2.1-incubating > > > Currently, YARN containers are created without consideration for specific > requirements of a task, just the job configuration. > As a pipeline developer, I would like to be able to override the > specifications (v-cores or memory) of a specific task inside a flow. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AMATERASU-6) Long running actions/pipelines
[ https://issues.apache.org/jira/browse/AMATERASU-6?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eyal Ben Ivri reassigned AMATERASU-6: - Assignee: Eyal Ben Ivri > Long running actions/pipelines > --- > > Key: AMATERASU-6 > URL: https://issues.apache.org/jira/browse/AMATERASU-6 > Project: AMATERASU > Issue Type: New Feature >Affects Versions: 0.3.0-incubating >Reporter: Yaniv Rodenski > Assignee: Eyal Ben Ivri >Priority: Major > Fix For: 0.3.0-incubating > > > As a pipeline developer, I can define pipelines and actions to be > long-running, for workloads such as stream processing etc. > Example maki.yml > job-name:amaterasu-test > type: long-running > flow: >- name: action1 > type: long-running > runner: > group: spark > type: scala > file: src/file.scala > exports: > oddData: parquet -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AMATERASU-22) Improve the way yarn containers are allocated to consider specific task requirments
Eyal Ben Ivri created AMATERASU-22: -- Summary: Improve the way yarn containers are allocated to consider specific task requirments Key: AMATERASU-22 URL: https://issues.apache.org/jira/browse/AMATERASU-22 Project: AMATERASU Issue Type: Improvement Reporter: Eyal Ben Ivri Assignee: Eyal Ben Ivri Fix For: 0.2.1-incubating Currently, YARN containers are created without consideration for specific requirements of a task, just the job configuration. As a Flow developer, i would like to be able to override the specifications (vcores or memory) of a specific task inside a flow. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AMATERASU-18) Containers are not influenced by framework configuration
[ https://issues.apache.org/jira/browse/AMATERASU-18?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eyal Ben Ivri resolved AMATERASU-18. Resolution: Fixed > Containers are not influenced by framework configuration > > > Key: AMATERASU-18 > URL: https://issues.apache.org/jira/browse/AMATERASU-18 > Project: AMATERASU > Issue Type: Bug >Reporter: Yaniv Rodenski > Assignee: Eyal Ben Ivri >Priority: Major > Fix For: 0.2.0-incubating > > > Currently, the containers sizing (YARN and Mesos) is derived from the default > configuration and not from the configuration of the framework. We need to > connect the spark properties to the container setup -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Project status
Hi to all, Sorry for not replying sooner, but I have been following the thread closely. It is quite obvious we need to find more people that will be active in the Amaterasu project. Yaniv and myself are often communicating through private channels (and I believe other developers are doing the same), and as an improvement, we need to make Amaterasu related discussions through this mailing list. A lot of the features and milestones discussions (some of them are mentioned here like the Travis build) should be discussed here, and that is something all of the active developers of the project should start to do, in my opinion. Besides that, i agree with all the points raised here, and do think that the main “beyond-the-code” goal should be growing the community around Amaterasu. Thank you, Eyal On 27. March 2018 at 07:12:04, Davor Bonaci (da...@apache.org) wrote: Anybody else? On Tue, Mar 20, 2018 at 5:06 PM, Yaniv Rodenski wrote: > Thanks Davor, > > Points taken, we will learn and improve on those. > > Just one clearification, I was not blaming the mentor I myself was more > focused on working with Guy on automating the build than following up. > Rereading my own response I can see that was unclear. > > > On Wed, 21 Mar 2018 at 11:02 am, Davor Bonaci wrote: > > > Thanks for a great response. Some comments inline. > > > > * In the last month we have been working on automating the release > process > > > via Travis, we are still trying to enable Travis build for the > Amaterasu > > > repo, which is taking ridiculously long. We need one of the mentors to > > just > > > enable it via their account (I've already talked a couple of times to > one > > > of the mentors about it). > > > > > > > Searching for 'travis' in the mailing list archives doesn't yield any > > discussion threads. > > > > Mentors don't have permission to do this themselves. Infra JIRA is the > way > > to do it, but I couldn't find such JIRA ticket filed. > > > > Emailing one mentor directly (or any other community member) isn't a way > to > > build the community. Things need to be discussed in public whenever > > possible. > > > > Given the above, blaming a mentor (whomever you may be referring to) > > doesn't make sense. > > > > * We are ready to release version 0.2.0-incubating, the reason it took > us a > > > month to initiate the process is the above automated build, which I > > > suggested in prior discussion and had no rejections. We will complete > > this > > > once build is enabled. > > > > > > > The release itself is a great milestone, but not the purpose to itself. > > > > > > > * as for community growth, we are working with two organizations on > > running > > > POCs (which will hopefully grow the user base) one of them is due to > > start > > > very soon. I don't want to name them (first of all it's too early, and > > also > > > it is for them to decide if they want to share) but a representative > from > > > at least one of those organisations is on the list and is welcomed to > > share > > > :) > > > > > > > Great! > > > > > > > * This year I've seen contributions from 4 contributors (not much more > > than > > > 3, I know) but one of them is new (Guy Peleg) and AFAIK additional > > > longer-term work is done by one more contributor on his local fork > (Nadav > > > Har-Tzvi) > > > > > > > I think this is the crux of the problem. Why is longer-term work going on > > in a local fork? > > > > > > > * We should be presenting more, and growing the community more which is > > > hard to do starting out as a tiny community. Any advice given there > would > > > be appreciated. > > > > > > > The first thing has to be do the basics well: on-list communication, open > > discussions, no side channels, etc. > > > -- > Yaniv Rodenski > > +61 477 778 405 > ya...@shinto.io >
