Re: Time To Merge Spark 2.0 Support PR

[dusenberrymw]

Let's cut a 0.12 branch tomorrow, and then submit it for the release process on 
Friday. Thoughts?

--

Mike Dusenberry
GitHub: github.com/dusenberrymw
LinkedIn: linkedin.com/in/mikedusenberry

Sent from my iPhone.


> On Jan 9, 2017, at 12:40 PM, Arvind Surve  wrote:
> 
> ok, Thanks
> 
>  Arvind SurveSpark Technology Centerhttp://www.spark.tc/
> 
>  From: Luciano Resende 
> To: dev@systemml.incubator.apache.org 
> Sent: Monday, January 9, 2017 12:33 PM
> Subject: Re: Time To Merge Spark 2.0 Support PR
> 
>> On Mon, Jan 9, 2017 at 12:28 PM,  wrote:
>> 
>> Right, so we can cut a 0.12 release branch now, and then release from
>> that, while work moves forward on the master branch, including 2.0 support.
>> 
>> 
> Exactly, 0.12 release will come from a brach that we will create and Spark
> 2.0 support gets merged into Master.
> 
> 
> -- 
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
> 
> 

Re: Maven build failing

[Deron Eriksson]

Hi Sandeep,

That is really a great question, and thank you for bringing up this issue.
I would like to see the main SystemML jar (and -sources.jar and
-javadoc.jar) in future releases so that we have 8 Apache-approved
artifacts (including the new Python artifact) in the maven central
repository (in addition to the 4 at
http://www.apache.org/dyn/closer.lua/incubator/systemml/0.11.0-incubating).
This would allow users to do things such as use SystemML as a library using
standard Maven mechanisms. Also, the sources jar and javadoc jar are very
useful for developers.

For some background, our 0.10.0 release included 10 artifacts, which I
believe caused confusion because it was hard to tell which artifact for
users to download and use. For 0.11.0, we had quite a bit of release
difficulties, so we ended up with 4 artifacts (2 binary, 2 source), which
allowed us to conform to Apache incubator release candidate validation
while releasing a huge number of improvements to the project. In future
releases, I believe 8 artifacts is probably ideal. These would be: 1 main
jar, 1 sources jar, 1 javadoc jar, 2 binary releases, 2 source releases,
and 1 python jar.

Deron



On Mon, Jan 9, 2017 at 2:40 PM,  wrote:

> Glad we were able to sort out the original issue, and thanks for the
> follow up question. The download page offers a single release binary in
> either a zip or tar archive format (additionally you will find a sources
> folder). Once you unzip or untar the release archive, you will find a
> SystemML JAR file within the folder. This is exactly the same as the
> `SystemML.jar` file, but has been renamed to include the release version
> number.
>
> --
>
> Mike Dusenberry
> GitHub: github.com/dusenberrymw
> LinkedIn: linkedin.com/in/mikedusenberry
>
> Sent from my iPhone.
>
>
> > On Jan 9, 2017, at 2:21 PM, Narayanaswami, Sandeep <
> sandeep.narayanasw...@capitalone.com> wrote:
> >
> > Thank you all so much for your responses. I really appreciate it.
> > Felix was correct: It did turn out to be a proxy issue. I was able to
> hit the repository from a browser, so I was initially confused as to why it
> was breaking. I ended up changing my maven settings.xml to include the
> proxy info and that worked.
> >
> > Felix’s response leads me to another question: I’m probably missing
> something obvious here, but it seems to me like the SystemML download page
> only offers binaries for the standalone version, and there seems to be no
> way (besides building from source) to get my hands on the SystemML.jar. Is
> this correct?
> >
> > On 1/6/17, 17:31, "Mike Dusenberry"  wrote:
> >
> >Hi Sandeep,
> >
> >Thanks for reaching out!  The error message looks like a local Maven
> >issue.  In addition to the suggestion from Felix, another possibility
> is to
> >remove (or rename) your Maven cache, typically located at `~/.m2`,
> and then
> >try building again with a fresh start.
> >
> >- Mike
> >
> >
> >--
> >
> >Michael W. Dusenberry
> >GitHub: github.com/dusenberrymw
> >LinkedIn: linkedin.com/in/mikedusenberry
> >
> >>On Fri, Jan 6, 2017 at 5:26 PM,  wrote:
> >>
> >> Hi Sandeep,
> >>
> >> it seems like you can't connect to the maven repository. Can you open
> the
> >> link to the repository in a browser? (https://repo1.maven.org/maven2/)
> >> Are you behind a proxy maybe?
> >>
> >> If nothing else, you can download a binary release from our website to
> get
> >> started working with SystemML: http://systemml.apache.org/download.html
> >> The beginners guide (http://systemml.apache.org/get-started) has
> another
> >> way of getting started if you don't need to build the source yourself.
> >>
> >> - Felix
> >>
> >>
> >> Am 07.01.2017 02:09 schrieb Narayanaswami, Sandeep:
> >>
> >>> Dear SystemML community,
> >>>
> >>> I attempted the following (as per the Beginners' Guide for Python
> >>> users >>> guide-python.html>):
> >>>
> >>> git checkout clone https://github.com/apache/incubator-systemml.git
> >>> cd incubator-systemml
> >>> mvn clean package -P distribution
> >>>
> >>> This resulted in:
> >>>
> >>> [INFO] Scanning for projects...
> >>> Downloading: https://repo1.maven.org/maven2/org/apache/apache/18/
> apache-
> >>> 18.pom
> >>> Downloading:
> >>> https://raw.github.com/niketanpansare/mavenized-jcuda/mvn-
> >>> repo/org/apache/apache/18/apache-18.pom
> >>> [ERROR] [ERROR] Some problems were encountered while processing the
> POMs:
> >>> [FATAL] Non-resolvable parent POM for
> >>> org.apache.systemml:systemml:0.12.0-incubating-SNAPSHOT: Could not
> >>> transfer artifact org.apache:apache:pom:18 from/to central
> >>> (https://repo1.maven.org/maven2): Connect to repo1.maven.org:443
> >>> [repo1.maven.org/151.101.32.209] failed: Operation timed out
> >>> (Connection timed out) and 'parent.relativePath' points at wrong local
> >>> POM @ line 22, column 10
> >>> @
> >>> [ERROR] The build could not read 1 project -> [Help 1]
> >>> [ER

Re: Maven build failing

[dusenberrymw]

Glad we were able to sort out the original issue, and thanks for the follow up 
question. The download page offers a single release binary in either a zip or 
tar archive format (additionally you will find a sources folder). Once you 
unzip or untar the release archive, you will find a SystemML JAR file within 
the folder. This is exactly the same as the `SystemML.jar` file, but has been 
renamed to include the release version number.

--

Mike Dusenberry
GitHub: github.com/dusenberrymw
LinkedIn: linkedin.com/in/mikedusenberry

Sent from my iPhone.


> On Jan 9, 2017, at 2:21 PM, Narayanaswami, Sandeep 
>  wrote:
> 
> Thank you all so much for your responses. I really appreciate it. 
> Felix was correct: It did turn out to be a proxy issue. I was able to hit the 
> repository from a browser, so I was initially confused as to why it was 
> breaking. I ended up changing my maven settings.xml to include the proxy info 
> and that worked. 
> 
> Felix’s response leads me to another question: I’m probably missing something 
> obvious here, but it seems to me like the SystemML download page only offers 
> binaries for the standalone version, and there seems to be no way (besides 
> building from source) to get my hands on the SystemML.jar. Is this correct?
> 
> On 1/6/17, 17:31, "Mike Dusenberry"  wrote:
> 
>Hi Sandeep,
> 
>Thanks for reaching out!  The error message looks like a local Maven
>issue.  In addition to the suggestion from Felix, another possibility is to
>remove (or rename) your Maven cache, typically located at `~/.m2`, and then
>try building again with a fresh start.
> 
>- Mike
> 
> 
>--
> 
>Michael W. Dusenberry
>GitHub: github.com/dusenberrymw
>LinkedIn: linkedin.com/in/mikedusenberry
> 
>>On Fri, Jan 6, 2017 at 5:26 PM,  wrote:
>> 
>> Hi Sandeep,
>> 
>> it seems like you can't connect to the maven repository. Can you open the
>> link to the repository in a browser? (https://repo1.maven.org/maven2/)
>> Are you behind a proxy maybe?
>> 
>> If nothing else, you can download a binary release from our website to get
>> started working with SystemML: http://systemml.apache.org/download.html
>> The beginners guide (http://systemml.apache.org/get-started) has another
>> way of getting started if you don't need to build the source yourself.
>> 
>> - Felix
>> 
>> 
>> Am 07.01.2017 02:09 schrieb Narayanaswami, Sandeep:
>> 
>>> Dear SystemML community,
>>> 
>>> I attempted the following (as per the Beginners' Guide for Python
>>> users>> guide-python.html>):
>>> 
>>> git checkout clone https://github.com/apache/incubator-systemml.git
>>> cd incubator-systemml
>>> mvn clean package -P distribution
>>> 
>>> This resulted in:
>>> 
>>> [INFO] Scanning for projects...
>>> Downloading: https://repo1.maven.org/maven2/org/apache/apache/18/apache-
>>> 18.pom
>>> Downloading:
>>> https://raw.github.com/niketanpansare/mavenized-jcuda/mvn-
>>> repo/org/apache/apache/18/apache-18.pom
>>> [ERROR] [ERROR] Some problems were encountered while processing the POMs:
>>> [FATAL] Non-resolvable parent POM for
>>> org.apache.systemml:systemml:0.12.0-incubating-SNAPSHOT: Could not
>>> transfer artifact org.apache:apache:pom:18 from/to central
>>> (https://repo1.maven.org/maven2): Connect to repo1.maven.org:443
>>> [repo1.maven.org/151.101.32.209] failed: Operation timed out
>>> (Connection timed out) and 'parent.relativePath' points at wrong local
>>> POM @ line 22, column 10
>>> @
>>> [ERROR] The build could not read 1 project -> [Help 1]
>>> [ERROR]
>>> [ERROR]   The project
>>> org.apache.systemml:systemml:0.12.0-incubating-SNAPSHOT
>>> (/Users/ioz814/Projects/oss/incubator-systemml/pom.xml) has 1 error
>>> [ERROR] Non-resolvable parent POM for
>>> org.apache.systemml:systemml:0.12.0-incubating-SNAPSHOT: Could not
>>> transfer artifact org.apache:apache:pom:18 from/to central
>>> (https://repo1.maven.org/maven2): Connect to repo1.maven.org:443
>>> [repo1.maven.org/151.101.32.209] failed: Operation timed out
>>> (Connection timed out) and 'parent.relativePath' points at wrong local
>>> POM @ line 22, column 10 -> [Help 2]
>>> [ERROR]
>>> [ERROR] To see the full stack trace of the errors, re-run Maven with
>>> the -e switch.
>>> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
>>> [ERROR]
>>> [ERROR] For more information about the errors and possible solutions,
>>> please read the following articles:
>>> [ERROR] [Help 1]
>>> http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
>>> [ERROR] [Help 2]
>>> http://cwiki.apache.org/confluence/display/MAVEN/Unresolvabl
>>> eModelException
>>> 
>>> I’d appreciate any help with resolving this issue.
>>> 
>>> Thanks,
>>> Sandeep
>>> 
>>> 
>>> The information contained in this e-mail is confidential and/or
>>> proprietary to Capital One and/or its affiliates and may only be used
>>> solely 

Re: Maven build failing

[Narayanaswami, Sandeep]

Thank you all so much for your responses. I really appreciate it. 
Felix was correct: It did turn out to be a proxy issue. I was able to hit the 
repository from a browser, so I was initially confused as to why it was 
breaking. I ended up changing my maven settings.xml to include the proxy info 
and that worked. 

Felix’s response leads me to another question: I’m probably missing something 
obvious here, but it seems to me like the SystemML download page only offers 
binaries for the standalone version, and there seems to be no way (besides 
building from source) to get my hands on the SystemML.jar. Is this correct?

On 1/6/17, 17:31, "Mike Dusenberry"  wrote:

Hi Sandeep,

Thanks for reaching out!  The error message looks like a local Maven
issue.  In addition to the suggestion from Felix, another possibility is to
remove (or rename) your Maven cache, typically located at `~/.m2`, and then
try building again with a fresh start.

- Mike


--

Michael W. Dusenberry
GitHub: github.com/dusenberrymw
LinkedIn: linkedin.com/in/mikedusenberry

On Fri, Jan 6, 2017 at 5:26 PM,  wrote:

> Hi Sandeep,
>
> it seems like you can't connect to the maven repository. Can you open the
> link to the repository in a browser? (https://repo1.maven.org/maven2/)
> Are you behind a proxy maybe?
>
> If nothing else, you can download a binary release from our website to get
> started working with SystemML: http://systemml.apache.org/download.html
> The beginners guide (http://systemml.apache.org/get-started) has another
> way of getting started if you don't need to build the source yourself.
>
> - Felix
>
>
> Am 07.01.2017 02:09 schrieb Narayanaswami, Sandeep:
>
>> Dear SystemML community,
>>
>> I attempted the following (as per the Beginners' Guide for Python
>> users> guide-python.html>):
>>
>> git checkout clone https://github.com/apache/incubator-systemml.git
>> cd incubator-systemml
>> mvn clean package -P distribution
>>
>> This resulted in:
>>
>> [INFO] Scanning for projects...
>> Downloading: https://repo1.maven.org/maven2/org/apache/apache/18/apache-
>> 18.pom
>> Downloading:
>> https://raw.github.com/niketanpansare/mavenized-jcuda/mvn-
>> repo/org/apache/apache/18/apache-18.pom
>> [ERROR] [ERROR] Some problems were encountered while processing the POMs:
>> [FATAL] Non-resolvable parent POM for
>> org.apache.systemml:systemml:0.12.0-incubating-SNAPSHOT: Could not
>> transfer artifact org.apache:apache:pom:18 from/to central
>> (https://repo1.maven.org/maven2): Connect to repo1.maven.org:443
>> [repo1.maven.org/151.101.32.209] failed: Operation timed out
>> (Connection timed out) and 'parent.relativePath' points at wrong local
>> POM @ line 22, column 10
>> @
>> [ERROR] The build could not read 1 project -> [Help 1]
>> [ERROR]
>> [ERROR]   The project
>> org.apache.systemml:systemml:0.12.0-incubating-SNAPSHOT
>> (/Users/ioz814/Projects/oss/incubator-systemml/pom.xml) has 1 error
>> [ERROR] Non-resolvable parent POM for
>> org.apache.systemml:systemml:0.12.0-incubating-SNAPSHOT: Could not
>> transfer artifact org.apache:apache:pom:18 from/to central
>> (https://repo1.maven.org/maven2): Connect to repo1.maven.org:443
>> [repo1.maven.org/151.101.32.209] failed: Operation timed out
>> (Connection timed out) and 'parent.relativePath' points at wrong local
>> POM @ line 22, column 10 -> [Help 2]
>> [ERROR]
>> [ERROR] To see the full stack trace of the errors, re-run Maven with
>> the -e switch.
>> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
>> [ERROR]
>> [ERROR] For more information about the errors and possible solutions,
>> please read the following articles:
>> [ERROR] [Help 1]
>> http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
>> [ERROR] [Help 2]
>> http://cwiki.apache.org/confluence/display/MAVEN/Unresolvabl
>> eModelException
>>
>> I’d appreciate any help with resolving this issue.
>>
>> Thanks,
>> Sandeep
>> 
>>
>> The information contained in this e-mail is confidential and/or
>> proprietary to Capital One and/or its affiliates and may only be used
>> solely in performance of work or services for Capital One. The
>> information transmitted herewith is intended only for use by the
>> individual or entity to which it is addressed. If the reader of this
>> message is not the intended recipient, you are hereby notified that
>> any review, retransmission, dissemination, distribution, copying or
>> other use of, or taking of any action in reliance upon this
>> inf

Re: Time To Merge Spark 2.0 Support PR

[Arvind Surve]

ok, Thanks

 Arvind SurveSpark Technology Centerhttp://www.spark.tc/

  From: Luciano Resende 
 To: dev@systemml.incubator.apache.org 
 Sent: Monday, January 9, 2017 12:33 PM
 Subject: Re: Time To Merge Spark 2.0 Support PR
   
On Mon, Jan 9, 2017 at 12:28 PM,  wrote:

> Right, so we can cut a 0.12 release branch now, and then release from
> that, while work moves forward on the master branch, including 2.0 support.
>
>
Exactly, 0.12 release will come from a brach that we will create and Spark
2.0 support gets merged into Master.


-- 
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/


   

Re: Time To Merge Spark 2.0 Support PR

[Luciano Resende]

On Mon, Jan 9, 2017 at 12:28 PM,  wrote:

> Right, so we can cut a 0.12 release branch now, and then release from
> that, while work moves forward on the master branch, including 2.0 support.
>
>
Exactly, 0.12 release will come from a brach that we will create and Spark
2.0 support gets merged into Master.


-- 
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/

Re: Time To Merge Spark 2.0 Support PR

[dusenberrymw]

Right, so we can cut a 0.12 release branch now, and then release from that, 
while work moves forward on the master branch, including 2.0 support.

--

Mike Dusenberry
GitHub: github.com/dusenberrymw
LinkedIn: linkedin.com/in/mikedusenberry

Sent from my iPhone.


> On Jan 9, 2017, at 12:26 PM, Acs S  wrote:
> 
> As I already mentioned we need to have proper way for Spark 1.6 users to use 
> SystemML Python DSL. Pip Install Artifact was missing from SystemML 0.11 
> release and it needs to be added in SystemML 0.12 release.
> Arvind SurveSpark Technology Centerhttp://www.spark.tc
>  From: "dusenberr...@gmail.com" 
> To: dev@systemml.incubator.apache.org 
> Sent: Monday, January 9, 2017 12:18 PM
> Subject: Re: Time To Merge Spark 2.0 Support PR
> 
> Just to be clear, instead of creating a branch to merge the 2.0 support, we 
> will want to merge the 2.0 support into the master branch.
> 
> --
> 
> Mike Dusenberry
> GitHub: github.com/dusenberrymw
> LinkedIn: linkedin.com/in/mikedusenberry
> 
> Sent from my iPhone.
> 
> 
>> On Jan 9, 2017, at 12:02 PM, Acs S  wrote:
>> 
>> Based on discussion thread we will start creating SystemML release based on 
>> Spark 2.0.
>> There are bunch of activities need to be completed and we need volunteer for 
>> most of them.
>> Activity 
>>
>> Volunteer1. Create a branch based on SystemML 0.12 release to merge Spark 
>> 2.0 codeLuciano2. Get Spark 2.0 PR merged to 
>> this new branch. 
>>Glenn3. Do build changes to have both Spark 1.6 and 2.0 
>> builds for release and PR.  (Someone needs to work 
>> with Alan)
>> 4. Setup Spark 2.0 cluster (One of the Almaden cluster updated with Spark 
>> 2.0)5. Create Release Candidate  
>> Glenn, 
>> Deron, Arvind6. Performance Testing7. Notebook testing   
>>  
>>   Arvind8. Python DSL verification (2.x and 3.x)9. 
>> Scala DSL verification10. Artifacts verification11. Documentation update.
>> 
>> -Arvind SurveSpark Technology Centerhttp://www.spark.tc
>> 
>>   From: Niketan Pansare 
>> To: dev@systemml.incubator.apache.org 
>> Sent: Friday, January 6, 2017 1:12 PM
>> Subject: Re: Time To Merge Spark 2.0 Support PR
>> 
>> I am fine with creating a branch for Spark 1.6 support and merging Spark 2.0 
>> PR then. Like Luciano said, we can creating a release 0.12 from our Spark 
>> 1.6 branch. 
>> 
>> Overriding previous release is common practice for pip installer, however 
>> pypi does maintain the history of releases. Once a release candidate 0.12 is 
>> created, the user can install SystemML python package in three ways:
>> 1. From source by checking out the branch and executing: mvn package -P 
>> distribution, followed by pip install 
>> target/systemml-0.12.0-incubating-python.tgz
>> 2. From Apache site, pip install 
>> http://www.apache.org/dyn/closer.lua/incubator/systemml/0.12.0-incubating/systemml-0.12.0-incubating-python.tgz
>> 3. From pypi by specifying the version, pip install -I 
>> systemml_incubating==0.12
>> 
>> As long as we ensure that version of the python package on pypi matches our 
>> release version and we document the Spark support in our release notes, 
>> there should not be any confusion on usage :)
>> 
>> Thanks,
>> 
>> Niketan Pansare
>> IBM Almaden Research Center
>> E-mail: npansar At us.ibm.com
>> http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar
>> 
>> Acs S ---01/06/2017 12:57:53 PM---I would agree to create a branch and add 
>> Spark 2.0 to it, while still releasing SystemML 0.12 releas
>> 
>> From: Acs S 
>> To: "dev@systemml.incubator.apache.org" 
>> Date: 01/06/2017 12:57 PM
>> Subject: Re: Time To Merge Spark 2.0 Support PR
>> 
>> 
>> 
>> I would agree to create a branch and add Spark 2.0 to it, while still 
>> releasing SystemML 0.12 release with Pip Install Artifact.
>> Regarding comment from Mike, that new SystemML release will update PyPy 
>> package.Shouldn't it be tagged with version #? Otherwise every release will 
>> override previous one.Niketan, any comments?
>> -Arvind
>> 
>>   From: Matthias Boehm 
>> To: dev@systemml.incubator.apache.org 
>> Sent: Friday, January 6, 2017 12:52 PM
>> Subject: Re: Time To Merge Spark 2.0 Support PR
>>   
>> +1 on moving to Spark 2.x - I think we delayed this way too long now and 
>> there will always be some awesome feature that we'd want to support on 
>> older Spark versions too.
>> 
>> Regards,
>> Matthias
>> 
>>> On 1/6/2017 9:41 PM, Mike Dusenberry wrote:
>>> Well to be fair, a user can still use the Pyth

Re: Time To Merge Spark 2.0 Support PR

[Acs S]

As I already mentioned we need to have proper way for Spark 1.6 users to use 
SystemML Python DSL. Pip Install Artifact was missing from SystemML 0.11 
release and it needs to be added in SystemML 0.12 release.
Arvind SurveSpark Technology Centerhttp://www.spark.tc
  From: "dusenberr...@gmail.com" 
 To: dev@systemml.incubator.apache.org 
 Sent: Monday, January 9, 2017 12:18 PM
 Subject: Re: Time To Merge Spark 2.0 Support PR
   
Just to be clear, instead of creating a branch to merge the 2.0 support, we 
will want to merge the 2.0 support into the master branch.

--

Mike Dusenberry
GitHub: github.com/dusenberrymw
LinkedIn: linkedin.com/in/mikedusenberry

Sent from my iPhone.


> On Jan 9, 2017, at 12:02 PM, Acs S  wrote:
> 
> Based on discussion thread we will start creating SystemML release based on 
> Spark 2.0.
> There are bunch of activities need to be completed and we need volunteer for 
> most of them.
> Activity                                                                      
>                                                                           
> Volunteer1. Create a branch based on SystemML 0.12 release to merge Spark 2.0 
> code                                Luciano2. Get Spark 2.0 PR merged to this 
> new branch.                                                                   
>              Glenn3. Do build changes to have both Spark 1.6 and 2.0 builds 
> for release and PR.                          (Someone needs to work with Alan)
> 4. Setup Spark 2.0 cluster (One of the Almaden cluster updated with Spark 
> 2.0)5. Create Release Candidate                                               
>                                                                Glenn, Deron, 
> Arvind6. Performance Testing7. Notebook testing                               
>                                                                               
>                  Arvind8. Python DSL verification (2.x and 3.x)9. Scala DSL 
> verification10. Artifacts verification11. Documentation update.
> 
> -Arvind SurveSpark Technology Centerhttp://www.spark.tc
> 
>      From: Niketan Pansare 
> To: dev@systemml.incubator.apache.org 
> Sent: Friday, January 6, 2017 1:12 PM
> Subject: Re: Time To Merge Spark 2.0 Support PR
> 
> I am fine with creating a branch for Spark 1.6 support and merging Spark 2.0 
> PR then. Like Luciano said, we can creating a release 0.12 from our Spark 1.6 
> branch. 
> 
> Overriding previous release is common practice for pip installer, however 
> pypi does maintain the history of releases. Once a release candidate 0.12 is 
> created, the user can install SystemML python package in three ways:
> 1. From source by checking out the branch and executing: mvn package -P 
> distribution, followed by pip install 
> target/systemml-0.12.0-incubating-python.tgz
> 2. From Apache site, pip install 
> http://www.apache.org/dyn/closer.lua/incubator/systemml/0.12.0-incubating/systemml-0.12.0-incubating-python.tgz
> 3. From pypi by specifying the version, pip install -I 
> systemml_incubating==0.12
> 
> As long as we ensure that version of the python package on pypi matches our 
> release version and we document the Spark support in our release notes, there 
> should not be any confusion on usage :)
> 
> Thanks,
> 
> Niketan Pansare
> IBM Almaden Research Center
> E-mail: npansar At us.ibm.com
> http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar
> 
> Acs S ---01/06/2017 12:57:53 PM---I would agree to create a branch and add 
> Spark 2.0 to it, while still releasing SystemML 0.12 releas
> 
> From: Acs S 
> To: "dev@systemml.incubator.apache.org" 
> Date: 01/06/2017 12:57 PM
> Subject: Re: Time To Merge Spark 2.0 Support PR
> 
> 
> 
> I would agree to create a branch and add Spark 2.0 to it, while still 
> releasing SystemML 0.12 release with Pip Install Artifact.
> Regarding comment from Mike, that new SystemML release will update PyPy 
> package.Shouldn't it be tagged with version #? Otherwise every release will 
> override previous one.Niketan, any comments?
> -Arvind
> 
>      From: Matthias Boehm 
> To: dev@systemml.incubator.apache.org 
> Sent: Friday, January 6, 2017 12:52 PM
> Subject: Re: Time To Merge Spark 2.0 Support PR
>  
> +1 on moving to Spark 2.x - I think we delayed this way too long now and 
> there will always be some awesome feature that we'd want to support on 
> older Spark versions too.
> 
> Regards,
> Matthias
> 
>> On 1/6/2017 9:41 PM, Mike Dusenberry wrote:
>> Well to be fair, a user can still use the Python DSL with the SystemML 0.11
>> release by using `pip install -e src/main/python`.  We just didn't place a
>> separate Python binary on the release website.  Keep in mind as well that
>> once we release the next release with Spark 2.x support, a Spark 1.6 will
>> not be able to use `pip install systemml` anyway, as that PyPy package will
>> have been updated to the latest Spark 2.0 release.
>> 
>> I'm concerned that we are moving too s

Re: Time To Merge Spark 2.0 Support PR

[dusenberrymw]

Just to be clear, instead of creating a branch to merge the 2.0 support, we 
will want to merge the 2.0 support into the master branch.

--

Mike Dusenberry
GitHub: github.com/dusenberrymw
LinkedIn: linkedin.com/in/mikedusenberry

Sent from my iPhone.


> On Jan 9, 2017, at 12:02 PM, Acs S  wrote:
> 
> Based on discussion thread we will start creating SystemML release based on 
> Spark 2.0.
> There are bunch of activities need to be completed and we need volunteer for 
> most of them.
> Activity  
>   
> Volunteer1. Create a branch based on SystemML 0.12 release to merge Spark 2.0 
> codeLuciano2. Get Spark 2.0 PR merged to this 
> new branch.   
>  Glenn3. Do build changes to have both Spark 1.6 and 2.0 builds 
> for release and PR.  (Someone needs to work with Alan)
> 4. Setup Spark 2.0 cluster (One of the Almaden cluster updated with Spark 
> 2.0)5. Create Release Candidate   
> Glenn, Deron, 
> Arvind6. Performance Testing7. Notebook testing   
>   
>  Arvind8. Python DSL verification (2.x and 3.x)9. Scala DSL 
> verification10. Artifacts verification11. Documentation update.
> 
> -Arvind SurveSpark Technology Centerhttp://www.spark.tc
> 
>  From: Niketan Pansare 
> To: dev@systemml.incubator.apache.org 
> Sent: Friday, January 6, 2017 1:12 PM
> Subject: Re: Time To Merge Spark 2.0 Support PR
> 
> I am fine with creating a branch for Spark 1.6 support and merging Spark 2.0 
> PR then. Like Luciano said, we can creating a release 0.12 from our Spark 1.6 
> branch. 
> 
> Overriding previous release is common practice for pip installer, however 
> pypi does maintain the history of releases. Once a release candidate 0.12 is 
> created, the user can install SystemML python package in three ways:
> 1. From source by checking out the branch and executing: mvn package -P 
> distribution, followed by pip install 
> target/systemml-0.12.0-incubating-python.tgz
> 2. From Apache site, pip install 
> http://www.apache.org/dyn/closer.lua/incubator/systemml/0.12.0-incubating/systemml-0.12.0-incubating-python.tgz
> 3. From pypi by specifying the version, pip install -I 
> systemml_incubating==0.12
> 
> As long as we ensure that version of the python package on pypi matches our 
> release version and we document the Spark support in our release notes, there 
> should not be any confusion on usage :)
> 
> Thanks,
> 
> Niketan Pansare
> IBM Almaden Research Center
> E-mail: npansar At us.ibm.com
> http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar
> 
> Acs S ---01/06/2017 12:57:53 PM---I would agree to create a branch and add 
> Spark 2.0 to it, while still releasing SystemML 0.12 releas
> 
> From: Acs S 
> To: "dev@systemml.incubator.apache.org" 
> Date: 01/06/2017 12:57 PM
> Subject: Re: Time To Merge Spark 2.0 Support PR
> 
> 
> 
> I would agree to create a branch and add Spark 2.0 to it, while still 
> releasing SystemML 0.12 release with Pip Install Artifact.
> Regarding comment from Mike, that new SystemML release will update PyPy 
> package.Shouldn't it be tagged with version #? Otherwise every release will 
> override previous one.Niketan, any comments?
> -Arvind
> 
>  From: Matthias Boehm 
> To: dev@systemml.incubator.apache.org 
> Sent: Friday, January 6, 2017 12:52 PM
> Subject: Re: Time To Merge Spark 2.0 Support PR
>   
> +1 on moving to Spark 2.x - I think we delayed this way too long now and 
> there will always be some awesome feature that we'd want to support on 
> older Spark versions too.
> 
> Regards,
> Matthias
> 
>> On 1/6/2017 9:41 PM, Mike Dusenberry wrote:
>> Well to be fair, a user can still use the Python DSL with the SystemML 0.11
>> release by using `pip install -e src/main/python`.  We just didn't place a
>> separate Python binary on the release website.  Keep in mind as well that
>> once we release the next release with Spark 2.x support, a Spark 1.6 will
>> not be able to use `pip install systemml` anyway, as that PyPy package will
>> have been updated to the latest Spark 2.0 release.
>> 
>> I'm concerned that we are moving too slowly as a project without a clear
>> set of users holding us back on 1.6.  Thoughts?
>> 
>> 
>> --
>> 
>> Michael W. Dusenberry
>> GitHub: github.com/dusenberrymw
>> LinkedIn: linkedin.com/in/mikedusenberry
>> 
>>> On Fri, Jan 6, 2017 at 12:33 PM, Acs S  wrote:
>>> 
>>> With SystemML 0.11 release we don't have Python DSL support on Spark
>>> 1.6.To have Python DSL on Spark 1.6 support we have to release SystemML
>>> with Pip Insta

Re: Time To Merge Spark 2.0 Support PR

[Acs S]

Based on discussion thread we will start creating SystemML release based on 
Spark 2.0.
There are bunch of activities need to be completed and we need volunteer for 
most of them.
Activity                                                                        
                                                                        
Volunteer1. Create a branch based on SystemML 0.12 release to merge Spark 2.0 
code                                Luciano2. Get Spark 2.0 PR merged to this 
new branch.                                                                     
           Glenn3. Do build changes to have both Spark 1.6 and 2.0 builds for 
release and PR.              (Someone needs to work with Alan)
4. Setup Spark 2.0 cluster (One of the Almaden cluster updated with Spark 
2.0)5. Create Release Candidate                                                 
                                                              Glenn, Deron, 
Arvind6. Performance Testing7. Notebook testing                                 
                                                                                
             Arvind8. Python DSL verification (2.x and 3.x)9. Scala DSL 
verification10. Artifacts verification11. Documentation update.

-Arvind SurveSpark Technology Centerhttp://www.spark.tc

  From: Niketan Pansare 
 To: dev@systemml.incubator.apache.org 
 Sent: Friday, January 6, 2017 1:12 PM
 Subject: Re: Time To Merge Spark 2.0 Support PR
  
I am fine with creating a branch for Spark 1.6 support and merging Spark 2.0 PR 
then. Like Luciano said, we can creating a release 0.12 from our Spark 1.6 
branch. 

Overriding previous release is common practice for pip installer, however pypi 
does maintain the history of releases. Once a release candidate 0.12 is 
created, the user can install SystemML python package in three ways:
1. From source by checking out the branch and executing: mvn package -P 
distribution, followed by pip install 
target/systemml-0.12.0-incubating-python.tgz
2. From Apache site, pip install 
http://www.apache.org/dyn/closer.lua/incubator/systemml/0.12.0-incubating/systemml-0.12.0-incubating-python.tgz
3. From pypi by specifying the version, pip install -I systemml_incubating==0.12

As long as we ensure that version of the python package on pypi matches our 
release version and we document the Spark support in our release notes, there 
should not be any confusion on usage :)

Thanks,

Niketan Pansare
IBM Almaden Research Center
E-mail: npansar At us.ibm.com
http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar

Acs S ---01/06/2017 12:57:53 PM---I would agree to create a branch and add 
Spark 2.0 to it, while still releasing SystemML 0.12 releas

From: Acs S 
To: "dev@systemml.incubator.apache.org" 
Date: 01/06/2017 12:57 PM
Subject: Re: Time To Merge Spark 2.0 Support PR



I would agree to create a branch and add Spark 2.0 to it, while still releasing 
SystemML 0.12 release with Pip Install Artifact.
Regarding comment from Mike, that new SystemML release will update PyPy 
package.Shouldn't it be tagged with version #? Otherwise every release will 
override previous one.Niketan, any comments?
-Arvind

      From: Matthias Boehm 
 To: dev@systemml.incubator.apache.org 
 Sent: Friday, January 6, 2017 12:52 PM
 Subject: Re: Time To Merge Spark 2.0 Support PR
   
+1 on moving to Spark 2.x - I think we delayed this way too long now and 
there will always be some awesome feature that we'd want to support on 
older Spark versions too.

Regards,
Matthias

On 1/6/2017 9:41 PM, Mike Dusenberry wrote:
> Well to be fair, a user can still use the Python DSL with the SystemML 0.11
> release by using `pip install -e src/main/python`.  We just didn't place a
> separate Python binary on the release website.  Keep in mind as well that
> once we release the next release with Spark 2.x support, a Spark 1.6 will
> not be able to use `pip install systemml` anyway, as that PyPy package will
> have been updated to the latest Spark 2.0 release.
>
> I'm concerned that we are moving too slowly as a project without a clear
> set of users holding us back on 1.6.  Thoughts?
>
>
> --
>
> Michael W. Dusenberry
> GitHub: github.com/dusenberrymw
> LinkedIn: linkedin.com/in/mikedusenberry
>
> On Fri, Jan 6, 2017 at 12:33 PM, Acs S  wrote:
>
>> With SystemML 0.11 release we don't have Python DSL support on Spark
>> 1.6.To have Python DSL on Spark 1.6 support we have to release SystemML
>> with Pip Install artifact, which was planned for SystemML 0.12.
>> I understand Spark 2.0 has released, at the same time there are Spark 1.6
>> users who will not able to use SystemML with Python DSL if we simply move
>> to Spark 2.0 without this artifact as planned. Understand SystemML 0.12 has
>> been delayed due to holiday period, but lets not get panicked and jump the
>> hops.
>> -Arvind
>>
>>      From: Deron Eriksson 
>>  To: dev@systemml.incubator.apache.org
>> Cc: Acs S 
>>  Sent: Friday,

Re: Release cadence

[Acs S]

We need to release SystemML on more frequent basis to get community engaged. It 
will provide us more feedback on functionality we add.While releasing SystemML 
on monthly basis is challenge due to longer phase of validation process we need 
to find a way to be quicker.
I can propose options to get closer to monthly release if acceptable.
Make every two releases available on monthly basis and third on two months 
basis. This cycle will continue.
1. Do minimal testing on two releases (minor releases) and release them on 
monthly basis. Performance testing is one of the major time consuming activity 
especially for larger data size. We can limit testing only upto 80GB. We can do 
code freeze (other than fixes) at the end of third week and do verification on 
last week. If we find any issues we can still release the code with limitation 
documented unless issue breaks major functionality.2. Do comprehensive testing 
on third release.   This will include performance testing for all data size and 
every other testing we do. We can do code freeze (other than fixes) at the end 
of third week and start verification of code. All issues found will be 
addressed. Exception will be considered.

Meantime we need to start automating testing pieces.

Arvind SurveSpark Technology Centerhttp://www.spark.tc/

  From: Berthold Reinwald 
 To: dev@systemml.incubator.apache.org 
 Sent: Saturday, January 7, 2017 1:35 AM
 Subject: Re: Release cadence
   
I think that a 2 month cycle would be a good compromise for major/minor 
releases. Fixpack release could be at a 1 month cycle.


Regards,
Berthold Reinwald
IBM Almaden Research Center
office: (408) 927 2208; T/L: 457 2208
e-mail: reinw...@us.ibm.com



From:  Deron Eriksson 
To:    dev@systemml.incubator.apache.org
Date:  01/05/2017 02:14 PM
Subject:        Re: Release cadence



+1 for trying out a 1 month release cycle.

However, I highly agree with Matthias that there is a lot of overhead with
releases, so it would be good if we can work to streamline/automate the
process as much as possible. Also, it would be good to distribute the 
tasks
around as much as possible. This can result in cross-training and help
avoid overburdening the same contributors each month.

If the overhead slows us down too much, then we can go to a slower release
cycle.

Deron




On Thu, Jan 5, 2017 at 1:50 PM,  wrote:

> +1 for adopting a 1 month release cycle.
>
> --
>
> Mike Dusenberry
> GitHub: github.com/dusenberrymw
> LinkedIn: linkedin.com/in/mikedusenberry
>
> Sent from my iPhone.
>
>
> > On Jan 5, 2017, at 1:35 PM, Luciano Resende 
> wrote:
> >
> > On Thu, Jan 5, 2017 at 6:05 AM, Matthias Boehm 

> > wrote:
> >
> >> In general, I like the idea of aiming for consistent release cycles.
> >> However, every month is just too much, at least for me. There is a
> >> considerable overhead associated with each release for end-to-end
> >> performance tests, tests on different environments, code freeze for 
new
> >> features, etc. Hence, a too short release cycle would not be "agile" 
but
> >> would actually slow us down. From my perspective, a realistic release
> >> cadence would be 2-3 months, maybe a bit more for major releases.
> >>
> >>
> > 2-3 months of release cadence for an open source is probably a long
> > stretch, particular for a project that does not have very large set of
> 3rd
> > party dependencies.
> >
> > As for some of the overhead issues you mentioned, they are probably 
easy
> to
> > workaround:
> >
> > - code-freeze timeframe can be resolved with branches
> > - end-to-end performance regressions can be avoided by better code
> review,
> > and if you were willing to go with 2-3 months without performing these
> > tests, we could perform them only for major releases, and proactively
> > quickly build a minor release with the patch when a user report any
> > performance regression.
> >
> >
> > Anyway, I would really like to see SystemML more agile with regards to
> its
> > release process because, as I mentioned before, the release early,
> release
> > often mantra is good to increase community interest, generate more
> traffic
> > to the list as developers discuss the roadmap and release blockers, 
and
> > also enable users to provide feedback sooner on the areas we are
> developing.
> >
> >
> >
> > --
> > Luciano Resende
> > http://twitter.com/lresende1975
> > http://lresende.blogspot.com/
>



-- 
Deron Eriksson
Spark Technology Center
http://www.spark.tc/