Re: Hadoop is not working in dev environment on windows [since 2.7.7 update]. Thanks.
Please take a look at our setup instructions for windows that we created a while back for our KDD Tutorial. You may need to download a newer version of winutils from https://github.com/steveloughran/winutils, and don't forget to chmod the permissions. The attached instructions are from 2017, so you need to adjust your versions. From: Janardhan To: Niketan Pansare , dev@systemml.apache.org, Matthias Boehm Date: 03/05/2019 07:54 PM Subject:Hadoop is not working in dev environment on windows [since 2.7.7 update]. Thanks. Hi, Since 2.7.7 update, my hadoop and winutils [prebuilt winutils.exe not available] is not working because of file permissions. As a workaround I have changed hadoop source locally to bypass the access check. But, is there any way one could run the tests without hadoop. spark-submit is working fine for me. Thanks, Janardhan 1. Java === The Java version should be > 1.8. > java -version Set JAVA_HOME environment variable and include %JAVA_HOME%\bin in the environment variable PATH > ls "%JAVA_HOME%" 2. Spark Download and extract Spark from https://spark.apache.org/downloads.html, > tar -xzf spark-2.1.0-bin-hadoop2.7.tgz and set environment variable SPARK_HOME to point to the extracted directory. Next step, install winutils: - Download winutils.exe from http://github.com/steveloughran/winutils/raw/master/hadoop-2.6.0/bin/winutils.exe - Place it in c:\winutils\bin - Set environment variable HADOOP_HOME to point to c:\winutils - Add c:\winutils\bin to the environment variable PATH. - Finally, modify permission of hive directory that will be used by spark > winutils.exe chmod 777 /tmp/hive Finally, check if Spark is correctly installed: > %SPARK_HOME%\bin\spark-shell > %SPARK_HOME%\bin\pyspark 3. Python and Jupyter = Download and install Anaconda Python 2.7 from https://www.continuum.io/downloads#macos (includes jupyter, and pip) 4. Libraries used in this tutorial == 4.1 Graphviz To check if Graphviz is installed on your system, > dot --help If you get an error, - Download and install http://www.graphviz.org/pub/graphviz/stable/windows/graphviz-2.38.msi - Please ensure that the C:\Program Files (x86)\Graphviz2.38\bin folder in added to the PATH environment variable. 5. Apache SystemML == cd to tutorial folder, and install this version of Apache SystemML, > pip install ./systemml-1.0.0-SNAPSHOT-python.tgz and start pyspark/Jupyter > set PYSPARK_DRIVER_PYTHON=jupyter > set PYSPARK_DRIVER_PYTHON_OPTS=notebook > %SPARK_HOME%\bin\pyspark --driver-memory 8g
upgrading hadoop version
Folks, we still have a dependency to Hadoop 2.6.0 which has several vulnerabilities. We checked Hadoop 2.7.7 where many vulnerabilities have been fixed. Does anyone object upgrading to Hadoop version 2.7.7? Regards, Berthold Reinwald
Re: New committer: Guobao Li
Congrats, Guobao! Regards, Berthold Reinwald IBM Almaden Research Center office: (408) 927 2208; T/L: 457 2208 e-mail: reinw...@us.ibm.com From: Deron Eriksson To: dev@systemml.apache.org Date: 09/05/2018 11:47 AM Subject:Re: New committer: Guobao Li Welcome and congratulations Guobao! Deron On Wed, Sep 5, 2018 at 1:28 AM Guobao Li wrote: > Thanks everyone! It's a great pleasure to join the community. > > Regards, > Guobao > > On Wed, Sep 5, 2018 at 7:02 AM Niketan Pansare wrote: > > > Welcome, Guobao. > > > > > On Sep 4, 2018, at 4:09 PM, dusenberr...@gmail.com wrote: > > > > > > Welcome, Guobao > > > > >
[ANNOUNCE] Apache SystemML 1.2.0 released
The Apache SystemML team is pleased to announce the release of Apache SystemML version 1.2.0. Apache SystemML provides declarative, large-scale machine learning (ML) and deep learning (DL) that aims at a flexible specification of algorithms and automatic generation of hybrid runtime execution plans ranging from single-node, in-memory computations, to distributed computations on Apache Hadoop MapReduce and Apache Spark. For detailed information about the updates, please access the release notes: http://systemml.apache.org/release-notes/systemml-release-notes-1.2.0.html To download the distribution, and learn more about SystemML project, please go to: http://systemml.apache.org/ The Apache SystemML Team Regards, Berthold Reinwald IBM Almaden Research Center office: (408) 927 2208; T/L: 457 2208 e-mail: reinw...@us.ibm.com
[RESULT][VOTE] Apache SystemML 1.2.0 (RC1)
Voting has passed with +1 binding votes from 5 PMC members and +1 vote from 2 contributors. Thanks everyone! PMC MEMBERS: +1 Glenn Weidner Berthold Reinwald Niketan Pansare Deron Eriksson Matthias Boehm CONTRIBUTORS: +1 Anthony Thomas GuoBao Li Regards, Berthold Reinwald IBM Almaden Research Center office: (408) 927 2208; T/L: 457 2208 e-mail: reinw...@us.ibm.com From: "Berthold Reinwald" To: dev@systemml.apache.org Date: 08/16/2018 11:41 PM Subject:[VOTE] Apache SystemML 1.2.0 (RC1) Please vote on releasing the following candidate as Apache SystemML version 1.2.0 The vote is open for at least 72 hours and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache SystemML 1.2.0 [ ] -1 Do not release this package because ... To learn more about Apache SystemML, please see http://systemml.apache.org/ The tag to be voted on is v1.2.0-rc1 ( a1a05e29f6ee78f3c33fea355f62c78ce21766ee): https://github.com/apache/systemml/tree/v1.2.0-rc1 The release artifacts can be found at: https://dist.apache.org/repos/dist/dev/systemml/1.2.0-rc1/ The maven release artifacts, including signatures, digests, etc. can be found at: https://repository.apache.org/content/repositories/orgapachesystemml-1030/org/apache/systemml/systemml/1.2.0/ === == Apache Release policy == === http://www.apache.org/legal/release-policy.html === == How can I help test this release? == === If you are a SystemML user, you can help us test this release by taking an existing Algorithm or workload and running on this release candidate, then reporting any regressions. == What justifies a -1 vote for this release? == -1 votes should only occur for significant stop-ship bugs or legal related issues (e.g. wrong license, missing header files, etc). Minor bugs or regressions should not block this release. Regards, Berthold Reinwald IBM Almaden Research Center office: (408) 927 2208; T/L: 457 2208 e-mail: reinw...@us.ibm.com
Re: [VOTE] Apache SystemML 1.2.0 (RC1)
+1 Ran verification tests, and multiple unit tests on MacOS. Regards, Berthold Reinwald IBM Almaden Research Center office: (408) 927 2208; T/L: 457 2208 e-mail: reinw...@us.ibm.com From: Deron Eriksson To: dev@systemml.apache.org Date: 08/21/2018 02:18 PM Subject:Re: [VOTE] Apache SystemML 1.2.0 (RC1) +1 Ran basic tests to verify bin artifacts run and that artifacts can be built and run using source artifact. License files in artifacts appeared to be correct to me. Note that I believe the lite jar should be removed from https://dist.apache.org/repos/dist/dev/systemml/1.2.0-rc1/ Deron On Tue, Aug 21, 2018 at 11:51 AM Anthony Thomas wrote: > +1 > > I ran the Python test suite on Red Hat Linux under Spark 2.2.0 (Python > 2.7.5) and encountered no errors. > > Regards, > Anthony > > On Tue, Aug 21, 2018 at 7:49 AM Guobao Li wrote: > > > +1 > > > > As an initiator and user of paramserv func, I just launched several tests > > on local pc with a script using paramserv func without mkl. And no bug is > > observed. > > > > Regards, > > Guobao > > > > On Sun, Aug 19, 2018 at 8:09 PM Matthias Boehm > wrote: > > > > > +1 > > > > > > I ran the perftest suite multiple times up to 80GB with and without > > > codegen. After fixing all the issues and regressions, the entire suite > > > ran successfully against Spark 2.2 and 2.3 and all use cases showed > > > equal or better performance compared to SystemML 1.1. > > > > > > Regards, > > > Matthias > > > > > > On Fri, Aug 17, 2018 at 8:41 AM, Berthold Reinwald < > reinw...@us.ibm.com> > > > wrote: > > > > Please vote on releasing the following candidate as Apache SystemML > > > > version 1.2.0 > > > > > > > > The vote is open for at least 72 hours and passes if a majority of at > > > > least 3 +1 PMC votes are cast. > > > > > > > > [ ] +1 Release this package as Apache SystemML 1.2.0 > > > > [ ] -1 Do not release this package because ... > > > > > > > > To learn more about Apache SystemML, please see > > > > http://systemml.apache.org/ > > > > > > > > > > > > The tag to be voted on is v1.2.0-rc1 ( > > > > a1a05e29f6ee78f3c33fea355f62c78ce21766ee): > > > > https://github.com/apache/systemml/tree/v1.2.0-rc1 > > > > > > > > > > > > The release artifacts can be found at: > > > > https://dist.apache.org/repos/dist/dev/systemml/1.2.0-rc1/ > > > > > > > > > > > > The maven release artifacts, including signatures, digests, etc. can > be > > > > found at: > > > > > > > > > > https://repository.apache.org/content/repositories/orgapachesystemml-1030/org/apache/systemml/systemml/1.2.0/ > > > > > > > > > > > > > > > > === > > > > == Apache Release policy == > > > > === > > > > http://www.apache.org/legal/release-policy.html > > > > > > > > > > > > === > > > > == How can I help test this release? == > > > > === > > > > If you are a SystemML user, you can help us test this release by > taking > > > an > > > > > > > > > > > > existing Algorithm or workload and running on this release candidate, > > > then > > > > > > > > > > > > reporting any regressions. > > > > > > > > > > > > == What justifies a -1 vote for this release? == > > > > > > > > -1 votes should only occur for significant stop-ship bugs or legal > > > related > > > > > > > > > > > > issues (e.g. wrong license, missing header files, etc). Minor bugs or > > > > regressions should not block this release. > > > > > > > > > > > > > > > > Regards, > > > > Berthold Reinwald > > > > IBM Almaden Research Center > > > > office: (408) 927 2208; T/L: 457 2208 > > > > e-mail: reinw...@us.ibm.com > > > > > > > > > >
[VOTE] Apache SystemML 1.2.0 (RC1)
Please vote on releasing the following candidate as Apache SystemML version 1.2.0 The vote is open for at least 72 hours and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache SystemML 1.2.0 [ ] -1 Do not release this package because ... To learn more about Apache SystemML, please see http://systemml.apache.org/ The tag to be voted on is v1.2.0-rc1 ( a1a05e29f6ee78f3c33fea355f62c78ce21766ee): https://github.com/apache/systemml/tree/v1.2.0-rc1 The release artifacts can be found at: https://dist.apache.org/repos/dist/dev/systemml/1.2.0-rc1/ The maven release artifacts, including signatures, digests, etc. can be found at: https://repository.apache.org/content/repositories/orgapachesystemml-1030/org/apache/systemml/systemml/1.2.0/ === == Apache Release policy == === http://www.apache.org/legal/release-policy.html === == How can I help test this release? == === If you are a SystemML user, you can help us test this release by taking an existing Algorithm or workload and running on this release candidate, then reporting any regressions. == What justifies a -1 vote for this release? == -1 votes should only occur for significant stop-ship bugs or legal related issues (e.g. wrong license, missing header files, etc). Minor bugs or regressions should not block this release. Regards, Berthold Reinwald IBM Almaden Research Center office: (408) 927 2208; T/L: 457 2208 e-mail: reinw...@us.ibm.com
Re: which() Function in SystemML
You may also look at example 2 "Replace NaN with mode" in our "DML Tips
and Tricks" examples notebook. Will add a which() example shortly.
https://github.com/apache/systemml/raw/master/samples/jupyter-notebooks/DML%20Tips%20and%20Tricks%20(aka%20Fun%20With%20DML).ipynb
Regards,
Berthold Reinwald
IBM Almaden Research Center
office: (408) 927 2208; T/L: 457 2208
e-mail: reinw...@us.ibm.com
From: "Niketan Pansare"
To: dev@systemml.apache.org
Date: 07/05/2018 10:26 AM
Subject:Re: which() Function in SystemML
Hi Kuno,
Thanks for trying out SystemML. One way to implement your logic in
SystemML is as follows:
X = matrix("1 4 -99 2 5 -99", rows=6, cols=1)
Y = removeEmpty(target=(X > 2)*seq(1, nrow(X)), margin="rows")
print("\nIndexes:\n" + toString(Y))
For handling NAs, you can replace NAs by certain value (in this case -99
or one of the builtin constants, i.e. Inf or NaN ... see
https://github.com/apache/systemml/blob/master/src/test/scripts/org/apache/sysml/api/mlcontext/builtin-constants-test.dml
) using the "na.strings" parameter of the transform builtin function (
http://apache.github.io/systemml/dml-language-reference.html#transforming-frames
).
Hope this helps.
Regards,
Niketan Pansare
IBM Almaden Research Center
E-mail: npansar At us.ibm.com
http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar
"Baeriswyl Kuno (IT-SWE-CC1-JV6)" ---07/05/2018 05:29:53 AM---Hello, I'm a
developer want to use SystemMl for running R-Code from our business people
on a Spark c
From: "Baeriswyl Kuno (IT-SWE-CC1-JV6)"
To: "dev@systemml.apache.org"
Date: 07/05/2018 05:29 AM
Subject: which() Function in SystemML
Hello,
I'm a developer want to use SystemMl for running R-Code from our business
people on a Spark cluster.
I've studied http://apache.github.io/systemml/dml-language-reference ,
however, haven't found a implementation of the R function "which" or any
alternative functionality. Has anyone an idea how I could
# Given
v = c(1,4,NA,2, 5, NA)
# Expect indexes where value meets condition = int[] 2 5
v2 = which(v>2)
# Expect indexes where is.na returns TRUE = int[] 3 6
v3 = which(is.na(v))
I've already considered the functions replace() and removeEmpty(), but
they don't exactly meets my needs.
Thanks a lot in advance
Kuno
Re: Release Planning SystemML 1.2
either one would be great but having the GPU integration is a plus. Why wouldn't that be possible for Centos? Regards, Berthold Reinwald IBM Almaden Research Center office: (408) 927 2208; T/L: 457 2208 e-mail: reinw...@us.ibm.com From: Janardhan To: "dev@systemml.apache.org" Date: 06/24/2018 05:56 AM Subject:Re: Release Planning SystemML 1.2 One more thing, I am already building a docker image. But, which image do you prefer 1. CentOS 7 or 2. Ubuntu - Later extensible to GPU very easily. This decreases setup time to six minutes on any platform, even for non devs. Thank you Janardhan On Sunday, June 24, 2018, Matthias Boehm wrote: > given the current status of open tasks and the delay with regard to > QA, I think we need to push this release out by a couple of weeks. > Does mid to end July sound good to everyone? > > Regards, > Matthias > > On Wed, Jun 6, 2018 at 11:28 PM, Matthias Boehm wrote: > > thanks Berthold - that sounds good. We'll probably start a little > > earlier with the tests to ensure the python test suite gives the same > > results for the set of overlapping algorithms. > > > > Regards, > > Matthias > > > > On Wed, Jun 6, 2018 at 11:08 PM, Berthold Reinwald > wrote: > >> +1. Happy to kick off the release scripts the week of June 25th. We > should > >> probably kick off the python perf test suite the week before to avoid > >> hiccups. > >> > >> Regards, > >> Berthold Reinwald > >> IBM Almaden Research Center > >> office: (408) 927 2208; T/L: 457 2208 > >> e-mail: reinw...@us.ibm.com > >> > >> > >> > >> From: Krishna Kalyan > >> To: dev@systemml.apache.org > >> Date: 06/05/2018 10:09 PM > >> Subject:Re: Release Planning SystemML 1.2 > >> > >> > >> > >> +1 > >> > >> I am completely available to help with the QA cycle and help with > >> switching > >> to new perf test suite. > >> > >> Cheers, > >> Krishna > >> > >> On Tue, Jun 5, 2018 at 8:59 AM, Matthias Boehm > wrote: > >> > >>> Hi all, > >>> > >>> given our current release cadence of about 3 month, we should start > >>> talking about our SystemML 1.2 release. There have been many new > >>> features, improvements over all backends, and various critical fixes. > >>> I know there are still some tasks that should go into 1.2, and for > >>> this QA cycle we'd like to switch to the new python perf test suite. > >>> How about we aim for an RC1 by end of June which would give us more > >>> than 3 weeks from now? > >>> > >>> Regards, > >>> Matthias > >>> > >> > >> > >> > >> >
Re: Release Planning SystemML 1.2
+1. Happy to kick off the release scripts the week of June 25th. We should probably kick off the python perf test suite the week before to avoid hiccups. Regards, Berthold Reinwald IBM Almaden Research Center office: (408) 927 2208; T/L: 457 2208 e-mail: reinw...@us.ibm.com From: Krishna Kalyan To: dev@systemml.apache.org Date: 06/05/2018 10:09 PM Subject:Re: Release Planning SystemML 1.2 +1 I am completely available to help with the QA cycle and help with switching to new perf test suite. Cheers, Krishna On Tue, Jun 5, 2018 at 8:59 AM, Matthias Boehm wrote: > Hi all, > > given our current release cadence of about 3 month, we should start > talking about our SystemML 1.2 release. There have been many new > features, improvements over all backends, and various critical fixes. > I know there are still some tasks that should go into 1.2, and for > this QA cycle we'd like to switch to the new python perf test suite. > How about we aim for an RC1 by end of June which would give us more > than 3 weeks from now? > > Regards, > Matthias >
Re: GSoC 2018 Student Guobao Li
Welcome, Guobao! Regards, Berthold Reinwald IBM Almaden Research Center office: (408) 927 2208; T/L: 457 2208 e-mail: reinw...@us.ibm.com From: Krishna Kalyan To: dev@systemml.apache.org Cc: Guobao Li Date: 05/01/2018 11:50 PM Subject:Re: GSoC 2018 Student Guobao Li Congratulations Guobao Li. On Wed, May 2, 2018, 6:49 AM Matthias Boehm wrote: > Hi all, > > please join me in welcoming Guobao Li as a GSoC 2018 student, who will > be working on SYSTEMML-2083 (Language and runtime for parameter > servers) this summer. We're currently in the community bonding phase, > but the project will start May 14. > > Krishna already kindly volunteered (on the dev list and in an offline > discussion) to act as a co-mentor when needed. Additionally, since > this is a major feature, I encourage everybody to provide early > feedback on the design documents and PRs, even if it's just for > specific aspects. > > @Guobao: Please reach out as needed and let's aim to have all > technical discussions, except for our weekly calls, in public. A good > place to start would be to restructure the current JIRA tasks in the > next weeks. > > Regards, > Matthias >
[ANNOUNCE] Apache SystemML 1.1.0 released.
The Apache SystemML team is pleased to announce the release of Apache SystemML version 1.1.0. Apache SystemML provides declarative, large-scale machine learning (ML) and deep learning (DL) that aims at a flexible specification of algorithms and automatic generation of hybrid runtime execution plans ranging from single-node, in-memory computations, to distributed computations on Apache Hadoop MapReduce and Apache Spark. For detailed information about the updates, please access the release notes: http://systemml.apache.org/release-notes/systemml-release-notes-1.1.0.html To download the distribution, and learn more about SystemML project, please go to: http://systemml.apache.org/ The Apache SystemML Team Regards, Berthold Reinwald IBM Almaden Research Center office: (408) 927 2208; T/L: 457 2208 e-mail: reinw...@us.ibm.com
Draft Release Notes 1.1.0
Please add/update below release note draft prior to putting it on our website. Please also go to the JIRA release notes link at the end, and update version/status if necessary. Thanks. * Release Notes - SystemML - Version SystemML 1.1 New Capabilities/Features - Dense matrix blocks >16GB, and operations - Support bitwise operators not, and, or, xor, & LShift, Rshift - Additional ParFor result aggregation operations - UDFs callable in expressions - zero rws/columns matrices - Extended Caffe2DML and Keras2DML APIs New Builtin Functions - ifelse() Performance Improvements - Ultra-sparse operations Internals - Single-precision support for native conv2d and mm operations. - Consolidate replicated compilation chain - Generalize Binary Operations to (vector, matrix) Operands - Use common thread pool Bug Fixes - in APIs, performance bugs, optimizer, runtime, GPU backend, Spark backend. Experimental - Codegen Deprecate - Support for Spark 2.1/2.2 (make switch to newer ANTLR version) Detailed JIRA release notes are here: https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12342282&styleName=Text&projectId=12319522&Create=Create&atl_token=A5KQ-2QAV-T4JA-FDED%7C2e97f76f75a466564dba2a7d0bec5ca06bffd66e%7Clout Regards, Berthold Reinwald IBM Almaden Research Center office: (408) 927 2208; T/L: 457 2208 e-mail: reinw...@us.ibm.com
[RESULT] [VOTE] Apache SystemML 1.1.0 (RC2)
Voting has passed with +1 binding votes from 4 PMC members, +1 vote from 1 committer, and +1 vote from 1 contributor. Thanks everyone! PMC MEMBERS: +1 Matthias Boehm, Glenn Weidner, Berthold Reinwald, Niketan Pansare Committer +1 Krishna Kalyan Contributor +1 Ted Yu Regards, Berthold Reinwald IBM Almaden Research Center office: (408) 927 2208; T/L: 457 2208 e-mail: reinw...@us.ibm.com From: Berthold Reinwald/Almaden/IBM To: dev@systemml.apache.org Date: 03/23/2018 05:24 PM Subject:[VOTE] Apache SystemML 1.1.0 (RC2) Please vote on releasing the following candidate as Apache SystemML version 1.1.0 The vote is open for at least 72 hours and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache SystemML 1.1.0 [ ] -1 Do not release this package because ... To learn more about Apache SystemML, please see http://systemml.apache.org/ The tag to be voted on is v1.1.0-rc2 ( deddaee1fd3f8e87ff3a4403edf06cdb022ba949): https://github.com/apache/systemml/commit/deddaee1fd3f8e87ff3a4403edf06cdb022ba949 The release artifacts can be found at: https://dist.apache.org/repos/dist/dev/systemml/1.1.0-rc2/ The maven release artifacts, including signatures, digests, etc. can be found at: https://repository.apache.org/content/repositories/orgapachesystemml-1029/org/apache/systemml/systemml/1.1.0/ Regards, Berthold Reinwald IBM Almaden Research Center office: (408) 927 2208; T/L: 457 2208 e-mail: reinw...@us.ibm.com
Re: [VOTE] Apache SystemML 1.1.0 (RC2)
+1 - All Artifacts and Checksums present - Release Candidagte BUild - All Binaries Execute - Python tests for Spark 2 - Checked License and notice files - Src Artifact Builds and Tests Pass - Single-Node Standalone - Single-Node Spark - Did not verify notebooks, Performance Suite, NN Unit Test for GPU, GPU Unit Tests. Regards, Berthold Reinwald IBM Almaden Research Center office: (408) 927 2208; T/L: 457 2208 e-mail: reinw...@us.ibm.com From: "Glenn Weidner" To: dev@systemml.apache.org Date: 03/24/2018 08:08 PM Subject:Re: [VOTE] Apache SystemML 1.1.0 (RC2) +1 I installed release candidate in cloud environment. Successfully ran notebook containing different forms of Linear Regression (Direct Solve, Batch Gradient Descent and Conjugate Gradient) on Python 2.7 with Spark 2.1.2. Also ran tests for Logistic Regression, SVM, Naive Bayes. Regards, Glenn Weidner Matthias Boehm ---03/24/2018 07:50:41 PM---+1, I reran all tests mentioned before for RC2 as well without any issues. Regards, From: Matthias Boehm To: dev@systemml.apache.org Date: 03/24/2018 07:50 PM Subject: Re: [VOTE] Apache SystemML 1.1.0 (RC2) +1, I reran all tests mentioned before for RC2 as well without any issues. Regards, Matthias On Fri, Mar 23, 2018 at 5:26 PM, Berthold Reinwald wrote: > Please vote on releasing the following candidate as Apache SystemML > version 1.1.0 > > The vote is open for at least 72 hours and passes if a majority of at > least 3 +1 PMC votes are cast. > > [ ] +1 Release this package as Apache SystemML 1.1.0 > [ ] -1 Do not release this package because ... > > To learn more about Apache SystemML, please see > https://urldefense.proofpoint.com/v2/url?u=http-3A__systemml.apache.org_&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=we2L9Q2NOeniL5PbfdrO3CmQPNy_vnhoDpPDviJpnrU&m=yW2LvwpVHHF2gH86mNkepxlt6p-N7gKyF-7WuISxeaI&s=tTkfz8va3LBa6QGYSvAWzHWNduMOp_oulG50Tld7hac&e= > > The tag to be voted on is v1.1.0-rc2 ( > deddaee1fd3f8e87ff3a4403edf06cdb022ba949): > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_systemml_commit_deddaee1fd3f8e87ff3a4403edf06c&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=we2L9Q2NOeniL5PbfdrO3CmQPNy_vnhoDpPDviJpnrU&m=yW2LvwpVHHF2gH86mNkepxlt6p-N7gKyF-7WuISxeaI&s=Oe92U-HKVm7urCcDvxJnuPyN1WkSawjW4N_g4g4cicA&e= > db022ba949 > > The release artifacts can be found at: > https://urldefense.proofpoint.com/v2/url?u=https-3A__dist.apache.org_repos_dist_dev_systemml_1.1.0-2Drc2_&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=we2L9Q2NOeniL5PbfdrO3CmQPNy_vnhoDpPDviJpnrU&m=yW2LvwpVHHF2gH86mNkepxlt6p-N7gKyF-7WuISxeaI&s=xPViDmV63Nn1uzftEmLnJaB9NUpqK2xFFsxgWM6n20U&e= > > The maven release artifacts, including signatures, digests, etc. can be > found at: > https://urldefense.proofpoint.com/v2/url?u=https-3A__repository.apache.org_content_repositories_&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=we2L9Q2NOeniL5PbfdrO3CmQPNy_vnhoDpPDviJpnrU&m=yW2LvwpVHHF2gH86mNkepxlt6p-N7gKyF-7WuISxeaI&s=knMsCVw9y27HnVIsXsT3QRxR-_WvWm4sRXmt958e2qQ&e= > orgapachesystemml-1029/org/apache/systemml/systemml/1.1.0/ > > Regards, > Berthold Reinwald > IBM Almaden Research Center > office: (408) 927 2208; T/L: 457 2208 > e-mail: reinw...@us.ibm.com > >
[VOTE] Apache SystemML 1.1.0 (RC2)
Please vote on releasing the following candidate as Apache SystemML version 1.1.0 The vote is open for at least 72 hours and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache SystemML 1.1.0 [ ] -1 Do not release this package because ... To learn more about Apache SystemML, please see http://systemml.apache.org/ The tag to be voted on is v1.1.0-rc2 ( deddaee1fd3f8e87ff3a4403edf06cdb022ba949): https://github.com/apache/systemml/commit/deddaee1fd3f8e87ff3a4403edf06cdb022ba949 The release artifacts can be found at: https://dist.apache.org/repos/dist/dev/systemml/1.1.0-rc2/ The maven release artifacts, including signatures, digests, etc. can be found at: https://repository.apache.org/content/repositories/orgapachesystemml-1029/org/apache/systemml/systemml/1.1.0/ Regards, Berthold Reinwald IBM Almaden Research Center office: (408) 927 2208; T/L: 457 2208 e-mail: reinw...@us.ibm.com
[VOTE] Apache SystemML 1.1.0 (RC1)
Please vote on releasing the following candidate as Apache SystemML version 1.1.0 The vote is open for at least 72 hours and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache SystemML 1.1.0 [ ] -1 Do not release this package because ... To learn more about Apache SystemML, please see http://systemml.apache.org/ The tag to be voted on is v1.1.0-rc1 ( edfc293339f7a88db4ee407ac1194983d5f8c23f): https://github.com/apache/systemml/commit/edfc293339f7a88db4ee407ac1194983d5f8c23f The release artifacts can be found at: https://dist.apache.org/repos/dist/dev/systemml/1.1.0-rc1/ The maven release artifacts, including signatures, digests, etc. can be found at: https://repository.apache.org/content/repositories/orgapachesystemml-1028/org/apache/systemml/systemml/1.1.0/ === == Apache Release policy == === http://www.apache.org/legal/release-policy.html === == How can I help test this release? == === If you are a SystemML user, you can help us test this release by taking an existing Algorithm or workload and running on this release candidate, then reporting any regressions. == What justifies a -1 vote for this release? == -1 votes should only occur for significant stop-ship bugs or legal related issues (e.g. wrong license, missing header files, etc). Minor bugs or regressions should not block this release. Regards, Berthold Reinwald IBM Almaden Research Center office: (408) 927 2208; T/L: 457 2208 e-mail: reinw...@us.ibm.com
Re: Release Planning
thanks, Matthias. I will kick off the process on Monday morning PST. Regards, Berthold Reinwald IBM Almaden Research Center office: (408) 927 2208; T/L: 457 2208 e-mail: reinw...@us.ibm.com From: Matthias Boehm To: dev@systemml.apache.org Date: 03/16/2018 09:21 PM Subject:Re: Release Planning Thanks for the patience. Meanwhile, all known issues have been resolved and we're ready to cut an RC. Until this is done (or we have a 1.1 branch), I would recommend to limit all commits to critical fixes. Regards, Matthias On Tue, Mar 13, 2018 at 12:18 AM, Matthias Boehm wrote: > just a quick update: the ANTLR issue and most other things that came up > during QA are fixed now, except for performance issues with stratified > stats for which I need some more time (SYSTEMML-2181 tracks the open > issues). So it's probably a good idea to postpone the RC1 for a few days to > avoid unnecessary release efforts. > > Regards, > Matthias > > On Sun, Mar 11, 2018 at 5:43 PM, Matthias Boehm wrote: > >> well, after trying to run our perftest suite with Spark 2.3 and Spark 2.2 >> this seems to be more complicated. Although the version update from 4.5.3 >> to 4.7.1 solved the problem with Spark 2.3 (which uses 4.7.1), SystemML >> would no longer be backwards compatible with Spark 2.2 and 2.1 (which use >> 4.5.3) because ANTLR checks bidirectional mismatches.Unfortunately, >> removing ANTLR from our jar does not help because the above versions are >> binary incompatible. >> >> We need to hold off the release until we decided whether (1) we directly >> release for Spark 2.3 and drop 2.2 and 2.1, or (2) we release for Spark 2.2 >> and 2.1 with a subsequent ANTLR change and release for 2.3. >> >> Regards, >> Matthias >> >> On Thu, Mar 8, 2018 at 5:38 PM, Niketan Pansare >> wrote: >> >>> +1. That will preserve current behavior on older Spark versions too. >>> >>> > On Mar 8, 2018, at 5:24 PM, Ted Yu wrote: >>> > >>> > +1 on upgrading >>> > Original message From: Matthias Boehm < >>> mboe...@gmail.com> Date: 3/8/18 5:19 PM (GMT-08:00) To: >>> dev@systemml.apache.org Subject: Re: Release Planning >>> > related to Spark 2.3, we might want to update our ANTLR version because >>> > with Spark 2.3 every parsed DML script (i.e., multiple times with >>> imported >>> > DML files) produces the following warning: >>> > >>> > "ANTLR Tool version 4.5.3 used for code generation does not match the >>> > current runtime version 4.7" >>> > >>> > Regards, >>> > Matthias >>> > >>> >> On Thu, Mar 1, 2018 at 5:22 PM, Matthias Boehm >>> wrote: >>> >> >>> >> Hi all, >>> >> >>> >> I'm sure you've seen that Spark 2.3 just got released. This lines up >>> >> beautifully with our own SystemML 1.1 release. Accordingly, I would >>> >> recommend to use Spark 2.3 for our due diligence algorithm-level Q/A. >>> How >>> >> about we shoot for an RC1 by March 12? This should give enough time >>> to run >>> >> over reasonably large data and fix all related issues. >>> >> >>> >> Regards, >>> >> Matthias >>> >> >>> >>> On Tue, Feb 6, 2018 at 12:51 PM, Matthias Boehm >>> wrote: >>> >>> >>> >>> yes, absolutely. Here is a list of new features and improvements - >>> please >>> >>> feel free to extend as needed: >>> >>> >>> >>> 1) Extended Caffe2DML and Keras2DML APIs >>> >>> 2) Support for large-dense blocks >16GB in CP >>> >>> 3) New builtin functions ifelse, xor, as well as and/or/not over >>> matrices >>> >>> 4) Single-precision support for native conv2d and mm operations. >>> >>> 5) Performance features and correctness of ultra-sparse operations >>> >>> 6) Codegen: new plan cache, nary cbind >>> >>> 7) Parfor: result merge with accumulators +=, reduced initialization >>> >>> overhead >>> >>> 8) Compiler improvements: avoid unnecessary spark instructions, >>> corrected >>> >>> memory estimates >>> >>> >>> >>> Until the RC, I'd like to support all deep learning builtin >>> functions in >>> >>> codegen, add a couple of pending parfor impr
Re: Release Planning
sure. Makes sense. Codegen and Keras2DML made good progress, and many other fixes/improvements. What else do we want time/track/highlight for it? Regards, Berthold Reinwald IBM Almaden Research Center office: (408) 927 2208; T/L: 457 2208 e-mail: reinw...@us.ibm.com From: Matthias Boehm To: dev@systemml.apache.org Date: 02/05/2018 11:05 PM Subject:Release Planning Hi all, since our 1.0 release in Dec, we already got a number enhancements and new features in, so I think it would be good to discuss the timeline for our next SystemML 1.1 release. How about, we target mid March for a first RC? Also, Berthold would you be willing to serve again as the release manager? Regards, Matthias
[ANNOUNCE] Apache SystemML 1.0.0 released.
The Apache SystemML team is pleased to announce the release of Apache SystemML version 1.0.0. Apache SystemML provides declarative, large-scale machine learning (ML) and deep learning (DL) that aims at a flexible specification of algorithms and automatic generation of hybrid run time execution plans ranging from single-node, in-memory computations, to distributed computations on Apache Hadoop MapReduce and Apache Spark. For detailed information about the updates, please access the release notes: http://systemml.apache.org/release-notes/systemml-release-notes-1.0.0.html To download the distribution, and learn more about SystemML project, please go to: http://systemml.apache.org/ The Apache SystemML Team
[Result] [VOTE] Apache SystemML 1.0.0 (RC2)
Voting has passed with +1 vote from 6 PMC members and 1 committer. Thanks everyone ! PPMC MEMBERS: Matthias Boehm Glenn Weidner Niketan Pansare Fred Reiss Arvind Surve Berthold Reinwald COMMITTER: Krishna Kalyan Regards, Berthold Reinwald IBM Almaden Research Center office: (408) 927 2208; T/L: 457 2208 e-mail: reinw...@us.ibm.com From: "Berthold Reinwald" To: dev@systemml.apache.org Date: 12/11/2017 04:18 PM Subject:Re: [VOTE] Apache SystemML 1.0.0 (RC2) Thanks, Luciano, for providing the right URL to the v1.0.0-rc2. Given that the testing happened on the artifacts ... unless I hear differently ... we will still count the votes. Regards, Berthold Reinwald IBM Almaden Research Center office: (408) 927 2208; T/L: 457 2208 e-mail: reinw...@us.ibm.com From: Luciano Resende To: dev@systemml.apache.org Date: 12/09/2017 04:40 AM Subject:Re: [VOTE] Apache SystemML 1.0.0 (RC2) On Fri, Dec 8, 2017 at 12:15 PM, Berthold Reinwald wrote: > Please vote on releasing the following candidate as Apache SystemML > version 1.0.0 > > The vote is open for at least 72 hours and passes if a majority of at > least 3 +1 PMC votes are cast. > > [ ] +1 Release this package as Apache SystemML 1.0.0 > [ ] -1 Do not release this package because ... > > To learn more about Apache SystemML, please see > https://urldefense.proofpoint.com/v2/url?u=http-3A__systemml.apache.org_&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=rKpXCxhAHRimLTQtp6EVnl3INiKbhTT17qTa33Vf73k&m=eou0r2zjKobBw6J6T0B8JA0uV5Hg9aOuGwyPXjUjD58&s=wdOwXfhHzgIVCqYR0yO3QnIJcRDW8qdK-oCTRsxIMRA&e= > > The tag to be voted on is v1.0.0-rc2 ( > 7b44380e49425ac20912c2d42de8603a06fa55a1): > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_systemml_commit_7b44380e49425ac20912c2d42de860&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=rKpXCxhAHRimLTQtp6EVnl3INiKbhTT17qTa33Vf73k&m=eou0r2zjKobBw6J6T0B8JA0uV5Hg9aOuGwyPXjUjD58&s=0DHX6hrY0Wk1zJJFeliE2fk3OErSP0UFCsLnPXlDcGY&e= > 3a06fa55a1 > > Looks like the tag hash should be 6519f3383f8fd43af93ecba8742ae4b2b28d4b35 And the link https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_systemml_tree_6519f3383f8fd43af93ecba8742ae4b2b28d4b35&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=rKpXCxhAHRimLTQtp6EVnl3INiKbhTT17qTa33Vf73k&m=eou0r2zjKobBw6J6T0B8JA0uV5Hg9aOuGwyPXjUjD58&s=knOmScBJ-6tdd7Gm9MNYKeeMS0G52RTsUlN_yn9qSTc&e= > > The release artifacts can be found at: > https://urldefense.proofpoint.com/v2/url?u=https-3A__dist.apache.org_repos_dist_dev_systemml_1.0.0-2Drc2_&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=rKpXCxhAHRimLTQtp6EVnl3INiKbhTT17qTa33Vf73k&m=eou0r2zjKobBw6J6T0B8JA0uV5Hg9aOuGwyPXjUjD58&s=tCa8NampBIxay_0dzH1-l0d-spKLxRKc1RUssMJVlN8&e= > > > The maven release artifacts, including signatures, digests, etc. can be > found at: > https://urldefense.proofpoint.com/v2/url?u=https-3A__repository.apache.org_content_repositories_&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=rKpXCxhAHRimLTQtp6EVnl3INiKbhTT17qTa33Vf73k&m=eou0r2zjKobBw6J6T0B8JA0uV5Hg9aOuGwyPXjUjD58&s=SPiv_akoJfsZuTYp3n0dhlhvBzC4Yw_snBzpH9Gd-kU&e= > orgapachesystemml-1026/org/apache/systemml/systemml/1.0.0/ > > > > === > == Apache Release policy == > === > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.apache.org_legal_release-2Dpolicy.html&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=rKpXCxhAHRimLTQtp6EVnl3INiKbhTT17qTa33Vf73k&m=eou0r2zjKobBw6J6T0B8JA0uV5Hg9aOuGwyPXjUjD58&s=ftel26ughEALqJoIE9_CZOhYL8psZCxDa75TaLe4McM&e= > > === > == How can I help test this release? == > === > If you are a SystemML user, you can help us test this release by taking an > existing Algorithm or workload and running on this release candidate, then > reporting any regressions. > > > == What justifies a -1 vote for this release? == > > -1 votes should only occur for significant stop-ship bugs or legal related > issues (e.g. wrong license, missing header files, etc). Minor bugs or > regressions should not block this release. > > -- Luciano Resende https://urldefense.proofpoint.com/v2/url?u=http-3A__twitter.com_lresende1975&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=rKpXCxhAHRimLTQtp6EVnl3INiKbhTT17qTa33Vf73k&m=eou0r2zjKobBw6J6T0B8JA0uV5Hg9aOuGwyPXjUjD58&s=mw_eeXRt-_qvXPMvkNLe0neN2HkKvCiRSZfwZN2Fsd0&e= https://urldefense.proofpoint.com/v2/url?u=http-3A__lresende.blogspot.com_&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=rKpXCxhAHRimLTQtp6EVnl3INiKbhTT17qTa33Vf73k&m=eou0r2zjKobBw6J6T0B8JA0uV5Hg9aOuGwyPXjUjD58&s=3udCTmnn9ztd4J3wRevm5_vH-kIuQOLq4D7KT-8FHeA&e=
Re: [VOTE] Apache SystemML 1.0.0 (RC2)
+1 Tested: all artifacts and checksums present for artifacts tested release candidate build ran test suite passes all binaries execute checked license and notice files Src artifact builds and tests pass ran Single-Node Standalone ran Singe-Node Spark ran Single-Node Hadoop ran jupyter notebooks Regards, Berthold Reinwald IBM Almaden Research Center office: (408) 927 2208; T/L: 457 2208 e-mail: reinw...@us.ibm.com From: Arvind Surve To: dev@systemml.apache.org Date: 12/12/2017 11:39 AM Subject:Re: [VOTE] Apache SystemML 1.0.0 (RC2) +1 Arvind Surve | Spark Technology Center | https://urldefense.proofpoint.com/v2/url?u=http-3A__www.spark.tc_&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=rKpXCxhAHRimLTQtp6EVnl3INiKbhTT17qTa33Vf73k&m=PRbQ2gy3ZggwGa4d6gWRNtYuNkULDzPZFQ62hrW1LsE&s=wvb3Icsr3YHFtEpsiUN6AzOmJVFKx2g2U_pOOC0ICUQ&e= On Tuesday, December 12, 2017, 4:35:32 AM GMT+5:30, Niketan Pansare wrote: +1. Ran GPU tests as well as sanity test for Python APIs. Thanks, Niketan Pansare IBM Almaden Research Center E-mail: npansar At us.ibm.com http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar "Glenn Weidner" ---12/11/2017 09:49:48 AM---+1 I ran Linear Regression, Logistic Regression, SVM, Naive Bayes Python tests From: "Glenn Weidner" To: dev@systemml.apache.org Date: 12/11/2017 09:49 AM Subject: Re: [VOTE] Apache SystemML 1.0.0 (RC2) +1 I ran Linear Regression, Logistic Regression, SVM, Naive Bayes Python tests with Spark 2.1 in cloud environment. All tests passed with Python 2.7. Note I also attempted to repeat the tests with Python 3.5 'experimental' kernel but encountered issue: .local/lib/python3.5/site-packages/systemml/mllearn/estimators.py", line 887 def __init__(self, sparkSession, keras_model, input_shape, transferUsingDF=False, weights=None, labels=None): SyntaxError: import * only allowed at module level" I don't believe this is release blocking issue but may want to discuss if other problems reported during RC2 validation. Regards, Glenn Matthias Boehm ---12/09/2017 03:13:43 PM---+1 I ran the perftest suite with the artifact on Spark 2.2 up to 80GB without From: Matthias Boehm To: dev@systemml.apache.org Date: 12/09/2017 03:13 PM Subject: Re: [VOTE] Apache SystemML 1.0.0 (RC2) +1 I ran the perftest suite with the artifact on Spark 2.2 up to 80GB without any failures or performance issues. On earlier versions, I also ran the perftest suite with Spark 2.1 and 2.2, w/ and w/o codegen, and w/ auto compression up to 800GB without remaining issues. As a minor nitpick (to be resolved in any additional RCs or the next release), the root directory of the artifact includes a SystemML-config.xml file that only has a small subset of configuration parameters. We might want to either delete this or pull in a copy of SystemML-config.xml.template from the conf directory. Regards, Matthias On Sat, Dec 9, 2017 at 1:53 AM, Luciano Resende wrote: > On Fri, Dec 8, 2017 at 12:15 PM, Berthold Reinwald > wrote: > > > Please vote on releasing the following candidate as Apache SystemML > > version 1.0.0 > > > > The vote is open for at least 72 hours and passes if a majority of at > > least 3 +1 PMC votes are cast. > > > > [ ] +1 Release this package as Apache SystemML 1.0.0 > > [ ] -1 Do not release this package because ... > > > > To learn more about Apache SystemML, please see > > https://urldefense.proofpoint.com/v2/url?u=http-3A__systemml.apache.org_&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=we2L9Q2NOeniL5PbfdrO3CmQPNy_vnhoDpPDviJpnrU&m=Ud62VUVZNsp0I0PlFMRr-OOvxUC5pssmMZdUB4I3g5w&s=KKHMDbNiFLfVSqUYXqIZLGxx-yif5PAGAQQLnnMfXR8&e= > > > > The tag to be voted on is v1.0.0-rc2 ( > > 7b44380e49425ac20912c2d42de8603a06fa55a1): > > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_systemml_commit_7b44380e49425ac20912c2d42de860&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=we2L9Q2NOeniL5PbfdrO3CmQPNy_vnhoDpPDviJpnrU&m=Ud62VUVZNsp0I0PlFMRr-OOvxUC5pssmMZdUB4I3g5w&s=k8A_OU7p9Wa4sKxpiY5v9gzbLrJg8rGTSiSEZ-hFgTA&e= > > 3a06fa55a1 > > > > > Looks like the tag hash should be > 6519f3383f8fd43af93ecba8742ae4b2b28d4b35 > > And the link > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_systemml_tree_6519f3383f8fd43af93ecba8742ae4&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=we2L9Q2NOeniL5PbfdrO3CmQPNy_vnhoDpPDviJpnrU&m=Ud62VUVZNsp0I0PlFMRr-OOvxUC5pssmMZdUB4I3g5w&s=ntv-qbRObkRBbs_a2UlTXf-Oe1zcYp7itwRbclOi0d4&e= > b2b28d4b35 > > > > > > The release artifacts can be found at: > > https://urldefense.proofpoint.com/v2/url?u=https-3A__dist.apache.org_repos_dist_dev_systemml_1.0.0-2Drc2_&d=DwIBaQ&c=jf_iaSHvJ
Re: [VOTE] Apache SystemML 1.0.0 (RC2)
Thanks, Luciano, for providing the right URL to the v1.0.0-rc2. Given that the testing happened on the artifacts ... unless I hear differently ... we will still count the votes. Regards, Berthold Reinwald IBM Almaden Research Center office: (408) 927 2208; T/L: 457 2208 e-mail: reinw...@us.ibm.com From: Luciano Resende To: dev@systemml.apache.org Date: 12/09/2017 04:40 AM Subject:Re: [VOTE] Apache SystemML 1.0.0 (RC2) On Fri, Dec 8, 2017 at 12:15 PM, Berthold Reinwald wrote: > Please vote on releasing the following candidate as Apache SystemML > version 1.0.0 > > The vote is open for at least 72 hours and passes if a majority of at > least 3 +1 PMC votes are cast. > > [ ] +1 Release this package as Apache SystemML 1.0.0 > [ ] -1 Do not release this package because ... > > To learn more about Apache SystemML, please see > https://urldefense.proofpoint.com/v2/url?u=http-3A__systemml.apache.org_&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=rKpXCxhAHRimLTQtp6EVnl3INiKbhTT17qTa33Vf73k&m=eou0r2zjKobBw6J6T0B8JA0uV5Hg9aOuGwyPXjUjD58&s=wdOwXfhHzgIVCqYR0yO3QnIJcRDW8qdK-oCTRsxIMRA&e= > > The tag to be voted on is v1.0.0-rc2 ( > 7b44380e49425ac20912c2d42de8603a06fa55a1): > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_systemml_commit_7b44380e49425ac20912c2d42de860&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=rKpXCxhAHRimLTQtp6EVnl3INiKbhTT17qTa33Vf73k&m=eou0r2zjKobBw6J6T0B8JA0uV5Hg9aOuGwyPXjUjD58&s=0DHX6hrY0Wk1zJJFeliE2fk3OErSP0UFCsLnPXlDcGY&e= > 3a06fa55a1 > > Looks like the tag hash should be 6519f3383f8fd43af93ecba8742ae4b2b28d4b35 And the link https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_systemml_tree_6519f3383f8fd43af93ecba8742ae4b2b28d4b35&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=rKpXCxhAHRimLTQtp6EVnl3INiKbhTT17qTa33Vf73k&m=eou0r2zjKobBw6J6T0B8JA0uV5Hg9aOuGwyPXjUjD58&s=knOmScBJ-6tdd7Gm9MNYKeeMS0G52RTsUlN_yn9qSTc&e= > > The release artifacts can be found at: > https://urldefense.proofpoint.com/v2/url?u=https-3A__dist.apache.org_repos_dist_dev_systemml_1.0.0-2Drc2_&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=rKpXCxhAHRimLTQtp6EVnl3INiKbhTT17qTa33Vf73k&m=eou0r2zjKobBw6J6T0B8JA0uV5Hg9aOuGwyPXjUjD58&s=tCa8NampBIxay_0dzH1-l0d-spKLxRKc1RUssMJVlN8&e= > > > The maven release artifacts, including signatures, digests, etc. can be > found at: > https://urldefense.proofpoint.com/v2/url?u=https-3A__repository.apache.org_content_repositories_&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=rKpXCxhAHRimLTQtp6EVnl3INiKbhTT17qTa33Vf73k&m=eou0r2zjKobBw6J6T0B8JA0uV5Hg9aOuGwyPXjUjD58&s=SPiv_akoJfsZuTYp3n0dhlhvBzC4Yw_snBzpH9Gd-kU&e= > orgapachesystemml-1026/org/apache/systemml/systemml/1.0.0/ > > > > === > == Apache Release policy == > === > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.apache.org_legal_release-2Dpolicy.html&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=rKpXCxhAHRimLTQtp6EVnl3INiKbhTT17qTa33Vf73k&m=eou0r2zjKobBw6J6T0B8JA0uV5Hg9aOuGwyPXjUjD58&s=ftel26ughEALqJoIE9_CZOhYL8psZCxDa75TaLe4McM&e= > > === > == How can I help test this release? == > === > If you are a SystemML user, you can help us test this release by taking an > existing Algorithm or workload and running on this release candidate, then > reporting any regressions. > > > == What justifies a -1 vote for this release? == > > -1 votes should only occur for significant stop-ship bugs or legal related > issues (e.g. wrong license, missing header files, etc). Minor bugs or > regressions should not block this release. > > -- Luciano Resende https://urldefense.proofpoint.com/v2/url?u=http-3A__twitter.com_lresende1975&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=rKpXCxhAHRimLTQtp6EVnl3INiKbhTT17qTa33Vf73k&m=eou0r2zjKobBw6J6T0B8JA0uV5Hg9aOuGwyPXjUjD58&s=mw_eeXRt-_qvXPMvkNLe0neN2HkKvCiRSZfwZN2Fsd0&e= https://urldefense.proofpoint.com/v2/url?u=http-3A__lresende.blogspot.com_&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=rKpXCxhAHRimLTQtp6EVnl3INiKbhTT17qTa33Vf73k&m=eou0r2zjKobBw6J6T0B8JA0uV5Hg9aOuGwyPXjUjD58&s=3udCTmnn9ztd4J3wRevm5_vH-kIuQOLq4D7KT-8FHeA&e=
[VOTE] Apache SystemML 1.0.0 (RC2)
Please vote on releasing the following candidate as Apache SystemML version 1.0.0 The vote is open for at least 72 hours and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache SystemML 1.0.0 [ ] -1 Do not release this package because ... To learn more about Apache SystemML, please see http://systemml.apache.org/ The tag to be voted on is v1.0.0-rc2 ( 7b44380e49425ac20912c2d42de8603a06fa55a1): https://github.com/apache/systemml/commit/7b44380e49425ac20912c2d42de8603a06fa55a1 The release artifacts can be found at: https://dist.apache.org/repos/dist/dev/systemml/1.0.0-rc2/ The maven release artifacts, including signatures, digests, etc. can be found at: https://repository.apache.org/content/repositories/orgapachesystemml-1026/org/apache/systemml/systemml/1.0.0/ === == Apache Release policy == === http://www.apache.org/legal/release-policy.html === == How can I help test this release? == === If you are a SystemML user, you can help us test this release by taking an existing Algorithm or workload and running on this release candidate, then reporting any regressions. == What justifies a -1 vote for this release? == -1 votes should only occur for significant stop-ship bugs or legal related issues (e.g. wrong license, missing header files, etc). Minor bugs or regressions should not block this release.
[VOTE] Apache SystemML 1.0.0 (RC1)
Please vote on releasing the following candidate as Apache SystemML version 1.0.0 The vote is open for at least 72 hours and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache SystemML 0.15.0 [ ] -1 Do not release this package because ... To learn more about Apache SystemML, please see http://systemml.apache.org/ The tag to be voted on is v1.0.0-rc1 ( 054b85efdec92b8f66a125e014798076843e3492) https://github.com/apache/systemml/commit/054b85efdec92b8f66a125e014798076843e3492 The release artifacts can be found at : https://dist.apache.org/repos/dist/dev/systemml/1.0.0-rc1/ The maven release artifacts, including signatures, digests, etc. can be found at: https://repository.apache.org/content/repositories/orgapachesystemml-1025/org/apache/systemml/systemml/1.0.0/ = == Apache Incubator release policy == = Please find below the guide to release management during incubation: http://incubator.apache.org/guides/releasemanagement.html === == How can I help test this release? == === If you are a SystemML user, you can help us test this release by taking an existing Algorithm or workload and running on this release candidate, then reporting any regressions. == What justifies a -1 vote for this release? == -1 votes should only occur for significant stop-ship bugs or legal related issues (e.g. wrong license, missing header files, etc). Minor bugs or regressions should not block this release. Regards, Berthold Reinwald IBM Almaden Research Center office: (408) 927 2208; T/L: 457 2208 e-mail: reinw...@us.ibm.com
Re: dev environment on windows
Please make sure that you specify a JRE/JDK of 1.8 when you use a maven
run configuration.
If I switch the runtime JRE in my maven configuration back to jdk1.7, then
I observe the same caffe2dml compilation error. caffe2dml randomly seems
to be the 1st class being compiled to throw the error.
Regards,
Berthold Reinwald
IBM Almaden Research Center
office: (408) 927 2208; T/L: 457 2208
e-mail: reinw...@us.ibm.com
From: Dylan Hutchison
To: dev@systemml.apache.org
Date: 12/02/2017 01:19 PM
Subject:Re: dev environment on windows
I have used Intellij on windows and had no problems. Did you open a new
project from SystemML's pom.xml? If so, it should automatically configure
itself to match the pom. Then you can use the "Maven projects" tab on the
right side of Intellij to run "mvn package".
On Sat, Dec 2, 2017 at 1:13 PM, Matthias Boehm wrote:
> I'm using eclipse on win and simply defined two maven run configurations
> for
>
> (1) the default build (base=${workspace_loc:/systemml}, goals=package),
> and
> (2) the distribution build ((base=${workspace_loc:/systemml},
> goals=package, profiles=distribution).
>
> I'm sure intellij provides a similar builtin maven support, but maybe an
> intellij user can comment on that as well.
>
> Regards,
> Matthias
>
>
> On Sat, Dec 2, 2017 at 10:08 AM, Janardhan Pulivarthi <
> janardhan.pulivar...@gmail.com> wrote:
>
> > Hi all,
> >
> > 1. How to build systemml on windows, as there is a backslash problem.
> >
> > 2. When build systemml on intellij idea, it has the compilation error
> with
> > caffe2dml class.
> >
> >
> > Thanks.
> >
>
Re: [DISCUSS] Support for lower precision in SystemML
+1. Staging it to gain experience, and starting with the GPU backend is the right approach. Regards, Berthold Reinwald IBM Almaden Research Center office: (408) 927 2208; T/L: 457 2208 e-mail: reinw...@us.ibm.com From: Matthias Boehm To: dev@systemml.apache.org Date: 10/24/2017 09:07 PM Subject:Re: [DISCUSS] Support for lower precision in SystemML +1 this is really great. I like the semantics of a best-effort use of single-precision (when configured we're free to use single precision but can fall back to double precision if certain operations or backends don't support it yet). This allows us to add single precision support incrementally one component at-a-time. For the java backends, this will entail creating a good internal abstraction and well, reimplementing most of the operations. Hence, it's probably a good idea to create an umbrella jira and add the individual design docs as we go. Regards, Matthias On Tue, Oct 24, 2017 at 8:26 AM, Niketan Pansare wrote: > Hi all, > > We are in process of adding support for lower precision and wanted to give > everyone heads up. By lower precision, I mean support storing matrices in > float array (or half precision array) and performing operations using float > kernels. Initial experiments suggest that we can get up to 2x improvements > in terms of performance for Deep Learning algorithms. Also, this reduces > the memory requirements by 2x. > > Please provide any concerns or suggestions. > > The high-level plan is as follows: > 1. Support lower precision on GPU. Please see https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=rKpXCxhAHRimLTQtp6EVnl3INiKbhTT17qTa33Vf73k&m=zWUKaXp5UoL0_2Ox7fxvfE2N6H37Pcy0xssylnuRwww&s=l7YQ4c3HAYq4nDXiuQkxWXkZr1NPxLN6c50WflVo9OY&e= > systemml/pull/688 > 2. Support lower precision with native BLAS. > 3. Support lower precision on CP/Spark. This includes writing float matrix > in binary format and updating memory estimation in hops. > 4. Extend Python APIs to support lower precision. > > The first two steps requires the conversion of double array to float/half > precision array. > > Thanks > > Niketan. >
Re: [VOTE] Apache SystemML 0.15.0 (RC2)
+1 verified Jupyter notebooks under samples and scripts against Spark 2.1.0 on Mac. All fine but was not able to verifiy ALS_python_demo.ipynb as I couldn't find nf_prize_dataset.tar.gz Regards, Berthold Reinwald IBM Almaden Research Center office: (408) 927 2208; T/L: 457 2208 e-mail: reinw...@us.ibm.com From: "Glenn Weidner" To: dev@systemml.apache.org Date: 09/11/2017 09:07 PM Subject:Re: [VOTE] Apache SystemML 0.15.0 (RC2) +1 Verified Zeppelin sample notebooks against Spark 2.1 on Linux and completed artifact validation on another platform. Website updates ready pending release candidate approval. --Glenn Matthias Boehm ---09/10/2017 10:55:09 PM---+1 I ran the perftest suite up to 8GB (all categories) with and without From: Matthias Boehm To: dev@systemml.apache.org, Arvind Surve Date: 09/10/2017 10:55 PM Subject: Re: [VOTE] Apache SystemML 0.15.0 (RC2) +1 I ran the perftest suite up to 8GB (all categories) with and without codegen. With RC2, both configurations ran without any issues and codegen showed consistent improvements (without slowdown on a single algorithm). Regards, Matthias On Fri, Sep 8, 2017 at 4:57 PM, Arvind Surve wrote: > Please vote on releasing the following candidate as Apache SystemML > version 0.15.0 > > The vote is open for at least 72 hours and passes if a majority of at > least 3 +1 PMC votes are cast. > > [ ] +1 Release this package as Apache SystemML 0.15.0 > [ ] -1 Do not release this package because ... > > To learn more about Apache SystemML, please see https://urldefense.proofpoint.com/v2/url?u=http-3A__systemml.apache&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=we2L9Q2NOeniL5PbfdrO3CmQPNy_vnhoDpPDviJpnrU&m=xi4xkBrskP6gEtPIg9ddw_8x2Aw0tmIM9DOfEPRPwxI&s=s-3awGzSgaqBimBeWu6jP5zziYGSzrz_sC2XJMKAtcI&e= . > org/ > > The tag to be voted on is v0.15.0-rc2 (ccaaa18769ae045fd31068dead2565 > 844b4cee47) > > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_systemml_commit_ccaaa18769ae045fd31068dead2565&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=we2L9Q2NOeniL5PbfdrO3CmQPNy_vnhoDpPDviJpnrU&m=xi4xkBrskP6gEtPIg9ddw_8x2Aw0tmIM9DOfEPRPwxI&s=DJLxEmtymcNIf6RAcmKPPBGsm6AMzDUyRTwizkFyluU&e= > 844b4cee47 > > The release artifacts can be found at : > https://urldefense.proofpoint.com/v2/url?u=https-3A__dist.apache.org_repos_dist_dev_systemml_0.15.0-2Drc2_&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=we2L9Q2NOeniL5PbfdrO3CmQPNy_vnhoDpPDviJpnrU&m=xi4xkBrskP6gEtPIg9ddw_8x2Aw0tmIM9DOfEPRPwxI&s=eN3qCuAtc_wBcuHveqBj3KR3O7yomfb8EC9yavmlIIU&e= > > > The maven release artifacts, including signatures, digests, etc. can > be found at: > https://urldefense.proofpoint.com/v2/url?u=https-3A__repository.apache.org_content_repositories_&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=we2L9Q2NOeniL5PbfdrO3CmQPNy_vnhoDpPDviJpnrU&m=xi4xkBrskP6gEtPIg9ddw_8x2Aw0tmIM9DOfEPRPwxI&s=E-wWFSMFg8tDqQWjBcMdOi5HaUjjCI-Am1BRdOrJ_CA&e= > orgapachesystemml-1024/org/apache/systemml/systemml/0.15.0/ > > = > == Apache Incubator release policy == > = > Please find below the guide to release management during incubation: > https://urldefense.proofpoint.com/v2/url?u=http-3A__incubator.apache.org_guides_releasemanagement.html&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=we2L9Q2NOeniL5PbfdrO3CmQPNy_vnhoDpPDviJpnrU&m=xi4xkBrskP6gEtPIg9ddw_8x2Aw0tmIM9DOfEPRPwxI&s=ORBtt51elPd2MjUYnv_12kUlx0k4UnsVOD6grC_IS2M&e= > > === > == How can I help test this release? == > === > If you are a SystemML user, you can help us test this release by taking > an existing Algorithm or workload and running on this release candidate, > then > reporting any regressions. > > > == What justifies a -1 vote for this release? == > > -1 votes should only occur for significant stop-ship bugs or legal > related issues (e.g. wrong license, missing header files, etc). Minor bugs > or regressions should not block this release. > -Arvind > > Arvind Surve | Spark Technology Center | https://urldefense.proofpoint.com/v2/url?u=http-3A__www.spark.tc_&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=we2L9Q2NOeniL5PbfdrO3CmQPNy_vnhoDpPDviJpnrU&m=xi4xkBrskP6gEtPIg9ddw_8x2Aw0tmIM9DOfEPRPwxI&s=OWZkjhiXCT5Ghi5AQ7bGlKFcISF53DcUEibALJhd9g8&e=
Re: List of SystemML research papers?
... here we go. Regards, Berthold Reinwald IBM Almaden Research Center office: (408) 927 2208; T/L: 457 2208 e-mail: reinw...@us.ibm.com From: Deron Eriksson To: dev@systemml.apache.org Date: 06/01/2017 11:27 AM Subject:List of SystemML research papers? Currently the SystemML website does not have a research page, but we should probably add one, similar to Apache Spark's research page at http://spark.apache.org/research.html. Could someone respond with a list of SystemML research papers that could be added to a research page? Thanks, Deron -- Deron Eriksson Spark Technology Center http://www.spark.tc/ Research Papers.pptx Description: MS-Powerpoint 2007 presentation
