from:"Matthias Boehm"

Re: Performance Question

2020-06-19 Thread Matthias Boehm

Thanks for the question and the detailed inputs - this is an effect of 
simplification rewrites that only apply in one of the cases. 
Specifically, (t(N)%*%t(M))[1,1] is rewritten to t(N)[1,] %*% t(M)[,1], 
which is a form of selection pushdown. You can do the following, for 
benchmarking*: introduce an explicit program block cut to separate the 
indexing operations - as most simplification rewrites (including this 
one) are only applied at DAG level, you will run the full matrix 
multiplications.


res = t(N)%*%t(M)
while(FALSE){}
print(as.scalar(res[1,1]))

Note that there is also further simplification potential we don't 
exploit yet: the indexing could be pushed into the transpose (in both 
cases) as well. For benchmarking MM, it's probably also a good idea to 
compare that with our native BLAS integration (e.g., OpenBLAS or MKL), 
which you can enable in our configuration file and by passing the 
library path to the JVM.


* we don't remove while loops to allow these kinds of explicit cuts and 
because there is not a lot of potential in real scripts.


Regards,
Matthias

On 6/19/2020 6:00 AM, Rana Alotaibi wrote:

Hi SystemML developers,

I have been experimenting with SystemML lately. I found some interesting
numbers when evaluating these two expressions: *t(M%*%N)* and *t(N)%*% t(M)*
(The size of matrix M is 20Kx100 and N is 100x20K). The pipeline *t(N)%*%
t(M) *drastically outperforms* t(M%*%N)*.

- *t(M%*%N)*
- The multiplication took ~2.53 secs
   - The transpose took ~ 1.41 (The size of the intermediate *M%*%N *is
   20Kx20K)
- *t(N)%*% t(M) *
- The multiplication took ~0.002 secs (given that the size of t(N) is
   20Kx100 and t(M) is 100x20K )
   - The total transpose time for both matrices is 0.023 secs

So, I'm interested to know why the matrix multiplication in this
pipeline *t(N)%*%
t(M)* is faster than *t(M%*%N)*? and in general why *t(N)%*% t(M)* is
faster than *t(M%*%N)*?

*Experiments Configurations**: *

- I'm using  systemml-1.2.0 (Binary tgz) and working on a single node.
- CPU E5-2640 v4 @ 2.40GHz, 40Cores
- RAM : 123GB

*t(N)%*%t(M) DML Script*
 | time = externalFunction(Integer i) return (Double B)
 |   implemented in
(classname="org.apache.sysml.udf.lib.TimeWrapper", exectype="mem");
 |  times = matrix(0.0, rows = 5, cols = 1)
 |  M = rand(rows = 2, cols = 100, pdf = 'uniform')
 |  N = rand(rows = 100, cols = 2, pdf = 'uniform')
 | for (ix in 1:5) {
 |   start = time(1)
 |   print("StartTime: "+start)
 |   res = t(N)%*%t(M)
 |   print(as.scalar(res[1,1]))
 |   stop = time(1)
 |   print("EndTime:- "+stop)
 |   pipTime= (stop - start) / 1000
 |   print ("Time:"+ pipTime)
 |   times[ix,1] = pipTime
 | }
 | times = t(times)

 *t(M%*%N) DML Script*
 | time = externalFunction(Integer i) return (Double B)
 |   implemented in
(classname="org.apache.sysml.udf.lib.TimeWrapper", exectype="mem");
 |  times = matrix(0.0, rows = 5, cols = 1)
 |  M = rand(rows = 2, cols = 100, pdf = 'uniform')
 |  N = rand(rows = 100, cols = 2, pdf = 'uniform')
 | for (ix in 1:5) {
 |   start = time(1)
 |   print("StartTime: "+start)
 |   res = t(M%*%N)
 |   print(as.scalar(res[1,1]))
 |   end = time(1)
 |   print("EndTime: "+end)
 |   pipTime = (end - start)/1000
 |   print ("Time:"+ pipTime)
 |   times[ix,1] = pipTime
 | }
 | times = t(times)

*t(N)%*%t(M) RUN TIME EXPLAIN *
# Memory Budget local/remote = 63716MB/140MB/140MB
# Degree of Parallelism (vcores) local/remote = 40/1/1
PROGRAM ( size CP/MR = 58/0 )
--FUNCTIONS
FUNCTION CALL GRAPH
--MAIN PROGRAM
.defaultNS::time
EXTERNAL FUNCTION .defaultNS::time
--MAIN PROGRAM
GENERIC (lines 4-6) [recompile=false]
--CP createvar _mVar0 scratch_space//_p32870_192.168.250.59//_t0/temp0
true MATRIX binaryblock 5 1 1000 1000 0 copy
--CP rand 5.SCALAR.INT.true 1.SCALAR.INT.true 1000 1000 0.0 0.0 1.0 -1
uniform 1.0 40 _mVar0.MATRIX.DOUBLE
--CP createvar _mVar1 scratch_space//_p32870_192.168.250.59//_t0/temp1
true MATRIX binaryblock 2 100 1000 1000 200 copy
--CP rand 2.SCALAR.INT.true 100.SCALAR.INT.true 1000 1000 0.0 1.0
1.0 -1 uniform 1.0 40 _mVar1.MATRIX.DOUBLE
--CP createvar _mVar2 scratch_space//_p32870_192.168.250.59//_t0/temp2
true MATRIX binaryblock 100 2 1000 1000 200 copy
--CP rand 100.SCALAR.INT.true 2.SCALAR.INT.true 1000 1000 0.0 1.0
1.0 -1 uniform 1.0 40 _mVar2.MATRIX.DOUBLE
--CP cpvar _mVar0 times
--CP cpvar _mVar1 M
--CP cpvar _mVar2 N
--CP rmvar _mVar0 _mVar1 _mVar2
FOR (lines 0-0) [in-place=[times]]
--CP assignvar 1.SCALAR.INT.true __pred.SCALAR.INT
--CP assignvar 5.SCALAR.INT.true __pred.SCALAR.INT
--CP assignvar 1.SCALAR.INT.true __pred.SCALAR.INT
--GENERIC (lines 8-8) [recompile=false]
CP extfunct .defaultNS time 1 1

Re: [DISCUSSION] Website stack `systemml.apache.org`. Thanks.

2020-06-18 Thread Matthias Boehm

don't worry about - it's all fine. Leaving it like that (for a single 
commit) is MUCH better than trying to forcefully rewrite the history.


Regards,
Matthias

On 6/18/2020 5:35 PM, Baunsgaard, Sebastian wrote:

Hi Dev,


I merged in the documentation PR, this was an slight oversight, while trying to 
merge something else. That said nothing bad happened except you did not get a 
chance to look through the PR, and it is progress in the direction discussed in 
this thread.


Best regards

Sebastian



From: Matthias Boehm 
Sent: Saturday, June 6, 2020 7:03:19 PM
To: dev@systemml.apache.org
Subject: Re: [DISCUSSION] Website stack `systemml.apache.org`. Thanks.

from my perspective it would be very important to have all builtin
functions in a single markdown file to allow users to search for things
and users don't care how a builtin function is internally implemented.
So we might want to use this opportunity to consolidate the already
documented builtin functions into the new list and add everything that
is not yet documented as I'm sure many functions are missing.

Btw, in the INFRA ticket for renaming our repositories [1], I also asked
to disable sending all these github comments and PR interactions to our
dev list as I'm sure people get overwhelmed by these auto-generated
emails which leads to important messages being missed. Related to
documentation, we might want to group categories of builtin functions
into larger PRs instead of having a PR per builtin function as its
primarily a copy of existing documentation so they can be reviewed more
efficiently together to avoid unnecessary overhead.

[1] https://issues.apache.org/jira/browse/INFRA-20362

Regards,
Matthias

On 6/6/2020 5:23 PM, Janardhan wrote:

Hi Arnab,

We have all the Java-based builtin functions documented at [1].
Most of these functions are already documented!

[1]
http://apache.github.io/systemml/dml-language-reference.html#built-in-functions


Thank you,
Janardhan

On Fri, Jun 5, 2020 at 4:37 PM arnab phani  wrote:


That's a good suggestion. One thing I want to point out that Builtins.java
[1] contains all the dml-bodied and java-bodied built-ins. For the dml
built-ins, it is easier to parse the corresponding dml script (if
documentation is available there), but for other built-ins, I am not aware
of any particular source today where they are documented -- a few of them
are documented as a part of commit message but not all.
Though it is certainly possible to slice the dml builtins from [1] out, but
it will be awesome to generate docs for all of those.

[1]

https://github.com/apache/systemml/blob/master/src/main/java/org/apache/sysds/common/Builtins.java

Regards,
Arnab..

On Thu, Jun 4, 2020 at 12:44 PM Baunsgaard, Sebastian <
baunsga...@tugraz.at>
wrote:


Hi dev team!


Update / suggestion.


First of all, i think it is great that we have people working on our
documentation. It is really important. But I can't help to note that the
improvements, are going in an unmaintainable direction where we have to
maintain the documentation at multiple locations.


For instance our new toOneHot function already had some basic
documentation inside its build-in script already [1], it would be much
nicer if we modified that and then had some automation to generate the

docs

made in commit [2]. Now we have to maintain two locations, resulting in
high risk of either being outdated.


We need to ask ourselves where do we want to make the edit of
documentation. I would argue that it should be located as close to the
internal definition of a function, because then when making or modifying
that function the documentation is also reviewed.


To achieve this i suggest that we instead relocate the documentation
source and generation to an appropriately named file inside [3], that
simply loops through all function definitions from [4], and generate
markdown files for our /docs/ folder.


Shifting to such an implementation enables us to systematically cover all
functions defined in the system. It also is a full list of the systems
current build-in functions thereby giving us the exact file paths to
extract the source documentation from the comments and generate the docs
for those files.

In the future it could be extended to verify and detecting which input

and

output types each function definition has, such that we have even less
manual documentation needs.


Best regards

Sebastian


[1]


https://github.com/apache/systemml/blob/master/scripts/builtin/toOneHot.dml

[2]


https://github.com/apache/systemml/commit/859ad3a72906c67b7300c9980251da4cde9ed8f8

[3]


https://github.com/apache/systemml/tree/master/src/main/java/org/apache/sysds/common

[4]


https://github.com/apache/systemml/blob/master/src/main/java/org/apache/sysds/common/Builtins.java



From: arnab phani 
Sent: Sunday, May 31, 2020 3:15:44 PM
To: dev@systemml.apache.org
Subject: Re: [DISCUSSION] Website

Re: [DISCUSSION] Website stack `systemml.apache.org`. Thanks.

2020-06-06 Thread Matthias Boehm

from my perspective it would be very important to have all builtin 
functions in a single markdown file to allow users to search for things 
and users don't care how a builtin function is internally implemented. 
So we might want to use this opportunity to consolidate the already 
documented builtin functions into the new list and add everything that 
is not yet documented as I'm sure many functions are missing.


Btw, in the INFRA ticket for renaming our repositories [1], I also asked 
to disable sending all these github comments and PR interactions to our 
dev list as I'm sure people get overwhelmed by these auto-generated 
emails which leads to important messages being missed. Related to 
documentation, we might want to group categories of builtin functions 
into larger PRs instead of having a PR per builtin function as its 
primarily a copy of existing documentation so they can be reviewed more 
efficiently together to avoid unnecessary overhead.


[1] https://issues.apache.org/jira/browse/INFRA-20362

Regards,
Matthias

On 6/6/2020 5:23 PM, Janardhan wrote:

Hi Arnab,

We have all the Java-based builtin functions documented at [1].
Most of these functions are already documented!

[1]
http://apache.github.io/systemml/dml-language-reference.html#built-in-functions


Thank you,
Janardhan

On Fri, Jun 5, 2020 at 4:37 PM arnab phani  wrote:


That's a good suggestion. One thing I want to point out that Builtins.java
[1] contains all the dml-bodied and java-bodied built-ins. For the dml
built-ins, it is easier to parse the corresponding dml script (if
documentation is available there), but for other built-ins, I am not aware
of any particular source today where they are documented -- a few of them
are documented as a part of commit message but not all.
Though it is certainly possible to slice the dml builtins from [1] out, but
it will be awesome to generate docs for all of those.

[1]

https://github.com/apache/systemml/blob/master/src/main/java/org/apache/sysds/common/Builtins.java

Regards,
Arnab..

On Thu, Jun 4, 2020 at 12:44 PM Baunsgaard, Sebastian <
baunsga...@tugraz.at>
wrote:


Hi dev team!


Update / suggestion.


First of all, i think it is great that we have people working on our
documentation. It is really important. But I can't help to note that the
improvements, are going in an unmaintainable direction where we have to
maintain the documentation at multiple locations.


For instance our new toOneHot function already had some basic
documentation inside its build-in script already [1], it would be much
nicer if we modified that and then had some automation to generate the

docs

made in commit [2]. Now we have to maintain two locations, resulting in
high risk of either being outdated.


We need to ask ourselves where do we want to make the edit of
documentation. I would argue that it should be located as close to the
internal definition of a function, because then when making or modifying
that function the documentation is also reviewed.


To achieve this i suggest that we instead relocate the documentation
source and generation to an appropriately named file inside [3], that
simply loops through all function definitions from [4], and generate
markdown files for our /docs/ folder.


Shifting to such an implementation enables us to systematically cover all
functions defined in the system. It also is a full list of the systems
current build-in functions thereby giving us the exact file paths to
extract the source documentation from the comments and generate the docs
for those files.

In the future it could be extended to verify and detecting which input

and

output types each function definition has, such that we have even less
manual documentation needs.


Best regards

Sebastian


[1]


https://github.com/apache/systemml/blob/master/scripts/builtin/toOneHot.dml


[2]


https://github.com/apache/systemml/commit/859ad3a72906c67b7300c9980251da4cde9ed8f8


[3]


https://github.com/apache/systemml/tree/master/src/main/java/org/apache/sysds/common


[4]


https://github.com/apache/systemml/blob/master/src/main/java/org/apache/sysds/common/Builtins.java




From: arnab phani 
Sent: Sunday, May 31, 2020 3:15:44 PM
To: dev@systemml.apache.org
Subject: Re: [DISCUSSION] Website stack `systemml.apache.org`. Thanks.

Hi,

Another advantage of markdown syntax is that it gives more freedom to

add a

description as needed. Java methods and builtins might not fall in the

same

bucket and different builtins might need different ways to describe it.
For example, glm needs more details than other built-ins. Too much
standardization can be a bad idea.

Regards,
Arnab.

On Sun, May 31, 2020 at 9:55 AM Baunsgaard, Sebastian <
baunsga...@tugraz.at>
wrote:


Hi Janardhan,


That would be one option, lets call it option 1, I have another

suggestion:



   1.  Markdown syntax with out commented individual lines.
   2.  Jdoc like syntax with annotations

Pros and cons

   *   1 Pros

Re: Fwd: Help me improve my documentation search :)

2020-06-01 Thread Matthias Boehm

Janardhan, thanks for the initiative but please refrain from sending 
such advertisements to our dev mailing list and making any promises in 
the name of SystemDS/SystemML on potential inclusion of such 
dependencies into future releases.


Right now we did not yet decide what our target documentation scheme and 
tools will be. Personally, I would prefer not creating a dependency to 
external tools if possible but stick to plain markdown files - all up 
for discussion right now.


Regards,
Matthias

On 6/1/2020 7:40 PM, Janardhan wrote:

-- Forwarded message -
From: DocSearch Support 
Date: Mon, 1 Jun, 2020, 21:55
Subject: Re: Help me improve my documentation search :)
To: 


Hi there,

The DocSearch team has a news to share with you: We are on the process to
release a new version of the search User Interface.

We are doing a pre-release on Docusaurus to iterate faster and improving
the current sate of this UI. We will then release a vanilla version so that
anyone can integrate it. We will share a blog post and a migration guide to
got you covered.

You're invited to our next online event for developers: the Algolia
Community Party! The topic will be around Developer Experience,
Documentation and Open-Source. Join us on Thursday, June 4th at 8am PST /
5pm CEST (Europe)

.

We will present you the current state of DocSearch and show you the ins and
outs of this new release. In addition to this talk, we're proud to bring a
speaker from Facebook, live from Seattle. The talks will show the technical
behind-the-scenes of documentation, like exploring built-in themes with
plugins, and even creating one during the event!

Schedule:
- DocSearch: Enhancing the developer experience of open-source
documentation - Sylvain Pace, Full-Stack Software Engineer at Algolia
- Documentation made easy with Docusaurus - Dmitry Vinnik, Open-source
Developer Advocate at Facebook.

Open time for Q RSVP HERE


So grab a cup of tea, of coffee or a glass of lemonade, get comfortable and
join us at our next Algolia Community Party online!
Cheers,

The DocSearch team



How would you rate my reply?
Great

Okay

Not Good



Sylvain Pace
Software Engineer
  |  www.algolia.com |  @algolia
 [image: algolia]

{#HS:791339348-126209#}
On Wed, Mar 27, 2019 at 0:07:24 CET, DocSearch Support <
documentationsea...@algolia.com> wrote:


Hi Janardhan,

Thank you for the great news,

Let us know if you need anything,

Have a great day,

Cheers



Sylvain Pace
Software Engineer
  |  www.algolia.com |  @algolia
 [image: algolia]

On Mon, Mar 25, 2019 at 16:52:09 CET,  wrote:


Hi Sylvian Pace,

Thanks a lot for the configure. We have integrated the search utility in
our snapshot version and will be published with our next release.

Link here: http://apache.github.io/systemml/

Janardhan
Bridge Design Engineer

On Mon, Mar 25, 2019 at 14:55:18 CET, DocSearch Support <
documentationsea...@algolia.com> wrote:


Hi there,

Following up on my last email about DocSearch. I was wondering if you
had a chance to look at it in detail and integrate the code snippet on your
documentation pages? Is there anything I could help you with?

Looking forward to hearing back from you soon,



Sylvain Pace
Software Engineer
  |  www.algolia.com |  @algolia
 [image: algolia]

On Mon, Mar 4, 2019 at 16:02:57 CET, DocSearch Support <
documentationsea...@algolia.com> wrote:


Hi there,

Congratulations, your search is now ready!
I've successfully configured the underlying crawler and it will now run
every 24h.

You're now a few steps away from having it working on your website:

- Copy the following CSS/JS snippets and add them to your page


https://cdn.jsdelivr.net/npm/docsearch.js@2/dist/cdn/docsearch.min.css;
/>


https://cdn.jsdelivr.net/npm/docsearch.js@2/dist/cdn/docsearch.min.js";>

 docsearch({
apiKey: '78c19564c220d4642a41197baae304ef',
indexName: 'apache_systemml',
inputSelector: '### REPLACE ME ',
debug: false // Set debug to true if you want to inspect the dropdown
});


- Add a search input in your page if you don't have any yet. Then
update the inputSelector value in JS snippet to a CSS selector that targets
your search input field.

- Optionally customize the look and feel by following the DocSearch
documentation

Re: [DISCUSSION] Documentation dev & user along with builtins.

2020-05-23 Thread Matthias Boehm

thanks for the initiative and moving this discussion to the dev list. I 
think this would be very valuable. Regarding how to start, I would 
recommend to focus on external functionality first by documenting all 
dml-bodied and native builtin functions (see 
org.apache.sysds.common.Builtins) and operators.


We started with Kevin Innerebner and others on this a while ago: 
https://github.com/apache/systemml/blob/master/docs/builtins-reference.md, 
but it's far from complete. Also, Sebastian Baunsgaard has a couple of 
ideas on the topic.


Regards,
Matthias

On 5/23/2020 3:01 PM, Janardhan wrote:

Hi all,

We have few people with us, willing to work on documentation (no code!)

Ideas on the parts of the codebase lacking/improvisation of documentation.
(1) what and how we want to document (user-facing builtins, internal dev,
etc),
(2) which tools[1] we should consistently use.

[1]
https://mail-archives.apache.org/mod_mbox/systemml-dev/202005.mbox/%3CCAObMueX1LA7D_Lwxd-wkXiFWKt6Pem4p3H9ZmjKj%3DGSvzwghrw%40mail.gmail.com%3E

Thank you,
Janardhan

Re: Docker Container Organisation

2020-05-17 Thread Matthias Boehm

thanks for following-up on this. @Berthold: do I remember correctly that 
you looked into a similar setup a while ago?


Regards,
Matthias

On 5/15/2020 7:46 PM, Baunsgaard, Sebastian wrote:

Hi SystemDS developers.


Currently we have some docker containers that are associated with my  personal 
docker user.

It has been brought to my attention that this is not appropriate, and I would 
therefore suggest we make a Docker organization.

But this rises some questions:


- Is there a normal procedure for Apache Projects for docker containers?

- If not, then furthermore what should we call this docker organization?

- and how do we/I distribute rights to maintain this docker organization to the 
committers?


For notice of how important the docker container is, it has currently been 
downloaded and installed over 10k times for testing our system, and for each 
instance it has saved 1 min of testing time on GitHub Workflows (1 week total 
time so far).


Furthermore we need to find a solution hopefully soon so that we can set 
automatic updates of our docker image up, since when updating dependencies at 
the moment, require me to manually update the docker image. This is for 
instance required for a current PR (https://github.com/apache/systemml/pull/914)


Searching a little on for Apache docker images of different projects it looks like 
other persons have repositories that contain images for specific systems. Like 
Microsoft having a Spark image 
(https://hub.docker.com/_/microsoft-mmlspark-release).


best regards

Sebastian

Re: [#904] Can the ONNX-SystemDS implementation be reviewed. Thanks.

2020-05-12 Thread Matthias Boehm

yes this is the classic blocking issue - we only held back because you 
commented you want to review it in detail. We'll take care of it now.


Regards,
Matthias

On 5/12/2020 8:05 AM, Janardhan wrote:

Hi,

@lukas-jkl have implemented ONNX support for SystemDS, also well
documented (in code and in readme)

Any expert would like to provide comments/review for this PR.
PR Link - https://github.com/apache/systemml/pull/904

Thank you,
Janardhan

Welcome 4 New Committers

2020-05-01 Thread Matthias Boehm

The Project Management Committee (PMC) for Apache SystemML (SystemDS) 
has invited Arnab Phani, Mark Dokter, Shafaq Siddiqi, and Kevin 
Innerebner to become committers, and we are pleased to announce that all 
four have accepted.


The new committers cover many important areas of current and future 
development such as (1) lineage tracing and reuse, (2) GPU backend and 
code generation, (3) data cleaning pipelines and built-in functions, as 
well as (4) data tensors and federated data preparation and learning.


Congratulations and welcome Arnab, Mark, Shafaq, and Kevin - this is 
well deserved.


Regards,
Matthias

Re: Roadmap Merge and Rename SystemDS

2020-04-10 Thread Matthias Boehm

yes, all that will be covered, but there official processes to follow:

ad 1) there yesterday, the podling for the suitable name search has been
approved with additional comments to the PMC.

https://issues.apache.org/jira/projects/PODLINGNAMESEARCH/issues/PODLINGNAMESEARCH-179?filter=allissues

ad 2) we follow the official new committer process, and there are still
additional steps to do

http://community.apache.org/newcommitter.html

ad 3) It's still not decided yet, if we go directly to 2.0 in order to
avoid tag conflicts or 0.3. Feel free to express your opinion here too.

Regards,
Matthias

On 4/10/2020 4:39 AM, Janardhan wrote:

Hi Matthias,

Would you be so kind as to announce the following:
1. Apache Infra jira ticket for name change
2. new committers (welcome!) and of course contributors.
3. New release version number (is it SYSTEMDS-0.3.0-SNAPSHOT)

Thank you,
Janardhan

On Tue, Mar 24, 2020 at 6:28 PM Matthias Boehm wrote:

that's a good point Henry. Yes, with SystemDS 0.1.0, we removed the
MapReduce compiler and runtime backend, the pydml parser and language
support, the Java-UDF framework, and the script-level debugger. We are
concentrating on local, spark, GPU, and federated backends now, added
new language bindings including an initial Python binding. However, the
script-level operation support remains intact and is even largely
extended by builtins for algorithms, data cleaning, and debugging.

Accordingly, it might be good to deprecate the removed things while
merging the code in and then make the next Apache SystemDS (pending
approval) release a major release which allows us to break external APIs.

Regards,
Matthias

On 3/24/2020 2:07 AM, Henry Saputra wrote:

Thanks for starting this discussions, Matthias.

Are there any features from SystemML that could be be removed or

deprecated

when SystemDS being merged to SystemML repository?

- Henry

On Sat, Mar 21, 2020 at 2:47 PM Matthias Boehm

wrote:

just FYI, we created a ticket for the suitable name search, and shared
the related results [1]. So from my perspective, it really boils down to
the question if we accept the closeness to 'Linux systemd'. Back in 2018
(when starting SystemDS), I came to the conclusion that it's fine
because of the very different objectives and because SystemDS reflects
both the origin from SystemML and its new focus on data science

pipelines.

[1]

https://issues.apache.org/jira/projects/PODLINGNAMESEARCH/issues/PODLINGNAMESEARCH-179?filter=allissues

Regards,
Matthias

On 3/9/2020 6:37 PM, Matthias Boehm wrote:

Hi all,

as you're probably aware, development activities of Apache SystemML
significantly slowed down and were virtually non-existing in the last
year for various reasons. Part of that was that my team and I [1]
decided to start SystemDS [2,3] as a fork of SystemML in 09/2018 with a
new vision and roadmap for the future.

During PMC discussions regarding the retirement of SystemML, we came to
the conclusions that the best path forward -- for the entire community
-- would be to merge SystemDS back into Apache SystemML, rename it to
SystemDS, and continue jointly. Before doing so, I want to share the
plan with the entire community.

themselves

based on machine learning. Our main objectives are to support effective
and efficient data preparation, ML training and debugging at scale,
something that cannot be composed from existing libraries. The game

plan

includes three major parts:

1) DSL-based, High-level Abstractions: We aim to provide a hierarchy of
abstractions for the different lifecycle tasks as well as users with
different expertise (ML researchers, data scientists, domain experts),
based on our DSL for ML training and scoring. Exploratory data science
interleaves data preparation, ML training, scoring, and debugging in an
iterative process; and once these tasks are expressed in dense or

sparse

linear algebra, we expect very good performance.

2) Hybrid Runtime Plans and Optimizing Compiler: To support the wide
variety of algorithm classes, we will continue to provide different
parallelization strategies, enriched by a new backend for federated ML
and privacy enhancing technologies. Since the hierarchy of language
abstractions inevitably leads to redundancy, we further aim to improve
the automatic optimization capabilities of the compiler and underlying
runtime.

Re: Roadmap Merge and Rename SystemDS

2020-03-24 Thread Matthias Boehm

Regards,
Matthias

On 3/24/2020 2:07 AM, Henry Saputra wrote:

Thanks for starting this discussions, Matthias.

Are there any features from SystemML that could be be removed or deprecated
when SystemDS being merged to SystemML repository?

- Henry

On Sat, Mar 21, 2020 at 2:47 PM Matthias Boehm wrote:

[1]

https://issues.apache.org/jira/projects/PODLINGNAMESEARCH/issues/PODLINGNAMESEARCH-179?filter=allissues

Regards,
Matthias

On 3/9/2020 6:37 PM, Matthias Boehm wrote:

Hi all,

SystemDS aims at providing better systems support for the end-to-end
data science lifecycle, with a special focus on ML pipelines from data
integration, cleaning, and preparation, over efficient ML model
training, to model debugging and serving. A key observation is that
state-of-the-art data integration and cleaning primitives are themselves
based on machine learning. Our main objectives are to support effective
and efficient data preparation, ML training and debugging at scale,
something that cannot be composed from existing libraries. The game plan
includes three major parts:

3) Data Model - Heterogeneous Tensors: To support data integration and
cleaning primitives in linear algebra programs requires a more generic
data model for handling heterogeneous and structured data. In contrast
to existing ML systems, our central data model are heterogeneous
tensors. Thus, we generalize SystemML's FP64 matrices to
multi-dimensional arrays where one dimension may have a schema including
JSON strings to represent nested data.

Admin: We intend to create the SystemDS 0.2 release in March. Afterwards
we would then rebase all our commits (369) back onto the SystemML
codeline. Subsequently, we will rename Apache SystemML to Apache
SystemDS and continue our development under Apache umbrella. I just went
through the Apache name search guidelines and we'll perform a 'suitable
name search' accordingly and then transfer SystemDS. The existing PMC
and committer status stays of course intact unless people want to leave.
Shortly after the merge, I will nominate the four most active
contributors of the last year to become committers. Regarding releases
(and JIRA numbers), it's up for discussion but both, continuing with
SystemML versions (i.e., 1.3) or SystemDS versions (0.3) seem fine to me.

Roadmap: At technical level

Re: Roadmap Merge and Rename SystemDS

2020-03-21 Thread Matthias Boehm

[1]
https://issues.apache.org/jira/projects/PODLINGNAMESEARCH/issues/PODLINGNAMESEARCH-179?filter=allissues

Regards,
Matthias

On 3/9/2020 6:37 PM, Matthias Boehm wrote:

Hi all,

Roadmap: At technical level, SystemDS will continue to support all
operations and algorithms SystemML provided but significantly extent the
scope and functionality via the mentioned hierarchy of language
abstractions (in form of builtin functions). However, during the fork we
already removed old baggage like the MR backend, the scrip-level
debugger, the PyDML frontend and several other things [4]. Major new
internals are native support for lineage tracing and reuse, the data
model of heterogeneous tensors, and a new federated backend.

[1] https://damslab.github.io/
[2] https://github.com/tugraz-isds/systemds
[3] http://cidrdb.org/cidr2020/papers/p22-boehm-cidr20.pdf
[4] https://github.com/tugraz-isds/systemds/releases/tag/v0.1.0

Regards,
Matthias

Re: DML scripts under scripts/staging

2019-10-31 Thread Matthias Boehm


Hi Remy,

generally these scripts are not much different from the ones found in 
scripts/algorithms. However, the staging scripts were either in 
development, or did not receive enough testing to move to algorithms yet.


If you have a specific algorithm in mind, let us know and we help 
determine if it's useful to compare against it.


Regards,
Matthias

On 10/31/2019 3:43 AM, Remy Wang wrote:

Hello,

I'm running some performance tests using the DML scripts, and found the
systemml/scripts/staging directory. What does staging mean here, and how
does the scripts here differ from other dml scripts under systemml/scripts
?

Thanks,
Remy Wang

Re: SYSTEMML PyPi Statistics over the last six months.

2019-08-10 Thread Matthias Boehm

great - thank you so much for the summary. It would be awesome to get it 
for a longer history as well and aggregate it with the release/maven 
downloads.


Regards,
Matthias

On 10/08/2019 16:23, Janardhan wrote:

Hi,

The following are the queried SystemML download statistics to understand
the present user base.
Retrived from `the-psf.pypi.downloads`, processed with BiqQuery.

Month num_downloads
Aug-19 1205
Jul-19 3512
Jun-19 3582
May-19 1586
Apr-19 1076
Mar-19 1029
Feb-19 792
[image: image.png]

Thank you,
Janardhan

Re: upgrading hadoop version

2019-02-20 Thread Matthias Boehm

Raising the minimum version to 2.7.x is a good idea and fine by me. I 
would suggest updating the pom to version 2.7.7 but only mention 2.7 as 
a minimum requirement in the README because as far as I know no APIs 
changed that wouldn't allow running SystemML on older versions as well.


Regards,
Matthias

On 20/02/2019 01:54, Berthold Reinwald wrote:

Folks,

we still have a dependency to Hadoop 2.6.0 which has several
vulnerabilities. We checked Hadoop 2.7.7 where many vulnerabilities have
been fixed.

Does anyone object upgrading to Hadoop version 2.7.7?

Regards,
Berthold Reinwald

Re: Autoencoder codegen testing with R

2019-01-15 Thread Matthias Boehm

yes, you're absolutely right - in this form, results would always 
differ. Even if we feed a seed to both R and DML scripts, the 
implementation of our rand is very different as we need to ensure that, 
given a seed, we generate the same data in local and distributed 
operations. Accordingly, we derive seeds for each 1K x 1K block from the 
initial seed.


For testing purposes, we usually use one of the following two 
approaches: (1) use constant matrices or sequences (matrix/seq), or (2) 
generate the random data outside and read it into both scripts. 
Accordingly, I would recommend to create a copy of the dml script, place 
it inside the test directory, and modify the script accordingly.


Regards,
Matthias

On 15/01/2019 05:40, Janardhan wrote:

Hi,

I have replicated AutoEncoder-2layer script in R. But, can the output files
(W1, b1, ...) will match the R script. (not matching in my implementation).

Because, we have used Rand functions without seed in dml script also R
doesn't provide any Rand(.., seed=..) function. I have just used

*IN DML : Rand(m, n, min=-1, max=1)*
*IN R: runif(n, min=-1, max=1) %*% matrix(1, m, 1) *

Should the results from Autoencoder-2layer.dml will match with that of R
script.

Thanks,
Janardhan

New committer: Guobao Li

2018-09-04 Thread Matthias Boehm

The Project Management Committee (PMC) for Apache SystemML has invited
Guobao Li to become a committer and we are pleased to announce that he
has accepted.

Guobao was instrumental in creating language and runtime support for
parameter servers in SystemML. This includes local and distributed
runtime backends for data-parallel parameter servers as well as
various data partitioning schemes, update modes, and update
frequencies. This feature has already shipped in SystemML 1.2.

Congratulations and welcome Guobao - this is well deserved.

Regards,
Matthias

Fwd: [ComDev] High resolution project logos wanted!

2018-08-23 Thread Matthias Boehm

Could someone with access to our logos please commit them into the
mentioned repo? @Deron: I remember you gave me once the archive with
all versions of our logos. Thanks.

Regards,
Matthias


-- Forwarded message --
From: Daniel Gruno 
Date: Thu, Aug 23, 2018 at 1:52 PM
Subject: [ComDev] High resolution project logos wanted!
To: d...@community.apache.org


Hi awesome PMCs!

This is a request for your project logo(s) in the best possible format
you have. I'm on a crazy mission to gather up all project logos in the
foundation and put them somewhere central and browseable.

To that end, I need your help to speed up things!

If you have or know of a high resolution or scalable version of your
project's logo (or logos!), if you could please do the following, that
would be really helpful:

- Either commit your logo to
https://svn.apache.org/repos/asf/comdev/project-logos/originals/
(everyone should be able to write here)
- Or send it to me at humbed...@apache.org


Some guidelines:

- Logos should be either scalable (SVG, EPS, PDF etc) or high
resolution images, as good as you can get 'em, preferably PNG or high
quality JPEGs/TIFFs.
- Please name the file $project.$ext, e.g. cassandra.svg, httpd.png or
netbeans.eps (use the subdomain/ldap name of your project). If you
have multiple logos or multiple versions of your one logo, you can
append '-1', '-2' etc to the name, for example tomee-1.svg,
tomee-2.svg.
- Do not include 'incubator-' in your project name, we'll treat
podlings and projects like equals for this exercise.
- If you have sub-projects, treat them like regular projects. thus,
rat should be rat.svg rather than creadur-rat.
- IF someone else beat you to it, but you have a better version of the
logo, please add it as a new file, using the same naming convention as
above (-1, -2 etc)

For those that submit high res bitmaps, I'll try my best to convert to
scalable formats where possible.

If all goes well and enough projects collaborate on this, we should be
able to present a web site for finding and downloading project logos
from across the foundation.

With warm regards,
Daniel on behalf of the Community Development Project.

PS: reply-to is set to my personal email, in case of out-of-office
replies that people wouldn't want public.

Re: [VOTE] Apache SystemML 1.2.0 (RC1)

2018-08-19 Thread Matthias Boehm

+1

I ran the perftest suite multiple times up to 80GB with and without
codegen. After fixing all the issues and regressions, the entire suite
ran successfully against Spark 2.2 and 2.3 and all use cases showed
equal or better performance compared to SystemML 1.1.

Regards,
Matthias

On Fri, Aug 17, 2018 at 8:41 AM, Berthold Reinwald  wrote:
> Please vote on releasing the following candidate as Apache SystemML
> version 1.2.0
>
> The vote is open for at least 72 hours and passes if a majority of at
> least 3 +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache SystemML 1.2.0
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache SystemML, please see
> http://systemml.apache.org/
>
>
> The tag to be voted on is v1.2.0-rc1 (
> a1a05e29f6ee78f3c33fea355f62c78ce21766ee):
> https://github.com/apache/systemml/tree/v1.2.0-rc1
>
>
> The release artifacts can be found at:
> https://dist.apache.org/repos/dist/dev/systemml/1.2.0-rc1/
>
>
> The maven release artifacts, including signatures, digests, etc. can be
> found at:
> https://repository.apache.org/content/repositories/orgapachesystemml-1030/org/apache/systemml/systemml/1.2.0/
>
>
>
> ===
> == Apache Release policy ==
> ===
> http://www.apache.org/legal/release-policy.html
>
>
> ===
> == How can I help test this release? ==
> ===
> If you are a SystemML user, you can help us test this release by taking an
>
>
> existing Algorithm or workload and running on this release candidate, then
>
>
> reporting any regressions.
>
> 
> == What justifies a -1 vote for this release? ==
> 
> -1 votes should only occur for significant stop-ship bugs or legal related
>
>
> issues (e.g. wrong license, missing header files, etc). Minor bugs or
> regressions should not block this release.
>
>
>
> Regards,
> Berthold Reinwald
> IBM Almaden Research Center
> office: (408) 927 2208; T/L: 457 2208
> e-mail: reinw...@us.ibm.com
>

Re: Release Planning SystemML 1.2

2018-08-11 Thread Matthias Boehm

Well, this took a little longer than expected but now we have resolved
all issues and performance regressions. Therefore, I think we should
be ready to cut an RC1 for SystemML 1.2 early next week. So please
hold off any major new features for now to minimize the chances of
regressions during this RC phase.

Regards,
Matthias

On Sun, Jun 24, 2018 at 4:26 PM, Janardhan  wrote:
> Nvidia have official support for prebuilt docker image(CUDA + CuDNN) only
> for Ubuntu. This will be convenient for us.
>
> Nvidia drivers are simply hanging my machine (CentOS 7), but it is easier
> to change this at a later date.
>
> Thank you,
> Janardhan
>
> On Monday, June 25, 2018, Berthold Reinwald  wrote:
>
>> either one would be great but having the GPU integration is a plus. Why
>> wouldn't that be possible for Centos?
>>
>>
>> Regards,
>> Berthold Reinwald
>> IBM Almaden Research Center
>> office: (408) 927 2208; T/L: 457 2208
>> e-mail: reinw...@us.ibm.com
>>
>>
>>
>> From:   Janardhan 
>> To: "dev@systemml.apache.org" 
>> Date:   06/24/2018 05:56 AM
>> Subject:Re: Release Planning SystemML 1.2
>>
>>
>>
>> One more thing, I am already building a docker image. But, which image do
>> you prefer
>>
>> 1. CentOS 7 or
>> 2. Ubuntu - Later extensible to GPU very easily.
>>
>> This decreases setup time to six minutes on any platform, even for non
>> devs.
>>
>> Thank you
>> Janardhan
>>
>> On Sunday, June 24, 2018, Matthias Boehm  wrote:
>>
>> > given the current status of open tasks and the delay with regard to
>> > QA, I think we need to push this release out by a couple of weeks.
>> > Does mid to end July sound good to everyone?
>> >
>> > Regards,
>> > Matthias
>> >
>> > On Wed, Jun 6, 2018 at 11:28 PM, Matthias Boehm 
>> wrote:
>> > > thanks Berthold - that sounds good. We'll probably start a little
>> > > earlier with the tests to ensure the python test suite gives the same
>> > > results for the set of overlapping algorithms.
>> > >
>> > > Regards,
>> > > Matthias
>> > >
>> > > On Wed, Jun 6, 2018 at 11:08 PM, Berthold Reinwald
>> 
>> > wrote:
>> > >> +1. Happy to kick off the release scripts the week of June 25th. We
>> > should
>> > >> probably kick off the python perf test suite the week before to avoid
>> > >> hiccups.
>> > >>
>> > >> Regards,
>> > >> Berthold Reinwald
>> > >> IBM Almaden Research Center
>> > >> office: (408) 927 2208; T/L: 457 2208
>> > >> e-mail: reinw...@us.ibm.com
>> > >>
>> > >>
>> > >>
>> > >> From:   Krishna Kalyan 
>> > >> To: dev@systemml.apache.org
>> > >> Date:   06/05/2018 10:09 PM
>> > >> Subject:Re: Release Planning SystemML 1.2
>> > >>
>> > >>
>> > >>
>> > >> +1
>> > >>
>> > >> I am completely available to help with the QA cycle and help with
>> > >> switching
>> > >> to new perf test suite.
>> > >>
>> > >> Cheers,
>> > >> Krishna
>> > >>
>> > >> On Tue, Jun 5, 2018 at 8:59 AM, Matthias Boehm 
>> > wrote:
>> > >>
>> > >>> Hi all,
>> > >>>
>> > >>> given our current release cadence of about 3 month, we should start
>> > >>> talking about our SystemML 1.2 release. There have been many new
>> > >>> features, improvements over all backends, and various critical
>> fixes.
>> > >>> I know there are still some tasks that should go into 1.2, and for
>> > >>> this QA cycle we'd like to switch to the new python perf test suite.
>> > >>> How about we aim for an RC1 by end of June which would give us more
>> > >>> than 3 weeks from now?
>> > >>>
>> > >>> Regards,
>> > >>> Matthias
>> > >>>
>> > >>
>> > >>
>> > >>
>> > >>
>> >
>>
>>
>>
>>
>>

Re: [DISCUSS] Adding SystemML to OSS Fuzz

2018-07-19 Thread Matthias Boehm

Hi Janardhan,

instead of continuing the discussion on the other thread of
google/oss-fuzz, let us arrive at a project-wide conclusion first. As
I said before, I don't think that creating fuzz tests for our native
kernels is a pressing issue right now because they only receive
internally constructed intermediates. Instead it would be better to
harden our external entry points (read/programmatic APIs) which are
all Java.

However, I don't want to hold you back. If you're interested in this
project and you're willing to do the work, please come up with a
concrete plan of action items that we can discuss here. If I don't
here back in 3 days, I'll recommend to close the issue at
google/oss-fuzz.

Regards,
Matthias

On Mon, May 21, 2018 at 5:29 PM, Matthias Boehm  wrote:
> Well, in general this can be interesting. Apart from our default
> testsuite, we occasionally ran static code analysis tools. Having
> additional tests for partially valid scripts and inputs can help to
> find more issues.
>
> That being said, I don't think we currently qualify as a project with
> "significant user base and/or be critical to the global IT
> infrastructure". Also, without Java support these tests would only
> apply to our native and GPU operations, which do not directly deal
> with external inputs.
>
> So Janardhan, which fuzz targets to you have in mind? Looking over the
> existing projects we would have to provide build scripts that
> reference C/C++ entry points for fuzz testing. Although I can see
> applications (e.g., corrupted column indexes in sparse matrices), I'm
> not sure if it's a good idea to perform checks for valid inputs on
> every operation instead of simply hardening the code path for external
> inputs.
>
> Regards,
> Matthias
>
> On Mon, May 21, 2018 at 10:41 AM, Janardhan  wrote:
>> They accepted( google/oss-fuzz ), SystemML project for fuzz testing.
>>
>> PR link: https://github.com/google/oss-fuzz/pull/1429
>>
>> - Janardhan
>>
>> On Mon, May 21, 2018 at 11:46 AM, Janardhan  wrote:
>>
>>> Hi all,
>>>
>>> 
>>> To find various programming errors (mostly detectable such as buffer
>>> overflow), a fuzz testing can be of great help.
>>>
>>> 
>>> Merits:
>>> 1. It will easily detects common programming errors, which we might have
>>> missed or not unit tested.
>>> 2. Improves the quality of our code.
>>>
>>> ---
>>> Demerits:
>>> 1.  If a bug is found, it will be made public after 90 + 15 (grace period)
>>> days. So, we must fix it before three months, if there is bug.
>>> 2. For now only C and C++ are supported, Java will be supported soon.
>>>
>>> Please use this PR for discussion https://github.com/
>>> google/oss-fuzz/pull/1429 , for adding our project's CPP part for fuzzing.
>>>
>>> Once you approve, I will try to build a docker image of SystemML and
>>> configure with help. The results of the test will be CC'ed to private
>>> mailing list, only.
>>>
>>>
>>> Thank you,
>>> Janardhan
>>>

Re: Release Planning SystemML 1.2

2018-06-24 Thread Matthias Boehm

given the current status of open tasks and the delay with regard to
QA, I think we need to push this release out by a couple of weeks.
Does mid to end July sound good to everyone?

Regards,
Matthias

On Wed, Jun 6, 2018 at 11:28 PM, Matthias Boehm  wrote:
> thanks Berthold - that sounds good. We'll probably start a little
> earlier with the tests to ensure the python test suite gives the same
> results for the set of overlapping algorithms.
>
> Regards,
> Matthias
>
> On Wed, Jun 6, 2018 at 11:08 PM, Berthold Reinwald  
> wrote:
>> +1. Happy to kick off the release scripts the week of June 25th. We should
>> probably kick off the python perf test suite the week before to avoid
>> hiccups.
>>
>> Regards,
>> Berthold Reinwald
>> IBM Almaden Research Center
>> office: (408) 927 2208; T/L: 457 2208
>> e-mail: reinw...@us.ibm.com
>>
>>
>>
>> From:   Krishna Kalyan 
>> To: dev@systemml.apache.org
>> Date:   06/05/2018 10:09 PM
>> Subject:Re: Release Planning SystemML 1.2
>>
>>
>>
>> +1
>>
>> I am completely available to help with the QA cycle and help with
>> switching
>> to new perf test suite.
>>
>> Cheers,
>> Krishna
>>
>> On Tue, Jun 5, 2018 at 8:59 AM, Matthias Boehm  wrote:
>>
>>> Hi all,
>>>
>>> given our current release cadence of about 3 month, we should start
>>> talking about our SystemML 1.2 release. There have been many new
>>> features, improvements over all backends, and various critical fixes.
>>> I know there are still some tasks that should go into 1.2, and for
>>> this QA cycle we'd like to switch to the new python perf test suite.
>>> How about we aim for an RC1 by end of June which would give us more
>>> than 3 weeks from now?
>>>
>>> Regards,
>>> Matthias
>>>
>>
>>
>>
>>

Re: Release Planning SystemML 1.2

2018-06-07 Thread Matthias Boehm

thanks Berthold - that sounds good. We'll probably start a little
earlier with the tests to ensure the python test suite gives the same
results for the set of overlapping algorithms.

Regards,
Matthias

On Wed, Jun 6, 2018 at 11:08 PM, Berthold Reinwald  wrote:
> +1. Happy to kick off the release scripts the week of June 25th. We should
> probably kick off the python perf test suite the week before to avoid
> hiccups.
>
> Regards,
> Berthold Reinwald
> IBM Almaden Research Center
> office: (408) 927 2208; T/L: 457 2208
> e-mail: reinw...@us.ibm.com
>
>
>
> From:   Krishna Kalyan 
> To: dev@systemml.apache.org
> Date:   06/05/2018 10:09 PM
> Subject:Re: Release Planning SystemML 1.2
>
>
>
> +1
>
> I am completely available to help with the QA cycle and help with
> switching
> to new perf test suite.
>
> Cheers,
> Krishna
>
> On Tue, Jun 5, 2018 at 8:59 AM, Matthias Boehm  wrote:
>
>> Hi all,
>>
>> given our current release cadence of about 3 month, we should start
>> talking about our SystemML 1.2 release. There have been many new
>> features, improvements over all backends, and various critical fixes.
>> I know there are still some tasks that should go into 1.2, and for
>> this QA cycle we'd like to switch to the new python perf test suite.
>> How about we aim for an RC1 by end of June which would give us more
>> than 3 weeks from now?
>>
>> Regards,
>> Matthias
>>
>
>
>
>

Re: [DISCUSS] Adding SystemML to OSS Fuzz

2018-05-21 Thread Matthias Boehm

Well, in general this can be interesting. Apart from our default
testsuite, we occasionally ran static code analysis tools. Having
additional tests for partially valid scripts and inputs can help to
find more issues.

That being said, I don't think we currently qualify as a project with
"significant user base and/or be critical to the global IT
infrastructure". Also, without Java support these tests would only
apply to our native and GPU operations, which do not directly deal
with external inputs.

So Janardhan, which fuzz targets to you have in mind? Looking over the
existing projects we would have to provide build scripts that
reference C/C++ entry points for fuzz testing. Although I can see
applications (e.g., corrupted column indexes in sparse matrices), I'm
not sure if it's a good idea to perform checks for valid inputs on
every operation instead of simply hardening the code path for external
inputs.

Regards,
Matthias

On Mon, May 21, 2018 at 10:41 AM, Janardhan  wrote:
> They accepted( google/oss-fuzz ), SystemML project for fuzz testing.
>
> PR link: https://github.com/google/oss-fuzz/pull/1429
>
> - Janardhan
>
> On Mon, May 21, 2018 at 11:46 AM, Janardhan  wrote:
>
>> Hi all,
>>
>> 
>> To find various programming errors (mostly detectable such as buffer
>> overflow), a fuzz testing can be of great help.
>>
>> 
>> Merits:
>> 1. It will easily detects common programming errors, which we might have
>> missed or not unit tested.
>> 2. Improves the quality of our code.
>>
>> ---
>> Demerits:
>> 1.  If a bug is found, it will be made public after 90 + 15 (grace period)
>> days. So, we must fix it before three months, if there is bug.
>> 2. For now only C and C++ are supported, Java will be supported soon.
>>
>> Please use this PR for discussion https://github.com/
>> google/oss-fuzz/pull/1429 , for adding our project's CPP part for fuzzing.
>>
>> Once you approve, I will try to build a docker image of SystemML and
>> configure with help. The results of the test will be CC'ed to private
>> mailing list, only.
>>
>>
>> Thank you,
>> Janardhan
>>

Re: SYSTEMML-447

2018-05-10 Thread Matthias Boehm

This particular JIRA is only partially related. Niketan and Nakul
worked out the details - the only reason I show up as the reporter is
that, if I remember correctly, we split a larger scoped JIRA for
low-level optimizations (GPU, codegen, compression) into individual
JIRAs and created the detailed tasks.

Overall, I believe that sparse GPU operations would be very valuable,
especially in the context of NLP, graphs, and structured data with
categorical features (which often become very sparse after dummy
coding) because in these ultra-sparse scenarios dense operations cause
unnecessary overheads of orders of magnitude (proportional to the
sparsity). However, creating efficient sparse GPU kernels is
challenging due to irregularities (e.g., sparsity skew). Compared to
CPU operations, there might still be benefit depending on the data
location of inputs/outputs, as well as higher memory bandwidth.

Even in the face of extending the codegen framework for GPUs (which is
still on the roadmap for this year), we would still need dense/sparse
kernels for the individual operations because we want to apply codegen
only if we can benefit from fusion. Right now we call existing
libraries such as cuBLAS and cuDNN and have dense kernels for a subset
of operations such as unary and binary, and unary aggregates.

Regarding ramping up on the GPU backend, maybe it's a good idea to
first start with missing dense operations. I'm thinking of statistical
functions (e.g., covariance, moment), parameterized builtin functions
(e.g., grouped aggregated), missing unary and binary operations (e.g.,
bitwise), missing reorg operations (e.g., reshape, sort - there should
be library for the latter), missing unary, binary and ternary
aggregates, missing nary (e.g., nary cbind/rbind), etc. Adding these
remaining operations would also help a lot. However, if you're more
interested in contributing to the development of sparse kernels, maybe
you could one or two dense operations, get comfortable, and then move
on to sparse operations. Apart from the kernels, a seamless support
for sparse operations would also require some integration work on how
we pass data, maintain nnz, preallocate sparse outputs, etc.

Regards,
Matthias

On Thu, May 10, 2018 at 8:47 PM, Janardhan  wrote:
> Hi Matthias,
>
> Was this related to long term plan for GPU codegen?
>
> Thank you,
> Janardhan

Re: Questions about MNIST LeNet example

2018-05-10 Thread Matthias Boehm

just FYI: we now have support for list and named-list data types in
SystemML, which allow passing the entire model as a single handle. For
example, you can define the following

l1 = list(W1, b1, W2, b2, W3, b3, W4, b4), or
l2 = list(a=W1, b=b1, c=W2, d=b2, e=W3, f=b3, g=W4, h=b4)

and access entries via l1[7] or l2['g'] accordingly. We're still
working on additional features to make the integration with IPA,
functions, and size/type propagation smoother, but the basic
functionality is already available.

Regards,
Matthias

On Sun, May 6, 2018 at 1:08 PM, Matthias Boehm <mboe...@gmail.com> wrote:
> Hi Guobao,
>
> that sounds very good. In general, the "model" refers to the
> collection of all weights and bias matrices of a given architecture.
> Similar to a classic regression model, we can view the weights as the
> "slope", i.e., multiplicative terms, while the biases are the
> "intercept", i.e., additive terms that shift the layer output. Both
> are subject to training and thus part of the model.
>
> This implies that the number of matrices in the model depends on the
> architecture. Hence, we have two choices here: (a) allow for a
> variable number of inputs and outputs, or (b) create a struct-like
> data type that allows passing the collection of matrices via a single
> handle. We've discussed the second option in other contexts as well
> because this would also be useful for reducing the number of
> parameters passed through function calls. I'm happy to help out
> integrating these struct-like data types if needed.
>
> Great to see that you're in the process of updating the related JIRAs.
> Let us know whenever you think you're ready with an initial draft -
> then I'd make a detailed pass over it.
>
> Furthermore, I would recommend to experiment with running these
> existing mnist lenet examples (which is one of our baselines moving
> forward):
> * Download the "infinite MNIST" data generator
> (http://leon.bottou.org/projects/infimnist), and generate a moderately
> sized dataset (e.g., 256K instances).
> * Convert the input into SystemML's binary block format. The generator
> produces the data in libsvm format and we provide a data converter
> (see RDDConverterUtils.libsvmToBinaryBlock) to convert this into our
> internal binary representation.
> * Run the basic mnist lenet example for a few epochs.
> * Install the native BLAS libraries mkl or openblas and try using it
> for the above example to ensure its setup and configured correctly.
>
>
> Regards,
> Matthias
>
> On Sun, May 6, 2018 at 3:24 AM, Guobao Li <guobao1993...@gmail.com> wrote:
>> Hi Matthias,
>>
>> I'm currently reading the dml script MNIST LeNet example and got some
>> questions. I hope that you could help me out of them.
>>
>> 1) Is it possible to define a matrix containing the variables? Because I'm
>> wondering how to represent the model as a parameter for the "paramserv"
>> function.
>> 2) What is the role of bias? Why we need it?
>>
>> Additionally, I have added some updates in JIRA for SYSTEMML-2083 and hope
>> to get some feedback. Thanks!
>>
>> Regards,
>> Guobao
>>

Re: Questions about MNIST LeNet example

2018-05-06 Thread Matthias Boehm

Hi Guobao,

that sounds very good. In general, the "model" refers to the
collection of all weights and bias matrices of a given architecture.
Similar to a classic regression model, we can view the weights as the
"slope", i.e., multiplicative terms, while the biases are the
"intercept", i.e., additive terms that shift the layer output. Both
are subject to training and thus part of the model.

This implies that the number of matrices in the model depends on the
architecture. Hence, we have two choices here: (a) allow for a
variable number of inputs and outputs, or (b) create a struct-like
data type that allows passing the collection of matrices via a single
handle. We've discussed the second option in other contexts as well
because this would also be useful for reducing the number of
parameters passed through function calls. I'm happy to help out
integrating these struct-like data types if needed.

Great to see that you're in the process of updating the related JIRAs.
Let us know whenever you think you're ready with an initial draft -
then I'd make a detailed pass over it.

Furthermore, I would recommend to experiment with running these
existing mnist lenet examples (which is one of our baselines moving
forward):
* Download the "infinite MNIST" data generator
(http://leon.bottou.org/projects/infimnist), and generate a moderately
sized dataset (e.g., 256K instances).
* Convert the input into SystemML's binary block format. The generator
produces the data in libsvm format and we provide a data converter
(see RDDConverterUtils.libsvmToBinaryBlock) to convert this into our
internal binary representation.
* Run the basic mnist lenet example for a few epochs.
* Install the native BLAS libraries mkl or openblas and try using it
for the above example to ensure its setup and configured correctly.

Regards,
Matthias

On Sun, May 6, 2018 at 3:24 AM, Guobao Li  wrote:
> Hi Matthias,
>
> I'm currently reading the dml script MNIST LeNet example and got some
> questions. I hope that you could help me out of them.
>
> 1) Is it possible to define a matrix containing the variables? Because I'm
> wondering how to represent the model as a parameter for the "paramserv"
> function.
> 2) What is the role of bias? Why we need it?
>
> Additionally, I have added some updates in JIRA for SYSTEMML-2083 and hope
> to get some feedback. Thanks!
>
> Regards,
> Guobao
>

GSoC 2018 Student Guobao Li

2018-05-01 Thread Matthias Boehm

Hi all,

please join me in welcoming Guobao Li as a GSoC 2018 student, who will
be working on SYSTEMML-2083 (Language and runtime for parameter
servers) this summer. We're currently in the community bonding phase,
but the project will start May 14.

Krishna already kindly volunteered (on the dev list and in an offline
discussion) to act as a co-mentor when needed. Additionally, since
this is a major feature, I encourage everybody to provide early
feedback on the design documents and PRs, even if it's just for
specific aspects.

@Guobao: Please reach out as needed and let's aim to have all
technical discussions, except for our weekly calls, in public. A good
place to start would be to restructure the current JIRA tasks in the
next weeks.

Regards,
Matthias

Re: distributed cholesky on systemml

2018-04-23 Thread Matthias Boehm

you're welcome. Let me just make some additional remarks for other
people who might come over this thread. First, please don't
extrapolate characteristics from an experimental script and
unoptimized operations such as cholesky to SystemML in general.
Second, for compute-intensive algorithms, it's usually a good idea for
performance to leverage native BLAS libraries and our GPU backend
which are still disabled by default. Third, while data-parallel
operations indeed translate to individual RDD operations, there are
also language constructs like parfor (parallel for loops), where the
entire loop with all its iterations, nested loops and function calls
may be mapped to a single Spark job.

Regards,
Matthias


On Mon, Apr 23, 2018 at 12:42 AM, Qifan Pu <qifan...@gmail.com> wrote:
> You are right. Switching to Spark mode makes it even much slower...
> It seems that each operator will generate a Spark task converting things
> into RDD operators.
> Thanks so much for the patience and detailed instructions. I have a much
> better understanding of the system now.
>
> On Sun, Apr 22, 2018 at 7:47 PM, Matthias Boehm <mboe...@gmail.com> wrote:
>>
>> well, SystemML decides the execution type of each operation based on
>> its worst-case memory estimate and the available driver memory budget
>> to avoid unnecessary overheads for distributed operations. Maybe the
>> size of the matrix that is fed into cholesky is smaller than the
>> intermediates used for data generation? Btw, this is a characteristic
>> we often see for other scripts (e.g., t(X)%*%X in LinearRegDS
>> potentially runs over large matrices whereas the subsequent solve only
>> works over a features-by-features matrix) - this was the reason why
>> distributed operations for builtin functions like solve and cholesky
>> had low priority in the past.
>>
>> If you're interested in understanding the execution plan of your
>> scenarios better, you can use '-explain' or '-explain hops' to see the
>> memory budgets, estimates, and selected execution types. Also, if you
>> want to force all operations to spark, you can do so via '-exec
>> spark'. However, note that this should be done only for debugging
>> because forcing all operations to spark can be very counterproductive
>> for performance.
>>
>> Regards,
>> Matthias
>>
>> On Sun, Apr 22, 2018 at 5:07 PM, Qifan Pu <qifan...@gmail.com> wrote:
>> > and everything before (e.g., I generate the matrix using another DML)
>> > was
>> > indeed run by Spark and shows up on the UI.
>> >
>> > On Sun, Apr 22, 2018 at 5:05 PM, Qifan Pu <qifan...@gmail.com> wrote:
>> >>
>> >> Thanks Jeremy and Matthias. When I run the script, the cholesky or the
>> >> inv
>> >> is executed completely on the driver, and nothing shows up on Spark UI.
>> >> Is that the expected behavior?
>> >>
>> >> On Sun, Apr 22, 2018 at 3:34 PM, Jeremy Nilmeier <nilme...@us.ibm.com>
>> >> wrote:
>> >>>
>> >>> Yes, I also spoke with Sasha about this some time last year.  Thanks
>> >>> for
>> >>> following up.
>> >>>
>> >>> Cheers, J
>> >>>
>> >>>
>> >>> Jerome Nilmeier, PhD
>> >>> Data Scientist and Engineer
>> >>> IBM Spark Technology Center
>> >>> http://www.spark.tc/
>> >>>
>> >>>
>> >>>
>> >>> - Original message -
>> >>> From: Matthias Boehm <mboe...@gmail.com>
>> >>> To: dev@systemml.apache.org
>> >>> Cc: Qifan Pu <qifan...@gmail.com>, Jeremy Nilmeier
>> >>> <nilme...@us.ibm.com>
>> >>> Subject: Re: distributed cholesky on systemml
>> >>> Date: Sun, Apr 22, 2018 2:41 PM
>> >>>
>> >>> thanks for the context Jeremy - that helps. I also had an offline
>> >>> conversion with Sasha and he pointed me to a script that does exactly
>> >>> that (iterative invert_lower_triangular) combined with a parfor over
>> >>> independent blocks. We'll merge these scripts soon and I'll reach out
>> >>> individually as necessary. Thanks everybody for now.
>> >>>
>> >>> Regards,
>> >>> Matthias
>> >>>
>> >>> On Sun, Apr 22, 2018 at 12:40 PM, Jeremy Nilmeier
>> >>> <nilme...@us.ibm.com>
>> >>> wrote:
>> >>> > This may be a duplicate...it was bounced from the dev list.
>>

Re: distributed cholesky on systemml

2018-04-22 Thread Matthias Boehm

sure no problem - thanks again for catching this issue that was hidden
for a while.

Yes, the same depth-first characteristic applies to the Cholesky
function as well. In contrast to U_triangular_inv, however, there are
data dependencies between the blocks per level (at least in the
current algorithm formulation), which means we cannot use the approach
I described for U_triangular_inv.

L11 = Cholesky(A11, nb)
A22 = ... U_triangular_inv(t(L11))
L22 = Cholesky(A22, nb)

However, note that there are much fewer calls to Cholesky due to the
switch to the builtin cholesky according to the given min block size.
For example, in our new test for dimensions 1362 x 1362 and min size
of 200, we call Cholesky 15 times but U_triangular_inv 2539 times.

For sufficiently large min block size this might be ok for Cholesky,
because each level also does a number of matrix multiplies that will
exploit the available parallelism of your cluster. In that regard. you
might want to experiment with different block sizes and driver memory
budgets. If I get a chance, I will also run a number of experiments
and see if we can rewrite these scripts.

Regards,
Matthias

On Sun, Apr 22, 2018 at 12:48 AM, Qifan Pu <qifan...@gmail.com> wrote:
> Matthias,
>
> Thanks so much for taking time to fix. Really appreciated it.
> Does the same reasoning apply to the cholesky script? The recursive approach
> also looks inherently sequential.
>
> Best,
> Qifan
>
> On Sat, Apr 21, 2018 at 11:39 PM, Matthias Boehm <mboe...@gmail.com> wrote:
>>
>> just as a quick update: this issue has now been fixed in SystemML
>> master - it was essentially a missing guard for recursive functions
>> when checking for unary size-preserving functions during
>> inter-procedural analysis (IPA).
>>
>> However, while working with this recursive cholesky function I came to
>> the conclusion that it may need some rework. The current top-down,
>> depth-first, approach is inherently sequential. This is partially
>> unnecessary because for the used recursive function U_triangular_inv
>> (which is called many more times than cholesky), blocks per level are
>> independent. Therefore, we should look into a bottom-up, breadth-first
>> approach to parallelize over the blocks in each level, which could be
>> done via parfor at script level.
>>
>> Regards,
>> Matthias
>>
>> On Sat, Apr 21, 2018 at 6:59 PM, Matthias Boehm <mboe...@gmail.com> wrote:
>> > thanks for catching this - I just ran a toy example and this seems to
>> > be a rewrite issue (there are specific right indexing rewrites that
>> > collapse U[1:k,1:k] and U[1:k,k+1:n] into a single access to U which
>> > helps for large distributed matrices). As a workaround, you can set
>> > "sysml.optlevel" to 1 (instead of default 2, where 1 disables all
>> > rewrites), which worked fine for me. I'll fix this later today. Also
>> > I'll fix the naming from "Choleskey" to "Cholesky". Thanks again.
>> >
>> > Regards,
>> > Matthias
>> >
>> >
>> > On Sat, Apr 21, 2018 at 6:28 PM, Qifan Pu <qifan...@gmail.com> wrote:
>> >> Hi Matthias,
>> >>
>> >> Thanks for the fast response and detailed information. This is really
>> >> helpful.
>> >>
>> >> I just tried to run it, and was tracing down a indexing bug that can be
>> >> repeated by simply running the test script of triangle solve[1]
>> >> Caused by: org.apache.sysml.runtime.DMLRuntimeException: Invalid values
>> >> for
>> >> matrix indexing: [1667:,1:1666] must be within matrix dimensions
>> >> [1000,1000]
>> >>
>> >>
>> >> Am I missing some configuration here?
>> >>
>> >>
>> >> [1]
>> >>
>> >> https://github.com/apache/systemml/blob/master/scripts/staging/scalable_linalg/test/test_triangular_inv.dml
>> >>
>> >>
>> >> Best,
>> >> Qifan
>> >>
>> >>
>> >> On Sat, Apr 21, 2018 at 4:06 PM, Matthias Boehm <mboe...@gmail.com>
>> >> wrote:
>> >>>
>> >>> Hi Qifan,
>> >>>
>> >>> thanks for your feedback. You're right, the builtin functions
>> >>> cholesky, inverse, eigen, solve, svd, qr, and lu are currently only
>> >>> supported as single-node operations because they're still implemented
>> >>> via Apache commons.math.
>> >>>
>> >>> However, there is an experimental script for distributed cholesky [1]
>> >>&g

Fwd: Fw: Request for a beginner JIRA

2018-04-03 Thread Matthias Boehm

Thanks for your interest Daiki. I created two JIRAs SYSTEMML-2233 and
SYSTEMML-2232 that might me a good starting point. I would recommend
to begin with 2233 as a basic cleanup task, which is meant to get you
comfortable. The other task is then a bit more involved but would
improve our function namespace handling (e.g., for better statistics
outputs).

If you are mostly interested in working on the Python APIs, an awesome
project would be to include the Python MLContext API into our
testsuite (or a separate Python testsuite). @Niketan: Do you have more
thoughts on that? Our dev environment anyway requires Python for the
distribution build.

Regarding the development environment, you would typically clone the
repository, setup your preferred IDE and work on local branches for
new features. Once a change is ready for review, you submit a PR which
automatically triggers a run of our testsuite on our jenkins
infrastructure. You can also run these tests locally through maven or
directly through junit, which is especially useful for debugging. The
same applies for Python as well but as mentioned above the Python API
is not yet part of the automated testsuite, so yes changes there
involve semi-manual testing.

The issue you've encountered with Java 9 might be related to [1]. I
just gave it a try and core SystemML compiles and runs fine with Java
9 when build directly through the IDE. If it's related to Scala, we
should find and discuss a resolution for our next 1.2 release.

[1] https://stackoverflow.com/questions/49016074/systemml-build-failed-on-macos

Regards,
Matthias

From: Daiki Matsunaga/Japan/IBM
To: dev@systemml.apache.org, Matthias Boehm1/US/IBM@IBM
Date: 04/03/2018 12:46 AM
Subject: Request for a beginner JIRA


HI everyone,

My name is Daiki from IBM Japan and I'm just starting out with
building/going through some examples so I can get used to the project
before contributing.

Request for JIRA

I would like some guidance on choosing which JIRA issue to choose
from. I don't have any strong preference for which part of SystemML to
contribute,but

Questions

I apologize in advance if this is too simple but what are the basic
steps involved during the iteration of development, build, test,
especially for Python. For example, do you write out code in Java or
Python on the source (e.g. a feature branched out from master), build,
and pip install and then check on Jupyter notebook? Or do you directly
edit the Python code that is generated after pip install?

Some things I noticed during install/build

Problem: It says Java 8+ is required but Java 9 does not work. With
Java 9 installed on my machine(Mac OS Sierra 10.12.5. ), there was a
build failure with "mvn clean package" as well as in the pyspark shell
with "ml = MLContext(sc)". This would be an issue with the current
documentation since it automatically downloaded Java 9 with "brew
install Caskroom/cask/java".
Solution: Since I had both versions of Java, I set the bash profile
Java environment variable to the following and it worked:
export JAVA_HOME="$(/usr/libexec/java_home -v 1.8)"






Daiki Matsunaga
Cognitive AI
GBS, IBM Japan
Phone: 050-3150-7460 / 080-5915-7460
E-mail: e36...@jp.ibm.com

Re: Contribution to SystemML

2018-04-02 Thread Matthias Boehm

SYSTEMML-2226 might be a good choice because it requires touching several
layers from the language through the runtime, without being too involved.
Also, I will add several other unary aggregates in the next days which
could act as templates (but you don't have to wait for that). Simply leave
a comment on the JIRA once you start working on it and create PR whenever
you're ready. Thanks.

Regards,
Matthias


On Sun, Apr 1, 2018 at 11:38 PM, Govinda Malavipathirana <
mp.govi...@gmail.com> wrote:

> Hi'
> Still waiting for guidance to choose a task. Can I have some tips to
> select one?
> regards,
> Govinda
>
>
> On Thu, Mar 29, 2018 at 10:06 AM Matthias Boehm <mboe...@gmail.com> wrote:
>
>> -- Forwarded message --
>> From: Matthias Boehm <mboe...@gmail.com>
>> Date: Wed, Mar 28, 2018 at 9:34 PM
>> Subject: Re: Contribution to SystemML
>> To: Govinda Malavipathirana <mp.govi...@gmail.com>
>>
>>
>> well, first of all sorry that you wasted one of your proposals because it
>> would have been better to combine them into a single proposal (as
>> commented
>> on the earlier draft). Unfortunately, I didn't see these duplicates before
>> the deadline.
>>
>> Regarding your interest in getting familiar with the internals of
>> SystemML,
>> that's great to hear. In addition to the existing JIRAs (you're welcome to
>> leave a comment if you start working on one), I'll create some additional
>> beginner tasks related to your interests in the next days.
>>
>> Regards,
>> Matthias
>>
>> On Wed, Mar 28, 2018 at 12:14 AM, Govinda Malavipathirana <
>> mp.govi...@gmail.com> wrote:
>>
>> > Hi'
>> > I have submitted 2 proposals for Language .and Parameter Server project.
>> > In this proposal review period I would like to contribute to systemml
>> for
>> > further understanding the systemml internals. I'm looking for some
>> guidance
>> > to select issue that relates/more like to [SYSTEMML-2084] Language and
>> > Compiler Extension and [SYSTEMML-2089] Extended Keras2DML and Caffe2DML
>> > Script Generators. Thanks in advanced.
>> > Best regards,
>> > Govinda
>> >
>>
>

Re: Draft Release Notes 1.1.0

2018-03-28 Thread Matthias Boehm

thanks for the initial draft and extensions - I would remove internals
#2/#3 because they are still open, move the other internals to performance,
and include (or extend) the following:

* codegen extensions (operation support, extended optimizer, see
SYSTEMML-2065)
* new accumulator operator += (not just in parfor)
* matrix-matrix multiplication over compressed matrices
* zero row/column matrices and updated operations such as removeEmpty
* logical operator support over matrices AND/OR/NOT/XOR (besides the
bitwise ops)
* new second-order eval builtin function
* extended UDF framework
* performance: sparse left indexing, sparse reshape, ultra-sparse
transpose, ultra-sparse rand, binary in-place operations, sparse relu
backward, maxpooling, sparse im2col, ultra-sparse conv2d, read of
short-wide sparse matrices, avoid unnecessary evictions, lock-free
statistics maintenance, spark cpmm, spark aggregates, spark reshape, spark
binary ops, use common thread pool for multi-threaded cp operations, etc
* improved nnz maintenance, runtime propogation and memory management
* robustness for matrices with larger than int dimensions

This also remind me that we should probably make a pass over all new
builtin functions, operators and generalized/extended operations and update
our documentation accordingly.

Regards,
Matthias


On Wed, Mar 28, 2018 at 10:02 PM, Niketan Pansare 
wrote:

> Thanks Berthold. Overall, the draft looks good to me. Few minor additions:
>
> New Builtin Functions
> - Deep learning builtin functions: avg_pool and avg_pool_backward
> - assert
>
> New Layers in the NN library:
> - Average pooling
> - Upsampling
> - Low-rank fully connected
>
> Thanks,
>
> Niketan Pansare
> IBM Almaden Research Center
> E-mail: npansar At us.ibm.com
> http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar
>
> [image: Inactive hide details for "Berthold Reinwald" ---03/28/2018
> 04:02:46 PM---Please add/update below release note draft prior to p]"Berthold
> Reinwald" ---03/28/2018 04:02:46 PM---Please add/update below release note
> draft prior to putting it on our website. Please also go to th
>
> From: "Berthold Reinwald" 
> To: dev@systemml.apache.org
> Date: 03/28/2018 04:02 PM
> Subject: Draft Release Notes 1.1.0
> --
>
>
>
> Please add/update below release note draft prior to putting it on our
> website. Please also go to the JIRA release notes link at the end, and
> update version/status if necessary. Thanks.
>
>
> * Release Notes - SystemML - Version SystemML 1.1
>
> New Capabilities/Features
> - Dense matrix blocks >16GB, and operations
> - Support bitwise operators not, and, or, xor, & LShift, Rshift
> - Additional ParFor result aggregation operations
> - UDFs callable in expressions
> - zero rws/columns matrices
> - Extended Caffe2DML and Keras2DML APIs
>
> New Builtin Functions
> - ifelse()
>
> Performance Improvements
> - Ultra-sparse operations
>
> Internals
> - Single-precision support for native conv2d and mm operations.
> - Consolidate replicated compilation chain
> - Generalize Binary Operations to (vector, matrix) Operands
> - Use common thread pool
>
> Bug Fixes
> - in APIs, performance bugs, optimizer, runtime, GPU backend, Spark
> backend.
>
> Experimental
> - Codegen
>
> Deprecate
> - Support for Spark 2.1/2.2 (make switch to newer ANTLR version)
>
> Detailed JIRA release notes are here:
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.
> apache.org_jira_secure_ReleaseNote.jspa-3Fversion-
> 3D12342282-26styleName-3DText-26projectId-3D12319522-
> 26Create-3DCreate-26atl-5Ftoken-3DA5KQ-2D2QAV-2DT4JA-2DFDED-
> 257C2e97f76f75a466564dba2a7d0bec5ca06bffd66e-257Clout=
> DwIFAg=jf_iaSHvJObTbx-siA1ZOg=HzVC6v79boGYQrpc383_
> Kao_6a6SaOkZrfiSrYZVby0=fku28ex5cFV8MRqEKRuh-_I3UTefK_Qx_YXQD9UfoWU=
> K6GTOjWQiV6jki5_tkBF1tXLL2SM2Jta2ZXFdjBYbQw=
>
>
> Regards,
> Berthold Reinwald
> IBM Almaden Research Center
> office: (408) 927 2208 <%28408%29%20927-2208>; T/L: 457 2208
> e-mail: reinw...@us.ibm.com
>
>
>
>
>

Fwd: Contribution to SystemML

2018-03-28 Thread Matthias Boehm

-- Forwarded message --
From: Matthias Boehm <mboe...@gmail.com>
Date: Wed, Mar 28, 2018 at 9:34 PM
Subject: Re: Contribution to SystemML
To: Govinda Malavipathirana <mp.govi...@gmail.com>

well, first of all sorry that you wasted one of your proposals because it
would have been better to combine them into a single proposal (as commented
on the earlier draft). Unfortunately, I didn't see these duplicates before
the deadline.

Regarding your interest in getting familiar with the internals of SystemML,
that's great to hear. In addition to the existing JIRAs (you're welcome to
leave a comment if you start working on one), I'll create some additional
beginner tasks related to your interests in the next days.

Regards,
Matthias

On Wed, Mar 28, 2018 at 12:14 AM, Govinda Malavipathirana <
mp.govi...@gmail.com> wrote:

> Hi'
> I have submitted 2 proposals for Language .and Parameter Server project.
> In this proposal review period I would like to contribute to systemml for
> further understanding the systemml internals. I'm looking for some guidance
> to select issue that relates/more like to [SYSTEMML-2084] Language and
> Compiler Extension and [SYSTEMML-2089] Extended Keras2DML and Caffe2DML
> Script Generators. Thanks in advanced.
> Best regards,
> Govinda
>

Re: [SYSTEMML-2084][SYSTEMML-2085] Language and Compiler Extension, Basic Runtime Primitives.

2018-03-26 Thread Matthias Boehm

Well, SYSTEMML-2084 aims to integrate the new paramserv builtin function -
as described in the JIRA - into SystemML's semantic validation and
compiler. This would entail to first finalize the design of this builtin
function (e.g., function signature and semantics) and then integrate it
into the HOPs-LOPs-instruction compiler. You might want to take an existing
parameterized builtin function (e.g., aggregate) and trace through the
system, how we compile such a function.

In contrast, SYSTEMML-2085 would design and built common internal
primitives (e.g., data structures, and abstractions for update strategies)
that could be shared by all backends to it's mostly related to the runtime
side. If you're more interested in the language aspects, I would recommend
to focus on the language integration, maybe the local backend, and the
extensions for Keras2DML/Caffe2DML.

However, at the end of the day, it's your proposal so the scope and
approach is up to you, but we're happy to provide additional comments on
revised versions of your draft.

Regards,
Matthias

On Sat, Mar 24, 2018 at 11:20 PM, Govinda Malavipathirana <
mp.govi...@gmail.com> wrote:

> Hi'
> Apart from the extended script generators I would like to contribute one
> of these topic. I would like know the both tasks in more detail, So I can
> choose the task according to my interest.
> Regards,
> Govinda.
>

Re: [VOTE] Apache SystemML 1.1.0 (RC2)

2018-03-24 Thread Matthias Boehm

+1, I reran all tests mentioned before for RC2 as well without any issues.

Regards,
Matthias

On Fri, Mar 23, 2018 at 5:26 PM, Berthold Reinwald 
wrote:

> Please vote on releasing the following candidate as Apache SystemML
> version 1.1.0
>
> The vote is open for at least 72 hours and passes if a majority of at
> least 3 +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache SystemML 1.1.0
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache SystemML, please see
> http://systemml.apache.org/
>
> The tag to be voted on is v1.1.0-rc2 (
> deddaee1fd3f8e87ff3a4403edf06cdb022ba949):
> https://github.com/apache/systemml/commit/deddaee1fd3f8e87ff3a4403edf06c
> db022ba949
>
> The release artifacts can be found at:
> https://dist.apache.org/repos/dist/dev/systemml/1.1.0-rc2/
>
> The maven release artifacts, including signatures, digests, etc. can be
> found at:
> https://repository.apache.org/content/repositories/
> orgapachesystemml-1029/org/apache/systemml/systemml/1.1.0/
>
> Regards,
> Berthold Reinwald
> IBM Almaden Research Center
> office: (408) 927 2208; T/L: 457 2208
> e-mail: reinw...@us.ibm.com
>
>

Re: Apache SystemML 1.1.0 : Performance Test

2018-03-22 Thread Matthias Boehm

awesome - thanks for sharing Krishna.

Regards,
Matthias

On Wed, Mar 21, 2018 at 2:10 AM, Krishna Kalyan 
wrote:

> Hello All,
> Sharing a small Shiny App to visualize the runtime performance for System
> ML algorithms. This is work still in progress. Any feedback would be really
> appreciated :).
> https://kkalyan3.shinyapps.io/SystemML/
>
> (Currently, this app contains the total runtime on amazon c5.xlarge
> instance (standalone) on 10k_1k and 10k_100 dataset size, experiment
> repeated 4 times.)
>
> I also plan to refractor our python performance scripts
> - Make it object-oriented
> - Capture meta-data information like "hardware configuration", "Author" etc
> - Repeat performance tests easily
>
> Regards,
> Krishna
>

Re: [VOTE] Apache SystemML 1.1.0 (RC1)

2018-03-21 Thread Matthias Boehm

-1, sorry but I have to change my vote due to SYSTEMML-2201. The change
will be in master tomorrow - once this is done, we can cut another RC.

Regards,
Matthias

On Wed, Mar 21, 2018 at 4:12 PM, Matthias Boehm <mboe...@gmail.com> wrote:

> +1
>
> I ran the perftest suite up to 80GB with and without codegen and it ran
> just fine without errors or performance issues. There are two additional
> performance improvements that are not included in RC1 but it should be fine
> to postpone these till the next release.
>
> Regards,
> Matthias
>
> On Mon, Mar 19, 2018 at 1:51 PM, Berthold Reinwald <reinw...@us.ibm.com>
> wrote:
>
>> Please vote on releasing the following candidate as Apache SystemML
>> version 1.1.0
>>
>> The vote is open for at least 72 hours and passes if a majority of at
>> least 3 +1 PMC votes are cast.
>>
>> [ ] +1 Release this package as Apache SystemML 1.1.0
>> [ ] -1 Do not release this package because ...
>>
>> To learn more about Apache SystemML, please see
>> http://systemml.apache.org/
>>
>> The tag to be voted on is v1.1.0-rc1 (
>> edfc293339f7a88db4ee407ac1194983d5f8c23f):
>> https://github.com/apache/systemml/commit/edfc293339f7a88db4
>> ee407ac1194983d5f8c23f
>>
>>
>> The release artifacts can be found at:
>> https://dist.apache.org/repos/dist/dev/systemml/1.1.0-rc1/
>>
>>
>> The maven release artifacts, including signatures, digests, etc. can be
>> found at:
>> https://repository.apache.org/content/repositories/orgapache
>> systemml-1028/org/apache/systemml/systemml/1.1.0/
>>
>>
>>
>>
>> ===
>> == Apache Release policy ==
>> ===
>> http://www.apache.org/legal/release-policy.html
>>
>> ===
>> == How can I help test this release? ==
>> ===
>> If you are a SystemML user, you can help us test this release by taking an
>>
>> existing Algorithm or workload and running on this release candidate, then
>>
>> reporting any regressions.
>>
>> 
>> == What justifies a -1 vote for this release? ==
>> 
>> -1 votes should only occur for significant stop-ship bugs or legal related
>>
>> issues (e.g. wrong license, missing header files, etc). Minor bugs or
>> regressions should not block this release.
>>
>>
>>
>>
>> Regards,
>> Berthold Reinwald
>> IBM Almaden Research Center
>> office: (408) 927 2208; T/L: 457 2208
>> e-mail: reinw...@us.ibm.com
>>
>>
>

Fwd: Sub projects in Language and run time for parameter servers [SYSTEMML-2083]

2018-03-17 Thread Matthias Boehm

-- Forwarded message --
From: Matthias Boehm <mboe...@gmail.com>
Date: Sat, Mar 17, 2018 at 5:41 PM
Subject: Re: Sub projects in Language and run time for parameter servers
[SYSTEMML-2083]
To: Chamath Abeysinghe <abeysinghecham...@gmail.com>


great to see that you're making progress on your proposal. However and as a
general note to all students, please don't share your personal proposals
here. Instead submit your proposals through the official channel, the GSoC
website, and I will provide feedback there.

Having said that, if you have questions related to SystemML in general or
specifics of individual components, please, don't hesitate to ask them here.

Regards,
Matthias

On Fri, Mar 16, 2018 at 4:17 AM, Chamath Abeysinghe <
abeysinghecham...@gmail.com> wrote:

> Hi Matthias,
> After going through JIRA sub projects and references you provide I thought
> of drafting proposal focusing the Distributed spark backend
> <https://issues.apache.org/jira/browse/SYSTEMML-2087> project because it
> seems challenging and exciting area to explore :-).
> I have sketched a rough diagram for design and the implementation plan for
> the proposal, https://drive.google.com/file/d/1MTlYWvkkApe28vDOo
> dDR8hmxzVx9QwQX/view?usp=sharing
>
> My idea is making Paramserv runtime similar design to ParFor runtime, and
> as a extension it will handle parameter exchange. So there I will work on
> some primitives required by runtime to manage the PS and then in Spark I
> will implement a parameter server. Initially it will work using synchronous
> method and then if time allows I will experiment with other methods and
> performance factors.
>
> And also regarding the control program I have some concerns,
> In the project JIRA it was mentioned that "PS strategies will be selected
> by the user", does this include the architecture of the parameter server(#
> of workers and servers)  also  or does it need to be handled in the
> project?
>
> I hope this plan aligns with expectations of the community and does not
> conflicts with other GSoC candidates. Your feedback for this highly
> appreciated, if there is anything wrong please correct me. Thanks
>
> *PS : I am re sending the same mail because it seems previous mail with
> attachment was not delivered to the dev mailing list. *
>
> Regards,
> Chamath
>
>
> On Fri, Mar 9, 2018 at 2:19 PM, Matthias Boehm <mboe...@gmail.com> wrote:
>
>> Hi Chamath,
>>
>> ad 1: Yes, this is absolutely correct. However, it is important to
>> realize that within the workers, we want to run dml functions, and for
>> these we'll reuse our existing compiler, runtime, operations, and data
>> structures.
>>
>> ad 2: Yes, this is also correct. Indeed we can use an existing parfor
>> (with local execution mode) to emulate a local, synchronous parameter
>> server. However, it would be very hard - and conflicting with our
>> functional and thus, stateless execution semantics - to incorporate
>> asynchronous updates and strategies such as Hogwild!. Furthermore, such a
>> local parameter server might also have an application with very large
>> models and batches, because this would enable distributed data-parallel
>> operations spawn from each local worker.
>>
>> ad 3: Unfortunately, there is no one single detailed architecture diagram
>> because the system evolves over time. I would recommend to look at the
>> following two papers, where especially [1] (the parfor paper, and its
>> extensions for Spark in [2]) might give you a better idea of the parameter
>> server and its workers, which are primarily meant to handle the
>> orchestration and efficient parameter updates/exchange. if you're looking
>> for coarse-grained component, then [3], slide 8 might be a starting point.
>> At a high-level each operation and some constructs like parfor have
>> physical operators for CP, SPARK, MR, and some for GPU. Similarly this
>> project aims to introduce a new paramserv builtin function (most similar to
>> parfor) and its different physical operators.
>>
>> ad 4: Since this paramserv function has similarity with parfor, we will
>> be able to reuse key primitives for bringing up local/remote workers,
>> shipping the compiled functions, and input data. The major extensions will
>> be to call the shipped functions per batch, get the returned (i.e.,
>> updated) parameters and handle the exchange accordingly to the paramserv
>> configuration. However, since paramserv as an operation is implemented from
>> scratch, we can customize as needed and are not restricted by script-level
>> semantics which renders the problem simpler as the general-purpose parfor
>>

Fwd: Extending Codegen algorithm tests for heuristics

2018-03-13 Thread Matthias Boehm

-- Forwarded message --
From: Matthias Boehm <mboe...@gmail.com>
Date: Tue, Mar 13, 2018 at 1:00 PM
Subject: Re: Extending Codegen algorithm tests for heuristics
To: Chamath Abeysinghe <abeysinghecham...@gmail.com>


without debugging it's hard to tell, but usually something like this
happens if blocks are incorrectly aligned. So I would recommend to simply
do a mapToPair before the RDDAggregateUtils.mergeByKey and validate the
correctness of shifted block indexes. Maybe the newly introduced broadcasts
are not shifted into their target positions? For example, consider
cbind(A,B) - before aggregation, B needs to be shifted by ncol(A).
Furthermore, it would be great to avoid unnecessary aggregation if all but
one inputs are broadcasts.

Regards,
Matthias

On Tue, Mar 13, 2018 at 7:36 AM, Chamath Abeysinghe <
abeysinghecham...@gmail.com> wrote:

> Hi Matthias,
> I am working on SYSTEMML-2169 issue. I have sent a partially completed PR
> ( https://github.com/apache/systemml/pull/747 ). After those changes,
> some test cases in NaryRBindTest, are failing and I could not understand
> the reason.
> Test cases are failing with following error
> *Caused by: org.apache.sysml.runtime.DMLRuntimeException: Mismatched block
> sizes for: 280 101 1000 101*
> * at
> org.apache.sysml.runtime.instructions.spark.utils.RDDAggregateUtils$MergeBlocksFunction.call(RDDAggregateUtils.java:622)*
> * at
> org.apache.sysml.runtime.instructions.spark.utils.RDDAggregateUtils$MergeBlocksFunction.call(RDDAggregateUtils.java:596)*
>
> Even after debugging the whole process I could not find a reason for this.
> If you can give any suggestion that would be really helpful.
>
> If you have any other comment regarding the PR I could modify code
> according to that.
>
> Thanks,
> Chamath
>
>
> On Wed, Mar 7, 2018 at 11:44 AM, Matthias Boehm <mboe...@gmail.com> wrote:
>
>> -- Forwarded message --
>> From: Matthias Boehm <mboe...@gmail.com>
>> Date: Tue, Mar 6, 2018 at 10:14 PM
>> Subject: Re: Extending Codegen algorithm tests for heuristics
>> To: Chamath Abeysinghe <abeysinghecham...@gmail.com>
>>
>>
>> Hi Chamath,
>>
>> great thanks for your contribution - I left a couple of comments but we
>> should be ready to merge this in soon. If you want to get a better feeling
>> for the distributed spark backend as well, I created SYSTEMML-2169, which
>> aims to extend our recently added nary cbind/rbind operations to leverage
>> broadcasts when applicable.
>>
>> Regarding the proposal, most of the backends are rather independent, but
>> each backend depends on the language integration. We will help out where
>> necessary. So it depends on your interests and ideas. If you're more
>> interested in defining the language APIs, make this and a simple backend
>> the core of your proposal. If you're more interested in the runtime
>> backends, I would help and add a basic language integration in time, which
>> would allow you to immediate start working on the backends.
>>
>> Following the GSoC guidelines it's usually better to underscope the
>> project
>> than overscope it because you want to ensure that you're able to
>> successfully complete the project in the ambitious timeframe and there
>> will
>> always be unforeseen obstacles. I would recommend to define a core project
>> and potential extensions you will address if time allows. For example, the
>> local, multi-threaded backend can indeed be realized relatively quickly.
>> However, subsequently we can add and experiment with Hogwild! (i.e.,
>> unsynchronized updates) which is known to work well for sparse models,
>> replication and partitioning in NUMA settings, and potentially the
>> automatic selection of update strategies.
>>
>> Regards,
>> Matthias
>>
>>
>> On Tue, Mar 6, 2018 at 10:24 AM, Chamath Abeysinghe <
>> abeysinghecham...@gmail.com> wrote:
>>
>> > Hi,
>> > I have sent a pull request for this issue.
>> > As a next step, could you suggest any new issue? or anything I have to
>> do
>> > to familiarize with Language and run time for parameter servers project.
>> >
>> > And regarding writing the project proposal I have few questions.
>> > * In the epic there are few sub tasks, is it enough to focus on a single
>> > task through out the summer? Would it have enough work load or should I
>> go
>> > for multiple tasks?
>> > * What is the linkage between sub tasks? Do tasks like, Distributed
>> Spark
>> > Back-end or Local multi threaded b

Re: [DISCUSS] integrated testing for MLContext, SPARK, codegen.

2018-03-10 Thread Matthias Boehm

Hi Janardhan,

in general, we prefer to compare against R because it helps detecting
issues that are common across different optimizers and execution modes. So
for small scripts like PCA, I would recommend to simply create an R script,
which should be very similar to the dml script.

However, for more complex scripts with lots of table, removeEmpty, and
matrix-vector operations, creating the R scripts can be tedious and
error-prone because it requires additional operations such as vector
replications. For such case (including some existing tests), we could
indeed compare the different modes (e.g., w/ and w/o codegen). Let's decide
that case by case.

Regarding MLContext, yes it would be good to extend the algorithm test
coverage by simply reusing the dml and R scripts from the existing
application or codegen tests.

Regards,
Matthias

On Fri, Mar 9, 2018 at 9:52 PM, Janardhan Pulivarthi <
janardhan.pulivar...@gmail.com> wrote:

> Hi,
>
> Matthias -
> 1. We are checking the values of codegen algorithms with the R script, but
> can we compare
>
> ` pca script with [codegen_disabled] = pca script with [codegen_enabled]`
>
>
> Deron -
> 1. The same way can we compare the result through MLContext invocation
> with other mode of running the script.
>
>
> Thanks,
> Janardhan
>

Re: Sub projects in Language and run time for parameter servers [SYSTEMML-2083]

2018-03-09 Thread Matthias Boehm

Hi Chamath,

ad 1: Yes, this is absolutely correct. However, it is important to realize
that within the workers, we want to run dml functions, and for these we'll
reuse our existing compiler, runtime, operations, and data structures.

ad 2: Yes, this is also correct. Indeed we can use an existing parfor (with
local execution mode) to emulate a local, synchronous parameter server.
However, it would be very hard - and conflicting with our functional and
thus, stateless execution semantics - to incorporate asynchronous updates
and strategies such as Hogwild!. Furthermore, such a local parameter server
might also have an application with very large models and batches, because
this would enable distributed data-parallel operations spawn from each
local worker.

ad 3: Unfortunately, there is no one single detailed architecture diagram
because the system evolves over time. I would recommend to look at the
following two papers, where especially [1] (the parfor paper, and its
extensions for Spark in [2]) might give you a better idea of the parameter
server and its workers, which are primarily meant to handle the
orchestration and efficient parameter updates/exchange. if you're looking
for coarse-grained component, then [3], slide 8 might be a starting point.
At a high-level each operation and some constructs like parfor have
physical operators for CP, SPARK, MR, and some for GPU. Similarly this
project aims to introduce a new paramserv builtin function (most similar to
parfor) and its different physical operators.

ad 4: Since this paramserv function has similarity with parfor, we will be
able to reuse key primitives for bringing up local/remote workers, shipping
the compiled functions, and input data. The major extensions will be to
call the shipped functions per batch, get the returned (i.e., updated)
parameters and handle the exchange accordingly to the paramserv
configuration. However, since paramserv as an operation is implemented from
scratch, we can customize as needed and are not restricted by script-level
semantics which renders the problem simpler as the general-purpose parfor
construct. Both have their use cases.

In case this did not clarify your questions, let us known and we'll sort it
out.

[1] http://www.vldb.org/pvldb/vol7/p553-boehm.pdf, 2014
[2] http://www.vldb.org/pvldb/vol9/p1425-boehm.pdf, 2016
[3] http://boss.dima.tu-berlin.de/media/BOSS16-Tutorial-mboehm.pdf, 2016

Regards,
Matthias

On Thu, Mar 8, 2018 at 10:28 PM, Chamath Abeysinghe <
abeysinghecham...@gmail.com> wrote:

> Hi,
> I am trying to understand the purpose and work needed for different sub
> projects in SYSTEMML-2083. And I got few questions,
>
> * In the JIRA it was mentioned that we are not integrating off the shelf
> Parameter Server, but rather develop language and run time support from
> scratch. As far as I understand, this means creating syntax for DML to
> interact with the parameter server. And the parameter server implementation
> is in different back-ends. So for example in Spark back end we have to
> create a some kind of parameter server implementation with different
> strategies, and it should be invoked by the syntax in DML. Is this
> understanding correct?
>
> * In the JIRA there is a sub project for local multi threaded back-end. In
> this project does "local" mean executing on single node similar to
> ExecType.CP? If it is the case why use a parameter server for a single
> node?
>
> * I was unable to find a architecture diagram for SystemML, is there any
> that kind of diagram to understand the interaction between different
> back-ends and language API or can you point me to those classes?
>
> * And those new run times, are they going to be completely new separate
> run times or improvements to the existing ones?
>
> Please help me understand these issues. Thanks in advance.
>
> Regards,
> Chamath
>
>

Re: Release Planning

2018-02-06 Thread Matthias Boehm

yes, absolutely. Here is a list of new features and improvements - please
feel free to extend as needed:

1) Extended Caffe2DML and Keras2DML APIs
2) Support for large-dense blocks >16GB in CP
3) New builtin functions ifelse, xor, as well as and/or/not over matrices
4) Single-precision support for native conv2d and mm operations.
5) Performance features and correctness of ultra-sparse operations
6) Codegen: new plan cache, nary cbind
7) Parfor: result merge with accumulators +=, reduced initialization
overhead
8) Compiler improvements: avoid unnecessary spark instructions, corrected
memory estimates

Until the RC, I'd like to support all deep learning builtin functions in
codegen, add a couple of pending parfor improvements, and potentially do a
first cut of the compiler/runtime integration for parameter servers.

Regards,
Matthias

On Tue, Feb 6, 2018 at 12:36 PM, Niketan Pansare <npan...@us.ibm.com> wrote:

> +1.
>
> We should consider including single precision native BLAS in the release
> notes as well. If possible, we should add JNI wrappers for PowerPC, Windows
> and Mac too in this release.
>
> > On Feb 6, 2018, at 12:27 PM, Berthold Reinwald <reinw...@us.ibm.com>
> wrote:
> >
> > sure.
> >
> > Makes sense. Codegen and Keras2DML made good progress, and many other
> > fixes/improvements.
> >
> > What else do we want time/track/highlight for it?
> >
> > Regards,
> > Berthold Reinwald
> > IBM Almaden Research Center
> > office: (408) 927 2208; T/L: 457 2208
> > e-mail: reinw...@us.ibm.com
> >
> >
> >
> > From:   Matthias Boehm <mboe...@gmail.com>
> > To: dev@systemml.apache.org
> > Date:   02/05/2018 11:05 PM
> > Subject:Release Planning
> >
> >
> >
> > Hi all,
> >
> > since our 1.0 release in Dec, we already got a number enhancements and
> new
> > features in, so I think it would be good to discuss the timeline for our
> > next SystemML 1.1 release. How about, we target mid March for a first RC?
> > Also, Berthold would you be willing to serve again as the release
> manager?
> >
> > Regards,
> > Matthias
> >
> >
> >
> >
>
>

Re: [Discuss] GSoC 2018

2018-01-27 Thread Matthias Boehm

yes, but we as mentors will help out as needed, especially on the initial
language and compiler integration. Furthermore, it's a rather scalable
project in the sense that it could also accommodate multiple students. For
example, the different backends are fairly independent sub-projects.

Regards,
Matthias

On Sat, Jan 27, 2018 at 1:57 PM, Nakul Jindal <naku...@gmail.com> wrote:

> This is awesome!
> I am guessing the goal is to have this epic be a summer worth of
> mini-projects for a single GSoC student, isthat correct?
>
>
>
> On Fri, Jan 26, 2018 at 7:46 PM, Matthias Boehm <mboe...@gmail.com> wrote:
>
> > just FYI: I've created https://issues.apache.org/
> jira/browse/SYSTEMML-2083
> > with the gsoc2018 label. If you have additional project ideas, please
> file
> > the respective JIRAs. Thanks.
> >
> > Regards,
> > Matthias
> >
> > On Mon, Jan 22, 2018 at 12:13 AM, Matthias Boehm <mboe...@gmail.com>
> > wrote:
> >
> > > yes, that is a good idea and we should leverage this opportunity. I'm
> > > happy to mentor a project as well, specifically on parameter server
> > > architectures for distributed deep learning in SystemML.
> > >
> > > Right now we can emulate synchronous parameter servers with parfor, but
> > > there are other architectures like asynchronous, stale-synchronous,
> > > heterogeneity-aware, and decentralized, which we should support as well
> > in
> > > form of a dedicated runtime infrastructure similar to local/remote
> > parfor.
> > > If done right, we should be able to leverage common primitives in
> parfor
> > > and these parameter servers.
> > >
> > > Apart from better support for distributed deep learning, this would
> bring
> > > us much closer to the goal of a unified framework for large-scale
> machine
> > > learning as we would be the only framework that supports data-parallel
> > > (distributed ops), task-parallel (parfor), and model-parallel
> (parameter
> > > server) execution strategies in a single system.
> > >
> > > I'll create an epic with sub tasks later this week. The goal would be
> to
> > > integrate these parameter servers with the underlying data-parallel
> > > framework to have a common runtime for distributed operations and
> > > automatically support all underlying resource schedulers such as YARN,
> > > Mesos, and Kubernetes. Since the project is relatively large, we might
> > need
> > > to split it up into multiple GSoC projects along with multiple mentors.
> > >
> > > Regards,
> > > Matthias
> > >
> > > On 1/22/2018 8:53 AM, Nakul Jindal wrote:
> > >
> > >> Hi Krishna,
> > >>
> > >> That is a great idea. Thank you for offering to be a mentor/co-mentor.
> > >> I suggest that you think of a list of projects that you can mentor,
> > >> discuss
> > >> them on the mailing list and we can add them as JIRAs with the
> > appropriate
> > >> tags (as required by Apache).
> > >>
> > >> If there are students on this mailing list who want to suggest a
> topic,
> > >> that is very welcome too. If a mentor is willing to take it on, they
> > will
> > >> respond either here or on a JIRA.
> > >>
> > >> -Nakul
> > >>
> > >>
> > >>
> > >> On Sun, Jan 21, 2018 at 9:49 PM, Krishna Kalyan <
> > krishnakaly...@gmail.com
> > >> >
> > >> wrote:
> > >>
> > >> Hello All,
> > >>> I was wondering if system-ml community was planning to participate in
> > >>> GSoC
> > >>> this year. I had a wonderful time last year and learned a lot!.
> > >>>
> > >>> If we could have a couple of JIRAS for GSoC 2018 it would be great. I
> > am
> > >>> willing to help out as a mentor/ co-mentor if that is a bottleneck.
> > >>>
> > >>>
> > >>> [1] GSoC Website (https://summerofcode.withgoogle.com/)
> > >>> [2] GSoC Time Line (https://developers.google.
> > >>> com/open-source/gsoc/timeline)
> > >>> [3] GSoC 2017 JIRA (https://issues.apache.org/
> > jira/browse/SYSTEMML-1451)
> > >>>
> > >>> Regards,
> > >>> Krishna
> > >>>
> > >>>
> > >>
> >
>

Re: Passing a CoordinateMatrix to SystemML

2018-01-10 Thread Matthias Boehm

great - I'm glad to hear that. Thanks again for catching these issues
Anthony.

Regards,
Matthias

On Wed, Jan 10, 2018 at 11:09 AM, Anthony Thomas <ahtho...@eng.ucsd.edu>
wrote:

> Hey Matthias,
>
> Just wanted to confirm that patch above works for me - I'm now able to pass
> a dataframe of sparse vectors to a DML script without issue. Sorry for the
> slow confirmation on this - I've been out of the office for the last couple
> weeks. Thanks for your help debugging this!
>
> Best,
>
> Anthony
>
> On Mon, Dec 25, 2017 at 5:35 AM, Matthias Boehm <mboe...@gmail.com> wrote:
>
> > ok that was very helpful - I just pushed two additional fixes which
> should
> > resolve these issues. The underlying cause was an incorrect sparse row
> > preallocation (to reduce GC overhead), which resulted in resizing issues
> > for initial sizes of zero. These two patches fix the underlying issues,
> > make both MCSR and COO more robust for such ultra-sparse cases, and
> improve
> > the performance for converting ultra-sparse matrices. Thanks again for
> your
> > help Anthony.
> >
> > As a side note: our default block size is 1000 but converting to 1024 is
> > fine if you also set 'sysml.defaultblocksize' to 1024; otherwise there
> will
> > be an unnecessary reblock (with shuffle) from block size 1024 to 1000 on
> > the first access of this input.
> >
> > Regards,
> > Matthias
> >
> >
> > On 12/25/2017 3:07 AM, Anthony Thomas wrote:
> >
> >> Thanks Matthias - unfortunately I'm still running into an
> >> ArrayIndexOutOfBounds exception both in reading the file as IJV and when
> >> calling dataFrametoBinaryBlock. Just to confirm: I downloaded and
> compiled
> >> the latest version using:
> >>
> >> git clone https://github.com/apache/systemml
> >> cd systemml
> >> mvn clean package
> >>
> >> mvn -version
> >> Apache Maven 3.3.9
> >> Maven home: /usr/share/maven
> >> Java version: 1.8.0_151, vendor: Oracle Corporation
> >> Java home: /usr/lib/jvm/java-8-oracle/jre
> >> Default locale: en_US, platform encoding: UTF-8
> >> OS name: "linux", version: "4.4.0-103-generic", arch: "amd64", family:
> >> "unix"
> >>
> >> I have a simple driver script written in Scala which calls the API
> >> methods.
> >> I compile the script using SBT (version 1.0.4) and submit using
> >> spark-submit (spark version 2.2.0). Here's how I'm calling the methods:
> >>
> >> val x = spark.read.parquet(inputPath).select(featureNames)
> >> val mc = new MatrixCharacteristics(199563535L, 71403L, 1024,
> >> 1024,
> >> 2444225947L) // as far as I know 1024x1024 is default block size in
> sysml?
> >> println("Reading Direct")
> >> val xrdd = RDDConverterUtils.dataFrameToBinaryBlock(jsc, x, mc,
> >> false, true)
> >> xrdd.count
> >>
> >> here is the stacktrace from calling dataFrameToBinaryBlock:
> >>
> >>  java.lang.ArrayIndexOutOfBoundsException: 0
> >> at
> >> org.apache.sysml.runtime.matrix.data.SparseRowVector.append(
> >> SparseRowVector.java:196)
> >> at
> >> org.apache.sysml.runtime.matrix.data.SparseBlockMCSR.append(
> >> SparseBlockMCSR.java:267)
> >> at
> >> org.apache.sysml.runtime.matrix.data.MatrixBlock.appendValue
> >> (MatrixBlock.java:685)
> >> at
> >> org.apache.sysml.runtime.instructions.spark.utils.RDDConvert
> >> erUtils$DataFrameToBinaryBlockFunction.call(RDDConverterUtils.java:
> 1067)
> >> at
> >> org.apache.sysml.runtime.instructions.spark.utils.RDDConvert
> >> erUtils$DataFrameToBinaryBlockFunction.call(RDDConverterUtils.java:999)
> >> at
> >> org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$7$1.apply(
> >> JavaRDDLike.scala:186)
> >> at
> >> org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$7$1.apply(
> >> JavaRDDLike.scala:186)
> >> at
> >> org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$
> >> apply$23.apply(RDD.scala:797)
> >> at
> >> org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$
> >> apply$23.apply(RDD.scala:797)
> >> at
> >> org.apache.spark.rdd.MapPartitionsRDD.compute(
> MapPartitionsRDD.scala:38)
> >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:
>

Re: Can BITWISE_XOR be added. Thanks.

2018-01-01 Thread Matthias Boehm

Hi Janardhan,

sure - adding such bitwise operations is a nice addition.

There is still an open task (SYSTEMML-1931) to generalize the existing NOT,
AND, OR, and XOR to matrix arguments which should be straightforward as is
seamlessly fits into the existing binary operator. In a separate task, we
could then also add the bitwise versions of these operators. I would
recommend to stick to R's builtin function names though, to avoid
unnecessary divergence.

bitwNot(a)
bitwAnd(a, b)
bitwOr(a, b)
bitwXor(a, b)

bitwShiftL(a, n)
bitwShiftR(a, n)

Regards,
Matthias

On Mon, Jan 1, 2018 at 11:00 AM, Janardhan Pulivarthi <
janardhan.pulivar...@gmail.com> wrote:

> Hi all,
>
> I am using the following line code operation
> `
> x_tmp = bitwise_xor( v_index, as.scalar(V[v_index,]) )
> `
> Can this `bitwise_xor` be added?. I will try to take up the task, if there
> it fits into dml.
>
> Thanks,
> Janardhan
>

Re: Passing a CoordinateMatrix to SystemML

2017-12-25 Thread Matthias Boehm

AppendOnlyMap.changeValue(SizeTrackingAppendOnlyMap.scala:32)
at
org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:194)
at
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:63)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.scheduler.Task.run(Task.scala:108)
at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Best,

Anthony


On Sun, Dec 24, 2017 at 3:14 AM, Matthias Boehm <mboe...@gmail.com> wrote:


Thanks again for catching this issue Anthony - this IJV reblock issue with
large ultra-sparse matrices is now fixed in master. It likely did not show
up on the 1% sample because the data was small enough to read it directly
into the driver.

However, the dataFrameToBinaryBlock might be another issue that I could
not reproduce yet, so it would be very helpful if you could give it another
try. Thanks.

Regards,
Matthias


On 12/24/2017 9:57 AM, Matthias Boehm wrote:


Hi Anthony,

thanks for helping to debug this issue. There are no limits other than
the dimensions and number of non-zeros being of type long. It sounds
more like an issues of converting special cases of ultra-sparse
matrices. I'll try to reproduce this issue and give an update as soon as
I know more. In the meantime, could you please (a) also provide the
stacktrace of calling dataFrameToBinaryBlock with SystemML 1.0, and (b)
try calling your IJV conversion script via spark submit to exclude that
this issue is API-related? Thanks.

Regards,
Matthias

On 12/24/2017 1:40 AM, Anthony Thomas wrote:


Okay thanks for the suggestions - I upgraded to 1.0 and tried providing
dimensions and blocksizes to dataFrameToBinaryBlock both without
success. I
additionally wrote out the matrix to hdfs in IJV format and am still
getting the same error when calling "read()" directly in the DML.
However,
I created a 1% sample of the original data in IJV format and SystemML was
able to read the smaller file without any issue. This would seem to
suggest
that either there is some corruption in the full file or I'm running into
some limit. The matrix is on the larger side: 1.9e8 rows by 7e4 cols with
2.4e9 nonzero values, but this seems like it should be well within the
limits of what SystemML/Spark can handle. I also checked for obvious data
errors (file is not 1 indexed or contains blank lines). In case it's
helpful, the stacktrace from reading the data from hdfs in IJV format is
attached. Thanks again for your help - I really appreciate it.

 00:24:18 WARN TaskSetManager: Lost task 30.0 in stage 0.0 (TID 126,
10.11.10.13, executor 0): java.lang.ArrayIndexOutOfBoundsException
at java.lang.System.arraycopy(Native Method)
at
org.apache.sysml.runtime.matrix.data.SparseBlockCOO.shiftRig
htByN(SparseBlockCOO.java:594)

at
org.apache.sysml.runtime.matrix.data.SparseBlockCOO.set(
SparseBlockCOO.java:323)

at
org.apache.sysml.runtime.matrix.data.MatrixBlock.mergeIntoSp
arse(MatrixBlock.java:1790)

at
org.apache.sysml.runtime.matrix.data.MatrixBlock.merge(Matri
xBlock.java:1736)

at
org.apache.sysml.runtime.instructions.spark.utils.RDDAggrega
teUtils$MergeBlocksFunction.call(RDDAggregateUtils.java:627)

at
org.apache.sysml.runtime.instructions.spark.utils.RDDAggrega
teUtils$MergeBlocksFunction.call(RDDAggregateUtils.java:596)

at
org.apache.spark.api.java.JavaPairRDD$$anonfun$toScalaFuncti
on2$1.apply(JavaPairRDD.scala:1037)

at
org.apache.spark.util.collection.ExternalSorter$$anonfun$5.
apply(ExternalSorter.scala:189)

at
org.apache.spark.util.collection.ExternalSorter$$anonfun$5.
apply(ExternalSorter.scala:188)

at
org.apache.spark.util.collection.AppendOnlyMap.changeValue(
AppendOnlyMap.scala:150)

at
org.apache.spark.util.collection.SizeTrackingAppendOnlyMap.c
hangeValue(SizeTrackingAppendOnlyMap.scala:32)

at
org.apache.spark.util.collection.ExternalSorter.insertAll(
ExternalSorter.scala:194)

at
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortSh
uffleWriter.scala:63)

at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMap
Task.scala:96)

at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMap
Task.scala:53)

at org.apache.spark.scheduler.Task.run(Task.scala:108)
at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
Executor.java:1149)

at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
lExecutor.java:624)

at java.lang.Thread.run(Threa

Re: Passing a CoordinateMatrix to SystemML

2017-12-24 Thread Matthias Boehm

Hi Anthony,

thanks for helping to debug this issue. There are no limits other than
the dimensions and number of non-zeros being of type long. It sounds
more like an issues of converting special cases of ultra-sparse
matrices. I'll try to reproduce this issue and give an update as soon as
I know more. In the meantime, could you please (a) also provide the
stacktrace of calling dataFrameToBinaryBlock with SystemML 1.0, and (b)
try calling your IJV conversion script via spark submit to exclude that
this issue is API-related? Thanks.

Regards,
Matthias

On 12/24/2017 1:40 AM, Anthony Thomas wrote:

Okay thanks for the suggestions - I upgraded to 1.0 and tried providing
dimensions and blocksizes to dataFrameToBinaryBlock both without success. I
additionally wrote out the matrix to hdfs in IJV format and am still
getting the same error when calling "read()" directly in the DML. However,
I created a 1% sample of the original data in IJV format and SystemML was
able to read the smaller file without any issue. This would seem to suggest
that either there is some corruption in the full file or I'm running into
some limit. The matrix is on the larger side: 1.9e8 rows by 7e4 cols with
2.4e9 nonzero values, but this seems like it should be well within the
limits of what SystemML/Spark can handle. I also checked for obvious data
errors (file is not 1 indexed or contains blank lines). In case it's
helpful, the stacktrace from reading the data from hdfs in IJV format is
attached. Thanks again for your help - I really appreciate it.

00:24:18 WARN TaskSetManager: Lost task 30.0 in stage 0.0 (TID 126,
10.11.10.13, executor 0): java.lang.ArrayIndexOutOfBoundsException
at java.lang.System.arraycopy(Native Method)
at
org.apache.sysml.runtime.matrix.data.SparseBlockCOO.shiftRightByN(SparseBlockCOO.java:594)
at
org.apache.sysml.runtime.matrix.data.SparseBlockCOO.set(SparseBlockCOO.java:323)
at
org.apache.sysml.runtime.matrix.data.MatrixBlock.mergeIntoSparse(MatrixBlock.java:1790)
at
org.apache.sysml.runtime.matrix.data.MatrixBlock.merge(MatrixBlock.java:1736)
at
org.apache.sysml.runtime.instructions.spark.utils.RDDAggregateUtils$MergeBlocksFunction.call(RDDAggregateUtils.java:627)
at
org.apache.sysml.runtime.instructions.spark.utils.RDDAggregateUtils$MergeBlocksFunction.call(RDDAggregateUtils.java:596)
at
org.apache.spark.api.java.JavaPairRDD$$anonfun$toScalaFunction2$1.apply(JavaPairRDD.scala:1037)
at
org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:189)
at
org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:188)
at
org.apache.spark.util.collection.AppendOnlyMap.changeValue(AppendOnlyMap.scala:150)
at
org.apache.spark.util.collection.SizeTrackingAppendOnlyMap.changeValue(SizeTrackingAppendOnlyMap.scala:32)
at
org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:194)
at
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:63)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.scheduler.Task.run(Task.scala:108)
at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Anthony

On Sat, Dec 23, 2017 at 4:27 AM, Matthias Boehm <mboe...@gmail.com> wrote:

Given the line numbers from the stacktrace, it seems that you use a rather
old version of SystemML. Hence, I would recommend to upgrade to SystemML
1.0 or at least 0.15 first.

If the error persists or you're not able to upgrade, please try to call
dataFrameToBinaryBlock with provided matrix characteristics of dimensions
and blocksizes. The issue you've shown usually originates from incorrect
meta data (e.g., negative number of columns or block sizes), which prevents
the sparse rows from growing to the necessary sizes.

Regards,
Matthias

On 12/22/2017 10:42 PM, Anthony Thomas wrote:

Hi Matthias,

Thanks for the help! In response to your questions:

1. Sorry - this was a typo: the correct schema is: [y: int, features:
vector] - the column "features" was created using Spark's
VectorAssembler
and the underlying type is an org.apache.spark.ml.linalg.SparseVector.
Calling x.schema results in: org.apache.spark.sql.types.StructType =
StructType(StructField(features,org.apache.spark.ml.
linalg.VectorUDT@3bfc3ba7,true)
2. "y" converts fine - it appears the only issue is with X. The script
still crashes when running "print(sum(X))". The full stack trace is
attached at the end

Re: Passing a CoordinateMatrix to SystemML

2017-12-23 Thread Matthias Boehm

thodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(
DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$
deploy$SparkSubmit$$runMain(SparkSubmit.scala:755)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(
SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.sysml.api.mlcontext.MLContextException: Exception
occurred while executing runtime program
at org.apache.sysml.api.mlcontext.ScriptExecutor.executeRuntimeProgram(
ScriptExecutor.java:390)
at org.apache.sysml.api.mlcontext.ScriptExecutor.
execute(ScriptExecutor.java:298)
at org.apache.sysml.api.mlcontext.MLContext.execute(MLContext.java:303)
... 14 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException:
org.apache.sysml.runtime.DMLRuntimeException: ERROR: Runtime error in
program block generated from statement block between lines 1 and 2 -- Error
evaluating instruction: CP°uak+°X·MATRIX·DOUBLE°_Var0·SCALAR·STRING°24
at org.apache.sysml.runtime.controlprogram.Program.
execute(Program.java:130)
at org.apache.sysml.api.mlcontext.ScriptExecutor.executeRuntimeProgram(
ScriptExecutor.java:388)
... 16 more
...



On Fri, Dec 22, 2017 at 5:48 AM, Matthias Boehm <mboe...@gmail.com> wrote:


well, let's do the following to figure this out:

1) If the schema is indeed [label: Integer, features: SparseVector],
please change the third line to val y = input_data.select("label").

2) For debugging, I would recommend to use a simple script like
"print(sum(X));" and try converting X and y separately to isolate the
problem.

3) If it's still failing, it would be helpful to known (a) if it's an
issue of converting X, y, or both, as well as (b) the full stacktrace.

4) As a workaround you might also call our internal converter directly via:
RDDConverterUtils.dataFrameToBinaryBlock(jsc, df, mc, containsID,
isVector),
where jsc is the java spark context, df is the dataset, mc are matrix
characteristics (if unknown, simply use new MatrixCharacteristics()),
containsID indicates if the dataset contains a column "__INDEX" with the
row indexes, and isVector indicates if the passed datasets contains vectors
or basic types such as double.


Regards,
Matthias


On 12/22/2017 12:00 AM, Anthony Thomas wrote:


Hi SystemML folks,

I'm trying to pass some data from Spark to a DML script via the MLContext
API. The data is derived from a parquet file containing a dataframe with
the schema: [label: Integer, features: SparseVector]. I am doing the
following:

val input_data = spark.read.parquet(inputPath)
val x = input_data.select("features")
val y = input_data.select("y")
val x_meta = new MatrixMetadata(DF_VECTOR)
val y_meta = new MatrixMetadata(DF_DOUBLES)
val script = dmlFromFile(s"${script_path}/script.dml").
in("X", x, x_meta).
in("Y", y, y_meta)
...

However, this results in an error from SystemML:
java.lang.ArrayIndexOutOfBoundsException: 0
I'm guessing this has something to do with SparkML being zero indexed and
SystemML being 1 indexed. Is there something I should be doing differently
here? Note that I also tried converting the dataframe to a
CoordinateMatrix
and then creating an RDD[String] in IJV format. That too resulted in
"ArrayIndexOutOfBoundsExceptions." I'm guessing there's something simple
I'm doing wrong here, but I haven't been able to figure out exactly what.
Please let me know if you need more information (I can send along the full
error stacktrace if that would be helpful)!

Thanks,

Anthony

Re: Passing a CoordinateMatrix to SystemML

2017-12-22 Thread Matthias Boehm


well, let's do the following to figure this out:

1) If the schema is indeed [label: Integer, features: SparseVector], 
please change the third line to val y = input_data.select("label").


2) For debugging, I would recommend to use a simple script like 
"print(sum(X));" and try converting X and y separately to isolate the 
problem.


3) If it's still failing, it would be helpful to known (a) if it's an 
issue of converting X, y, or both, as well as (b) the full stacktrace.


4) As a workaround you might also call our internal converter directly via:
RDDConverterUtils.dataFrameToBinaryBlock(jsc, df, mc, containsID, 
isVector),
where jsc is the java spark context, df is the dataset, mc are matrix 
characteristics (if unknown, simply use new MatrixCharacteristics()), 
containsID indicates if the dataset contains a column "__INDEX" with the 
row indexes, and isVector indicates if the passed datasets contains 
vectors or basic types such as double.



Regards,
Matthias

On 12/22/2017 12:00 AM, Anthony Thomas wrote:

Hi SystemML folks,

I'm trying to pass some data from Spark to a DML script via the MLContext
API. The data is derived from a parquet file containing a dataframe with
the schema: [label: Integer, features: SparseVector]. I am doing the
following:

val input_data = spark.read.parquet(inputPath)
val x = input_data.select("features")
val y = input_data.select("y")
val x_meta = new MatrixMetadata(DF_VECTOR)
val y_meta = new MatrixMetadata(DF_DOUBLES)
val script = dmlFromFile(s"${script_path}/script.dml").
in("X", x, x_meta).
in("Y", y, y_meta)
...

However, this results in an error from SystemML:
java.lang.ArrayIndexOutOfBoundsException: 0
I'm guessing this has something to do with SparkML being zero indexed and
SystemML being 1 indexed. Is there something I should be doing differently
here? Note that I also tried converting the dataframe to a CoordinateMatrix
and then creating an RDD[String] in IJV format. That too resulted in
"ArrayIndexOutOfBoundsExceptions." I'm guessing there's something simple
I'm doing wrong here, but I haven't been able to figure out exactly what.
Please let me know if you need more information (I can send along the full
error stacktrace if that would be helpful)!

Thanks,

Anthony

Re: Jenkins build became unstable: SystemML-DailyTest #1424

2017-12-17 Thread Matthias Boehm

yes, that's indeed the case. The original PR was fine but I introduced 
this issue during some final cleanups while merging this PR.


Regards,
Matthias

On 12/17/2017 11:49 PM, Ted Yu wrote:

XorTest failure seems to be related to:

[SYSTEMML-1883] New xor builtin functions over scalars


On Sun, Dec 17, 2017 at 2:19 PM,  wrote:


See

Re: [VOTE] Apache SystemML 1.0.0 (RC2)

2017-12-12 Thread Matthias Boehm

 >
> > > "Glenn Weidner" ---12/11/2017 09:49:48 AM---+1 I ran Linear Regression,
> > > Logistic Regression, SVM, Naive Bayes Python tests
> > >
> > > From: "Glenn Weidner" <gweid...@us.ibm.com>
> > > To: dev@systemml.apache.org
> > > Date: 12/11/2017 09:49 AM
> > > Subject: Re: [VOTE] Apache SystemML 1.0.0 (RC2)
> > > --
> > >
> > >
> > >
> > > +1
> > >
> > > I ran Linear Regression, Logistic Regression, SVM, Naive Bayes Python
> > > tests with Spark 2.1 in cloud environment. All tests passed with Python
> > 2.7.
> > >
> > > Note I also attempted to repeat the tests with Python 3.5
> 'experimental'
> > > kernel but encountered issue:
> > >
> > > .local/lib/python3.5/site-packages/systemml/mllearn/estimators.py",
> line
> > > 887
> > >  def __init__(self, sparkSession, keras_model, input_shape,
> > > transferUsingDF=False, weights=None, labels=None):
> > > SyntaxError: import * only allowed at module level"
> > >
> > > I don't believe this is release blocking issue but may want to discuss
> if
> > > other problems reported during RC2 validation.
> > >
> > > Regards,
> > > Glenn
> > >
> > > Matthias Boehm ---12/09/2017 03:13:43 PM---+1 I ran the perftest suite
> > > with the artifact on Spark 2.2 up to 80GB without
> > >
> > > From: Matthias Boehm <mboe...@gmail.com>
> > > To: dev@systemml.apache.org
> > > Date: 12/09/2017 03:13 PM
> > > Subject: Re: [VOTE] Apache SystemML 1.0.0 (RC2)
> > > --
> > >
> > >
> > >
> > > +1
> > >
> > > I ran the perftest suite with the artifact on Spark 2.2 up to 80GB
> > without
> > > any failures or performance issues. On earlier versions, I also ran the
> > > perftest suite with Spark 2.1 and 2.2, w/ and w/o codegen, and w/ auto
> > > compression up to 800GB without remaining issues.
> > >
> > > As a minor nitpick (to be resolved in any additional RCs or the next
> > > release), the root directory of the artifact includes a
> > SystemML-config.xml
> > > file that only has a small subset of configuration parameters. We might
> > > want to either delete this or pull in a copy of
> > > SystemML-config.xml.template from the conf directory.
> > >
> > > Regards,
> > > Matthias
> > >
> > > On Sat, Dec 9, 2017 at 1:53 AM, Luciano Resende <luckbr1...@gmail.com>
> > > wrote:
> > >
> > > > On Fri, Dec 8, 2017 at 12:15 PM, Berthold Reinwald <
> > reinw...@us.ibm.com>
> > > > wrote:
> > > >
> > > > > Please vote on releasing the following candidate as Apache SystemML
> > > > > version 1.0.0
> > > > >
> > > > > The vote is open for at least 72 hours and passes if a majority of
> at
> > > > > least 3 +1 PMC votes are cast.
> > > > >
> > > > > [ ] +1 Release this package as Apache SystemML 1.0.0
> > > > > [ ] -1 Do not release this package because ...
> > > > >
> > > > > To learn more about Apache SystemML, please see
> > > > >
> > > *https://urldefense.proofpoint.com/v2/url?u=http-
> > 3A__systemml.apache.org_=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=
> > we2L9Q2NOeniL5PbfdrO3CmQPNy_vnhoDpPDviJpnrU=Ud62VUVZNsp0I0PlFMRr-
> > OOvxUC5pssmMZdUB4I3g5w=KKHMDbNiFLfVSqUYXqIZLGxx-yif5PAGAQQLnnMfXR8=*
> > > <https://urldefense.proofpoint.com/v2/url?u=http-
> > 3A__systemml.apache.org_=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=
> > we2L9Q2NOeniL5PbfdrO3CmQPNy_vnhoDpPDviJpnrU=Ud62VUVZNsp0I0PlFMRr-
> > OOvxUC5pssmMZdUB4I3g5w=KKHMDbNiFLfVSqUYXqIZLGxx-yif5PAGAQQLnnMfXR8=>
> > > > >
> > > > > The tag to be voted on is v1.0.0-rc2 (
> > > > > 7b44380e49425ac20912c2d42de8603a06fa55a1):
> > > > >
> > > *https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__github.com_apache_
> > systemml_commit_7b44380e49425ac20912c2d42de860
> =DwIBaQ=jf_iaSHvJObTbx-
> > siA1ZOg=we2L9Q2NOeniL5PbfdrO3CmQPNy_vnhoDpPDviJpnrU=
> > Ud62VUVZNsp0I0PlFMRr-OOvxUC5pssmMZdUB4I3g5w=k8A_
> > OU7p9Wa4sKxpiY5v9gzbLrJg8rGTSiSEZ-hFgTA=*
> > > <https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__github.com_apache_
> > systemml_commit_7b44380e49425ac20912c2d42de860
> =DwIBaQ=jf_iaSHvJObTbx-
&g

Re: [VOTE] Apache SystemML 1.0.0 (RC2)

2017-12-09 Thread Matthias Boehm

+1

I ran the perftest suite with the artifact on Spark 2.2 up to 80GB without
any failures or performance issues. On earlier versions, I also ran the
perftest suite with Spark 2.1 and 2.2, w/ and w/o codegen, and w/ auto
compression up to 800GB without remaining issues.

As a minor nitpick (to be resolved in any additional RCs or the next
release), the root directory of the artifact includes a SystemML-config.xml
file that only has a small subset of configuration parameters. We might
want to either delete this or pull in a copy of
SystemML-config.xml.template from the conf directory.

Regards,
Matthias

On Sat, Dec 9, 2017 at 1:53 AM, Luciano Resende 
wrote:

> On Fri, Dec 8, 2017 at 12:15 PM, Berthold Reinwald 
> wrote:
>
> > Please vote on releasing the following candidate as Apache SystemML
> > version 1.0.0
> >
> > The vote is open for at least 72 hours and passes if a majority of at
> > least 3 +1 PMC votes are cast.
> >
> > [ ] +1 Release this package as Apache SystemML 1.0.0
> > [ ] -1 Do not release this package because ...
> >
> > To learn more about Apache SystemML, please see
> > http://systemml.apache.org/
> >
> > The tag to be voted on is v1.0.0-rc2 (
> > 7b44380e49425ac20912c2d42de8603a06fa55a1):
> > https://github.com/apache/systemml/commit/7b44380e49425ac20912c2d42de860
> > 3a06fa55a1
> >
> >
> Looks like the tag hash should be
> 6519f3383f8fd43af93ecba8742ae4b2b28d4b35
>
> And the link
> https://github.com/apache/systemml/tree/6519f3383f8fd43af93ecba8742ae4
> b2b28d4b35
>
>
> >
> > The release artifacts can be found at:
> > https://dist.apache.org/repos/dist/dev/systemml/1.0.0-rc2/
> >
> >
> > The maven release artifacts, including signatures, digests, etc. can be
> > found at:
> > https://repository.apache.org/content/repositories/
> > orgapachesystemml-1026/org/apache/systemml/systemml/1.0.0/
> >
> >
> >
> > ===
> > == Apache Release policy ==
> > ===
> > http://www.apache.org/legal/release-policy.html
> >
> > ===
> > == How can I help test this release? ==
> > ===
> > If you are a SystemML user, you can help us test this release by taking
> an
> > existing Algorithm or workload and running on this release candidate,
> then
> > reporting any regressions.
> >
> > 
> > == What justifies a -1 vote for this release? ==
> > 
> > -1 votes should only occur for significant stop-ship bugs or legal
> related
> > issues (e.g. wrong license, missing header files, etc). Minor bugs or
> > regressions should not block this release.
> >
> >
>
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>

[DISCUSS] Roadmap SystemML 1.1 and beyond

2017-12-08 Thread Matthias Boehm

Hi all,

with our SystemML 1.0 release around the corner, I think we should start
the discussion on the roadmap for SystemML 1.1 and beyond. Below is an
initial list as a starting point, but please help to add relevant items,
especially for algorithms and APIs, which are barely covered so far.

1) Deep Learning
 * Full compiler integration GPU backend
 * Extended sparse operations on CPU/GPU
 * Extended single-precision support CPU
 * Distributed DL operations?

2) GPU Backend
 * Full support for sparse operations
 * Automatic decisions on CPU vs GPU operations
 * Graduate GPU backends (enable by default)

3) Code generation
 * Graduate code generation (enable by default)
 * Support for deep learning operations
 * Code generation for the heterogeneous HW, incl GPUs

4) Compressed Linear Algebra
 * Support for matrix-matrix multiplications
 * Support for deep learning operations
 * Improvements for ultra-sparse datasets

5) Misc Runtime
 * Large dense matrix blocks > 16GB
 * NUMA-awareness (thread pools, matrix partitioning)
 * Unified memory management (ops, bufferpool, RDDs/broadcasts)
 * Support feather format for matrices and frames
 * Parfor support for broadcasts
 * Extended support for multi-threaded operations
 * Boolean matrices

6) Misc Compiler
 * Support single-output UDFs in expressions
 * Consolidate replicated compilation chain (e.g., diff APIs)
 * Holistic sum-product optimization and operator fusion
 * Extended sparsity estimators
 * Rewrites and compiler improvements for mini-batching
 * Parfor optimizer support for shared reads

7) APIs
 * Python Binding for JMLC API
 * Consistency Python/Java APIs


Regards,
Matthias

Re: [VOTE] Apache SystemML 1.0.0 (RC1)

2017-12-07 Thread Matthias Boehm

-1 due to the issue mentioned by Niketan, as well as additional correctness
and performance issues fixed in the last couple of days.

Regards,
Matthias

On Tue, Dec 5, 2017 at 6:25 PM, Niketan Pansare  wrote:

> Soft -1 as GPU backend is in experimental mode. GPU matrix multiplication
> tests are failing. I plan to investigate and provide a fix tomorrow and we
> can cut a new RC then if it’s okay with others.
>
> > On Dec 5, 2017, at 2:47 PM, Berthold Reinwald 
> wrote:
> >
> > Please vote on releasing the following candidate as Apache SystemML
> > version
> > 1.0.0
> >
> > The vote is open for at least 72 hours and passes if a majority of at
> > least 3
> > +1 PMC votes are cast.
> >
> > [ ] +1 Release this package as Apache SystemML 0.15.0
> > [ ] -1 Do not release this package because ...
> >
> > To learn more about Apache SystemML, please see
> > https://urldefense.proofpoint.com/v2/url?u=http-3A__
> systemml.apache.org_=DwIFAg=jf_iaSHvJObTbx-siA1ZOg=
> HzVC6v79boGYQrpc383_Kao_6a6SaOkZrfiSrYZVby0=
> CRYYijXtRUSeHXzFAqhy0Yip2SFV_01vKhzCh3wrM7A=YvMCM0FPl8a6Se9aPwDU1-
> 8ImB6khJCUfZUQ1cyY70E=
> >
> > The tag to be voted on is v1.0.0-rc1 (
> > 054b85efdec92b8f66a125e014798076843e3492)
> >
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.
> com_apache_systemml_commit_054b85efdec92b8f66a125e0147980
> 76843e3492=DwIFAg=jf_iaSHvJObTbx-siA1ZOg=HzVC6v79boGYQrpc383_Kao_
> 6a6SaOkZrfiSrYZVby0=CRYYijXtRUSeHXzFAqhy0Yip2SFV_01vKhzCh3wrM7A=
> giFqvN1rtluEMc7KH7fQoMiW-7DO_jul_lqhOjPgIAU=
> >
> >
> >
> > The release artifacts can be found at :
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__dist.
> apache.org_repos_dist_dev_systemml_1.0.0-2Drc1_=DwIFAg=jf_iaSHvJObTbx-
> siA1ZOg=HzVC6v79boGYQrpc383_Kao_6a6SaOkZrfiSrYZVby0=
> CRYYijXtRUSeHXzFAqhy0Yip2SFV_01vKhzCh3wrM7A=
> y1myoSMN6i7ZC8WrJ1awUV97YJnHyYZAq8DpFCQN9Yg=
> >
> >
> > The maven release artifacts, including signatures, digests, etc. can
> > be found
> > at:
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__
> repository.apache.org_content_repositories_orgapachesystemml-2D1025_org_
> apache_systemml_systemml_1.0.0_=DwIFAg=jf_iaSHvJObTbx-
> siA1ZOg=HzVC6v79boGYQrpc383_Kao_6a6SaOkZrfiSrYZVby0=
> CRYYijXtRUSeHXzFAqhy0Yip2SFV_01vKhzCh3wrM7A=
> rxd0qIMgOBJ9XXQFm7oVgNpeUZSGosIDjyaGvVgUk1k=
> >
> >
> > =
> > == Apache Incubator release policy ==
> > =
> > Please find below the guide to release management during incubation:
> > https://urldefense.proofpoint.com/v2/url?u=http-3A__
> incubator.apache.org_guides_releasemanagement.html=
> DwIFAg=jf_iaSHvJObTbx-siA1ZOg=HzVC6v79boGYQrpc383_
> Kao_6a6SaOkZrfiSrYZVby0=CRYYijXtRUSeHXzFAqhy0Yip2SFV_01vKhzCh3wrM7A=
> glwAJClY6tTEZwAipXnNunNRDfrZBS8ctGKPtkH71qY=
> >
> > ===
> > == How can I help test this release? ==
> > ===
> > If you are a SystemML user, you can help us test this release by taking
> > an
> > existing Algorithm or workload and running on this release candidate,
> then
> > reporting any regressions.
> >
> > 
> > == What justifies a -1 vote for this release? ==
> > 
> > -1 votes should only occur for significant stop-ship bugs or legal
> > related
> > issues (e.g. wrong license, missing header files, etc). Minor bugs or
> > regressions should not block this release.
> >
> >
> > Regards,
> > Berthold Reinwald
> > IBM Almaden Research Center
> > office: (408) 927 2208; T/L: 457 2208
> > e-mail: reinw...@us.ibm.com
> >
>
>

Re: SystemML 1.0 release timeline

2017-12-01 Thread Matthias Boehm

After multiple runs of our perftest suite over 80MB-800GB with Spark 2.1
and 2.2, with and without codegen, as well as with and without compression,
we found and fixed quite a number of issues. However, there are a couple of
remaining performance issues with sparse maxpooling operations, so I would
recommend to defer the RC1 for a couple of days until these issues are
fixed as well.

Regards,
Matthias

On Tue, Nov 14, 2017 at 1:31 PM, Krishna Kalyan <krishnakaly...@gmail.com>
wrote:

> +1
>
> Regards,
> Krishna
>
> On Sun, Nov 12, 2017 at 5:53 AM, Matthias Boehm <mboe...@gmail.com> wrote:
>
> > just FYI: I've created SYSTEMML-2011 to track all QA-related tasks and
> > issues. The first run of our (old) perftest suite for 80MB-80GB ran fine
> > except for a data generation issue on stratstats for the 80GB case. Btw,
> I
> > think it would be good to run both the old and new perftest for
> additional
> > validation.
> >
> > Regards,
> > Matthias
> >
> > On Wed, Nov 8, 2017 at 8:30 AM, Glenn Weidner <gweid...@us.ibm.com>
> wrote:
> >
> > > +1
> > >
> > > Regards,
> > > Glenn
> > >
> > > [image: Inactive hide details for "Niketan Pansare" ---11/08/2017
> > 06:51:44
> > > AM---+1. Thanks,]"Niketan Pansare" ---11/08/2017 06:51:44 AM---+1.
> > Thanks,
> > >
> > > From: "Niketan Pansare" <npan...@us.ibm.com>
> > > To: dev@systemml.apache.org
> > > Date: 11/08/2017 06:51 AM
> > > Subject: Re: SystemML 1.0 release timeline
> > > --
> > >
> > >
> > >
> > >
> > > +1.
> > >
> > > Thanks,
> > >
> > > Niketan.
> > >
> > > > On Nov 7, 2017, at 10:50 PM, Matthias Boehm <mboe...@googlemail.com>
> > > wrote:
> > > >
> > > > Hi all,
> > > >
> > > > we made some good progress regarding deep learning support, code
> > > > generation, and low-latency scoring - so, I'm looking forward to our
> > > > upcoming 1.0 release. Since it's our first stable release, I think it
> > > would
> > > > be a good idea to allocate some extra time for QA. How about we shoot
> > for
> > > a
> > > > release candidate Dec 1?
> > > >
> > > > Regards,
> > > > Matthias
> > >
> > >
> > >
> > >
> >
>

Re: Distribution functions such as gamma, weibull etc.

2017-11-13 Thread Matthias Boehm

unfortunately, our cdf and invcdf currently only support the distributions
normal, exp, chisq, f, and t and scalar inputs. So you would have to
emulate this at script level. Extending the list of distribution functions
and adding matrix support would be a good addition though.

Regards,
Matthias

On Mon, Nov 13, 2017 at 4:51 AM, Janardhan Pulivarthi <
janardhan.pulivar...@gmail.com> wrote:

> Hi,
>
> I am wondering, whether we have builtin functions for distributions such as
> weibull, laplace, gamma. (looking at some top level algorithms, says that
> we do have!).
>
> Would like to use in the following case:
>
> ```
> lp = -gamma(k) - k*log(t) + (k-1)*lx - x/t;
>
> ```
>
> Thanks,
> Janardhan
>

SystemML 1.0 release timeline

2017-11-07 Thread Matthias Boehm

Hi all,

we made some good progress regarding deep learning support, code
generation, and low-latency scoring - so, I'm looking forward to our
upcoming 1.0 release. Since it's our first stable release, I think it would
be a good idea to allocate some extra time for QA. How about we shoot for a
release candidate Dec 1?

Regards,
Matthias

SystemML 1.0 release timeline

2017-11-07 Thread Matthias Boehm

Hi all,

we made some good progress regarding deep learning support, code
generation, and low-latency scoring - so, I'm looking forward to our
upcoming 1.0 release. Since it's our first stable release, I think it would
be a good idea to allocate some extra time for QA. How about we shoot for a
release candidate Dec 1?

Regards,
Matthias

Re: My life made easier, now!

2017-10-30 Thread Matthias Boehm

Janardhan, could you please elaborate a little what issues you faced?
SystemML itself does not require any specific installation. Also to be
productive, you might want to setup a dev environment, where you can run
tests locally directly from your IDE.

Regards,
Matthias

On Mon, Oct 30, 2017 at 4:59 AM, Janardhan Pulivarthi <
janardhan.pulivar...@gmail.com> wrote:

> Hi all,
>
> After 5 months of struggle for installation of Apache Systemml ( After,
> more than 150 failed installations...), I finally, got this message
> "Welcome to Apache SystemML!".
>
>
> 
> Now, I am not going to bother our Jenkins for testing my patches.
>
> Thanks everyone,
> Janardhan
>

Re: Get plans before and after rewrites

2017-10-25 Thread Matthias Boehm

Hi Nantia,

sure, let me clarify these two statements:

1) Optimization level: When I said O1, I was referring to SystemML's
configurable optimization level 1, which can be set through
'sysml.optlevel' (or 'optlevel' in SystemML 0.14) in a SystemML-config.xml
file or via setConfigProperty in MLContext. By default we use O2 with all
rewrites enabled, in O1, certain rewrites are disabled, which allows
comparing plans with and without rewrites. Furthermore, the quoted
statement below was more of a note for myself, to cleanup this handling of
remaining rewrites (that are not conditionally disabled yet) in an upcoming
SystemML release.

2) Log4j Configuration: You can take the log4j properties file from our
conf directory, put it into your classpath, and append the following line
to this file (alternatively with TRACE instead of DEBUG if you want to see
more details):
log4j.logger.org.apache.sysml.hops.rewrite=DEBUG

Regards,
Matthias

On Wed, Oct 25, 2017 at 4:24 AM, Nantia Makrynioti <nantiam...@gmail.com>
wrote:

> Hello Matthias,
>
> Thanks a lot for replying and sorry for my late response.
>
> I have two more questions regarding the steps you described.
>
>  I think it's a
> > good idea to clean this up (disable these remaining rewrites in O1 as
> > well), which would be useful for debugging.
>
>
> I don't understand what you mean by O1. So how can I disable these
> remaining rewrites?
>
> If you're debugging any specific issue and want to see which rewrites
> > trigger where, you can set the log level for package
> > 'org.apache.sysml.hops.rewrite' to DEBUG or TRACE in your log4j
> > configuration.
>
>
> I am working with SystemML 0.14.0. There is a log4j.properties file, but I
> cannot find anything relevant to  'org.apache.sysml.hops.rewrite'. Is
> there
> another file I should check?
>
> Thanks again,
> Nantia
>
> 2017-10-13 23:29 GMT+03:00 Matthias Boehm <mboe...@googlemail.com>:
>
> > Hi Nantia,
> >
> > in optimization level 1, we disable the following rewrites and the
> explain
> > hops or runtime output will show the resulting plan:
> > * Disable common-subexpression elimination
> > * Disable algebraic simplifications (static and dynamic)
> > * Disable inter-procedural analysis
> > * Disable branch removal and statement block merge
> > * Disable sum-product rewrites
> > * Disable update-in-place rewrites
> >
> > So, to compare plans without and with rewrites, you would need to run
> with
> > optimization level 1 and 2 (our default), and capture & compare these
> > outputs.
> >
> > Note that there are a number of rewrites such as matrix multiplication
> > chain optimization, operator selection, hop-lop rewrites that are always
> > applied and thus not affected by the optimization level. I think it's a
> > good idea to clean this up (disable these remaining rewrites in O1 as
> > well), which would be useful for debugging.
> >
> > If you're debugging any specific issue and want to see which rewrites
> > trigger where, you can set the log level for package
> > 'org.apache.sysml.hops.rewrite' to DEBUG or TRACE in your log4j
> > configuration. If you're compiling from sources, you can also simply set
> > ProgramRewriter.LDEBUG to true to do the same thing.
> >
> > Regards,
> > Matthias
> >
> > On Fri, Oct 13, 2017 at 2:04 AM, Nantia Makrynioti <nantiam...@gmail.com
> >
> > wrote:
> >
> > > Hello,
> > >
> > > I set optimization level to 1 in SystemML-config.xml, in order to get
> HOP
> > > and LOP plans before and after rewrites. However, I am still getting
> > just a
> > > single plan.
> > >
> > > I am using spark shell to execute the dml script.
> > >
> > > Thank you in advance,
> > > Nantia
> > >
> >
>

Re: Jenkins build is still unstable: SystemML-DailyTest #1297

2017-10-15 Thread Matthias Boehm

thanks Ted - this was my fault. A recent change toward a more aggressive
use of CSR revealed a number of hidden issues, which I already fixed last
night - so, the next build should be fine.

Regards,
Matthias

On Sun, Oct 15, 2017 at 6:36 AM, Ted Yu  wrote:

> *There seems to be some regression in LinearLogRegPyDMLTest.In build
> #1297:*
>
> *02:17:27* message: org.apache.sysml.runtime.DMLRuntimeException:
> org.apache.sysml.runtime.DMLRuntimeException: ERROR: Runtime error in
> program block generated from statement block between lines 32 and 0 --
> Error evaluating instruction:
> CP°r'°_mVar3898498·MATRIX·DOUBLE°_mVar3898500·MATRIX·DOUBLE°24
>
>
> In build #1296:
>
>
> *14:21:14* testLinearLogRegPyDml[2](org.apache.sysml.test.integration.
> applications.pydml.LinearLogRegPyDMLTest)
>  Time elapsed: 2.75 sec  <<< FAILURE!*14:21:14*
> java.lang.AssertionError: Detailed matrices characteristics:*14:21:14*
> !  wR<->wSYSTEMML # stored values in wSYSTEMML: 750*14:21:14* !
> wR<->wSYSTEMML # stored values in wR: 750*14:21:14* !  wR<->wSYSTEMML
> identical values(z=0): 0*14:21:14* !  wR<->wSYSTEMML wrong
> values(z=1.0E-14): 750*14:21:14* !  wR<->wSYSTEMML min error:
> 0.04402076708102141*14:21:14* !  wR<->wSYSTEMML max error:
> 0.08186434116577723
>
>
> On Sun, Oct 15, 2017 at 2:17 AM,  wrote:
>
> > See  >
> >
> >
>

Re: Minor script changes for SVM with `MLContext`, `spark_submit` etc.

2017-10-04 Thread Matthias Boehm

as mentioned on PR-673, I'm probably not the right person to comment on
algorithm or API-related changes, but I'll try to have a look tomorrow.

Regards,
Matthias


On Tue, Oct 3, 2017 at 6:52 AM, Janardhan Pulivarthi <
janardhan.pulivar...@gmail.com> wrote:

> Hi Matthias,
>
> Based on your comment here[https://github.com/apache/systemml/pull/529#
> issuecomment-316253791], I've updated the SVM scripts(I believe).
>
> Can you please have a look at whether the changeset[https://github.com/
> apache/systemml/pull/673/files] needs to be updated. :)
>
> Thanks,
> Janardhan
>

Re: file location for `import org.apache.sysml.parser.dml.DmlParser.WhileStatementContext`

2017-09-23 Thread Matthias Boehm

I guess this is a just a delayed message, right? Ted was right, it's a
generated class, and hence not in the repo. However, when you build
SystemML, the generated source ends up in the src directory of the
org.apache.sysml.parser.dml package as well.

Note that for new builtin functions such as xor(), there is no need to
extend the parser at all. You just need to handle your function in the
right expression. All other boolean operators are in BooleanExpression but
the parsing is specific to custom operators. Hence, please extend
BuiltinFunctionExpression to simply handle and validate your xor function.
At the end of the day, all these are eventually mapped to the hops UnaryOp
(for not) and BinaryOp otherwise.

Regards,
Matthias

On Thu, Sep 21, 2017 at 8:49 AM, Janardhan Pulivarthi <
janardhan.pulivar...@gmail.com> wrote:

> Hi,
>
> Can somebody point me out the `DmlParser` class?
>
> Thanks.
>

Re: [DISCUSS] R-Interface to SystemML

2017-09-21 Thread Matthias Boehm

I pretty much agree with Niketan and Deron. In general, it would be useful
to provide an R API as well. However, I'm a bit concerned for two reasons:

* Looking over the github repo, apparently R4ML is not under active
development/maintenance anymore (last commit Jul 20). So who would be
willing to maintain and extend it?

* Providing wrappers for our algorithm scripts would be just a start
because it hides our core value proposition of custom large-scale ML.
Hence, we would also need an MLContext equivalent that allows to execute
arbitrary DML scripts or R functions. Is there already a tentative design
of such an API and if not, who would like to take it over?

Regards,
Matthias


On Thu, Sep 21, 2017 at 3:43 PM, Deron Eriksson 
wrote:

> I agree with Niketan. An R interface definitely makes sense for SystemML.
> DML itself is based on R, so it's surprising we have Java/Scala/Python
> interfaces to SystemML but we don't have an R interface.
>
> Perhaps R4ML committers could supply a little more info? For instance:
> 1) Would they like to merge R4ML code into the main SystemML project
> itself? (Currently we have no modules.)
> 2) What would they like to merge?
> 3) If so, how do they propose to do so?
> 4) Who will do the majority of the work to add R4ML code to SystemML? Or
> who would like to volunteer to do this?
> 5) Who will maintain the contributed code? Or who would like to volunteer
> to do this?
> 6) Documentation is needed (fit in SystemML documentation framework).
> 7) Testing is needed (fit into SystemML testing framework).
> 8) How is this packaged?
>
> From a technology standpoint, I think an R interface totally makes sense.
> As for a minor criticism (which I apply to other parts of SystemML too), I
> see script wrappers at https://github.com/SparkTC/r4ml/tree/master/R4ML/R.
> This tightly binds the existing DML scripts to R, which means DML
> input/output modifications could potentially require modifications to R
> code.
>
> Deron
>
>
>
> On Thu, Sep 21, 2017 at 11:00 AM, Niketan Pansare 
> wrote:
>
> > Janardhan: I believe this is the R4ML repo: https://github.com/SparkTC/
> > r4ml . Arvind: please correct me if I am wrong.
> >
> > Overall, having a R interface for SystemML is an awesome idea. Since I am
> > not an R4ML expert, may be R4ML committers can comment on how they
> envision
> > "two code streams to work together".
> >
> > Also, comparing the features of R4ML with that of our Python APIs will be
> > useful as it might make a stronger case for R4ML.
> >
> > As an FYI, here are different ways Python users can use SystemML:
> > - Using MLContext to invoke DML script (http://apache.github.io/
> > systemml/beginners-guide-python#invoking-dmlpydml-
> scripts-using-mlcontext
> > and http://apache.github.io/systemml/spark-mlcontext-
> > programming-guide.html)
> > - Python algorithms wrappers (http://apache.github.io/
> > systemml/beginners-guide-python#invoke-systemmls-algorithms)
> > - (not important for R4ML discussion): Python DSL (
> > http://apache.github.io/systemml/beginners-guide-
> python#matrix-operations)
> >
> > Thanks,
> >
> > Niketan Pansare
> > IBM Almaden Research Center
> > E-mail: npansar At us.ibm.com
> > http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar
> >
> > [image: Inactive hide details for Janardhan ---09/21/2017 04:44:02
> AM---Hi
> > Arvind, This is a great idea. One question: the R4ML generat]Janardhan
> > ---09/21/2017 04:44:02 AM---Hi Arvind, This is a great idea. One
> question:
> > the R4ML generates any plan like the SystemML with `D
> >
> > From: Janardhan 
> > To: Arvind Surve , "dev@systemml.apache.org" <
> > dev@systemml.apache.org>
> > Date: 09/21/2017 04:44 AM
> > Subject: Re: [DISCUSS] R-Interface to SystemML
> > --
> >
> >
> >
> > Hi Arvind,
> >
> > This is a great idea. One question: the R4ML generates any plan like the
> > SystemML with `DML` Or with providing some interface we leverage this
> > feature ?. Community effort in the sense of collaborative algorithm
> > implementation.(?)
> >
> > Is this the Spark-R repo ( https://urldefense.proofpoint.
> > com/v2/url?u=https-3A__github.com_rstudio_sparklyr=DwIGaQ&
> > c=jf_iaSHvJObTbx-siA1ZOg=HzVC6v79boGYQrpc383_Kao_
> > 6a6SaOkZrfiSrYZVby0=uxG7P-4VuICwg6yatnAEX5JBdZ-
> PSwyvQzq5gwX1GL0=6VRs_
> > J7zXj9jZEavEP8iNvVfISAjDJeM8wFL2sBnb0g=  ) ?
> >
> > Thanks,
> > Janardhan
> >
> > Sent with [ProtonMail](https://urldefense.proofpoint.com/v2/
> > url?u=https-3A__protonmail.com=DwIGaQ=jf_iaSHvJObTbx-
> > siA1ZOg=HzVC6v79boGYQrpc383_Kao_6a6SaOkZrfiSrYZVby0=
> > uxG7P-4VuICwg6yatnAEX5JBdZ-PSwyvQzq5gwX1GL0=khkGV3oXz1W5m_
> > ueQRuKWlNMVOXXCVhV_ytNCINjJWY= ) Secure Email.
> >
> > >  Original Message 
> > > Subject: [DISCUSS] R-Interface to SystemML
> > > Local Time: September 20, 2017 12:50 PM
> > > UTC Time: September 20, 2017 4:50 PM
> > > From:

Re: Jenkins build became unstable: SystemML-DailyTest #1247

2017-09-20 Thread Matthias Boehm

just FYI: using the exact parameters of the failing test (randomly
generated), I was able to reproduce this now locally - so we should be able
to fix this flaky test once and for all.

Regards,
Matthias


On Wed, Sep 20, 2017 at 2:21 PM,  wrote:

> See 
>
>

Consistency SystemML configuration properties

2017-09-16 Thread Matthias Boehm

Currently, our SystemML configuration properties use an inconsistent prefix
scheme. For example, some properties use the prefix 'dml' (e.g.,
dml.yarn.appmaster), others 'systemml' (e.g., systemml.stats.finegrained),
and yet others no prefix at all (e.g., localtmpdir).

We discussed this before but we never actually made the change. I would
prefer the prefix 'sysml' for consistency with our package names. Are there
any other preferences? If not, I'll make this change end of next week.

Regards,
Matthias

Re: PCA data gen script, no output

2017-09-16 Thread Matthias Boehm

ok great - I also did some more debugging: (1) when the filename is
specified inside the dml script, the file is indeed written to a directory
~ in HDFS, but (2) when passed from command line the ~ is of course
resolved before it's even passed into SystemML.

So let's do the following: (1) use a small scenario of say 10K x 1K, (2)
run it with absolute file name, and see what happens. If this does not
work, I would suspect some permission issues next - maybe the bridge from
python to the jvm hides some error output. If this is also not the case,
please provide the -explain output and I have a closer look.

Regards,
Matthias

On Fri, Sep 15, 2017 at 11:52 PM, Krishna Kalyan <krishnakaly...@gmail.com>
wrote:

> Thank you so much for trying this Matthias. I will try this again with
> absolute path.
>
> Regards,
> Krishna
>
> On Sat, Sep 16, 2017 at 12:09 PM, Matthias Boehm <mboe...@googlemail.com>
> wrote:
>
> > ok, I just tried it with multiple different memory configurations (for
> 6GB
> > driver mem I got the same number of spark instructions as you reported)
> and
> > it ran just fine and produced the outputs. So please, give it a try
> without
> > the ~ (i.e., use an absolute or relative path).
> >
> > Also, even with 2GB mem, this data generation for 1M x 1K ran in about
> 60s
> > (including spark context creation) in my environment. Since your log
> shows
> > a runtime of 5000s, you might want to reduce the data size a bit.
> >
> > Regards,
> > Matthias
> >
> > On Fri, Sep 15, 2017 at 11:08 PM, Krishna Kalyan <
> krishnakaly...@gmail.com
> > >
> > wrote:
> >
> > > Thanks for the reply,
> > > I have tested with systemml-standalone.py too. I am still faced with
> the
> > > same problem. Currently my spark is configured to work on local fs
> > instead
> > > of HDFS hence I did not have a problem with the ~.
> > >
> > > Regards,
> > > Krishna
> > >
> > >
> > >
> > >
> > > On Sat, Sep 16, 2017 at 7:24 AM, Matthias Boehm <
> mboe...@googlemail.com>
> > > wrote:
> > >
> > > > well, I don't think any HDFS fs implementation resolves '~' - so it
> has
> > > > probably created a directory called '~/open-source/scripts/PCA_data'
> > in
> > > > your user path in HDFS or current directory in local FS.
> > > >
> > > > Regards,
> > > > Matthias
> > > >
> > > > On Fri, Sep 15, 2017 at 5:47 PM, Krishna Kalyan <
> > > krishnakaly...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hello,
> > > > > I using PCA
> > > > > <https://github.com/apache/systemml/blob/master/scripts/
> > > > > datagen/genRandData4PCA.dml>
> > > > > data
> > > > > generation scripts to generate data. Unfortunately they do not
> > produce
> > > > any
> > > > > output in the specified target directory.
> > > > >
> > > > > Command used:
> > > > >
> > > > > systemml/bin/systemml-spark-submit.py -f genRandData4PCA.dml
> -nvargs
> > > > > R=100 C=1000 OUT=~/open-source/scripts/PCA_data
> > > > >
> > > > > logs
> > > > > https://gist.github.com/krishnakalyan3/70796b13735743886e41d
> > 3da6b75d7
> > > d5
> > > > >
> > > > > This job also does not throw any errors during exection.
> > > > >
> > > > > Thank you so much,
> > > > > Krishna
> > > > >
> > > >
> > >
> >
>

Re: PCA data gen script, no output

2017-09-16 Thread Matthias Boehm

ok, I just tried it with multiple different memory configurations (for 6GB
driver mem I got the same number of spark instructions as you reported) and
it ran just fine and produced the outputs. So please, give it a try without
the ~ (i.e., use an absolute or relative path).

Also, even with 2GB mem, this data generation for 1M x 1K ran in about 60s
(including spark context creation) in my environment. Since your log shows
a runtime of 5000s, you might want to reduce the data size a bit.

Regards,
Matthias

On Fri, Sep 15, 2017 at 11:08 PM, Krishna Kalyan <krishnakaly...@gmail.com>
wrote:

> Thanks for the reply,
> I have tested with systemml-standalone.py too. I am still faced with the
> same problem. Currently my spark is configured to work on local fs instead
> of HDFS hence I did not have a problem with the ~.
>
> Regards,
> Krishna
>
>
>
>
> On Sat, Sep 16, 2017 at 7:24 AM, Matthias Boehm <mboe...@googlemail.com>
> wrote:
>
> > well, I don't think any HDFS fs implementation resolves '~' - so it has
> > probably created a directory called '~/open-source/scripts/PCA_data' in
> > your user path in HDFS or current directory in local FS.
> >
> > Regards,
> > Matthias
> >
> > On Fri, Sep 15, 2017 at 5:47 PM, Krishna Kalyan <
> krishnakaly...@gmail.com>
> > wrote:
> >
> > > Hello,
> > > I using PCA
> > > <https://github.com/apache/systemml/blob/master/scripts/
> > > datagen/genRandData4PCA.dml>
> > > data
> > > generation scripts to generate data. Unfortunately they do not produce
> > any
> > > output in the specified target directory.
> > >
> > > Command used:
> > >
> > > systemml/bin/systemml-spark-submit.py -f genRandData4PCA.dml -nvargs
> > > R=100 C=1000 OUT=~/open-source/scripts/PCA_data
> > >
> > > logs
> > > https://gist.github.com/krishnakalyan3/70796b13735743886e41d3da6b75d7
> d5
> > >
> > > This job also does not throw any errors during exection.
> > >
> > > Thank you so much,
> > > Krishna
> > >
> >
>

Re: Parfor optimizer getting stuck

2017-09-07 Thread Matthias Boehm

thanks again for catching these issues Rajarshi. I'd like to briefly
summarize their resolutions.

ad 1) Most likely this was caused by a configuration issue - specifically,
the default parallelism being set to less than the number of executors,
leading to the number of cores per executor getting zero and thus the
memory budget per core to INF. SystemML master now has a patch that
increases robustness for such cases and improves the parfor debug output to
log the relevant spark cluster configuration.

ad 2) After an offline discussion with Arvind, we decided to remove this
brittle rewrite on update-in-place for parfor intermediates because we
didn't want to stall the 0.15 release and the asymptotic behavior of this
rewrite was at least squared in the number of nodes and candidates. The
performance impact is limited because later we introduced a generic
update-in-place rewrite for all for, parfor, and while loops. However, for
the 1.0 release we'll reimplement this more advanced update-in-place
rewrite from scratch.

Regards,
Matthias

On Mon, Sep 4, 2017 at 12:46 AM, Matthias Boehm1 
wrote:

> thanks for sharing Rajarshi - well, this is definitely a bug and needs
> fixing before our 0.15 release.
>
> Looking over the output, there are really two different issues here:
>
> 1) The remote memory is Infinity, which causes the optimizer to go for
> REMOTE_SPARK despite the unknown sizes (and max memory estimates) and
> smaller degree of parallelism. This is because it mistakenly assumes all
> operations could be compiled to CP as they would "fit" into remote executor
> memory. We need to (a) make this decision more resilient and (b) find the
> root cause of the messed up memory budget.
>
> 2) Looking over the trace of the parfor optimizer, it seems to get stuck
> in "rewriteSetInPlaceResultIndexing". @Arvind: I remember that we had a
> similar issue when we introduced/extended this rewrite more than a year
> ago. I'll have a look into this tomorrow unless you want to handle it.
>
> Regards,
> Matthias
>
> [image: Inactive hide details for Rajarshi Bhadra ---09/03/2017 11:52:20
> PM---Hi, I am working on a custom tree based algorithm which I]Rajarshi
> Bhadra ---09/03/2017 11:52:20 PM---Hi, I am working on a custom tree based
> algorithm which I am trying to develop
>
> From: Rajarshi Bhadra 
> To: dev@systemml.apache.org
> Cc: Matthias Boehm1 
> Date: 09/03/2017 11:52 PM
> Subject: Parfor optimizer getting stuck
> --
>
>
>
> Hi,
>
> I am working on a custom tree based algorithm which I am trying to develop
> using SystemML. I am using Version 1.0.0-SNAPSHOT. Now my issue is the
> parfor statement is getting stuck somewhere and I am not getting any error
> report or output so I am unable to determine what the issue might be.
> However I have been able to get a log of the parfor which is as follows
>
>
> 17/09/04 06:59:12 DEBUG Optimizer: --- RULEBASED OPTIMIZER ---
> 17/09/04 06:59:12 DEBUG Optimizer: RULEBASED OPT: Optimize w/
> max_mem=63716MB/InfinityMB/InfinityMB, max_k=32/1/1).
> 17/09/04 06:59:12 DEBUG Optimizer: RULEBASED OPT: estimated mem (serial
> exec) M=52MB
> 17/09/04 06:59:12 DEBUG Optimizer: RULEBASED OPT: rewrite 'set data
> partitioner' - result=NONE ()
> 17/09/04 06:59:12 DEBUG Optimizer: RULEBASED OPT: rewrite 'remove
> unnecessary compare matrix' - result=false ()
> 17/09/04 06:59:12 DEBUG Optimizer: RULEBASED OPT: rewrite 'set result
> partitioning' - result=false
> 17/09/04 06:59:12 DEBUG Optimizer: RULEBASED OPT: estimated new mem
> (serial exec) M=52MB
> 17/09/04 06:59:12 DEBUG Optimizer: RULEBASED OPT: estimated new mem
> (serial exec, all CP) M=273068MB
> 17/09/04 06:59:12 DEBUG Optimizer: RULEBASED OPT: estimated new mem (cond
> partitioning) M=52MB
> 17/09/04 06:59:12 DEBUG Optimizer: RULEBASED OPT: rewrite 'set execution
> strategy' - result=REMOTE_SPARK (recompile=true
> )
> 17/09/04 06:59:12 DEBUG Optimizer: RULEBASED OPT: rewrite 'set operation
> exec type CP' - result=198
> 17/09/04 06:59:12 DEBUG Optimizer: RULEBASED OPT: rewrite 'enable data
> colocation' - result=false
> 17/09/04 06:59:12 DEBUG Optimizer: RULEBASED OPT: rewrite 'set partition
> replication factor' - result=false
> 17/09/04 06:59:12 DEBUG Optimizer: RULEBASED OPT: rewrite 'set export
> replication factor' - result=true (3)
> 17/09/04 06:59:12 DEBUG Optimizer: RULEBASED OPT: rewrite 'enable nested
> parallelism' - result=false
> 17/09/04 06:59:12 DEBUG Optimizer: RULEBASED OPT: rewrite 'set degree of
> parallelism' - result=(see EXPLAIN)
> 17/09/04 06:59:12 DEBUG Optimizer: RULEBASED OPT: rewrite 'set task
> partitioner' - result=NAIVE
> 17/09/04 06:59:12 DEBUG Optimizer: RULEBASED OPT: rewrite 'set fused data
> partitioning and execution' - result=false
> 17/09/04 06:59:12 DEBUG Optimizer: RULEBASED OPT: rewrite 'set transpose
> sparse vector operations' - result=false
>
> It would be great if someone

Re: [QUESTION] XOR operations in SystemML. Thanks.

2017-09-07 Thread Matthias Boehm

thanks Janardhan, in that case I would recommend to go with R syntax
because (1) it's actually one of our selling points that users don't have
to learn a new language, (2) it simplifies the porting of R scripts to DML
(and vice versa), and (3) I would think it's rather uncommon to have long
chains of xor operations.

Btw, a nice follow-up task - in case you're interested - would be to
generalize our AND, OR, NOT, (and XOR) operations to matrices, because
these are currently only supported over scalars. It would get you in touch
with the parser validation, compilation, and runtime of rather simple
operations.

Regards,
Matthias

On Mon, Sep 4, 2017 at 8:57 PM, Janardhan Pulivarthi <
janardhan.pulivar...@gmail.com> wrote:

> Hi,
>
> yes, no big reason to deviate. But, simplicity of `*a (+) b*` or `*a ><
> b*`like
> ` *a ^ 2*` (compared to `*xor(a, b)`*, which of the type` *pow(a, 2) `*),
> to be consistent with the other symbols of dml.
>
> In this simple case:
> 1. ` *a (+) b (+) c (+) d(+)...* `
> 2. ` *xor(xor(a, b), c)..)  ` (*sorry, if I written this syntax wrongly)
>
> Your word will be final.
>
> Thanks,
> Janardhan
>
>
> On Mon, Sep 4, 2017 at 6:46 PM, Matthias Boehm <mboe...@googlemail.com>
> wrote:
>
> > Could we please stick to R syntax (i.e., "xor(a, b)") here, unless there
> is
> > a good reason to deviate? Thanks.
> >
> > Regards,
> > Matthias
> >
> > On Mon, Sep 4, 2017 at 7:55 AM, Janardhan Pulivarthi <
> > janardhan.pulivar...@gmail.com> wrote:
> >
> > > Hi all, [XOR symbol]
> > >
> > > Now, I gave a sample try for the XOR operator, with caret ` ^ ` symbol.
> > > But, this have been reserved for exponentiation. So, another
> alternative
> > > would be
> > >
> > > 1. ` (+) `
> > > 2. ` >< `
> > > 3. ` >-< `
> > >
> > > Thanks,
> > > Janardhan
> > >
> > > On Thu, Aug 31, 2017 at 7:38 PM, Matthias Boehm <
> mboe...@googlemail.com>
> > > wrote:
> > >
> > >> From a scalar operation perspective, you could of course emulate XOR
> via
> > >> AND, OR, and negation. However, you might want to write anyway a
> > java-based
> > >> UDF to efficiently implement this recursive operator.
> > >>
> > >> Down the road, we can think about a generalization of our existing
> > >> cumulative operations such as cumsum, cumprod, cummax, to arbitrary
> cell
> > >> computations and aggregation functions, which would be useful for
> quite
> > a
> > >> number of applications.
> > >>
> > >> Regards,
> > >> Matthias
> > >>
> > >> On Thu, Aug 31, 2017 at 5:59 AM, Janardhan Pulivarthi <
> > >> janardhan.pulivar...@gmail.com> wrote:
> > >>
> > >>> Hi,
> > >>>
> > >>> The following is an equation (2.4) from the algorithm for the
> > generation
> > >>> of sobol sequences. The authors of the paper have utilized the
> bitwise
> > >>> operations of C++ to calculate this efficiently.
> > >>>
> > >>> *Now, the question is:* Can we do this at script level (in dml) or we
> > >>> should do it in the `java` itself as a builtin, function to generate
> > the
> > >>> numbers?.
> > >>>
> > >>>
> > >>> 
> > >>> Thanks,
> > >>> Janardhan
> > >>>
> > >>
> > >>
> > >
> >
>

Re: [QUESTION] XOR operations in SystemML. Thanks.

2017-09-04 Thread Matthias Boehm

Could we please stick to R syntax (i.e., "xor(a, b)") here, unless there is
a good reason to deviate? Thanks.

Regards,
Matthias

On Mon, Sep 4, 2017 at 7:55 AM, Janardhan Pulivarthi <
janardhan.pulivar...@gmail.com> wrote:

> Hi all, [XOR symbol]
>
> Now, I gave a sample try for the XOR operator, with caret ` ^ ` symbol.
> But, this have been reserved for exponentiation. So, another alternative
> would be
>
> 1. ` (+) `
> 2. ` >< `
> 3. ` >-< `
>
> Thanks,
> Janardhan
>
> On Thu, Aug 31, 2017 at 7:38 PM, Matthias Boehm <mboe...@googlemail.com>
> wrote:
>
>> From a scalar operation perspective, you could of course emulate XOR via
>> AND, OR, and negation. However, you might want to write anyway a java-based
>> UDF to efficiently implement this recursive operator.
>>
>> Down the road, we can think about a generalization of our existing
>> cumulative operations such as cumsum, cumprod, cummax, to arbitrary cell
>> computations and aggregation functions, which would be useful for quite a
>> number of applications.
>>
>> Regards,
>> Matthias
>>
>> On Thu, Aug 31, 2017 at 5:59 AM, Janardhan Pulivarthi <
>> janardhan.pulivar...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> The following is an equation (2.4) from the algorithm for the generation
>>> of sobol sequences. The authors of the paper have utilized the bitwise
>>> operations of C++ to calculate this efficiently.
>>>
>>> *Now, the question is:* Can we do this at script level (in dml) or we
>>> should do it in the `java` itself as a builtin, function to generate the
>>> numbers?.
>>>
>>>
>>> 
>>> Thanks,
>>> Janardhan
>>>
>>
>>
>

Re: Memory estimates equal to zero

2017-09-04 Thread Matthias Boehm

Hi Nantia,

that's a good question - if your input data is very small, there is nothing
to worry about. For the explain output, the memory estimates are rounded to
MB. Therefore, whenever your input is less than 65,536 cells (dense), you
will see a memory estimate of 0 MB (e.g., round(65,535*8/(1,024*1,024)) =
0).

Regards,
Matthias

On Mon, Sep 4, 2017 at 2:24 AM, Nantia Makrynioti 
wrote:

> Hello,
>
> I generated a HOP plan and memory estimates for the input data of HOP
> operators are all 0MB. The code runs locally and the execution type is CP.
> What could be the case where memory is estimated to 0MB?
>
> Thank you very much,
> Nantia
>

Re: SystemML 0.15 Release Candidate build (Not 1.0 release)

2017-09-02 Thread Matthias Boehm

As it turns out, SystemML master computes correct results, whereas SystemML
0.14 compiled invalid instructions for the initialization of centroids. The
details are in the jira and we can go ahead with the 0.15 release.

Regards,
Matthias

On Fri, Sep 1, 2017 at 11:40 PM, Matthias Boehm <mboe...@googlemail.com>
wrote:

> Arvind, regarding your question of open issues, there is one must-have
> open issue: SYSTEMML-1882. Kmeans shows different convergence behavior
> compared to the 0.14 release (and thus also substantially increased
> execution times). Since this is potentially an issue of incorrect results,
> I marked it as a blocker.
>
> Regards,
> Matthias
>
> On Fri, Sep 1, 2017 at 7:37 PM, Matthias Boehm <mboe...@googlemail.com>
> wrote:
>
>> +1 - I also think that a 0.15 release is a good idea. Many improvements
>> and fixes have been made since 0.14 but we're not quite ready for a 1.0
>> release.
>>
>> Regards,
>> Matthias
>>
>> On Fri, Sep 1, 2017 at 4:22 PM, Deron Eriksson <deroneriks...@gmail.com>
>> wrote:
>>
>>> +1 Arvind, thank you for getting things moving with a 0.15 release. It's
>>> very important with regards to the perceived health of the project. For
>>> my
>>> next project status report to the Apache Board of Directors, I look
>>> forward
>>> to reporting that we are about to publish a 0.15 release.
>>>
>>> Deron
>>>
>>>
>>> On Fri, Sep 1, 2017 at 1:35 AM, Arvind Surve <ac...@yahoo.com.invalid>
>>> wrote:
>>>
>>> > Hi,
>>> > As we could not complete some of the key features targeted for SystemML
>>> > 1.0, we will go next with release 0.15 instead of release 1.0.
>>> > If you have any critical fixes to be added for master repository (now
>>> > targeted for release 0.15) please let me know by EOD 09/01/2017 PST.
>>> > I will plan on starting SystemML 0.15 Release Candidate build by
>>> Tuesday
>>> > 09/05/2017 morning.
>>> > ThanksArvindArvind Surve | Spark Technology Center  |
>>> > http://www.spark.tc/
>>>
>>
>>
>

Re: SystemML 0.15 Release Candidate build (Not 1.0 release)

2017-09-02 Thread Matthias Boehm

Arvind, regarding your question of open issues, there is one must-have open
issue: SYSTEMML-1882. Kmeans shows different convergence behavior compared
to the 0.14 release (and thus also substantially increased execution
times). Since this is potentially an issue of incorrect results, I marked
it as a blocker.

Regards,
Matthias

On Fri, Sep 1, 2017 at 7:37 PM, Matthias Boehm <mboe...@googlemail.com>
wrote:

> +1 - I also think that a 0.15 release is a good idea. Many improvements
> and fixes have been made since 0.14 but we're not quite ready for a 1.0
> release.
>
> Regards,
> Matthias
>
> On Fri, Sep 1, 2017 at 4:22 PM, Deron Eriksson <deroneriks...@gmail.com>
> wrote:
>
>> +1 Arvind, thank you for getting things moving with a 0.15 release. It's
>> very important with regards to the perceived health of the project. For my
>> next project status report to the Apache Board of Directors, I look
>> forward
>> to reporting that we are about to publish a 0.15 release.
>>
>> Deron
>>
>>
>> On Fri, Sep 1, 2017 at 1:35 AM, Arvind Surve <ac...@yahoo.com.invalid>
>> wrote:
>>
>> > Hi,
>> > As we could not complete some of the key features targeted for SystemML
>> > 1.0, we will go next with release 0.15 instead of release 1.0.
>> > If you have any critical fixes to be added for master repository (now
>> > targeted for release 0.15) please let me know by EOD 09/01/2017 PST.
>> > I will plan on starting SystemML 0.15 Release Candidate build by Tuesday
>> > 09/05/2017 morning.
>> > ThanksArvindArvind Surve | Spark Technology Center  |
>> > http://www.spark.tc/
>>
>
>

Re: Enabling CLA by default in SystemML 1.0

2017-08-31 Thread Matthias Boehm

no, not yet. Before enabling it via 'auto', I wanted to completely
integrate with the new code generator as well. This is mostly done except
for some performance issues of row-wise codegen operations that require
another pass.

Regards,
Matthias

On Thu, Aug 31, 2017 at 5:52 PM, Frederick R Reiss <frre...@us.ibm.com>
wrote:

> Did this happen?
>
> Fred
>
> [image: Inactive hide details for "Glenn Weidner" ---05/28/2017 09:38:54
> PM---+1 Glenn]"Glenn Weidner" ---05/28/2017 09:38:54 PM---+1 Glenn
>
> From: "Glenn Weidner" <gweid...@us.ibm.com>
> To: dev@systemml.apache.org
> Date: 05/28/2017 09:38 PM
> Subject: Re: Enabling CLA by default in SystemML 1.0
> --
>
>
>
> +1
>
> Glenn
>
> "Niketan Pansare" ---05/28/2017 02:54:07 PM---+1 for the change. This will
> give us enough time to test CLA with different setting/algorithms/data
>
> From: "Niketan Pansare" <npan...@us.ibm.com>
> To: dev@systemml.apache.org
> Date: 05/28/2017 02:54 PM
> Subject: Re: Enabling CLA by default in SystemML 1.0
> --
>
>
>
>
> +1 for the change. This will give us enough time to test CLA with different
> setting/algorithms/data characteristics before the release.
>
> > On May 28, 2017, at 2:42 PM, Matthias Boehm <mboe...@googlemail.com>
> wrote:
> >
> > Hi all,
> >
> > just a heads-up: I intend to enable compressed linear algebra (CLA) by
> > default in our SystemML 1.0 release. So far, CLA can be configured via
> > 'compressed.linalg' as true or false, and it's disabled by default. For
> the
> > SystemML 1.0 release, I'd like to extend this configuration to take
> either
> > true, false, or auto, where auto will be the default. This 'auto' option
> > indicates to compress out-of-core matrices (i.e., matrices that do not
> fit
> > in aggregate memory), iff we support all operations over these matrices
> in
> > the given program as compressed operations to avoid implicit
> decompression.
> >
> > If you have any concerns, please let me know. Otherwise, I would make
> that
> > change in the next couple of weeks in order to give sufficient time for
> > testing in the wild.
> >
> > Regards,
> > Matthias
>
>
>
>
>
>

Re: [QUESTION] XOR operations in SystemML. Thanks.

2017-08-31 Thread Matthias Boehm

>From a scalar operation perspective, you could of course emulate XOR via
AND, OR, and negation. However, you might want to write anyway a java-based
UDF to efficiently implement this recursive operator.

Down the road, we can think about a generalization of our existing
cumulative operations such as cumsum, cumprod, cummax, to arbitrary cell
computations and aggregation functions, which would be useful for quite a
number of applications.

Regards,
Matthias

On Thu, Aug 31, 2017 at 5:59 AM, Janardhan Pulivarthi <
janardhan.pulivar...@gmail.com> wrote:

> Hi,
>
> The following is an equation (2.4) from the algorithm for the generation
> of sobol sequences. The authors of the paper have utilized the bitwise
> operations of C++ to calculate this efficiently.
>
> *Now, the question is:* Can we do this at script level (in dml) or we
> should do it in the `java` itself as a builtin, function to generate the
> numbers?.
>
>
> 
> Thanks,
> Janardhan
>

Re: JMLC and l2-svm.dml

2017-08-28 Thread Matthias Boehm

ad 1: Yes, you're right - this specific function with String input is 
indeed missing. You have a couple of choices here but the easiest way 
would be to create an input stream from your String via 
IOUtilFunctions.toInputStream which can be fed into 
convertToDoubleMatrix or convertToMatrix. The latter avoids unnecessary 
conversions and improves memory efficiency for large models. 
Alternatively, could also specify the format as part of a json meta data 
argument to convertToDoubleMatrix or convertToMatrix.


ad 2: I was simply saying that the registered inputs for l2svm predict 
should be "new String[]{"w","X"}" not "String[]{"model","X"}" as in your 
original example.


Regards,
Matthias

On 8/28/2017 4:53 PM, Federico Wachs wrote:

Matthias, thanks for replying in such short notice.

A couple of things I wanted to know from your response:

1. conn.convertToDoubleMatrix(conn.readScript(model), rows, cols, "csv").
This takes in an InputStream and conn.readScript does not return that.
Which API should I use instead?

2. pstmt = conn.prepareScript(conn.readScript(dml), new String[] { "w", "X"
}, new String[] { "predicted_y" }, false): is the code I posted and I don't
understand how it differs from what you said on point 2:
"So please change it as follows:
conn.prepareScript(conn.readScript(dml), new String[] {"w", "X"}, new String[]
{"predicted_y"}, "

Thanks!




On Fri, Aug 25, 2017 at 4:06 PM Matthias Boehm <mboe...@googlemail.com>
wrote:


thanks for reaching out - let us quickly go through these issues to
demystify them:

1) Matrix text formats: Apart from binary representations, we support the
following text matrix formats: "mm" (matrix market), "text" (like matrix
market but without the meta data header), and "csv". The sparse "text"
format is our default. You had a csv representation but (without specifying
the format) you tried to read it as "text" which failed on parsing the
first column as row index. You can use something like that

conn.convertToDoubleMatrix(conn.readScript(model), rows, cols, "csv")

2) Parameter binding: Furthermore, note that matrix inputs (and any other
inputs) should be bound to the left-hand-side variable name. So please
change it as follows

conn.prepareScript(conn.readScript(dml), new String[] {"w", "X"}, new
String[] {"predicted_y"},

3) Model reuse: In order to ensure low-latency scoring even for relatively
large models, you might want to mark your model as reused input. You can do
that as follows while setting up your prepared statement. This converts the
model just once and reuses the internal representation over multiple
invocations of executeScript.

pstmt.setMatrix("w", wData, true);


Regards,
Matthias

On Fri, Aug 25, 2017 at 9:48 AM, Federico Wachs <
federico.wa...@sqlstream.com> wrote:


Nevermind. I actually found out that I could change the output to write

to

the model file from csv to text which actually writes the matrix.

Thanks,

On Fri, Aug 25, 2017 at 11:02 AM Federico Wachs <
federico.wa...@sqlstream.com> wrote:


Hi all,

I've been able to follow the steps showed here
<http://apache.github.io/systemml/standalone-guide#

training-and-testing-the-model> and

I want to use the created model from JMLC.

But when I try to load the model like this:






*String dml =
"/home/fwachs/work/systemml/scripts/algorithms/l2-svm-

predict.dml";String

model = "/home/fwachs/Downloads/l2-svm-model.csv";Connection conn = new
Connection();*


*PreparedScript pstmt = conn.prepareScript(conn.readScript(dml), new
String[] { "model", "X" }, new String[] { "predicted_y" },
false);double[][] W =

conn.convertToDoubleMatrix(conn.readScript(model),

1,

1);System.out.println(W);*

I get the following error:

17/08/25 07:01:02 WARN rewrite.RewriteRemovePersistentReadWrite:
Non-registered persistent read of variable 'w' (line 44).
Exception in thread "main" java.io.IOException: Matrix cell [0,0] out

of

overall matrix range [1:1,1:1].
at
org.apache.sysml.runtime.io.ReaderTextCell.

readRawTextCellMatrixFromInputStream(ReaderTextCell.java:236)

at
org.apache.sysml.runtime.io.ReaderTextCell.readMatrixFromInputStream(

ReaderTextCell.java:90)

at
org.apache.sysml.api.jmlc.Connection.convertToDoubleMatrix(

Connection.java:413)

at
org.apache.sysml.api.jmlc.Connection.convertToDoubleMatrix(

Connection.java:382)

at
org.apache.sysml.api.jmlc.Connection.convertToDoubleMatrix(

Connection.java:368)

at com.sqlstream.plugin.systemml.SystemmlUdx.main(SystemmlUdx.java:65)
Caused by: java.lang.NumberFormatException: For input string:
"-2.0851945931025258"

It seems the csv generated is not a matrix at all, since it contains

the

following content:

-2.0851945931025258
6.006153060967643
3.094544228625507
-0.4316259496396
1.0
0
0
4.0


Any ideas? I am so stuck on this...

Thanks,
--
Federico Wachs | sqlstream | +54 911 5748 5048


--
Federico Wachs | sqlstream | +54 911 5748 5048

Re: Gentle ping for help on all my PRs. Thanks.

2017-08-23 Thread Matthias Boehm

that's a fair point and we should all help to move these forward. I'll make
another pass over 3 and 4 today.

Regards,
Matthias

On Wed, Aug 23, 2017 at 9:30 AM, Janardhan Pulivarthi <
janardhan.pulivar...@gmail.com> wrote:

> Dear committers,
>
> I am feeling that my contributions are not in an ordered way. So, I am
> listing them here. And, also listed the help required from volunteers.
>
> 1. [SYSTEMML-1437] - factorization machines
>  [*in progress*]
>  - till now, I have implemented the `*fm.dml*` core module.
>  - *Help: *I am unclear as to how the `i/o` will be for the example
> implementations, such as regression. A sample script for this, might help
> me complete all the examples *regression*, *classification*, & *ranking*.
>
>
> 2. [SYSTEMML-1645] - Verify whether all scripts work with MLContext &
> automate  [*in progress*]
> - This PR tries to write the test scripts for all top level algorithms
> for the new MLContext.
> - I am working with *Jerome* on this. Once he verifies all the scripts,
> I will add the tests for them.
> - *Help: *Can any body help review this PR, and suggest what is missing
> in this PR. I am getting script execution failures.
>
> 3. [SYSTEMML-1444] - UDFs w/ single output in expressions
>  [*in progress*]
>- The objective is to make udf's callable from expressions. I've gone
> through all the Hop, Lop implementations, compiler, parser, api to have a
> clear picture.
>- I am still making my way through this.
>- *Help: *Hi Matthias, I tried implementing a lop
> *FunctionCallCPSingle.java
>  1fb71e441518b2859963b386b1869711>
>  *
>
> 4. [SYSTEMML-1216] - implement local svd( ) function
>  [*done*]
>- With previously implemented local svd(), I've added little
> improvements and tests.
>- *Help: *This is ready to be merged.  (I believe)
>
> 5. [SYSTEMML-1214] Implement scalable version of singular value
> decomposition  [*issue with
> the testing?*]
> - This PR depends on the above PR, this implements distributed svd
> based on already implemented distributed qr() and then calculates local
> svd() of then obtained R matrix.
> - *Help: *I have implemented it preliminarily, but how should I test
> for scalability, Can I do that on *Jenkins CI*. or Do I need to run that on
> any cluster.
>
> 6. [SYSTEMML-979] Add support for Bayesian Optimization.
>  [*a lot to be done*]
> - There's a lot of work in progress, but I implemented a skeleton with
> bad syntax. ( I'll improve this soon)
> - many improvements have to be done, better operation needs to be kept
> and loops needs to be completely eliminated.
>
> Thanks all for the support,
> Janardhan
>

Re: Numerical accuracy of DML.

2017-08-19 Thread Matthias Boehm

Good question - let me separate the somewhat orthogonal aspects to it.

First, for descriptive statistics such as sum, mean, skewness, or kurtosis,
we already use numerically stable implementations based on Kahan Plus (see
org.apache.sysml.runtime.functionobjects.KahanPlus if your interested).
However, for performance reasons, operations like matrix multiplication
rely on the basic multiply and adds (except for block aggregations of
distributed operations which also use KahanPlus).

Second, for comparisons, we do simply rely on Java's builtin operators.
Once we extend the rather limited NaN support, we should change that to
Double.compare accordingly. However, both of these alternatives check for
exact matches. Hence, for comparisons of equivalence on script level, it's
usually a better idea to compare with a tolerance as follows:
abs(1-val)<10e-4 instead of val==1. Doing something like this inside the
builtin operations would probably create more problems and confusion than
it helps.

Regards,
Matthias

On Fri, Aug 18, 2017 at 11:39 PM, Janardhan Pulivarthi <
janardhan.pulivar...@gmail.com> wrote:

> Dear committers,
>
> May I know the numerical accuracy of dml at present, and are you planning
> to increase it. It seems for comparison operators we have depended upon
> java numerical floating point accuracy.
>
> Thank you very much,
> Janardhan
>

Merging sequences of last-level statement blocks

2017-08-06 Thread Matthias Boehm

Hi all,

we see a lot of scripts where conditional statement blocks split DAGs of
operations. After constant folding of if predicates, unnecessary branches
are already removed (which is important for size propagation) but we don't
merge sequences of statement blocks yet. Consider the following example:


if (intercept == 2) {
   
}


If the script is invoked with intercept=0 or 1, the entire if block is
removed and we end up with a sequence of block 1 and block 2. This cut
unnecessarily hides optimization opportunities. I intend to add a rewrite
that merges such sequences under certain conditions.

Note that this renders the current debugging approach of explicit cuts via
"if(1==1){}" ineffective because we will anyway merge the resulting blocks.
You can use while(false) {} instead in the future.

Regards,
Matthias

Re: Training Failure in Clustering : Execution mode singlenode

2017-07-29 Thread Matthias Boehm

ok this has been fixed in master - it was essentially an issue of size
propagation for ctable with two sequence inputs.

Regards,
Matthias

On Sat, Jul 29, 2017 at 1:57 PM, Matthias Boehm <mboe...@googlemail.com>
wrote:

> Hi Krishna,
>
> I just gave Kmeans a try with your parameters and it runs fine with
> hybrid_spark (default) through spark submit. However, I'm able to reproduce
> the issue when forcing it into singlenode. Thanks for catching this - I'll
> take care of it.
>
> Regards,
> Matthias
>
>
> On Sat, Jul 29, 2017 at 9:34 AM, Krishna Kalyan <krishnakaly...@gmail.com>
> wrote:
>
>> Hello All,
>> I am faced with an error below while trying to train Kmeans algorithm.
>>  ERROR: Runtime error in program block generated from statement block
>> between lines 94 and 120
>>
>> Could someone please guide with the fix.
>>
>> Full Error Stack Trace :
>> https://gist.github.com/krishnakalyan3/1225efa5cc13a53a3b4f859b10aefe79
>>
>> Regards,
>> Krishna
>>
>
>

Re: Training Failure in Clustering : Execution mode singlenode

2017-07-29 Thread Matthias Boehm

Hi Krishna,

I just gave Kmeans a try with your parameters and it runs fine with
hybrid_spark (default) through spark submit. However, I'm able to reproduce
the issue when forcing it into singlenode. Thanks for catching this - I'll
take care of it.

Regards,
Matthias

On Sat, Jul 29, 2017 at 9:34 AM, Krishna Kalyan 
wrote:

> Hello All,
> I am faced with an error below while trying to train Kmeans algorithm.
>  ERROR: Runtime error in program block generated from statement block
> between lines 94 and 120
>
> Could someone please guide with the fix.
>
> Full Error Stack Trace :
> https://gist.github.com/krishnakalyan3/1225efa5cc13a53a3b4f859b10aefe79
>
> Regards,
> Krishna
>

Re: spark hybrid mode on HDFS

2017-07-17 Thread Matthias Boehm

well, at a high-level, resource negotiation and distributed storage are
orthogonal concepts. Yarn, Mesos, Standalone, and Kubernetes are resource
schedulers, which you can configure via master and a separate deploy mode
(client/cluster). Under the covers of the HDFS API, you can also use
various alternative file system implementations such as HDFS, local file,
object stores (e.g., swift/s3), etc. At a bare minimum, you need to have
some hadoop jars in your classpath, which would already allow you to run
local/standalone and the local file system implementation.

Regarding the attached error, it looks like your HDFS is configured with
local FS as the default file system implementation but you're trying to
write to a filename with prefix hdfs. It also looks like you're running a
stale version of SystemML (according to the given line numbers in your
stacktrace). Note that up until SystemML 0.14 (inclusive), we always used
the default file system implementation, but in master, we create the
correct file system according to the given file schemes (see
SYSTEMML-1696). So please try to (1) use a recent build of SystemML master,
or (2) reconfigure your hdfs-site.xml to use hdfs as the default fs
implementation.

Regards,
Matthias

On Sun, Jul 16, 2017 at 11:22 PM, Krishna Kalyan 
wrote:

> Hello All,
> I have some questions running systemml scripts on HDFS (with hybrid_spark
> execution mode).
>
> My Current Configuration:
> Standalone HDFS on OSX (version 2.8)
> and Spark Pre-Built for hadoop 2.7 (version 2.1.0)
>
> *jps* out from my system
> [image: Inline image 1]
>
>
> Both of them have been installed separately.
> As far as I understand, to enable hdfs support we need to run spark master
> on yarn-client | yarn-cluster. (Is this understanding correct?)
>
> My question:
> I dont have access to a cluster, is there a way to set up a yarn-client /
> yarn-cluster or my local system so that I can run systemml scripts on
> hybrid_spark mode with HDFS?. If yes could you please point to some
> documentation?.
>
> Thank you so much,
> Krishna
>
>
> PS : sysout of what I have tried already attached below.
>
>

Re: Apache SystemML 1.0 Release build

2017-07-12 Thread Matthias Boehm

here is the list of open tasks (disregarding open bugs), that I think would
be good to have in SystemML 1.0:

SYSTEMML-1284 Code generation
-> all remaining subtasks

SYSTEMML-1299 Language features 1.0
-> 1301, 1304/1307, 1306, 1316, 1426, 1444

SYSTEMML-1308 Runtime reatures 1.0
-> 1313, 1637

SYSTEMML-1321 Compiler features 1.0
-> 1324, 1325, 1349, 1663

SYSTEMML-1696 SystemML in cloud environments
-> 1765

Regards,
Matthias

On Wed, Jul 12, 2017 at 12:37 PM, Arvind Surve 
wrote:

> HI,
> As communicated earlier, we will plan to start SystemML 1.0 build sometime
> toward end of this week.
> Please let me know if you need additional time (approx how much)  if any
> of your MUST get in features for SystemML 1.0 are in intermediate state.
> -ArvindArvind Surve | Spark Technology Center  | http://www.spark.tc/
>
>
> On Thursday, May 18, 2017, 12:21:51 PM PDT, Arvind Surve 
> wrote:
>
> Hi,
>
> We have released SystemML 0.14 on May 9th.Considering coming release is
> SystemML 1.0 with bunch of changes in mind, proposing first Release
> Candidate Date to be week of July 10th  instead of week of June 12th
> earlier communicated.
> Lets discuss if there is any other suggestion.
>
> -Arvind
>
> Arvind Surve | Spark Technology Center  | http://www.spark.tc/
>
> On Wednesday, April 19, 2017, 9:59:42 PM PDT, Arvind Surve <
> ac...@yahoo.com> wrote:Hi,
> We have created branch branch-0.14 based on SytemML 0.14 RC4 tag.This
> branch will be used to develop SystemML 0.14 based code.
> Current master branch will be used to develop next major release (SystemML
> 1.0) code.If you add major/blocking fix for existing supported
> functionality, you need to add that fix both in branch-0.14 and master
> branch.Changes specific to new feature should be added only in master
> branch.
> Release candidate for SystemML 1.0 is planned for week of  June 12th 2017,
> to have release available by end of June 2017.
>
>
>  Arvind Surve | Spark Technology Center  | http://www.spark.tc/

Re: Spark Core

2017-07-12 Thread Matthias Boehm

Well, we explicitly cleanup all intermediates that are no longer used. You
can use -explain to output the runtime plan, which includes rmvar (remove
variable), cpvar (copy variable) and mvvar (move variable) instructions
that internally cleanup intermediates. This cleanup removes data from
memory, potentially evicted/exported variables, and created broadcasts and
rdds. However, we also keep lineage to guard against eager broadcast/rdd
cleanup if they are still used by other lazily evaluated rdds, but whenever
an rdd is not referenced anymore, we cleanup its inputs.

Regarding the comparison to R, please ensure you are running in
hybrid_spark and not forced spark execution mode. Otherwise the latency of
distributed jobs might dominate the execution time for operations over
small data. Also, note that the spark write to csv currently requires a
sort (and hence shuffle) to create the correct order of rows in the output
files. If you want to read this later into SystemML again, you would be
better off writing to text or binary.

Regards,
Matthias

On Wed, Jul 12, 2017 at 11:44 AM, arijit chakraborty 
wrote:

> Hi,
>
>
> Suppose I've this following code:
>
>
> a = matrix(seq(1,10), 10,1)
>
>
> for(i in 1:100){
>
>   b = a + 10
>
>   write (b, "path" + ".csv", format="csv")
>
> }
>
>
> So what I'm doing is for 100 items, I'm adding a constant to a matrix than
> outputting it. And this operation occurs in spark using multiple core of
> the system.
>
>
> My question is, after the operation is the value (here b) remains in that
> core (memory) of the system, so that it get piled up in the memory. Will
> this affect the performance of the process? If it is, how to clean the
> memory after each execution of loop?
>
>
> The reason for asking the question is, when I'm testing the code in R the
> performance is much better than systemML. Since R to systemML is almost
> one-to-one mapping, so I'm not sure where I'm making the mistake. And
> unfortunately at the stage of progress I can't share the exact code.
>
>
> Thanks you!
>
> Arijit
>
>
>

Re: Decaying performance of SystemML

2017-07-11 Thread Matthias Boehm

without any specifics of scripts or datasets, it's unfortunately, hard 
if not impossible to help you here. However, note that the memory 
configuration seems wrong. Why would you configure the driver and 
executors with 2TB if you only have 256GB per node. Maybe you observe an 
issue of swapping. Also note that the maxResultSize is irrelevant in 
case SystemML creates the spark context because we would anyway set it 
to unlimited.


Regarding generally recommend configurations, it's usually a good idea 
to use one executor per worker node with the number of cores set to the 
number of virtual cores. This allows maximum sharing of broadcasts 
across tasks and hence reduces memory pressure.


Regards,
Matthias

On 7/11/2017 9:36 AM, arijit chakraborty wrote:

Hi,


I'm creating a process using systemML. But after certain period of time, the 
performance decreases.


1) This warning message: WARN TaskSetManager: Stage 25254 contains a task of 
very large size (3954 KB). The maximum recommended task size is 100 KB.


2) For Spark, we are implementing this setting:

 spark.executor.memory 2048g

  spark.driver.memory 2048g

spark.driver.maxResultSize 2048

is this good enough, or we can do something else to improve the performance? WE 
tried the spark implementation suggested in the documentation. But it didn't 
help much.


3) We are running on a system with 244 gb ram 32 cores and 100 gb hard disk 
space.


it will be great if anyone can guide me how to improve the performance.


Thank you!

Arijit

Re: If a Vector is in Another Vector

2017-07-09 Thread Matthias Boehm

well, there are different flavors to this problem:

1) If you're only interested in integer values and in testing if all values
of A appear in B, you can do the following:

A = seq(3,4)
B = seq(2,11)
N = min(nrow(A),nrow(B))
M = min(max(A),max(B))
I1 = table(A,1,M,1)!=0;
I2 = table(B,1,M,1)!=0;
ret = sum(I1==I2) == N;
print(ret);

2) If you are interested in testing if the sequence of values in A appears
in B, then this is a classic sequence alignment problem and, if I remember
correctly, there are good dynamic programming algorithms for that. However,
in a semi-vectorized form in DML, you could still do the following (which
first determines the set of equal start values and subsequently probes only
matching candidate subsequences):

A = seq(3,4)
B = seq(2,11)
N = min(nrow(A),nrow(B))
I = (B == A[1,]) * seq(1,nrow(B));
S = removeEmpty(target=I, margin="rows");
ret = FALSE;
if( sum(S) > 0 ) {
  for(i in 1:nrow(S)) {
start = as.scalar(S[i,]);
ret = ret | (sum(B[start:start+N-1,]==A)==N)
  }
}
print(ret);

Regards,
Matthias

On Sun, Jul 9, 2017 at 3:08 AM, arijit chakraborty 
wrote:

> Hi,
>
>
> Suppose I've 2 vectors as
>
>
> A = matrix("2 3", 2, 1)
>
>
> B = matrix(seq(1, 10), 10, 1)
>
>
> And I want to check if A is in B or not. I can use a for-loop to do so.
> But in my case that's not efficient. Is there a way to solve this problem.
>
>
> I tried "in" as we do in R, but it's not working.
>
>
> Thank you!
>
> Arijit
>

Re: Built-in functions and UDFs

2017-06-30 Thread Matthias Boehm

yes, I absolutely agree - we should add special handling of functions with
single outputs.

Quick background: Functions are initially split into separate statement
blocks so they can bind their outputs (potentially many) directly to
logical variable names because Hop DAGs cannot represent multiple outputs,
just multiple consumers. Originally, function calls where not even
represented as hops but constructed as special instructions. Later we
defined a three phase transition for a seamless integration: (1) represent
function calls via FunctionOps in the Hop DAG, (2) allow arbitrary
expressions for inputs of FunctionOps, and (3) allow FunctionOps with
single outputs in expressions. Unfortunately, we only realized phase 1 and
2 but never came to 3. Note that even builtin functions with multiple
outputs like qr and eigen are currently mapped to exactly this function
call framework.

For anybody interested in taking this over, it would essentially entail
modifications to the parser (language validate, hop dag construction), and
optimizer (hop dag representation, inter-procedural analysis, rewrites for
splitting hop dags for recompilation, hop-lop translation, lop instruction
generation).

Regards,
Matthias

On Fri, Jun 30, 2017 at 12:18 PM,  wrote:

> Hi all,
>
> Currently, the usage of UDFs is restricted to simply being able to call
> the UDF and assign the results to a variable.  If the user wishes to use
> the UDF as input to a built-in op of any sort on the same line, an error is
> thrown.  This is confusing for users, and leads to messy code involving
> temporary variables.
>
> Here is a simple example that does not work, yet should:
>
> out = my_udf_func(x) + 4
>
>
> Can we please fix this?  UDFs should be treated like any other op.
>
> Thanks,
>
> - Mike
>
> --
>
> Mike Dusenberry
> GitHub: github.com/dusenberrymw
> LinkedIn: linkedin.com/in/mikedusenberry
>
> Sent from my iPhone.
>
>

Re: On the need for Parameter Server. ( A Model Parallel Construct )

2017-06-19 Thread Matthias Boehm

Well, at a high-level, we could emulate synchronous model parallelism via
our existing parfor construct out of the box. If this is sufficient from an
algorithm perspective, I would be in favor of making any necessary
improvements there instead of introducing a new construct for parameter
servers.

There are a couple of reasons for that. First, given the variety of
backends and potential execution plans, it's usually hard work to integrate
such a construct well with the rest of the system. Second, a custom
parameter server would need to be either integrated with Spark, or (if
implemented from scratch) with a number of different cluster resource
managers (e.g., YARN, Mesos, Kubernetes, etc). Third, extending the
existing parfor construct as necessary would potentially also benefit other
scripts.

Asynchronous model parallelism might also be possible to integrate into
parfor. I remember discussions on state exchange between parfor workers
(e.g., for KMeans to find out if at least one run converged already). Maybe
this is a good time to introduce this, which would allow the update and
broadcast of models in this context.

Regards,
Matthias

On Sun, Jun 18, 2017 at 10:16 PM, Janardhan Pulivarthi <
janardhan.pulivar...@gmail.com> wrote:

> Dear committers,
>
> Implementation/Integration of existing parameter server for the execution
> of algorithms in a distributed fashion both for the machine learning and
> deep learning.
>
> The following document covers a bit about whether we need one or not ?.
>
> My name is Janardhan, currently working on [SYSTEMML-1437] implementation
> of factorization machines, which are to be sparse-safe and scalable, to
> stick to this philosophy we might need a model parallel construct. I know
> very little about how systemml exactly works. If you find some *7 minutes*
> please have a look at this doc.
> 
>  Parameter Server: a model parallel construct
>  3VF51i6xAjNCEC9I/edit?usp=drive_web>
> 
>

Re: Parfor loop interdependencies

2017-06-16 Thread Matthias Boehm

ok thanks for catching this Mike - I just gave it a try and the new IPA
propagated scalars into this function, which allowed for constant folding
of the indexing expressions, and effectively hid the underlying issue. With
disabled IPA, I'm able to reproduce this issue. Interestingly, it only
shows up in spark but not hadoop execution modes.

Regards,
Matthias

On Fri, Jun 16, 2017 at 5:35 PM, <dusenberr...@gmail.com> wrote:

> On the latest master, versus the one from a few days ago, this issue
> appears to be "fixed".  The code is in PR 542, and the for-loop in the
> `nn/layers/conv2d_depthwise.dml::forward` function was the one that was
> previously resulting in null pointer exceptions when forced as a parfor
> loop.  The `check=0` is still needed, but the null pointer is no longer
> being thrown.
>
> --
>
> Mike Dusenberry
> GitHub: github.com/dusenberrymw
> LinkedIn: linkedin.com/in/mikedusenberry
>
> Sent from my iPhone.
>
>
> > On Jun 14, 2017, at 9:08 PM, Matthias Boehm <mboe...@googlemail.com>
> wrote:
> >
> > Generally, the parfor dependency analysis applies a series of tests
> > including traditional techniques from high-performance compilers combined
> > with additional rules for common cases. This dependency analysis tries to
> > proof that there are no loop-carried dependencies - so yes, false
> positives
> > can happen, in which case a user needs to take responsibility by
> disabling
> > the dependency analysis.
> >
> > Regarding your concrete problem - any null pointer exception is of course
> > bug and needs fixing. Could you please create a minimal scenario that
> > reproduces this issues and create a JIRA for it. I'm happy to help
> > resolving this.
> >
> > Regards,
> > Matthias
> >
> >> On Wed, Jun 14, 2017 at 4:24 PM, <dusenberr...@gmail.com> wrote:
> >>
> >> While working on depthwise convolution, I wanted to make use of a parfor
> >> loop to run multiple convolutions at once (there is a vectorized
> approach
> >> as well, but that is out of scope for this question), but I'm running
> into
> >> issues with false flags for loop interdependencies.  Within the parfor
> >> body, I am performing left-indexing over unique ranges to store results
> in
> >> a variable `out`, and the engine currently expresses concerns regarding
> >> this variable.  The range bounds are short expressions, and I have
> printed
> >> out the ranges to be absolutely sure that they are unique.  If I use
> >> `check=0`, then the engine dies with a null pointer exception.
> >>
> >> What is the current strategy used for determining inter-loop
> dependencies
> >> for parfor statements?  How can we improve this?
> >>
> >> -Mike
> >>
> >> --
> >>
> >> Mike Dusenberry
> >> GitHub: github.com/dusenberrymw
> >> LinkedIn: linkedin.com/in/mikedusenberry
> >>
> >> Sent from my iPhone.
> >>
> >>
>

Re: Unexpected Executor Crash

2017-06-16 Thread Matthias Boehm

this might indeed be an robustness issues of rmm which is a replication
based matrix multiply operator. I'll have a look. For the meantime, you can
increase your driver memory (you currently run w/ 1GB driver, resulting in
700MB local memory budget) to something like 10GB. This would allow a
broadcast-based matrix multiply operator (as the broadcast creation
requires twice the memory of a matrix, in your case 2.8GB).

Regards,
Matthias


On Fri, Jun 16, 2017 at 11:52 AM, Anthony Thomas <ahtho...@eng.ucsd.edu>
wrote:

> Hi Matthias and Glenn,
>
> Unfortunately I'm still running into this problem with executors crashing
> due to OOM. Here's the runtime plan generated by SystemML:
>
> 17/06/16 18:36:03 INFO DMLScript: EXPLAIN (RUNTIME):
>
> # Memory Budget local/remote = 628MB/42600MB/63900MB/3195MB
>
> # Degree of Parallelism (vcores) local/remote = 8/24
>
> PROGRAM ( size CP/SP = 17/4 )
>
> --MAIN PROGRAM
>
> GENERIC (lines 1-4) [recompile=true]
>
> --CP createvar pREADM /scratch/M5.csv false MATRIX csv 18750 18750 -1
> -1 351562500 copy false , true 0.0
>
> --CP createvar _mVar1 scratch_space//_p15260_172.31.3.116//_t0/temp0
> true MATRIX binaryblock 18750 18750 1000 1000 351562500 copy
>
> --SPARK csvrblk pREADM.MATRIX.DOUBLE _mVar1.MATRIX.DOUBLE 1000 1000
> false , true 0.0
>
> --CP createvar _mVar2 scratch_space//_p15260_172.31.3.116//_t0/temp1
> true MATRIX binaryblock 18750 18750 1000 1000 351562500 copy
>
> --SPARK chkpoint _mVar1.MATRIX.DOUBLE _mVar2.MATRIX.DOUBLE
> MEMORY_AND_DISK
>
> --CP rmvar _mVar1
>
> --CP createvar _mVar3 scratch_space//_p15260_172.31.3.116//_t0/temp2
> true MATRIX binaryblock 18750 18750 1000 1000 -1 copy
>
> --SPARK rmm _mVar2.MATRIX.DOUBLE _mVar2.MATRIX.DOUBLE
> _mVar3.MATRIX.DOUBLE
>
> --CP rmvar _mVar2
>
> --CP cpvar _mVar3 MN
>
> --CP rmvar _mVar3
>
> GENERIC (lines 7-7) [recompile=true]
>
> --CP createvar _mVar4 scratch_space//_p15260_172.31.3.116//_t0/temp3
> true MATRIX binaryblock 1 1 1000 1000 -1 copy
>
> --SPARK rangeReIndex MN.MATRIX.DOUBLE 1.SCALAR.INT.true
> 1.SCALAR.INT.true 1.SCALAR.INT.true 1.SCALAR.INT.true _mVar4.MATRIX.DOUBLE
> NONE
>
> --CP castdts _mVar4.MATRIX.DOUBLE.false _Var5.SCALAR.STRING
>
> --CP rmvar _mVar4
>
> --CP print _Var5.SCALAR.STRING.false _Var6.SCALAR.STRING
>
> --CP rmvar _Var5
>
> --CP rmvar _Var6
>
> --CP rmvar MN
>
> GENERIC (lines 9-9) [recompile=false]
>
> --CP print DONE!.SCALAR.STRING.true _Var7.SCALAR.STRING
>
> --CP rmvar _Var7
>
>
> The actual error reports by the executor is:
>
>
> # There is insufficient memory for the Java Runtime Environment to
> continue.
> # Native memory allocation (mmap) failed to map 481296384 bytes for
> committing reserved memory.
>
>
> I can send my Spark and YARN configurations as well if that would be
> useful. Thanks a lot for your help.
>
>
> Best,
>
>
> Anthony
>
> On Thu, Jun 15, 2017 at 3:00 PM, Anthony Thomas <ahtho...@eng.ucsd.edu>
> wrote:
>
> > Thanks Matthias and Glenn,
> >
> > I'll give these suggestions a try once I get back in the office tomorrow.
> >
> > Best,
> >
> > Anthony
> >
> >
> > On Jun 15, 2017 12:36 PM, "Matthias Boehm" <mboe...@googlemail.com>
> wrote:
> >
> > well, I think Anthony already used -exec spark here; I would recommend to
> > (1) fix the driver configuration via --driver-java-options "-Xmn2500m"
> (we
> > assume that the young generation does not exceed 10% of the max heap
> > configuration) - this will help if the OOM comes from the driver, and (2)
> > potentially increase the memory overhead of the executors (--conf
> > spark.yarn.executor.memoryOverhead=10240) if ran on yarn and the node
> > manager kill the executor processes because they exceed the container
> > limits. If this does not help, please provide the -explain output and we
> > have a closer look.
> >
> > Regards,
> > Matthias
> >
> > On Thu, Jun 15, 2017 at 10:15 AM, Glenn Weidner <gweid...@us.ibm.com>
> > wrote:
> >
> > > Hi Anthony,
> > >
> > > Could you retry your scenario without the '-exec spark' option? By
> > > default, SystemML will run in hybrid_spark mode which is more
> efficient.
> > >
> > > Thanks,
> > > Glenn
> > >
> > >
> > > [image: Inactive hide details for Anthony Thomas ---06/15/2017 09:50:15
> > > AM---Hi SystemML Developers, I'm running the following simple D]Anthony
> > &

Re: Parfor loop interdependencies

2017-06-14 Thread Matthias Boehm

Generally, the parfor dependency analysis applies a series of tests
including traditional techniques from high-performance compilers combined
with additional rules for common cases. This dependency analysis tries to
proof that there are no loop-carried dependencies - so yes, false positives
can happen, in which case a user needs to take responsibility by disabling
the dependency analysis.

Regarding your concrete problem - any null pointer exception is of course
bug and needs fixing. Could you please create a minimal scenario that
reproduces this issues and create a JIRA for it. I'm happy to help
resolving this.

Regards,
Matthias

On Wed, Jun 14, 2017 at 4:24 PM,  wrote:

> While working on depthwise convolution, I wanted to make use of a parfor
> loop to run multiple convolutions at once (there is a vectorized approach
> as well, but that is out of scope for this question), but I'm running into
> issues with false flags for loop interdependencies.  Within the parfor
> body, I am performing left-indexing over unique ranges to store results in
> a variable `out`, and the engine currently expresses concerns regarding
> this variable.  The range bounds are short expressions, and I have printed
> out the ranges to be absolutely sure that they are unique.  If I use
> `check=0`, then the engine dies with a null pointer exception.
>
> What is the current strategy used for determining inter-loop dependencies
> for parfor statements?  How can we improve this?
>
> -Mike
>
> --
>
> Mike Dusenberry
> GitHub: github.com/dusenberrymw
> LinkedIn: linkedin.com/in/mikedusenberry
>
> Sent from my iPhone.
>
>

Re: Rework inter-procedural analysis

2017-06-14 Thread Matthias Boehm

sure - I'll try to add some documentation of IPA, probably directly inlined
into the code. Unfortunately, a too verbose dev documentation quickly gets
outdated because nobody updates it - let's see if we find the sweet spot
that works for the project.

Regards,
Matthias


On Wed, Jun 14, 2017 at 4:15 PM, <dusenberr...@gmail.com> wrote:

> Agreed.  More documentation, especially within the optimizer portion of
> the engine, is quite useful.  Given that a large number of our bugs and
> performance issues stem from this area, it would be good for it to be clean
> and well documented so that future bug searches/fixes can be completed in a
> more expedient manner.
>
> --
>
> Mike Dusenberry
> GitHub: github.com/dusenberrymw
> LinkedIn: linkedin.com/in/mikedusenberry
>
> Sent from my iPhone.
>
>
> > On Jun 14, 2017, at 8:51 AM, Nakul Jindal <naku...@gmail.com> wrote:
> >
> > Hi Matthias,
> >
> > If its not too much trouble, could you please create a design document
> for
> > this change.
> > This will help the rest of the contributors work on this component as
> well.
> >
> > Thanks,
> > Nakul
> >
> >
> > On Wed, Jun 14, 2017 at 12:00 AM, Matthias Boehm <mboe...@googlemail.com
> >
> > wrote:
> >
> >> just a quick heads up: in the next couple of days, I'll rework our
> existing
> >> inter-procedural analysis (IPA) in order to (1) create well-defined IPA
> >> passes, (2) reuse functional call graphs across multiple rounds of IPA,
> and
> >> (3) introduce new IPA passes such as fine-grained literal propagation
> and
> >> replacements as well as inlining of functions with control structures.
> This
> >> will help improve the performance and debugging of scripts with complex
> >> function call patterns. However, since this is a rather disruptive
> change,
> >> we might experience temporarily some compiler issues - if that happens
> >> please file anything you encounter against SYSTEMML-1668.
> >>
> >> Regards,
> >> Matthias
> >>
>

Handling of File URI Schemes

2017-06-09 Thread Matthias Boehm

just FYI: in the process of making SystemML work smoothly w/ object stores,
we fixed our handling of file URI schemes. So now, you can read/write
from/to different file systems, independent of the configured default fs
implementations. For example you can now do something like this

A = read("file:///");
B = read("./");
C = read("swift:///");

which also allows you to read local files even if HDFS is the default fs
implementation. The same applies to our scratch_space configuration, which
however, requires some additional work.

Regards,
Matthias

Re: Comparing scikit-learn, Mahout Samsara and SystemML

2017-06-05 Thread Matthias Boehm

e needed to scale
> the algorithm in a language such as Scala. This process typically involved
> days or weeks per iteration, and errors would occur translating the
> algorithms to operate on big data. " ( https://en.wikipedia.org/wiki/
> Apache_SystemML )
>
> And the article starts stating that Apache SystemML has "algorithm
> customizability via [...] Python-like languages”.
>
> Mahout Samsara is based on Scala. PredictionIO (predictionio.incubator.
> apache.org) algorithms are based on Mahout Samsara and Scala.  I asked
> Mr. Matthias Boehm at a conference how one could compare Mahout Samsara to
> SystemML. From what I understood, Samsara needs "explicit declarations” in
> expressions for distributed computing, while SystemML doesn’t — please
> correct me if I’m wrong. Also, SystemML will optimize the entire script,
> while Samsara will optimize expressions — again, please correct me if I’m
> wrong.
>
> While my main criterion is scalability (cluster, GPU support etc), other
> criteria to evaluate these frameworks may be: a) public adoption, b) active
> dev community, c) quality of tools for development, d) backing of big
> companies e) simplicity working with clusters (delegating the complexities
> of clustering to the framework, “hiding” them from the user), f) quality of
> documentation, g) quality of the software itself
>
> ( My question was deleted from stats.stackexchange.com for being
> off-topic and deleted from Stack Overflow for being bound to get answers
> with "opinions rather than facts” [sic]. I’m very much interested in
> hearing balanced and insightful comments from the list. )
>
> Thank you,
>
> Gustavo

95 matches

Mail list logo