systemml git commit: [SYSTEMML-540] Allow for `allreduce_parallel_batches` for multi-GPU training

2017-06-16 Thread niketanpansare 
Repository: systemml
Updated Branches:
  refs/heads/gh-pages 2cfd77491 -> 7c4907095


[SYSTEMML-540] Allow for `allreduce_parallel_batches` for multi-GPU training

- Introduced `allreduce_parallel_batches` for multi-GPU training as per
  Mike's suggestion.
- Moved `train_algo` and `test_algo` from solver specification to Python API
  to conform with Caffe as per Berthold's suggestion.
- Updated the documentation for Caffe2DML.

Closes #543.


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/7c490709
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/7c490709
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/7c490709

Branch: refs/heads/gh-pages
Commit: 7c4907095eaec6f70da6884e20ef6888c64f7fd8
Parents: 2cfd774
Author: Niketan Pansare 
Authored: Fri Jun 16 10:14:44 2017 -0800
Committer: Niketan Pansare 
Committed: Fri Jun 16 11:14:44 2017 -0700

--
 beginners-guide-caffe2dml.md | 23 ++-
 1 file changed, 22 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/7c490709/beginners-guide-caffe2dml.md
--
diff --git a/beginners-guide-caffe2dml.md b/beginners-guide-caffe2dml.md
index b44b113..f15e025 100644
--- a/beginners-guide-caffe2dml.md
+++ b/beginners-guide-caffe2dml.md
@@ -94,6 +94,8 @@ lenet.setStatistics(True).setExplain(True)
 
 # If you want to force GPU execution. Please make sure the required dependency 
are available.  
 # lenet.setGPU(True).setForceGPU(True)
+# Example usage of train_algo, test_algo. Assume 2 gpus on driver
+# lenet.set(train_algo="allreduce_parallel_batches", test_algo="minibatch", 
parallel_batches=2)
 
 # (Optional but recommended) Enable native BLAS. 
 lenet.setConfigProperty("native.blas", "auto")
@@ -108,6 +110,16 @@ lenet.predict(X_test)
 
 For more detail on enabling native BLAS, please see the documentation for the 
[native backend](http://apache.github.io/systemml/native-backend).
 
+Common settings for `train_algo` and `test_algo` parameters:
+
+|  | 
PySpark script  
 | Changes to 
Network/Solver  |
+|--|--||
+| Single-node CPU execution (similar to Caffe with solver_mode: CPU)   | 
`caffe2dml.set(train_algo="minibatch", test_algo="minibatch")`  
 | Ensure that 
`batch_size` is set to appropriate value (for example: 64) |
+| Single-node single-GPU execution | 
`caffe2dml.set(train_algo="minibatch", 
test_algo="minibatch").setGPU(True).setForceGPU(True)`  
  | Ensure that `batch_size` is set to appropriate value (for 
example: 64) |
+| Single-node multi-GPU execution (similar to Caffe with solver_mode: GPU) | 
`caffe2dml.set(train_algo="allreduce_parallel_batches", test_algo="minibatch", 
parallel_batches=num_gpu).setGPU(True).setForceGPU(True)` | Ensure that 
`batch_size` is set to appropriate value (for example: 64) |
+| Distributed prediction   | 
`caffe2dml.set(test_algo="allreduce")`  
 |  
  |
+| Distributed synchronous training | 
`caffe2dml.set(train_algo="allreduce_parallel_batches", 
parallel_batches=num_cluster_cores)`
 | Ensure that `batch_size` is set to appropriate value (for example: 64) |
+
 ## Frequently asked questions
 
  What is the purpose of Caffe2DML API ?
@@ -282,4 +294,13 @@ train_df.write.parquet('kaggle-cats-dogs.parquet')
  Can I use Caffe2DML via Scala ?
 
 Though we recommend using Caffe2DML via its Python interfaces, it is possible 
to use it by creating an object of the class
-`org.apache.sysml.api.dl.Caffe2DML`. It is important to note that Caffe2DML's 
scala API is packaged in `systemml-*-extra.jar`.
\ No newline at end of file
+`org.apache.sysml.api.dl.Caffe2DML`. It is important to note that Caffe2DML's 
scala API is packaged in `systemml-*-extra.jar`.
+
+
+ How can I view the script generated

systemml git commit: [SYSTEMML-540] Allow for `allreduce_parallel_batches` for multi-GPU training

2017-06-16 Thread niketanpansare 
Repository: systemml
Updated Branches:
  refs/heads/master ddcb9e019 -> 933a17c5f


[SYSTEMML-540] Allow for `allreduce_parallel_batches` for multi-GPU training

- Introduced `allreduce_parallel_batches` for multi-GPU training as per
  Mike's suggestion.
- Moved `train_algo` and `test_algo` from solver specification to Python API
  to conform with Caffe as per Berthold's suggestion.
- Updated the documentation for Caffe2DML.

Closes #543.


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/933a17c5
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/933a17c5
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/933a17c5

Branch: refs/heads/master
Commit: 933a17c5f38ac71d2aa3e855da1e38797fe6603d
Parents: ddcb9e0
Author: Niketan Pansare 
Authored: Fri Jun 16 10:14:44 2017 -0800
Committer: Niketan Pansare 
Committed: Fri Jun 16 11:14:44 2017 -0700

--
 docs/beginners-guide-caffe2dml.md   |  23 +-
 src/main/proto/caffe/caffe.proto|  47 ++--
 src/main/python/systemml/mllearn/estimators.py  |   7 +-
 .../org/apache/sysml/api/dl/Caffe2DML.scala | 247 ++-
 4 files changed, 248 insertions(+), 76 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/933a17c5/docs/beginners-guide-caffe2dml.md
--
diff --git a/docs/beginners-guide-caffe2dml.md 
b/docs/beginners-guide-caffe2dml.md
index b44b113..f15e025 100644
--- a/docs/beginners-guide-caffe2dml.md
+++ b/docs/beginners-guide-caffe2dml.md
@@ -94,6 +94,8 @@ lenet.setStatistics(True).setExplain(True)
 
 # If you want to force GPU execution. Please make sure the required dependency 
are available.  
 # lenet.setGPU(True).setForceGPU(True)
+# Example usage of train_algo, test_algo. Assume 2 gpus on driver
+# lenet.set(train_algo="allreduce_parallel_batches", test_algo="minibatch", 
parallel_batches=2)
 
 # (Optional but recommended) Enable native BLAS. 
 lenet.setConfigProperty("native.blas", "auto")
@@ -108,6 +110,16 @@ lenet.predict(X_test)
 
 For more detail on enabling native BLAS, please see the documentation for the 
[native backend](http://apache.github.io/systemml/native-backend).
 
+Common settings for `train_algo` and `test_algo` parameters:
+
+|  | 
PySpark script  
 | Changes to 
Network/Solver  |
+|--|--||
+| Single-node CPU execution (similar to Caffe with solver_mode: CPU)   | 
`caffe2dml.set(train_algo="minibatch", test_algo="minibatch")`  
 | Ensure that 
`batch_size` is set to appropriate value (for example: 64) |
+| Single-node single-GPU execution | 
`caffe2dml.set(train_algo="minibatch", 
test_algo="minibatch").setGPU(True).setForceGPU(True)`  
  | Ensure that `batch_size` is set to appropriate value (for 
example: 64) |
+| Single-node multi-GPU execution (similar to Caffe with solver_mode: GPU) | 
`caffe2dml.set(train_algo="allreduce_parallel_batches", test_algo="minibatch", 
parallel_batches=num_gpu).setGPU(True).setForceGPU(True)` | Ensure that 
`batch_size` is set to appropriate value (for example: 64) |
+| Distributed prediction   | 
`caffe2dml.set(test_algo="allreduce")`  
 |  
  |
+| Distributed synchronous training | 
`caffe2dml.set(train_algo="allreduce_parallel_batches", 
parallel_batches=num_cluster_cores)`
 | Ensure that `batch_size` is set to appropriate value (for example: 64) |
+
 ## Frequently asked questions
 
  What is the purpose of Caffe2DML API ?
@@ -282,4 +294,13 @@ train_df.write.parquet('kaggle-cats-dogs.parquet')
  Can I use Caffe2DML via Scala ?
 
 Though we recommend using Caffe2DML via its Python interfaces, it is possible 
to use it by creating an object of the class
-`org.apache.sysml.api.dl.Caffe2DML`. It is important to note that Caffe2DML's 
scala API is packaged in

systemml git commit: [SYSTEMML-1704] Add commons-io to bin and standalone-jar artifacts

2017-06-16 Thread deron 
Repository: systemml
Updated Branches:
  refs/heads/master 933a17c5f -> ca04d7cdd


[SYSTEMML-1704] Add commons-io to bin and standalone-jar artifacts

The commons-io library can be required for certain actions, such as
writing matrices and metadata to the file system, so add commons-io
to the bin and standalone-jar artifacts.

Closes #547.


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/ca04d7cd
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/ca04d7cd
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/ca04d7cd

Branch: refs/heads/master
Commit: ca04d7cdd4ba6bc36d7cd47b5d9b0dcef8943ebc
Parents: 933a17c
Author: Deron Eriksson 
Authored: Fri Jun 16 13:12:15 2017 -0700
Committer: Deron Eriksson 
Committed: Fri Jun 16 13:12:15 2017 -0700

--
 src/assembly/bin.xml| 1 +
 src/assembly/bin/LICENSE| 1 +
 src/assembly/standalone-jar.xml | 1 +
 src/assembly/standalone-jar/LICENSE | 1 +
 4 files changed, 4 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/ca04d7cd/src/assembly/bin.xml
--
diff --git a/src/assembly/bin.xml b/src/assembly/bin.xml
index d816fd4..30c6ce8 100644
--- a/src/assembly/bin.xml
+++ b/src/assembly/bin.xml
@@ -102,6 +102,7 @@
*:commons-collections*
*:commons-configuration*
*:commons-httpclient*
+   *:commons-io*
*:commons-lang
*:commons-logging*
*:commons-math3*

http://git-wip-us.apache.org/repos/asf/systemml/blob/ca04d7cd/src/assembly/bin/LICENSE
--
diff --git a/src/assembly/bin/LICENSE b/src/assembly/bin/LICENSE
index 741a7f9..7b8fc4a 100644
--- a/src/assembly/bin/LICENSE
+++ b/src/assembly/bin/LICENSE
@@ -208,6 +208,7 @@ commons-cli-1.2.jar
 commons-collections-3.2.1.jar
 commons-configuration-1.6.jar
 commons-httpclient-3.1.jar
+commons-io-2.4.jar
 commons-lang-2.6.jar
 commons-logging-1.1.3.jar
 commons-math3-3.4.1.jar

http://git-wip-us.apache.org/repos/asf/systemml/blob/ca04d7cd/src/assembly/standalone-jar.xml
--
diff --git a/src/assembly/standalone-jar.xml b/src/assembly/standalone-jar.xml
index 746907d..e1a0dcd 100644
--- a/src/assembly/standalone-jar.xml
+++ b/src/assembly/standalone-jar.xml
@@ -79,6 +79,7 @@
*:commons-collections*
*:commons-configuration*
*:commons-httpclient*
+   *:commons-io*
*:commons-lang
*:commons-logging*
*:commons-math3*

http://git-wip-us.apache.org/repos/asf/systemml/blob/ca04d7cd/src/assembly/standalone-jar/LICENSE
--
diff --git a/src/assembly/standalone-jar/LICENSE 
b/src/assembly/standalone-jar/LICENSE
index a39d4c5..6e56381 100644
--- a/src/assembly/standalone-jar/LICENSE
+++ b/src/assembly/standalone-jar/LICENSE
@@ -209,6 +209,7 @@ commons-cli:commons-cli:1.2
 commons-collections:commons-collections:3.2.1
 commons-configuration:commons-configuration:1.6
 commons-httpclient:commons-httpclient:3.1
+commons-io:commons-io:2.4
 commons-lang:commons-lang:2.6
 commons-logging:commons-logging:1.1.3
 log4j:log4j:1.2.15

systemml git commit: [MINOR] Prevent lifecycle configuration warnings in Eclipse 4.6.3

2017-06-16 Thread deron 
Repository: systemml
Updated Branches:
  refs/heads/master 9d8fc723c -> dc3dfbe85


[MINOR] Prevent lifecycle configuration warnings in Eclipse 4.6.3

Add ignores to m2e lifecycle-mapping for maven-dependency-plugin and
build-helper-maven-plugin to avoid warnings in Eclipse 4.6.3


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/dc3dfbe8
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/dc3dfbe8
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/dc3dfbe8

Branch: refs/heads/master
Commit: dc3dfbe85296a2d7118997d308ac4c2b7c350dce
Parents: 9d8fc72
Author: Deron Eriksson 
Authored: Fri Jun 16 17:09:26 2017 -0700
Committer: Deron Eriksson 
Committed: Fri Jun 16 17:09:26 2017 -0700

--
 pom.xml | 28 
 1 file changed, 28 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/dc3dfbe8/pom.xml
--
diff --git a/pom.xml b/pom.xml
index bde1215..e3bf831 100644
--- a/pom.xml
+++ b/pom.xml
@@ -782,6 +782,20 @@






+   
org.apache.maven.plugins
+   
maven-dependency-plugin
+   
[2.10,)
+   

+   
copy-dependencies
+   

+   

+   

+   

+   

+   

+   

+   

+   


net.alchim31.maven

scala-maven-plugin

[3.2.2,)
@@ -796,6 +810,20 @@






+   

+   

+   
org.codehaus.mojo
+   
build-helper-maven-plugin
+   
[1.8,)
+   

+   
add-test-source
+   

+   

+   

+   

+   

+   

+

systemml git commit: [SYSTEMML-1679] Add a new threshold utility function

2017-06-16 Thread dusenberrymw 
Repository: systemml
Updated Branches:
  refs/heads/master ca04d7cdd -> 9d8fc723c


[SYSTEMML-1679] Add a new threshold utility function

This function accepts a matrix X and a threshold parameter thresh to get an 
indicator matrix with
values in {0, 1} depending on whether or not the values in X are above thresh. 
It can be used, for
example, to determine the predicted class in a binary classification problem 
given the output of
a sigmoid layer.

Closes #548.


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/9d8fc723
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/9d8fc723
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/9d8fc723

Branch: refs/heads/master
Commit: 9d8fc723cdaad5d47692ba0b04e566b2a7d9b1bc
Parents: ca04d7c
Author: Fei Hu 
Authored: Fri Jun 16 16:16:34 2017 -0700
Committer: Mike Dusenberry 
Committed: Fri Jun 16 16:16:34 2017 -0700

--
 scripts/nn/test/README.md |  4 ++--
 scripts/nn/test/run_tests.dml |  1 +
 scripts/nn/test/test.dml  | 22 ++
 scripts/nn/util.dml   | 16 
 4 files changed, 41 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/9d8fc723/scripts/nn/test/README.md
--
diff --git a/scripts/nn/test/README.md b/scripts/nn/test/README.md
index b714d50..0143752 100644
--- a/scripts/nn/test/README.md
+++ b/scripts/nn/test/README.md
@@ -26,7 +26,7 @@ limitations under the License.
  All layers are tested for correct derivatives ("gradient-checking"), and 
many layers also have correctness tests against simpler reference 
implementations.
 * `grad_check.dml` - Contains gradient-checks for all layers as individual DML 
functions.
 * `test.dml` - Contains correctness tests for several of the more complicated 
layers by checking against simple reference implementations, such as 
`conv_simple.dml`.  All tests are formulated as individual DML functions.
-* `tests.dml` - A DML script that runs all of the tests in `grad_check.dml` 
and `test.dml`.
+* `run_tests.dml` - A DML script that runs all of the tests in 
`grad_check.dml` and `test.dml`.
 
 ## Execution
-* `spark-submit SystemML.jar -f nn/test/tests.dml` from the base of the 
project.
+* `spark-submit SystemML.jar -f nn/test/run_tests.dml` from the base of the 
project.

http://git-wip-us.apache.org/repos/asf/systemml/blob/9d8fc723/scripts/nn/test/run_tests.dml
--
diff --git a/scripts/nn/test/run_tests.dml b/scripts/nn/test/run_tests.dml
index b48606c..c9b1b3e 100644
--- a/scripts/nn/test/run_tests.dml
+++ b/scripts/nn/test/run_tests.dml
@@ -92,6 +92,7 @@ test::im2col()
 test::max_pool2d()
 test::padding()
 test::tanh()
+test::threshold()
 
 print("---")
 print("Other tests complete -- look for any ERRORs or WARNINGs.")

http://git-wip-us.apache.org/repos/asf/systemml/blob/9d8fc723/scripts/nn/test/test.dml
--
diff --git a/scripts/nn/test/test.dml b/scripts/nn/test/test.dml
index 52fb063..cfb8c79 100644
--- a/scripts/nn/test/test.dml
+++ b/scripts/nn/test/test.dml
@@ -605,3 +605,25 @@ tanh = function() {
   }
 }
 
+threshold = function() {
+  /*
+   * Test for threshold function.
+   */
+  print("Testing the threshold function.")
+
+  # Generate data
+  X = matrix("0.31 0.24 0.87
+  0.45 0.66 0.65
+  0.24 0.91 0.13", rows=3, cols=3)
+  thresh = 0.5
+  target_matrix = matrix("0.0 0.0 1.0
+  0.0 1.0 1.0
+  0.0 1.0 0.0", rows=3, cols=3)
+
+  # Get the indicator matrix
+  indicator_matrix = util::threshold(X, thresh)
+
+  # Equivalency check
+  out = test_util::check_all_equal(indicator_matrix, target_matrix)
+}
+

http://git-wip-us.apache.org/repos/asf/systemml/blob/9d8fc723/scripts/nn/util.dml
--
diff --git a/scripts/nn/util.dml b/scripts/nn/util.dml
index 3a73f08..c4da16a 100644
--- a/scripts/nn/util.dml
+++ b/scripts/nn/util.dml
@@ -200,3 +200,19 @@ unpad_image = function(matrix[double] img_padded, int Hin, 
int Win, int padh, in
   }
 }
 
+threshold = function(matrix[double] X, double thresh)
+return (matrix[double] out) {
+  /*
+   * Computes an indicator matrix with values in {0, 1} depending on
+   * whether or not the values in X are above the input threshold
+   *
+   * Inputs:
+   *  - X: Inputs, of shape (any, any).
+   *  - thresh: Input threshold.
+   *
+   * Outputs:
+   *  - out: Outputs, of same shape as X.
+   */
+  out = X > thresh
+}
+

systemml git commit: [SYSTEMML-1647] Update StepLinearReg to work with MLContext

2017-06-16 Thread deron 
Repository: systemml
Updated Branches:
  refs/heads/master cf0877b43 -> 948943d17


[SYSTEMML-1647] Update StepLinearReg to work with MLContext

Address write statements and reordering of coefficients issues.

Closes #525.


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/948943d1
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/948943d1
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/948943d1

Branch: refs/heads/master
Commit: 948943d17f5dd97d0678cc9bd36224b41b0b9d97
Parents: cf0877b
Author: Imran Younus 
Authored: Fri Jun 16 09:57:55 2017 -0700
Committer: Deron Eriksson 
Committed: Fri Jun 16 09:57:55 2017 -0700

--
 scripts/algorithms/StepLinearRegDS.dml | 630 ++--
 1 file changed, 325 insertions(+), 305 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/948943d1/scripts/algorithms/StepLinearRegDS.dml
--
diff --git a/scripts/algorithms/StepLinearRegDS.dml 
b/scripts/algorithms/StepLinearRegDS.dml
index 00f50ee..05f4b26 100644
--- a/scripts/algorithms/StepLinearRegDS.dml
+++ b/scripts/algorithms/StepLinearRegDS.dml
@@ -25,20 +25,23 @@
 #
 # INPUT PARAMETERS:
 # 

-# NAMETYPEDEFAULTMEANING
+# NAMETYPEDEFAULTMEANING
 # 

-# X   String   ---  Location (on HDFS) to read the matrix X of 
feature vectors
-# Y   String   ---  Location (on HDFS) to read the 1-column matrix 
Y of response values
-# B   String   ---  Location to store estimated regression 
parameters (the betas)
-# S   String---  Location to write the selected features ordered 
as computed by the algorithm
-# O   String   " "  Location to write the printed statistics; by 
default is standard output
-# icptInt0   Intercept presence, shifting and rescaling the 
columns of X:
-#0 = no intercept, no shifting, no rescaling;
-#1 = add intercept, but neither shift nor rescale 
X;
-#2 = add intercept, shift & rescale X columns to 
mean = 0, variance = 1
-# thr Double0.01 Threshold to stop the algorithm: if the decrease 
in the value of AIC falls below thr
-#no further features are being checked and the 
algorithm stops 
-# fmt String   "text"Matrix output format for B (the betas) only, 
usually "text" or "csv"
+# X   String ---   Location (on HDFS) to read the matrix X of 
feature vectors
+# Y   String ---   Location (on HDFS) to read the 1-column matrix 
Y of response values
+# B   String ---   Location to store estimated regression 
parameters (the betas)
+# S   String ---   Location to write the selected features ordered 
as computed by the algorithm
+# O   String " "   Location to write the printed statistics; by 
default is standard output
+# icptInt0 Intercept presence, shifting and rescaling the 
columns of X:
+#  0 = no intercept, no shifting, no rescaling;
+#  1 = add intercept, but neither shift nor 
rescale X;
+#  2 = add intercept, shift & rescale X columns to 
mean = 0, variance = 1
+# thr Double0.01   Threshold to stop the algorithm: if the 
decrease in the value of AIC falls below thr
+#  no further features are being checked and the 
algorithm stops 
+# fmt String   "text"  Matrix output format for B (the betas) only, 
usually "text" or "csv"
+# write_beta  Boolean   TRUE   Should the beta's be returned?
+#  0 = no
+#  1 = yes
 # 

 # OUTPUT: Matrix of regression parameters (the betas) and its size depend on 
icpt input value:
 # OUTPUT SIZE:   OUTPUT CONTENTS:HOW TO PREDICT Y FROM 
X AND B:
@@ -70,20 +73,22 @@
 #
 # HOW TO INVOKE THIS SCRIPT - EXAMPLE:
 # hadoop jar SystemML.jar -f StepLinearRegDS.dml -nvargs X=INPUT_DIR/X 
Y=INPUT_DIR/Y B=OUTPUT_DIR/betas
-# O=OUTPUT_DIR/stats S=OUTPUT_DIR/selected icpt=2 thr=0.01 fmt=csv
+# O=OUTPUT_DIR/stats S=OUTPUT_DIR/selected icpt=2 thr=0.01 fmt=csv 
write_beta=TRUE
 
 fileX = $X;
 fileY = $Y;
 fileB = $B;
 fileS = $S;
 
+write_beta = ifdef($write_beta, TRUE);
+
 # currently only the forward selection strategy in

[1/2] systemml git commit: [SYSTEMML-1691, 1692] New IPA passes: literal replacement and rewrites

2017-06-16 Thread mboehm7 
Repository: systemml
Updated Branches:
  refs/heads/master 948943d17 -> ddcb9e019


[SYSTEMML-1691,1692] New IPA passes: literal replacement and rewrites

This patch introduces two new passes for inter-procedural analysis
(IPA): (1) literal propagation and replacement into functions, and (2)
static rewrites, which are both applied for any number of requires IPA
iterations. The internal abstraction for function call summaries has
been extended accordingly. The new literal propagation and replacement
works on a fine granularity of individual function parameters and
propagates any literals that are consistent across all function calls,
independent of remaining inputs. Together with the additional rewrites
pass, this allows rewrites such as constant folding and subsequent
removal of branches which can significantly cut down the program size
and number of distributed operations. 

For example, for GLM poisson.log, this change reduced the size of the
initial runtime program from 2132/153 to 1164/45 local/distributed
instructions, which is now much easier to debug and profile.


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/1e6639c7
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/1e6639c7
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/1e6639c7

Branch: refs/heads/master
Commit: 1e6639c754961f51bb53754c1fa8b6dce404294a
Parents: 948943d
Author: Matthias Boehm 
Authored: Thu Jun 15 20:01:12 2017 -0700
Committer: Matthias Boehm 
Committed: Fri Jun 16 10:01:57 2017 -0700

--
 .../sysml/hops/ipa/FunctionCallSizeInfo.java|  90 ++-
 .../java/org/apache/sysml/hops/ipa/IPAPass.java |   3 +-
 .../hops/ipa/IPAPassApplyStaticHopRewrites.java |  53 +++
 .../ipa/IPAPassFlagFunctionsRecompileOnce.java  |   2 +-
 .../ipa/IPAPassPropagateReplaceLiterals.java| 155 +++
 .../ipa/IPAPassRemoveConstantBinaryOps.java |   2 +-
 .../IPAPassRemoveUnnecessaryCheckpoints.java|   2 +-
 .../hops/ipa/IPAPassRemoveUnusedFunctions.java  |   2 +-
 .../sysml/hops/ipa/InterProceduralAnalysis.java | 104 +++--
 .../org/apache/sysml/parser/DMLTranslator.java  |  12 +-
 .../java/org/apache/sysml/utils/Explain.java|   5 +-
 ...antFoldingScalarVariablePropagationTest.java |  17 +-
 12 files changed, 360 insertions(+), 87 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/1e6639c7/src/main/java/org/apache/sysml/hops/ipa/FunctionCallSizeInfo.java
--
diff --git a/src/main/java/org/apache/sysml/hops/ipa/FunctionCallSizeInfo.java 
b/src/main/java/org/apache/sysml/hops/ipa/FunctionCallSizeInfo.java
index 20054a2..402e780 100644
--- a/src/main/java/org/apache/sysml/hops/ipa/FunctionCallSizeInfo.java
+++ b/src/main/java/org/apache/sysml/hops/ipa/FunctionCallSizeInfo.java
@@ -24,6 +24,7 @@ import java.util.HashMap;
 import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
+import java.util.Map.Entry;
 import java.util.Set;
 
 import org.apache.sysml.hops.FunctionOp;
@@ -52,10 +53,16 @@ public class FunctionCallSizeInfo
//to subsequent statement blocks and functions)
private final Set _fcandUnary;

-   //indicators for which function arguments it is safe to propagate nnz
+   //indicators for which function arguments of valid functions it 
+   //is safe to propagate the number of non-zeros 
//(mapping from function keys to set of function input HopIDs)
private final Map _fcandSafeNNZ;

+   //indicators which literal function arguments can be safely 
+   //propagated into and replaced in the respective functions 
+   //(mapping from function keys to set of function input positions)
+   private final Map _fSafeLiterals;
+   
/**
 * Constructs the function call summary for all functions
 * reachable from the main program. 
@@ -84,6 +91,7 @@ public class FunctionCallSizeInfo
_fcand = new HashSet();
_fcandUnary = new HashSet();
_fcandSafeNNZ =  new HashMap();
+   _fSafeLiterals = new HashMap();

constructFunctionCallSizeInfo();
}
@@ -169,17 +177,44 @@ public class FunctionCallSizeInfo
 * 
 * @param fkey function key
 * @param inputHopID hop ID of the input
-* @return true if nnz can safely be propageted
+* @return true if nnz can safely be propagated
 */
public boolean isSafeNnz(String fkey, long inputHopID) {
return _fcandSafeNNZ.containsKey(fkey)
&& _fcandSafeNNZ.get(fkey).contains(inputHopID);

[2/2] systemml git commit: [SYSTEMML-1702] Fix robustness codegen row-wise (unknowns, scalar/vect)

2017-06-16 Thread mboehm7 
[SYSTEMML-1702] Fix robustness codegen row-wise (unknowns, scalar/vect)

Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/ddcb9e01
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/ddcb9e01
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/ddcb9e01

Branch: refs/heads/master
Commit: ddcb9e0190989ea1837af8dd44676d52497c3e15
Parents: 1e6639c
Author: Matthias Boehm 
Authored: Thu Jun 15 23:57:44 2017 -0700
Committer: Matthias Boehm 
Committed: Fri Jun 16 10:01:58 2017 -0700

--
 .../sysml/hops/codegen/template/TemplateCell.java|  2 +-
 .../sysml/hops/codegen/template/TemplateRow.java | 15 ++-
 .../sysml/hops/codegen/template/TemplateUtils.java   |  5 +
 3 files changed, 16 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/ddcb9e01/src/main/java/org/apache/sysml/hops/codegen/template/TemplateCell.java
--
diff --git 
a/src/main/java/org/apache/sysml/hops/codegen/template/TemplateCell.java 
b/src/main/java/org/apache/sysml/hops/codegen/template/TemplateCell.java
index d6dcdf6..91c61c2 100644
--- a/src/main/java/org/apache/sysml/hops/codegen/template/TemplateCell.java
+++ b/src/main/java/org/apache/sysml/hops/codegen/template/TemplateCell.java
@@ -75,7 +75,7 @@ public class TemplateCell extends TemplateBase
 
@Override
public boolean open(Hop hop) {
-   return isValidOperation(hop)
+   return hop.dimsKnown() && isValidOperation(hop)
|| (hop instanceof IndexingOp && (((IndexingOp)hop)
.isColLowerEqualsUpper() || hop.getDim2()==1));
}

http://git-wip-us.apache.org/repos/asf/systemml/blob/ddcb9e01/src/main/java/org/apache/sysml/hops/codegen/template/TemplateRow.java
--
diff --git 
a/src/main/java/org/apache/sysml/hops/codegen/template/TemplateRow.java 
b/src/main/java/org/apache/sysml/hops/codegen/template/TemplateRow.java
index a0b4572..8364fba 100644
--- a/src/main/java/org/apache/sysml/hops/codegen/template/TemplateRow.java
+++ b/src/main/java/org/apache/sysml/hops/codegen/template/TemplateRow.java
@@ -73,9 +73,9 @@ public class TemplateRow extends TemplateBase

@Override
public boolean open(Hop hop) {
-   return (hop instanceof BinaryOp && 
hop.getInput().get(0).getDim2()>1 
+   return (hop instanceof BinaryOp && hop.dimsKnown() && 
hop.getInput().get(0).getDim2()>1 
&& hop.getInput().get(1).getDim2()==1 && 
TemplateCell.isValidOperation(hop)) 
-   || (hop instanceof AggBinaryOp && hop.getDim2()==1
+   || (hop instanceof AggBinaryOp && hop.dimsKnown() && 
hop.getDim2()==1
&& hop.getInput().get(0).getDim1()>1 && 
hop.getInput().get(0).getDim2()>1)
|| (hop instanceof AggUnaryOp && 
((AggUnaryOp)hop).getDirection()!=Direction.RowCol 
&& hop.getInput().get(0).getDim1()>1 && 
hop.getInput().get(0).getDim2()>1
@@ -164,7 +164,7 @@ public class TemplateRow extends TemplateBase
MemoTableEntry me = memo.getBest(hop.getHopID(), 
TemplateType.RowTpl);
for( int i=0; i 1 && 
hop.getInput().get(0).getDim2() > 1 )
+   if( (hop.getInput().get(0).getDim1() > 1 && 
hop.getInput().get(0).getDim2() > 1)
+   || (hop.getInput().get(1).getDim1() > 1 && 
hop.getInput().get(1).getDim2() > 1))
{
if( HopRewriteUtils.isBinary(hop, 
SUPPORTED_VECT_BINARY) ) {
if( TemplateUtils.isMatrix(cdata1) && 
TemplateUtils.isMatrix(cdata2) ) {
@@ -267,6 +268,8 @@ public class TemplateRow extends TemplateBase
}