[jira] [Closed] (SYSTEMML-2476) Unexpected mapreduce task
[
https://issues.apache.org/jira/browse/SYSTEMML-2476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Boehm closed SYSTEMML-2476.
Fix Version/s: SystemDS 2.0
Assignee: Matthias Boehm
Resolution: Won't Fix
Closing due to removed MR backend.
> Unexpected mapreduce task
> -
>
> Key: SYSTEMML-2476
> URL: https://issues.apache.org/jira/browse/SYSTEMML-2476
> Project: SystemML
> Issue Type: Bug
>Reporter: LI Guobao
>Assignee: Matthias Boehm
>Priority: Major
> Fix For: SystemDS 2.0
>
>
> When trying to use scalar casting to get element from a list, unexpected
> mapreduce tasks are launched instead of CP mode. The scenario is to replace
> *C = 1* with *C = as.scalar(hyperparams["C"])* inside the {{_gradient
> function_}} found in
> {{_src/test/scripts/functions/paramserv/mnist_lenet_paramserv.dml_}}. And
> then the problem could be reproduced by launching the method
> {{_testParamservBSPBatchDisjointContiguous_}} inside class
> _{{org.apache.sysml.test.integration.functions.paramserv.ParamservLocalNNTest}}_
> Here is the stack:
> {code:java}
> 18/07/31 22:10:27 INFO mapred.MapTask: numReduceTasks: 1
> 18/07/31 22:10:27 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
> 18/07/31 22:10:27 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
> 18/07/31 22:10:27 INFO mapred.MapTask: soft limit at 83886080
> 18/07/31 22:10:27 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
> 18/07/31 22:10:27 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
> 18/07/31 22:10:27 INFO mapreduce.Job: The url to track the job:
> http://localhost:8080/
> 18/07/31 22:10:27 INFO mapreduce.Job: Running job: job_local792652629_0008
> {code}
> [~mboehm7], if possible, could you take a look on this? And I've double
> checked the creation of execution context in
> {{ParamservBuiltinCPInstruction}}. But it is instance of ExecutionContext not
> SparkExecutionContext.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (SYSTEMML-2481) Missing cleanup of matrices/frames in list objects
[ https://issues.apache.org/jira/browse/SYSTEMML-2481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Boehm closed SYSTEMML-2481. Fix Version/s: SystemML 1.2 Assignee: Matthias Boehm Resolution: Fixed > Missing cleanup of matrices/frames in list objects > -- > > Key: SYSTEMML-2481 > URL: https://issues.apache.org/jira/browse/SYSTEMML-2481 > Project: SystemML > Issue Type: Bug >Reporter: Matthias Boehm >Assignee: Matthias Boehm >Priority: Major > Fix For: SystemML 1.2 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (SYSTEMML-2426) CP tsmm crashing on matrix >16GB
[
https://issues.apache.org/jira/browse/SYSTEMML-2426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Boehm closed SYSTEMML-2426.
Fix Version/s: SystemML 1.2
Assignee: Matthias Boehm
Resolution: Fixed
> CP tsmm crashing on matrix >16GB
>
>
> Key: SYSTEMML-2426
> URL: https://issues.apache.org/jira/browse/SYSTEMML-2426
> Project: SystemML
> Issue Type: Bug
>Reporter: Matthias Boehm
>Assignee: Matthias Boehm
>Priority: Major
> Fix For: SystemML 1.2
>
>
> {code}
> Caused by: java.util.concurrent.ExecutionException:
> java.lang.ArrayIndexOutOfBoundsException
> at java.util.concurrent.ForkJoinTask.get(ForkJoinTask.java:1006)
> at
> org.apache.sysml.runtime.matrix.data.LibMatrixMult.matrixMultTransposeSelf(LibMatrixMult.java:390)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (SYSTEMML-2410) Perftest for SystemML 1.2 release
[ https://issues.apache.org/jira/browse/SYSTEMML-2410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Boehm closed SYSTEMML-2410. Fix Version/s: SystemML 1.2 Assignee: Matthias Boehm Resolution: Done > Perftest for SystemML 1.2 release > - > > Key: SYSTEMML-2410 > URL: https://issues.apache.org/jira/browse/SYSTEMML-2410 > Project: SystemML > Issue Type: Bug >Reporter: Matthias Boehm >Assignee: Matthias Boehm >Priority: Major > Fix For: SystemML 1.2 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (SYSTEMML-2467) Wrong IPA size propagation w/ permuted named function arguments
[ https://issues.apache.org/jira/browse/SYSTEMML-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Boehm closed SYSTEMML-2467. Fix Version/s: SystemML 1.2 Assignee: Matthias Boehm Resolution: Fixed > Wrong IPA size propagation w/ permuted named function arguments > --- > > Key: SYSTEMML-2467 > URL: https://issues.apache.org/jira/browse/SYSTEMML-2467 > Project: SystemML > Issue Type: Bug >Reporter: Matthias Boehm >Assignee: Matthias Boehm >Priority: Major > Fix For: SystemML 1.2 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (SYSTEMML-2427) Incorrect nnz after rand of large dense block
[
https://issues.apache.org/jira/browse/SYSTEMML-2427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Boehm closed SYSTEMML-2427.
Fix Version/s: SystemML 1.2
Assignee: Matthias Boehm
Resolution: Fixed
> Incorrect nnz after rand of large dense block
> -
>
> Key: SYSTEMML-2427
> URL: https://issues.apache.org/jira/browse/SYSTEMML-2427
> Project: SystemML
> Issue Type: Bug
>Reporter: Matthias Boehm
>Assignee: Matthias Boehm
>Priority: Major
> Fix For: SystemML 1.2
>
>
> The following code gives correctly {{2.0E9}} with with {{N=2000}} but
> consistently {{2.4970522E9}} with {{N=2500}} independent of the runtime
> seed.
> {code}
> X = rand(rows=N, cols=100)
> if(sum(X)==0)
> X[7,] = matrix(0, 1, 100)
> print(sum(X!=0));
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (SYSTEMML-2475) Leftindexing of matrices into lists fails
[
https://issues.apache.org/jira/browse/SYSTEMML-2475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Boehm closed SYSTEMML-2475.
Fix Version/s: SystemML 1.2
Assignee: Matthias Boehm
Resolution: Fixed
> Leftindexing of matrices into lists fails
> -
>
> Key: SYSTEMML-2475
> URL: https://issues.apache.org/jira/browse/SYSTEMML-2475
> Project: SystemML
> Issue Type: Bug
>Reporter: Matthias Boehm
>Assignee: Matthias Boehm
>Priority: Major
> Fix For: SystemML 1.2
>
>
> The following code currently fails for two reasons: (1) the data type of the
> unary aggregate rowMeans is inferred as list from the target, and (2) there
> is no support for list-matrix left indexing.
> {code}
> parfor(i in 1:length(L))
> L[i] = rowMeans(as.matrix(L[i]))
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (SYSTEMML-2175) Parfor spark fair scheduler pool leak
[ https://issues.apache.org/jira/browse/SYSTEMML-2175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Boehm reassigned SYSTEMML-2175: Assignee: Matthias Boehm > Parfor spark fair scheduler pool leak > - > > Key: SYSTEMML-2175 > URL: https://issues.apache.org/jira/browse/SYSTEMML-2175 > Project: SystemML > Issue Type: Bug >Reporter: Matthias Boehm >Assignee: Matthias Boehm >Priority: Major > > In an algorithm with an overall convergence loop and a parfor loop over large > data in each outer iteration, we encountered a leak of parfor fair scheduler > pools. Specifically, this issue seems to arise whenever the parfor loop has > fewer iterations and thus workers than locally available threads. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (SYSTEMML-2175) Parfor spark fair scheduler pool leak
[ https://issues.apache.org/jira/browse/SYSTEMML-2175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Boehm closed SYSTEMML-2175. Fix Version/s: SystemML 1.2 Resolution: Fixed > Parfor spark fair scheduler pool leak > - > > Key: SYSTEMML-2175 > URL: https://issues.apache.org/jira/browse/SYSTEMML-2175 > Project: SystemML > Issue Type: Bug >Reporter: Matthias Boehm >Assignee: Matthias Boehm >Priority: Major > Fix For: SystemML 1.2 > > > In an algorithm with an overall convergence loop and a parfor loop over large > data in each outer iteration, we encountered a leak of parfor fair scheduler > pools. Specifically, this issue seems to arise whenever the parfor loop has > fewer iterations and thus workers than locally available threads. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (SYSTEMML-2485) Parfor optimizer failing due to non-existing parent vars
[
https://issues.apache.org/jira/browse/SYSTEMML-2485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Boehm reassigned SYSTEMML-2485:
Assignee: Matthias Boehm
> Parfor optimizer failing due to non-existing parent vars
>
>
> Key: SYSTEMML-2485
> URL: https://issues.apache.org/jira/browse/SYSTEMML-2485
> Project: SystemML
> Issue Type: Bug
>Reporter: Matthias Boehm
>Assignee: Matthias Boehm
>Priority: Major
>
> Encountered in old apply transform script.
> {code}
> Caused by: java.lang.NullPointerException
> at
> org.apache.sysml.runtime.controlprogram.parfor.opt.OptimizerRuleBased.rewriteSetDataPartitioner(OptimizerRuleBased.java:422)
> at
> org.apache.sysml.runtime.controlprogram.parfor.opt.OptimizerRuleBased.optimize(OptimizerRuleBased.java:231)
> at
> org.apache.sysml.runtime.controlprogram.parfor.opt.OptimizationWrapper.optimize(OptimizationWrapper.java:243)
> at
> org.apache.sysml.runtime.controlprogram.parfor.opt.OptimizationWrapper.optimize(OptimizationWrapper.java:116)
> at
> org.apache.sysml.runtime.controlprogram.ParForProgramBlock.execute(ParForProgramBlock.java:599)
> at
> org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:116)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (SYSTEMML-2485) Parfor optimizer failing due to non-existing parent vars
[
https://issues.apache.org/jira/browse/SYSTEMML-2485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Boehm closed SYSTEMML-2485.
Fix Version/s: SystemML 1.2
Resolution: Fixed
> Parfor optimizer failing due to non-existing parent vars
>
>
> Key: SYSTEMML-2485
> URL: https://issues.apache.org/jira/browse/SYSTEMML-2485
> Project: SystemML
> Issue Type: Bug
>Reporter: Matthias Boehm
>Assignee: Matthias Boehm
>Priority: Major
> Fix For: SystemML 1.2
>
>
> Encountered in old apply transform script.
> {code}
> Caused by: java.lang.NullPointerException
> at
> org.apache.sysml.runtime.controlprogram.parfor.opt.OptimizerRuleBased.rewriteSetDataPartitioner(OptimizerRuleBased.java:422)
> at
> org.apache.sysml.runtime.controlprogram.parfor.opt.OptimizerRuleBased.optimize(OptimizerRuleBased.java:231)
> at
> org.apache.sysml.runtime.controlprogram.parfor.opt.OptimizationWrapper.optimize(OptimizationWrapper.java:243)
> at
> org.apache.sysml.runtime.controlprogram.parfor.opt.OptimizationWrapper.optimize(OptimizationWrapper.java:116)
> at
> org.apache.sysml.runtime.controlprogram.ParForProgramBlock.execute(ParForProgramBlock.java:599)
> at
> org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:116)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (SYSTEMML-2530) Integer casting produces incorrect results for negative numbers
[ https://issues.apache.org/jira/browse/SYSTEMML-2530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Boehm closed SYSTEMML-2530. Fix Version/s: SystemDS 2.0 Assignee: Matthias Boehm Resolution: Fixed > Integer casting produces incorrect results for negative numbers > --- > > Key: SYSTEMML-2530 > URL: https://issues.apache.org/jira/browse/SYSTEMML-2530 > Project: SystemML > Issue Type: Bug >Reporter: Matthias Boehm >Assignee: Matthias Boehm >Priority: Major > Fix For: SystemDS 2.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (SYSTEMML-2526) Date.getTime() can be changed to System.currentTimeMillis()
[ https://issues.apache.org/jira/browse/SYSTEMML-2526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Boehm closed SYSTEMML-2526. Fix Version/s: SystemDS 2.0 Resolution: Fixed > Date.getTime() can be changed to System.currentTimeMillis() > --- > > Key: SYSTEMML-2526 > URL: https://issues.apache.org/jira/browse/SYSTEMML-2526 > Project: SystemML > Issue Type: Bug >Reporter: bd2019us >Assignee: Matthias Boehm >Priority: Major > Fix For: SystemDS 2.0 > > Attachments: 1.patch > > > Hello, > I found that System.currentTimeMillis() can be used here instead of new > Date.getTime(). > Since new Date() is a thin wrapper of light method > System.currentTimeMillis(). The performance will be greatly damaged if it is > invoked too much times. > According to my local testing at the same environment, > System.currentTimeMillis() can achieve a speedup to 5 times (435 ms vs 2073 > ms), when these two methods are invoked 5,000,000 times. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (SYSTEMML-2526) Date.getTime() can be changed to System.currentTimeMillis()
[ https://issues.apache.org/jira/browse/SYSTEMML-2526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Boehm reassigned SYSTEMML-2526: Assignee: Matthias Boehm > Date.getTime() can be changed to System.currentTimeMillis() > --- > > Key: SYSTEMML-2526 > URL: https://issues.apache.org/jira/browse/SYSTEMML-2526 > Project: SystemML > Issue Type: Bug >Reporter: bd2019us >Assignee: Matthias Boehm >Priority: Major > Attachments: 1.patch > > > Hello, > I found that System.currentTimeMillis() can be used here instead of new > Date.getTime(). > Since new Date() is a thin wrapper of light method > System.currentTimeMillis(). The performance will be greatly damaged if it is > invoked too much times. > According to my local testing at the same environment, > System.currentTimeMillis() can achieve a speedup to 5 times (435 ms vs 2073 > ms), when these two methods are invoked 5,000,000 times. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (SYSTEMML-2538) Input CSV file is renamed by simple load and save script
[
https://issues.apache.org/jira/browse/SYSTEMML-2538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Boehm closed SYSTEMML-2538.
Fix Version/s: SystemML 1.3
Resolution: Fixed
Thanks again for catching this. In detail, it was indeed an issue of invalid
renames for csv/mm/textcell, which only occurred in special cases of forced
singlenode computation (as done in the example run script) while in cluster
environments and our testsuite, we run in hybrid by default (where
distributed/in-memory reblocks always create new metadata objects). The issue
has been fixed with
https://github.com/apache/systemml/commit/47924e6aced3dac0768756c7dfec932d696b6a3f.
> Input CSV file is renamed by simple load and save script
>
>
> Key: SYSTEMML-2538
> URL: https://issues.apache.org/jira/browse/SYSTEMML-2538
> Project: SystemML
> Issue Type: Bug
> Components: APIs, Compiler
>Affects Versions: SystemML 1.3
>Reporter: Samuel Kogler
>Assignee: Matthias Boehm
>Priority: Minor
> Fix For: SystemML 1.3
>
> Attachments: rename_bug.dml
>
>
> A simple script that reads a CSV file and writes it to an output file just
> renames the input file to the output file.
> This optimization should not be applied to input files but only to temporary
> scratch files.
> Judging from quick local checks, this bug does not seem to affect
> data_type='frame' inputs, but only data_type='matrix'. Test script is
> attached.
> Tested on current master git branch
> (ef8b551103085b73d0b471db9df6c5c0748f7d94) on the file
> {noformat}
> src/test/scripts/functions/io/csv/in/transfusion_1.csv
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (SYSTEMML-2538) Input CSV file is renamed by simple load and save script
[
https://issues.apache.org/jira/browse/SYSTEMML-2538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Boehm reassigned SYSTEMML-2538:
Assignee: Matthias Boehm
> Input CSV file is renamed by simple load and save script
>
>
> Key: SYSTEMML-2538
> URL: https://issues.apache.org/jira/browse/SYSTEMML-2538
> Project: SystemML
> Issue Type: Bug
> Components: APIs, Compiler
>Affects Versions: SystemML 1.3
>Reporter: Samuel Kogler
>Assignee: Matthias Boehm
>Priority: Minor
> Attachments: rename_bug.dml
>
>
> A simple script that reads a CSV file and writes it to an output file just
> renames the input file to the output file.
> This optimization should not be applied to input files but only to temporary
> scratch files.
> Judging from quick local checks, this bug does not seem to affect
> data_type='frame' inputs, but only data_type='matrix'. Test script is
> attached.
> Tested on current master git branch
> (ef8b551103085b73d0b471db9df6c5c0748f7d94) on the file
> {noformat}
> src/test/scripts/functions/io/csv/in/transfusion_1.csv
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (SYSTEMML-2530) Integer casting produces incorrect results for negative numbers
Matthias Boehm created SYSTEMML-2530: Summary: Integer casting produces incorrect results for negative numbers Key: SYSTEMML-2530 URL: https://issues.apache.org/jira/browse/SYSTEMML-2530 Project: SystemML Issue Type: Bug Reporter: Matthias Boehm -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Closed] (SYSTEMML-2479) Support for operations other than mm
[ https://issues.apache.org/jira/browse/SYSTEMML-2479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Boehm closed SYSTEMML-2479. Resolution: Fixed Assignee: Matthias Boehm Fix Version/s: SystemML 1.3 > Support for operations other than mm > > > Key: SYSTEMML-2479 > URL: https://issues.apache.org/jira/browse/SYSTEMML-2479 > Project: SystemML > Issue Type: Sub-task >Reporter: Matthias Boehm >Assignee: Matthias Boehm >Priority: Major > Fix For: SystemML 1.3 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SYSTEMML-2521) Integrate MNC sparsity estimator in mmchain optimization
Matthias Boehm created SYSTEMML-2521: Summary: Integrate MNC sparsity estimator in mmchain optimization Key: SYSTEMML-2521 URL: https://issues.apache.org/jira/browse/SYSTEMML-2521 Project: SystemML Issue Type: Sub-task Reporter: Matthias Boehm -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (SYSTEMML-2291) Sparsity estimators based on layered graphs
[ https://issues.apache.org/jira/browse/SYSTEMML-2291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Boehm closed SYSTEMML-2291. Resolution: Fixed Assignee: Matthias Boehm Fix Version/s: SystemML 1.3 > Sparsity estimators based on layered graphs > --- > > Key: SYSTEMML-2291 > URL: https://issues.apache.org/jira/browse/SYSTEMML-2291 > Project: SystemML > Issue Type: Sub-task >Reporter: Matthias Boehm >Assignee: Matthias Boehm >Priority: Major > Fix For: SystemML 1.3 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (SYSTEMML-2292) Exploiting structural properties for corrections
[ https://issues.apache.org/jira/browse/SYSTEMML-2292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Boehm closed SYSTEMML-2292. Resolution: Fixed Assignee: Matthias Boehm Fix Version/s: SystemML 1.3 > Exploiting structural properties for corrections > - > > Key: SYSTEMML-2292 > URL: https://issues.apache.org/jira/browse/SYSTEMML-2292 > Project: SystemML > Issue Type: Sub-task >Reporter: Matthias Boehm >Assignee: Matthias Boehm >Priority: Major > Fix For: SystemML 1.3 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (SYSTEMML-2468) Extended matrix histograms for matrix product chains
[ https://issues.apache.org/jira/browse/SYSTEMML-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Boehm closed SYSTEMML-2468. Resolution: Fixed Assignee: Matthias Boehm Fix Version/s: SystemML 1.3 > Extended matrix histograms for matrix product chains > > > Key: SYSTEMML-2468 > URL: https://issues.apache.org/jira/browse/SYSTEMML-2468 > Project: SystemML > Issue Type: Sub-task >Reporter: Matthias Boehm >Assignee: Matthias Boehm >Priority: Major > Fix For: SystemML 1.3 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (SYSTEMML-2289) Sparsity estimators based on hashing
[ https://issues.apache.org/jira/browse/SYSTEMML-2289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Boehm closed SYSTEMML-2289. Resolution: Fixed Assignee: Matthias Boehm Fix Version/s: SystemML 1.3 > Sparsity estimators based on hashing > > > Key: SYSTEMML-2289 > URL: https://issues.apache.org/jira/browse/SYSTEMML-2289 > Project: SystemML > Issue Type: Sub-task >Reporter: Matthias Boehm >Assignee: Matthias Boehm >Priority: Major > Fix For: SystemML 1.3 > > > Rasmus Resen Amossen, Andrea Campagna, Rasmus Pagh: Better Size Estimation > for Sparse Matrix Products. Algorithmica 69(3): 741-757 (2014) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (SYSTEMML-2511) Classcast exception in bitset estimator on large data
[
https://issues.apache.org/jira/browse/SYSTEMML-2511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Boehm closed SYSTEMML-2511.
Resolution: Fixed
Assignee: Matthias Boehm
Fix Version/s: SystemML 1.3
> Classcast exception in bitset estimator on large data
> -
>
> Key: SYSTEMML-2511
> URL: https://issues.apache.org/jira/browse/SYSTEMML-2511
> Project: SystemML
> Issue Type: Bug
>Reporter: Matthias Boehm
>Assignee: Matthias Boehm
>Priority: Major
> Fix For: SystemML 1.3
>
>
> A previous refactoring to reuse synopsis, created an issue of incorrectly
> allocating BitsetMatrix1 (the default option, independent of the input size).
> {code}
> Caused by: java.lang.ClassCastException:
> org.apache.sysml.hops.estim.EstimatorBitsetMM$BitsetMatrix1 cannot be cast to
> org.apache.sysml.hops.estim.EstimatorBitsetMM$BitsetMatrix2
> at
> org.apache.sysml.hops.estim.EstimatorBitsetMM$BitsetMatrix2.matMultIntern(EstimatorBitsetMM.java:474)
> at
> org.apache.sysml.hops.estim.EstimatorBitsetMM$BitsetMatrix.matMult(EstimatorBitsetMM.java:172)
> at
> org.apache.sysml.hops.estim.EstimatorBitsetMM.estimInternal(EstimatorBitsetMM.java:99)
> at
> org.apache.sysml.hops.estim.EstimatorBitsetMM.estim(EstimatorBitsetMM.java:71)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (SYSTEMML-2511) Classcast exception in bitset estimator on large data
Matthias Boehm created SYSTEMML-2511:
Summary: Classcast exception in bitset estimator on large data
Key: SYSTEMML-2511
URL: https://issues.apache.org/jira/browse/SYSTEMML-2511
Project: SystemML
Issue Type: Bug
Reporter: Matthias Boehm
A previous refactoring to reuse synopsis, created an issue of incorrectly
allocating BitsetMatrix1 (the default option, independent of the input size).
{code}
Caused by: java.lang.ClassCastException:
org.apache.sysml.hops.estim.EstimatorBitsetMM$BitsetMatrix1 cannot be cast to
org.apache.sysml.hops.estim.EstimatorBitsetMM$BitsetMatrix2
at
org.apache.sysml.hops.estim.EstimatorBitsetMM$BitsetMatrix2.matMultIntern(EstimatorBitsetMM.java:474)
at
org.apache.sysml.hops.estim.EstimatorBitsetMM$BitsetMatrix.matMult(EstimatorBitsetMM.java:172)
at
org.apache.sysml.hops.estim.EstimatorBitsetMM.estimInternal(EstimatorBitsetMM.java:99)
at
org.apache.sysml.hops.estim.EstimatorBitsetMM.estim(EstimatorBitsetMM.java:71)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (SYSTEMML-2509) Missing transformencode suport for binning over frames
Matthias Boehm created SYSTEMML-2509: Summary: Missing transformencode suport for binning over frames Key: SYSTEMML-2509 URL: https://issues.apache.org/jira/browse/SYSTEMML-2509 Project: SystemML Issue Type: Bug Reporter: Matthias Boehm Assignee: Matthias Boehm 1) Support binning in transformencode over frames 2) Support meta data serialization into frames 3) Distributed binning over frames on spark -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (SYSTEMML-2508) Improved compilation of single row block cum agg
[ https://issues.apache.org/jira/browse/SYSTEMML-2508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Boehm closed SYSTEMML-2508. Resolution: Fixed Assignee: Matthias Boehm Fix Version/s: SystemML 1.3 > Improved compilation of single row block cum agg > > > Key: SYSTEMML-2508 > URL: https://issues.apache.org/jira/browse/SYSTEMML-2508 > Project: SystemML > Issue Type: Task >Reporter: Matthias Boehm >Assignee: Matthias Boehm >Priority: Major > Fix For: SystemML 1.3 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SYSTEMML-2508) Improved compilation of single row block cum agg
Matthias Boehm created SYSTEMML-2508: Summary: Improved compilation of single row block cum agg Key: SYSTEMML-2508 URL: https://issues.apache.org/jira/browse/SYSTEMML-2508 Project: SystemML Issue Type: Task Reporter: Matthias Boehm -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (SYSTEMML-2507) New rewrites for cumulative aggregates
[ https://issues.apache.org/jira/browse/SYSTEMML-2507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Boehm closed SYSTEMML-2507. Resolution: Fixed Assignee: Matthias Boehm Fix Version/s: SystemML 1.3 > New rewrites for cumulative aggregates > -- > > Key: SYSTEMML-2507 > URL: https://issues.apache.org/jira/browse/SYSTEMML-2507 > Project: SystemML > Issue Type: Task >Reporter: Matthias Boehm >Assignee: Matthias Boehm >Priority: Major > Fix For: SystemML 1.3 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SYSTEMML-2507) New rewrites for cumulative aggregates
Matthias Boehm created SYSTEMML-2507: Summary: New rewrites for cumulative aggregates Key: SYSTEMML-2507 URL: https://issues.apache.org/jira/browse/SYSTEMML-2507 Project: SystemML Issue Type: Task Reporter: Matthias Boehm -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SYSTEMML-2506) Improved cumagg operator selection (memory)
Matthias Boehm created SYSTEMML-2506: Summary: Improved cumagg operator selection (memory) Key: SYSTEMML-2506 URL: https://issues.apache.org/jira/browse/SYSTEMML-2506 Project: SystemML Issue Type: Task Reporter: Matthias Boehm -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (SYSTEMML-2501) Sparse aggreate communication in spark cumagg
[ https://issues.apache.org/jira/browse/SYSTEMML-2501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Boehm closed SYSTEMML-2501. Resolution: Fixed Assignee: Matthias Boehm Fix Version/s: SystemML 1.3 > Sparse aggreate communication in spark cumagg > - > > Key: SYSTEMML-2501 > URL: https://issues.apache.org/jira/browse/SYSTEMML-2501 > Project: SystemML > Issue Type: Task >Reporter: Matthias Boehm >Assignee: Matthias Boehm >Priority: Major > Fix For: SystemML 1.3 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (SYSTEMML-2504) In-place CP cumulative aggregate, incl compiler integration
[ https://issues.apache.org/jira/browse/SYSTEMML-2504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Boehm closed SYSTEMML-2504. Resolution: Fixed Assignee: Matthias Boehm Fix Version/s: SystemML 1.3 > In-place CP cumulative aggregate, incl compiler integration > --- > > Key: SYSTEMML-2504 > URL: https://issues.apache.org/jira/browse/SYSTEMML-2504 > Project: SystemML > Issue Type: Task >Reporter: Matthias Boehm >Assignee: Matthias Boehm >Priority: Major > Fix For: SystemML 1.3 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (SYSTEMML-2500) Async matrix block allocation on collect
[ https://issues.apache.org/jira/browse/SYSTEMML-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Boehm closed SYSTEMML-2500. Resolution: Fixed Assignee: Matthias Boehm Fix Version/s: SystemML 1.3 > Async matrix block allocation on collect > > > Key: SYSTEMML-2500 > URL: https://issues.apache.org/jira/browse/SYSTEMML-2500 > Project: SystemML > Issue Type: Task >Reporter: Matthias Boehm >Assignee: Matthias Boehm >Priority: Major > Fix For: SystemML 1.3 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (SYSTEMML-2502) Avoid unnecessary data copy on cumagg offset aggregation
[ https://issues.apache.org/jira/browse/SYSTEMML-2502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Boehm closed SYSTEMML-2502. Resolution: Fixed Assignee: Matthias Boehm Fix Version/s: SystemML 1.3 > Avoid unnecessary data copy on cumagg offset aggregation > > > Key: SYSTEMML-2502 > URL: https://issues.apache.org/jira/browse/SYSTEMML-2502 > Project: SystemML > Issue Type: Task >Reporter: Matthias Boehm >Assignee: Matthias Boehm >Priority: Major > Fix For: SystemML 1.3 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SYSTEMML-2504) In-place CP cumulative aggregate, incl compiler integration
Matthias Boehm created SYSTEMML-2504: Summary: In-place CP cumulative aggregate, incl compiler integration Key: SYSTEMML-2504 URL: https://issues.apache.org/jira/browse/SYSTEMML-2504 Project: SystemML Issue Type: Task Reporter: Matthias Boehm -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SYSTEMML-2503) Exploit hash partitioning in spark cumoff operations
Matthias Boehm created SYSTEMML-2503: Summary: Exploit hash partitioning in spark cumoff operations Key: SYSTEMML-2503 URL: https://issues.apache.org/jira/browse/SYSTEMML-2503 Project: SystemML Issue Type: Task Reporter: Matthias Boehm -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SYSTEMML-2502) Avoid unnecessary data copy on cumagg offset aggregation
Matthias Boehm created SYSTEMML-2502: Summary: Avoid unnecessary data copy on cumagg offset aggregation Key: SYSTEMML-2502 URL: https://issues.apache.org/jira/browse/SYSTEMML-2502 Project: SystemML Issue Type: Task Reporter: Matthias Boehm -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SYSTEMML-2501) Sparse aggreate communication in spark cumagg
Matthias Boehm created SYSTEMML-2501: Summary: Sparse aggreate communication in spark cumagg Key: SYSTEMML-2501 URL: https://issues.apache.org/jira/browse/SYSTEMML-2501 Project: SystemML Issue Type: Task Reporter: Matthias Boehm -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SYSTEMML-2500) Async matrix block allocation on collect
[ https://issues.apache.org/jira/browse/SYSTEMML-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Boehm updated SYSTEMML-2500: - Issue Type: Task (was: Sub-task) Parent: (was: SYSTEMML-2493) > Async matrix block allocation on collect > > > Key: SYSTEMML-2500 > URL: https://issues.apache.org/jira/browse/SYSTEMML-2500 > Project: SystemML > Issue Type: Task >Reporter: Matthias Boehm >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SYSTEMML-2500) Async matrix block allocation on collect
Matthias Boehm created SYSTEMML-2500: Summary: Async matrix block allocation on collect Key: SYSTEMML-2500 URL: https://issues.apache.org/jira/browse/SYSTEMML-2500 Project: SystemML Issue Type: Sub-task Reporter: Matthias Boehm -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SYSTEMML-2498) Codegen compile issues with LSTM nn function
Matthias Boehm created SYSTEMML-2498: Summary: Codegen compile issues with LSTM nn function Key: SYSTEMML-2498 URL: https://issues.apache.org/jira/browse/SYSTEMML-2498 Project: SystemML Issue Type: Task Reporter: Matthias Boehm -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (SYSTEMML-2490) Improve rewrite update-in-place (data dependent reads)
[ https://issues.apache.org/jira/browse/SYSTEMML-2490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Boehm closed SYSTEMML-2490. Resolution: Fixed Assignee: Matthias Boehm Fix Version/s: SystemML 1.3 > Improve rewrite update-in-place (data dependent reads) > -- > > Key: SYSTEMML-2490 > URL: https://issues.apache.org/jira/browse/SYSTEMML-2490 > Project: SystemML > Issue Type: Sub-task >Reporter: Matthias Boehm >Assignee: Matthias Boehm >Priority: Major > Fix For: SystemML 1.3 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (SYSTEMML-2489) Improvements for update-in-place
[ https://issues.apache.org/jira/browse/SYSTEMML-2489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Boehm closed SYSTEMML-2489. Resolution: Fixed Assignee: Matthias Boehm > Improvements for update-in-place > > > Key: SYSTEMML-2489 > URL: https://issues.apache.org/jira/browse/SYSTEMML-2489 > Project: SystemML > Issue Type: Task >Reporter: Matthias Boehm >Assignee: Matthias Boehm >Priority: Major > Fix For: SystemML 1.3 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (SYSTEMML-2491) Avoid unnecessary instruction overheads
[ https://issues.apache.org/jira/browse/SYSTEMML-2491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Boehm closed SYSTEMML-2491. Resolution: Fixed Assignee: Matthias Boehm Fix Version/s: SystemML 1.3 > Avoid unnecessary instruction overheads > --- > > Key: SYSTEMML-2491 > URL: https://issues.apache.org/jira/browse/SYSTEMML-2491 > Project: SystemML > Issue Type: Sub-task >Reporter: Matthias Boehm >Assignee: Matthias Boehm >Priority: Major > Fix For: SystemML 1.3 > > > 1) Parsing of scalar literal arguments > 2) Createvar reuse of string builders > 3) Avoid stream allocation on rmvar reference analysis -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (SYSTEMML-2493) Performance cumulative aggregates
[ https://issues.apache.org/jira/browse/SYSTEMML-2493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Boehm closed SYSTEMML-2493. Resolution: Fixed Assignee: Matthias Boehm Fix Version/s: SystemML 1.3 > Performance cumulative aggregates > - > > Key: SYSTEMML-2493 > URL: https://issues.apache.org/jira/browse/SYSTEMML-2493 > Project: SystemML > Issue Type: Epic >Reporter: Matthias Boehm >Assignee: Matthias Boehm >Priority: Major > Fix For: SystemML 1.3 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (SYSTEMML-2492) Broadcast-based spark cumulative aggregates
[ https://issues.apache.org/jira/browse/SYSTEMML-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Boehm closed SYSTEMML-2492. Resolution: Fixed Assignee: Matthias Boehm Fix Version/s: SystemML 1.3 > Broadcast-based spark cumulative aggregates > --- > > Key: SYSTEMML-2492 > URL: https://issues.apache.org/jira/browse/SYSTEMML-2492 > Project: SystemML > Issue Type: Task >Reporter: Matthias Boehm >Assignee: Matthias Boehm >Priority: Major > Fix For: SystemML 1.3 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (SYSTEMML-2494) Caching in multi-level spark cumulative aggregates
[ https://issues.apache.org/jira/browse/SYSTEMML-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Boehm closed SYSTEMML-2494. Resolution: Fixed Assignee: Matthias Boehm Fix Version/s: SystemML 1.3 > Caching in multi-level spark cumulative aggregates > -- > > Key: SYSTEMML-2494 > URL: https://issues.apache.org/jira/browse/SYSTEMML-2494 > Project: SystemML > Issue Type: Task >Reporter: Matthias Boehm >Assignee: Matthias Boehm >Priority: Major > Fix For: SystemML 1.3 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (SYSTEMML-2495) Adjust number of partitions in spark cumulative aggregate forward
[ https://issues.apache.org/jira/browse/SYSTEMML-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Boehm closed SYSTEMML-2495. Resolution: Fixed Assignee: Matthias Boehm Fix Version/s: SystemML 1.3 > Adjust number of partitions in spark cumulative aggregate forward > - > > Key: SYSTEMML-2495 > URL: https://issues.apache.org/jira/browse/SYSTEMML-2495 > Project: SystemML > Issue Type: Task >Reporter: Matthias Boehm >Assignee: Matthias Boehm >Priority: Major > Fix For: SystemML 1.3 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SYSTEMML-2495) Adjust number of partitions in spark cumulative aggregate forward
Matthias Boehm created SYSTEMML-2495: Summary: Adjust number of partitions in spark cumulative aggregate forward Key: SYSTEMML-2495 URL: https://issues.apache.org/jira/browse/SYSTEMML-2495 Project: SystemML Issue Type: Task Reporter: Matthias Boehm -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SYSTEMML-2494) Caching in multi-level spark cumulative aggregates
[ https://issues.apache.org/jira/browse/SYSTEMML-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Boehm updated SYSTEMML-2494: - Summary: Caching in multi-level spark cumulative aggregates (was: Caching in multi-level cumulative aggregates) > Caching in multi-level spark cumulative aggregates > -- > > Key: SYSTEMML-2494 > URL: https://issues.apache.org/jira/browse/SYSTEMML-2494 > Project: SystemML > Issue Type: Task >Reporter: Matthias Boehm >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SYSTEMML-2494) Caching in multi-level cumulative aggregates
Matthias Boehm created SYSTEMML-2494: Summary: Caching in multi-level cumulative aggregates Key: SYSTEMML-2494 URL: https://issues.apache.org/jira/browse/SYSTEMML-2494 Project: SystemML Issue Type: Task Reporter: Matthias Boehm -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SYSTEMML-2493) Performance cumulative aggregates
Matthias Boehm created SYSTEMML-2493: Summary: Performance cumulative aggregates Key: SYSTEMML-2493 URL: https://issues.apache.org/jira/browse/SYSTEMML-2493 Project: SystemML Issue Type: Epic Reporter: Matthias Boehm -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SYSTEMML-2492) Broadcast-based spark cumulative aggregates
Matthias Boehm created SYSTEMML-2492: Summary: Broadcast-based spark cumulative aggregates Key: SYSTEMML-2492 URL: https://issues.apache.org/jira/browse/SYSTEMML-2492 Project: SystemML Issue Type: Task Reporter: Matthias Boehm -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SYSTEMML-2491) Avoid unnecessary instruction overheads
[ https://issues.apache.org/jira/browse/SYSTEMML-2491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Boehm updated SYSTEMML-2491: - Description: 1) Parsing of scalar literal arguments 2) Createvar reuse of string builders 3) Avoid stream allocation on rmvar reference analysis was: 1) Parsing of scalar literal arguments 2) > Avoid unnecessary instruction overheads > --- > > Key: SYSTEMML-2491 > URL: https://issues.apache.org/jira/browse/SYSTEMML-2491 > Project: SystemML > Issue Type: Sub-task >Reporter: Matthias Boehm >Priority: Major > > 1) Parsing of scalar literal arguments > 2) Createvar reuse of string builders > 3) Avoid stream allocation on rmvar reference analysis -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SYSTEMML-2491) Avoid unnecessary instruction overheads
Matthias Boehm created SYSTEMML-2491: Summary: Avoid unnecessary instruction overheads Key: SYSTEMML-2491 URL: https://issues.apache.org/jira/browse/SYSTEMML-2491 Project: SystemML Issue Type: Sub-task Reporter: Matthias Boehm 1) Parsing of scalar literal arguments 2) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SYSTEMML-2490) Improve rewrite update-in-place (data dependent reads)
Matthias Boehm created SYSTEMML-2490: Summary: Improve rewrite update-in-place (data dependent reads) Key: SYSTEMML-2490 URL: https://issues.apache.org/jira/browse/SYSTEMML-2490 Project: SystemML Issue Type: Sub-task Reporter: Matthias Boehm -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SYSTEMML-2489) Improvements for update-in-place
Matthias Boehm created SYSTEMML-2489: Summary: Improvements for update-in-place Key: SYSTEMML-2489 URL: https://issues.apache.org/jira/browse/SYSTEMML-2489 Project: SystemML Issue Type: Task Reporter: Matthias Boehm Fix For: SystemML 1.3 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SYSTEMML-2488) Function output binding via multi-assignment fails on validate
Matthias Boehm created SYSTEMML-2488:
Summary: Function output binding via multi-assignment fails on
validate
Key: SYSTEMML-2488
URL: https://issues.apache.org/jira/browse/SYSTEMML-2488
Project: SystemML
Issue Type: Sub-task
Reporter: Matthias Boehm
{code}
foo = function(Matrix[Double] A) return (Matrix[Double] B) {
#B = A;
[w, B] = eigen(A)
}
X = matrix(0.1, rows=100, cols=100);
Y = foo(X);
print(toString(Y))
{code}
just outputs {{No parse issue message.}} without script execution or any error.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (SYSTEMML-2410) Perftest for SystemML 1.2 release
[ https://issues.apache.org/jira/browse/SYSTEMML-2410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Boehm updated SYSTEMML-2410: - Issue Type: Bug (was: Task) > Perftest for SystemML 1.2 release > - > > Key: SYSTEMML-2410 > URL: https://issues.apache.org/jira/browse/SYSTEMML-2410 > Project: SystemML > Issue Type: Bug >Reporter: Matthias Boehm >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (SYSTEMML-2487) Native Dnn operations crashing in over-provisioned parfor
[
https://issues.apache.org/jira/browse/SYSTEMML-2487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Boehm closed SYSTEMML-2487.
Resolution: Fixed
Assignee: Matthias Boehm
Fix Version/s: SystemML 1.2
> Native Dnn operations crashing in over-provisioned parfor
> -
>
> Key: SYSTEMML-2487
> URL: https://issues.apache.org/jira/browse/SYSTEMML-2487
> Project: SystemML
> Issue Type: Bug
>Reporter: Matthias Boehm
>Assignee: Matthias Boehm
>Priority: Major
> Fix For: SystemML 1.2
>
>
> In case parfor does not consume all the available parallelism, we propagate
> this parallelism down to individual operations with slight (max 50%)
> overprovisioning. For example, if we have 80vcores, and parfor is assigned
> k=47, we still assign k=2 to individual operations.
> However, with native DNN operations this causes JVM crashes as follows:
> {code}
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGFPE (0x8) at pc=0x7f5de21902d6, pid=335027, tid=0x7f5df8bcb700
> #
> # JRE version: OpenJDK Runtime Environment (8.0_161-b14) (build 1.8.0_161-b14)
> # Java VM: OpenJDK 64-Bit Server VM (25.161-b14 mixed mode linux-amd64 )
> # Problematic frame:
> # C [libmkl_avx512.so+0x206d2d6][thread 140041622857472 also had an error]
> mkl_dnn_avx512_bkdGemmDirectConv_F64+0x276
> {code}
> Hence, when native BLAS or DNN libraries are loaded, we should be more
> conservative and not over-provision at all.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (SYSTEMML-2487) Native Dnn operations crashing in over-provisioned parfor
Matthias Boehm created SYSTEMML-2487:
Summary: Native Dnn operations crashing in over-provisioned parfor
Key: SYSTEMML-2487
URL: https://issues.apache.org/jira/browse/SYSTEMML-2487
Project: SystemML
Issue Type: Bug
Reporter: Matthias Boehm
In case parfor does not consume all the available parallelism, we propagate
this parallelism down to individual operations with slight (max 50%)
overprovisioning. For example, if we have 80vcores, and parfor is assigned
k=47, we still assign k=2 to individual operations.
However, with native DNN operations this causes JVM crashes as follows:
{code}
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGFPE (0x8) at pc=0x7f5de21902d6, pid=335027, tid=0x7f5df8bcb700
#
# JRE version: OpenJDK Runtime Environment (8.0_161-b14) (build 1.8.0_161-b14)
# Java VM: OpenJDK 64-Bit Server VM (25.161-b14 mixed mode linux-amd64 )
# Problematic frame:
# C [libmkl_avx512.so+0x206d2d6][thread 140041622857472 also had an error]
mkl_dnn_avx512_bkdGemmDirectConv_F64+0x276
{code}
Hence, when native BLAS or DNN libraries are loaded, we should be more
conservative and not over-provision at all.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (SYSTEMML-2486) Performance features sparsity estimators
Matthias Boehm created SYSTEMML-2486: Summary: Performance features sparsity estimators Key: SYSTEMML-2486 URL: https://issues.apache.org/jira/browse/SYSTEMML-2486 Project: SystemML Issue Type: Sub-task Reporter: Matthias Boehm This includes features such as: * Multi-threaded sketch construction * Multi-threaded estimation * Reduced memory footprint and special cases -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SYSTEMML-2458) Add experiment on spark paramserv
[ https://issues.apache.org/jira/browse/SYSTEMML-2458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16569601#comment-16569601 ] Matthias Boehm commented on SYSTEMML-2458: -- Thanks - the adagrad results are in the repo; currently adam and sgd are running. One observation is that ASP-batch is much slower than BSP-batch. It's understandable because for BSP-batch we simply accure gradients and perform one update for all workers but this effect should not be that pronounced. > Add experiment on spark paramserv > - > > Key: SYSTEMML-2458 > URL: https://issues.apache.org/jira/browse/SYSTEMML-2458 > Project: SystemML > Issue Type: Sub-task >Reporter: LI Guobao >Assignee: LI Guobao >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SYSTEMML-2458) Add experiment on spark paramserv
[ https://issues.apache.org/jira/browse/SYSTEMML-2458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16569372#comment-16569372 ] Matthias Boehm commented on SYSTEMML-2458: -- OK I just kicked of a run for LOCAL experiments with MKL. However, note that the SystemML-config.xml file needs to be in each of the subdirectories otherwise it's not picked up correctly. Also, the Intel MKL's direct conv2d still runs into segmentation faults on this new architecture whenever the batchsize larger than 64 and hence I limited it to max 64. Tomorrow, I will kickoff baseline runs (e.g., without parameter server, varying number of workers, and with our java backend operations). The distributed experiments will follow subsequently. > Add experiment on spark paramserv > - > > Key: SYSTEMML-2458 > URL: https://issues.apache.org/jira/browse/SYSTEMML-2458 > Project: SystemML > Issue Type: Sub-task >Reporter: LI Guobao >Assignee: LI Guobao >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SYSTEMML-2458) Add experiment on spark paramserv
[ https://issues.apache.org/jira/browse/SYSTEMML-2458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16569318#comment-16569318 ] Matthias Boehm commented on SYSTEMML-2458: -- Sure, I'm happy to kickoff additional rounds for local and distributed experiments. For the presentation, it would also be important to have baseline comparisons. Could you please add the baseline without paramserv to the experiments. Furthermore, I'll run these experiments with MKL so please double check that the native BLAS configuration is correctly set for distributed spark workers as well (see remote parfor worker setup) > Add experiment on spark paramserv > - > > Key: SYSTEMML-2458 > URL: https://issues.apache.org/jira/browse/SYSTEMML-2458 > Project: SystemML > Issue Type: Sub-task >Reporter: LI Guobao >Assignee: LI Guobao >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SYSTEMML-2485) Parfor optimizer failing due to non-existing parent vars
Matthias Boehm created SYSTEMML-2485:
Summary: Parfor optimizer failing due to non-existing parent vars
Key: SYSTEMML-2485
URL: https://issues.apache.org/jira/browse/SYSTEMML-2485
Project: SystemML
Issue Type: Bug
Reporter: Matthias Boehm
Encountered in old apply transform script.
{code}
Caused by: java.lang.NullPointerException
at
org.apache.sysml.runtime.controlprogram.parfor.opt.OptimizerRuleBased.rewriteSetDataPartitioner(OptimizerRuleBased.java:422)
at
org.apache.sysml.runtime.controlprogram.parfor.opt.OptimizerRuleBased.optimize(OptimizerRuleBased.java:231)
at
org.apache.sysml.runtime.controlprogram.parfor.opt.OptimizationWrapper.optimize(OptimizationWrapper.java:243)
at
org.apache.sysml.runtime.controlprogram.parfor.opt.OptimizationWrapper.optimize(OptimizationWrapper.java:116)
at
org.apache.sysml.runtime.controlprogram.ParForProgramBlock.execute(ParForProgramBlock.java:599)
at
org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:116)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (SYSTEMML-2484) AutoEncoder codegen failing w/ enabled rewrites
[
https://issues.apache.org/jira/browse/SYSTEMML-2484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Boehm closed SYSTEMML-2484.
Resolution: Fixed
Assignee: Matthias Boehm
Fix Version/s: SystemML 1.2
> AutoEncoder codegen failing w/ enabled rewrites
> ---
>
> Key: SYSTEMML-2484
> URL: https://issues.apache.org/jira/browse/SYSTEMML-2484
> Project: SystemML
> Issue Type: Bug
>Reporter: Matthias Boehm
>Assignee: Matthias Boehm
>Priority: Major
> Fix For: SystemML 1.2
>
>
> {code}
> public final class TMP13 extends SpoofRowwise {
> public TMP13() {
> super(RowType.NO_AGG, -1, false, 2);
> }
> protected void genexec(double[] a, int ai, SideInput[] b, double[] scalars,
> double[] c, int ci, int len, long grix, int rix) {
> double TMP5 = getValue(b[0], n, rix, cix);
> double TMP6 = getValue(b[1], n, rix, cix);
> double TMP7 = TMP5 - scalars[0] * TMP6;
> double[] TMP8 = LibSpoofPrimitives.vectPlusWrite(a, TMP7, ai, len);
> LibSpoofPrimitives.vectWrite(TMP8, c, ci, TMP8.length);
> }
> protected void genexec(double[] avals, int[] aix, int ai, SideInput[] b,
> double[] scalars, double[] c, int ci, int alen, int len, long grix, int rix)
> {
> double TMP9 = getValue(b[0], n, rix, cix);
> double TMP10 = getValue(b[1], n, rix, cix);
> double TMP11 = TMP9 - scalars[0] * TMP10;
> double[] TMP12 = LibSpoofPrimitives.vectPlusWrite(avals, TMP11, aix, ai,
> alen, len);
> LibSpoofPrimitives.vectWrite(TMP12, c, ci, TMP12.length);
> }
> }
> {code}
> {code}
> Caused by: org.apache.sysml.runtime.DMLRuntimeException: Failed to compile
> class codegen.TMP13.
> at
> org.apache.sysml.runtime.codegen.CodegenUtils.compileClassJanino(CodegenUtils.java:185)
> at
> org.apache.sysml.runtime.codegen.CodegenUtils.compileClass(CodegenUtils.java:80)
> at
> org.apache.sysml.hops.codegen.SpoofCompiler.optimize(SpoofCompiler.java:402)
> ... 40 more
> Caused by: org.codehaus.commons.compiler.CompileException: Line 13, Column
> 35: Unknown variable or type "n"
> at
> org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:11821)
> at org.codehaus.janino.UnitCompiler.getType2(UnitCompiler.java:6521)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (SYSTEMML-2484) AutoEncoder codegen failing w/ enabled rewrites
Matthias Boehm created SYSTEMML-2484:
Summary: AutoEncoder codegen failing w/ enabled rewrites
Key: SYSTEMML-2484
URL: https://issues.apache.org/jira/browse/SYSTEMML-2484
Project: SystemML
Issue Type: Bug
Reporter: Matthias Boehm
{code}
public final class TMP13 extends SpoofRowwise {
public TMP13() {
super(RowType.NO_AGG, -1, false, 2);
}
protected void genexec(double[] a, int ai, SideInput[] b, double[] scalars,
double[] c, int ci, int len, long grix, int rix) {
double TMP5 = getValue(b[0], n, rix, cix);
double TMP6 = getValue(b[1], n, rix, cix);
double TMP7 = TMP5 - scalars[0] * TMP6;
double[] TMP8 = LibSpoofPrimitives.vectPlusWrite(a, TMP7, ai, len);
LibSpoofPrimitives.vectWrite(TMP8, c, ci, TMP8.length);
}
protected void genexec(double[] avals, int[] aix, int ai, SideInput[] b,
double[] scalars, double[] c, int ci, int alen, int len, long grix, int rix) {
double TMP9 = getValue(b[0], n, rix, cix);
double TMP10 = getValue(b[1], n, rix, cix);
double TMP11 = TMP9 - scalars[0] * TMP10;
double[] TMP12 = LibSpoofPrimitives.vectPlusWrite(avals, TMP11, aix, ai,
alen, len);
LibSpoofPrimitives.vectWrite(TMP12, c, ci, TMP12.length);
}
}
{code}
{code}
Caused by: org.apache.sysml.runtime.DMLRuntimeException: Failed to compile
class codegen.TMP13.
at
org.apache.sysml.runtime.codegen.CodegenUtils.compileClassJanino(CodegenUtils.java:185)
at
org.apache.sysml.runtime.codegen.CodegenUtils.compileClass(CodegenUtils.java:80)
at
org.apache.sysml.hops.codegen.SpoofCompiler.optimize(SpoofCompiler.java:402)
... 40 more
Caused by: org.codehaus.commons.compiler.CompileException: Line 13, Column 35:
Unknown variable or type "n"
at
org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:11821)
at org.codehaus.janino.UnitCompiler.getType2(UnitCompiler.java:6521)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (SYSTEMML-2483) Failing matrix market to binary reblock with zero rows/columns
[
https://issues.apache.org/jira/browse/SYSTEMML-2483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Boehm closed SYSTEMML-2483.
Resolution: Fixed
Assignee: Matthias Boehm
Fix Version/s: SystemML 1.2
> Failing matrix market to binary reblock with zero rows/columns
> --
>
> Key: SYSTEMML-2483
> URL: https://issues.apache.org/jira/browse/SYSTEMML-2483
> Project: SystemML
> Issue Type: Bug
>Reporter: Matthias Boehm
>Assignee: Matthias Boehm
>Priority: Major
> Fix For: SystemML 1.2
>
>
> {code}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.sysml.runtime.matrix.mapred.ReblockBuffer.appendCell(ReblockBuffer.java:68)
> at
> org.apache.sysml.runtime.instructions.spark.utils.RDDConverterUtils$TextToBinaryBlockFunction.call(RDDConverterUtils.java:544)
> at
> org.apache.sysml.runtime.instructions.spark.utils.RDDConverterUtils$TextToBinaryBlockFunction.call(RDDConverterUtils.java:1)
> at
> org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$7$1.apply(JavaRDDLike.scala:186)
> at
> org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$7$1.apply(JavaRDDLike.scala:186)
> at
> org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:796)
> at
> org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:796)
> at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
> at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:105)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
> at org.apache.spark.scheduler.Task.run(Task.scala:99)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (SYSTEMML-2483) Failing matrix market to binary reblock with zero rows/columns
Matthias Boehm created SYSTEMML-2483:
Summary: Failing matrix market to binary reblock with zero
rows/columns
Key: SYSTEMML-2483
URL: https://issues.apache.org/jira/browse/SYSTEMML-2483
Project: SystemML
Issue Type: Bug
Reporter: Matthias Boehm
{code}
Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
at
org.apache.sysml.runtime.matrix.mapred.ReblockBuffer.appendCell(ReblockBuffer.java:68)
at
org.apache.sysml.runtime.instructions.spark.utils.RDDConverterUtils$TextToBinaryBlockFunction.call(RDDConverterUtils.java:544)
at
org.apache.sysml.runtime.instructions.spark.utils.RDDConverterUtils$TextToBinaryBlockFunction.call(RDDConverterUtils.java:1)
at
org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$7$1.apply(JavaRDDLike.scala:186)
at
org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$7$1.apply(JavaRDDLike.scala:186)
at
org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:796)
at
org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:796)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:105)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (SYSTEMML-2480) Too aggressive reuse of vectors in codegen row templates
[ https://issues.apache.org/jira/browse/SYSTEMML-2480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Boehm closed SYSTEMML-2480. Resolution: Fixed Assignee: Matthias Boehm Fix Version/s: SystemML 1.2 > Too aggressive reuse of vectors in codegen row templates > > > Key: SYSTEMML-2480 > URL: https://issues.apache.org/jira/browse/SYSTEMML-2480 > Project: SystemML > Issue Type: Bug >Reporter: Matthias Boehm >Assignee: Matthias Boehm >Priority: Major > Fix For: SystemML 1.2 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (SYSTEMML-2067) Add support for convolution operations (forward/backward)
[ https://issues.apache.org/jira/browse/SYSTEMML-2067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Boehm closed SYSTEMML-2067. Resolution: Fixed Fix Version/s: SystemML 1.2 > Add support for convolution operations (forward/backward) > - > > Key: SYSTEMML-2067 > URL: https://issues.apache.org/jira/browse/SYSTEMML-2067 > Project: SystemML > Issue Type: Sub-task >Reporter: Matthias Boehm >Assignee: Matthias Boehm >Priority: Major > Fix For: SystemML 1.2 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SYSTEMML-2482) Unexpected cleanup of list object
[
https://issues.apache.org/jira/browse/SYSTEMML-2482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16567464#comment-16567464
]
Matthias Boehm commented on SYSTEMML-2482:
--
OK I think you were talking about unnecessary evictions. The reason was that
paramserv relies on the status of list objects which is no longer used or
maintained. However, I now temporarily enabled it again for paramserv to
restore the original behavior. Down the road we should completely remove this
status and all related usage.
> Unexpected cleanup of list object
> -
>
> Key: SYSTEMML-2482
> URL: https://issues.apache.org/jira/browse/SYSTEMML-2482
> Project: SystemML
> Issue Type: Bug
>Reporter: LI Guobao
>Priority: Major
>
> Some unexpected overhead occurred when running the
> {{*testParamservASPEpochDisjointContiguous*}} in test
> {{*org.apache.sysml.test.integration.functions.paramserv.ParamservSparkNNTest*}}.
> It took more time to finish the test in the case that the output of
> instruction is a list which will be cleaned up after the execution. However,
> the matrices referenced by the list should be pinned to avoid being cleaned
> up. And this issue is related to
> [SYSTEMML-2481|https://issues.apache.org/jira/browse/SYSTEMML-2481]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SYSTEMML-2482) Unexpected cleanup of list object
[
https://issues.apache.org/jira/browse/SYSTEMML-2482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16567381#comment-16567381
]
Matthias Boehm commented on SYSTEMML-2482:
--
Could you please be more specific what issue you encountered and how it
manifests itself as it's important to properly cleanup these lists; otherwise
there are unnecessary evictions in the general case. In any case, we might need
to adapt the cleanup logic in paramserv accordingly.
> Unexpected cleanup of list object
> -
>
> Key: SYSTEMML-2482
> URL: https://issues.apache.org/jira/browse/SYSTEMML-2482
> Project: SystemML
> Issue Type: Bug
>Reporter: LI Guobao
>Priority: Major
>
> Some unexpected overhead occurred when running the
> {{*testParamservASPEpochDisjointContiguous*}} in test
> {{*org.apache.sysml.test.integration.functions.paramserv.ParamservSparkNNTest*}}.
> It took more time to finish the test in the case that the output of
> instruction is a list which will be cleaned up after the execution. However,
> the matrices referenced by the list should be pinned to avoid being cleaned
> up. And this issue is related to
> [SYSTEMML-2481|https://issues.apache.org/jira/browse/SYSTEMML-2481]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (SYSTEMML-2481) Missing cleanup of matrices/frames in list objects
Matthias Boehm created SYSTEMML-2481: Summary: Missing cleanup of matrices/frames in list objects Key: SYSTEMML-2481 URL: https://issues.apache.org/jira/browse/SYSTEMML-2481 Project: SystemML Issue Type: Bug Reporter: Matthias Boehm -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SYSTEMML-2480) Too aggressive reuse of vectors in codegen row templates
Matthias Boehm created SYSTEMML-2480: Summary: Too aggressive reuse of vectors in codegen row templates Key: SYSTEMML-2480 URL: https://issues.apache.org/jira/browse/SYSTEMML-2480 Project: SystemML Issue Type: Bug Reporter: Matthias Boehm -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (SYSTEMML-2461) Utililty for determining exact output nnz w/o matrix product
[ https://issues.apache.org/jira/browse/SYSTEMML-2461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Boehm closed SYSTEMML-2461. Resolution: Fixed Assignee: Matthias Boehm Fix Version/s: SystemML 1.2 > Utililty for determining exact output nnz w/o matrix product > > > Key: SYSTEMML-2461 > URL: https://issues.apache.org/jira/browse/SYSTEMML-2461 > Project: SystemML > Issue Type: Sub-task >Reporter: Matthias Boehm >Assignee: Matthias Boehm >Priority: Major > Fix For: SystemML 1.2 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SYSTEMML-2479) Support for operations other than mm
Matthias Boehm created SYSTEMML-2479: Summary: Support for operations other than mm Key: SYSTEMML-2479 URL: https://issues.apache.org/jira/browse/SYSTEMML-2479 Project: SystemML Issue Type: Sub-task Reporter: Matthias Boehm -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SYSTEMML-2478) Overhead when using parfor in update func
[
https://issues.apache.org/jira/browse/SYSTEMML-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16566118#comment-16566118
]
Matthias Boehm commented on SYSTEMML-2478:
--
Well, first of all we're not executing MR but SPARK instructions here. Second,
yes, there seems to be an issue but I was not able to reproduce yet because
(even after fixing the order of model entries to allow indexed access) there
are still some incorrect lookups that ultimately result in dimension mismatches
on aggregation with ADAM. So let's use the sequential aggregation for now and I
have to come back to this later.
> Overhead when using parfor in update func
> -
>
> Key: SYSTEMML-2478
> URL: https://issues.apache.org/jira/browse/SYSTEMML-2478
> Project: SystemML
> Issue Type: Bug
>Reporter: LI Guobao
>Priority: Major
>
> When using parfor inside update function, some MR tasks are launched to write
> the output of task. And it took more time to finish the paramserv run than
> without parfor in update function. The scenario is to launch the ASP Epoch DC
> spark paramserv test.
> Here is the stack:
> {code:java}
> Total elapsed time: 101.804 sec.
> Total compilation time: 3.690 sec.
> Total execution time: 98.114 sec.
> Number of compiled Spark inst:302.
> Number of executed Spark inst:540.
> Cache hits (Mem, WB, FS, HDFS): 57839/0/0/240.
> Cache writes (WB, FS, HDFS): 14567/58/61.
> Cache times (ACQr/m, RLS, EXP): 42.346/0.064/4.761/20.280 sec.
> HOP DAGs recompiled (PRED, SB): 0/144.
> HOP DAGs recompile time: 0.507 sec.
> Functions recompiled: 16.
> Functions recompile time: 0.064 sec.
> Spark ctx create time (lazy): 1.376 sec.
> Spark trans counts (par,bc,col):270/1/240.
> Spark trans times (par,bc,col): 0.573/0.197/42.255 secs.
> Paramserv total num workers: 3.
> Paramserv setup time: 1.559 secs.
> Paramserv grad compute time: 105.701 secs.
> Paramserv model update time: 56.801/47.193 secs.
> Paramserv model broadcast time: 23.872 secs.
> Paramserv batch slice time: 0.000 secs.
> Paramserv RPC request time: 105.159 secs.
> ParFor loops optimized: 1.
> ParFor optimize time: 0.040 sec.
> ParFor initialize time: 0.434 sec.
> ParFor result merge time: 0.005 sec.
> ParFor total update in-place: 0/7/7
> Total JIT compile time: 68.384 sec.
> Total JVM GC count: 1120.
> Total JVM GC time:22.338 sec.
> Heavy hitter instructions:
> # Instruction Time(s) Count
> 1 paramserv97.221 1
> 2 conv2d_bias_add 60.581614
> 3 *54.990 12447
> 4 sp_- 20.625240
> 5 -17.979 7287
> 6 +14.191 12824
> 7 r'5.636 1200
> 8 conv2d_backward_filter5.123600
> 9 max 4.985907
> 10 ba+* 4.591 1814
> {code}
> Here is the polished update func:
> {code:java}
> aggregation = function(list[unknown] model,
>list[unknown] gradients,
>list[unknown] hyperparams)
>return (list[unknown] modelResult) {
> lr = as.double(as.scalar(hyperparams["lr"]))
> mu = as.double(as.scalar(hyperparams["mu"]))
> modelResult = model
> # Optimize with SGD w/ Nesterov momentum
> parfor(i in 1:8, check=0) {
>P = as.matrix(model[i])
>dP = as.matrix(gradients[i])
>vP = as.matrix(model[8+i])
>[P, vP] = sgd_nesterov::update(P, dP, lr, mu, vP)
>modelResult[i] = P
>modelResult[8+i] = vP
> }
>}
> {code}
> [~mboehm7], in fact, I have no idea where the cause comes from? It seems that
> it tried to write the parfor task output into HDFS. So is it the normal
> behavior?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SYSTEMML-2476) Unexpected mapreduce task
[
https://issues.apache.org/jira/browse/SYSTEMML-2476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564750#comment-16564750
]
Matthias Boehm commented on SYSTEMML-2476:
--
thanks for catching this [~Guobao]. Let me demystify this my explaining the
three overlapping issues here:
* You see MR instead of SPARK jobs because the tests did not set SPARK hybrid
mode and hence we're running in hybrid (i.e., CP and MR).
* These distributed operations are caused by a missing literal replacement for
scalar lookups into lists which make C unknown and because the output sizes of
operations in the same DAG depend on C we compile conservative distributed
operations. I have an extension of the recompiler that fixes these unnecessary
distributed operations.
* However, there is a remaining issue. Specifically C comes out of the list
with value type STRING. I made the runtime robust enough to handle this but we
should also fix the root cause. I can have a look into this remaining issue
tomorrow. Until then please leave the JIRA open.
> Unexpected mapreduce task
> -
>
> Key: SYSTEMML-2476
> URL: https://issues.apache.org/jira/browse/SYSTEMML-2476
> Project: SystemML
> Issue Type: Bug
>Reporter: LI Guobao
>Priority: Major
>
> When trying to use scalar casting to get element from a list, unexpected
> mapreduce tasks are launched instead of CP mode. The scenario is to replace
> *C = 1* with *C = as.scalar(hyperparams["C"])* inside the {{_gradient
> function_}} found in
> {{_src/test/scripts/functions/paramserv/mnist_lenet_paramserv.dml_}}. And
> then the problem could be reproduced by launching the method
> {{_testParamservBSPBatchDisjointContiguous_}} inside class
> _{{org.apache.sysml.test.integration.functions.paramserv.ParamservLocalNNTest}}_
> Here is the stack:
> {code:java}
> 18/07/31 22:10:27 INFO mapred.MapTask: numReduceTasks: 1
> 18/07/31 22:10:27 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
> 18/07/31 22:10:27 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
> 18/07/31 22:10:27 INFO mapred.MapTask: soft limit at 83886080
> 18/07/31 22:10:27 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
> 18/07/31 22:10:27 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
> 18/07/31 22:10:27 INFO mapreduce.Job: The url to track the job:
> http://localhost:8080/
> 18/07/31 22:10:27 INFO mapreduce.Job: Running job: job_local792652629_0008
> {code}
> [~mboehm7], if possible, could you take a look on this? And I've double
> checked the creation of execution context in
> {{ParamservBuiltinCPInstruction}}. But it is instance of ExecutionContext not
> SparkExecutionContext.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (SYSTEMML-2475) Leftindexing of matrices into lists fails
Matthias Boehm created SYSTEMML-2475:
Summary: Leftindexing of matrices into lists fails
Key: SYSTEMML-2475
URL: https://issues.apache.org/jira/browse/SYSTEMML-2475
Project: SystemML
Issue Type: Bug
Reporter: Matthias Boehm
The following code currently fails for two reasons: (1) the data type of the
unary aggregate rowMeans is inferred as list from the target, and (2) there is
no support for list-matrix left indexing.
{code}
parfor(i in 1:length(L))
L[i] = rowMeans(as.matrix(L[i]))
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (SYSTEMML-2474) List as result variables in parfor
Matthias Boehm created SYSTEMML-2474: Summary: List as result variables in parfor Key: SYSTEMML-2474 URL: https://issues.apache.org/jira/browse/SYSTEMML-2474 Project: SystemML Issue Type: Sub-task Reporter: Matthias Boehm -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SYSTEMML-2473) Side effect between IPA inlining/dead code and rewrites
Matthias Boehm created SYSTEMML-2473:
Summary: Side effect between IPA inlining/dead code and rewrites
Key: SYSTEMML-2473
URL: https://issues.apache.org/jira/browse/SYSTEMML-2473
Project: SystemML
Issue Type: Task
Reporter: Matthias Boehm
For special cases IPA rewrites showed side effects with IPA function inlining
and dead code elimination leading to a temporarily corrupted function call
graph and thus a null pointer exception
{code}
Caused by: java.lang.NullPointerException
at
org.apache.sysml.hops.ipa.FunctionCallGraph.removeFunctionCall(FunctionCallGraph.java:182)
at
org.apache.sysml.hops.ipa.IPAPassEliminateDeadCode.findAndRemoveDeadCode(IPAPassEliminateDeadCode.java:83)
at
org.apache.sysml.hops.ipa.IPAPassEliminateDeadCode.rewriteProgram(IPAPassEliminateDeadCode.java:57)
at
org.apache.sysml.hops.ipa.InterProceduralAnalysis.analyzeProgram(InterProceduralAnalysis.java:207)
at
org.apache.sysml.parser.DMLTranslator.rewriteHopsDAG(DMLTranslator.java:272)
at org.apache.sysml.api.DMLScript.execute(DMLScript.java:447)
at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:272)
... 10 more
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (SYSTEMML-2472) Failed parsing of named function arguments in special cases
Matthias Boehm created SYSTEMML-2472:
Summary: Failed parsing of named function arguments in special
cases
Key: SYSTEMML-2472
URL: https://issues.apache.org/jira/browse/SYSTEMML-2472
Project: SystemML
Issue Type: Task
Reporter: Matthias Boehm
{code}
debug = function (String msg){
verbosePrint("INFO: " + msg)
}
verbosePrint = function (String msg){
verbose = ifdef($verbose, FALSE)
if (verbose)
print(msg)
}
p = ifdef($p, 2)
debug ("p= " + p)
{code}
{code}
PROGRAM ( size CP/MR = 1/0 )
--FUNCTIONS
FUNCTION CALL GRAPH
--MAIN PROGRAM
.defaultNS::debug
FUNCTION .defaultNS::debug [recompile=false]
--MAIN PROGRAM
GENERIC (lines 36-37) [recompile=false]
--CP extfunct .defaultNS debug 1 0 msg=p= 2.SCALAR.STRING.true
{code}
{code}
Caused by: java.lang.NumberFormatException: For input string: "p"
at
sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043)
at sun.misc.FloatingDecimal.parseDouble(FloatingDecimal.java:110)
at java.lang.Double.parseDouble(Double.java:538)
at
org.apache.sysml.runtime.instructions.cp.ScalarObjectFactory.createScalarObject(ScalarObjectFactory.java:33)
at
org.apache.sysml.runtime.controlprogram.context.ExecutionContext.getScalarInput(ExecutionContext.java:443)
at
org.apache.sysml.runtime.controlprogram.context.ExecutionContext.getScalarInput(ExecutionContext.java:438)
at
org.apache.sysml.runtime.controlprogram.context.ExecutionContext.getVariable(ExecutionContext.java:165)
at
org.apache.sysml.runtime.instructions.cp.FunctionCallCPInstruction.processInstruction(FunctionCallCPInstruction.java:133)
at
org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:252)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (SYSTEMML-2209) Support functions with default parameters
[ https://issues.apache.org/jira/browse/SYSTEMML-2209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Boehm closed SYSTEMML-2209. Resolution: Fixed Assignee: Matthias Boehm Fix Version/s: SystemML 1.2 > Support functions with default parameters > - > > Key: SYSTEMML-2209 > URL: https://issues.apache.org/jira/browse/SYSTEMML-2209 > Project: SystemML > Issue Type: Task >Reporter: Matthias Boehm >Assignee: Matthias Boehm >Priority: Major > Fix For: SystemML 1.2 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SYSTEMML-2470) New cumulative aggregate cumsumprod
Matthias Boehm created SYSTEMML-2470: Summary: New cumulative aggregate cumsumprod Key: SYSTEMML-2470 URL: https://issues.apache.org/jira/browse/SYSTEMML-2470 Project: SystemML Issue Type: Task Reporter: Matthias Boehm -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SYSTEMML-2469) Large distributed paramserv overheads
Matthias Boehm created SYSTEMML-2469:
Summary: Large distributed paramserv overheads
Key: SYSTEMML-2469
URL: https://issues.apache.org/jira/browse/SYSTEMML-2469
Project: SystemML
Issue Type: Bug
Reporter: Matthias Boehm
Initial runs with the distributed paramserv implementation on a small cluster
revealed that it is working correctly while exhibiting large overheads. Below
are the stats for mnist lenet, 10 epochs, ASP, update per EPOCH, on a cluster
of 1+6 nodes (24 cores per worker node).
{code}
otal elapsed time: 687.743 sec.
Total compilation time: 3.815 sec.
Total execution time: 683.928 sec.
Number of compiled Spark inst: 330.
Number of executed Spark inst: 0.
Cache hits (Mem, WB, FS, HDFS): 176210/0/0/2.
Cache writes (WB, FS, HDFS):29856/5271/0.
Cache times (ACQr/m, RLS, EXP): 1.178/0.087/198.892/0.000 sec.
HOP DAGs recompiled (PRED, SB): 0/1629.
HOP DAGs recompile time:4.878 sec.
Functions recompiled: 1.
Functions recompile time: 0.097 sec.
Spark ctx create time (lazy): 22.222 sec.
Spark trans counts (par,bc,col):2/1/0.
Spark trans times (par,bc,col): 0.390/0.242/0.000 secs.
Paramserv total num workers:144.
Paramserv setup time: 68.259 secs.
Paramserv grad compute time:6952.163 secs.
Paramserv model update time:2453.448/422.955 secs.
Paramserv model broadcast time: 24.982 secs.
Paramserv batch slice time: 0.204 secs.
Paramserv RPC request time: 51611.210 secs.
ParFor loops optimized: 1.
ParFor optimize time: 0.462 sec.
ParFor initialize time: 0.049 sec.
ParFor result merge time: 0.028 sec.
ParFor total update in-place: 0/188/188
Total JIT compile time: 98.786 sec.
Total JVM GC count: 68.
Total JVM GC time: 25.858 sec.
Heavy hitter instructions:
# Instruction Time(s) Count
1 paramserv665.479 1
2 +182.410 18636
3 conv2d_bias_add 150.938376
4 sqrt 69.768 11528
5 / 54.836 11732
6 ba+* 45.901376
7 * 38.046 11727
8 - 37.428 12096
9 ^235.533 6344
10 exp 21.022188
{code}
There seem to be three distinct issues:
* Too larger number of tasks on assembling the distributed input data (in the
number of rows, i.e., >50,000 tasks), which makes the distributed data
partitioning very slow (multiple minutes).
* Evictions from the buffer pool at the driver node (see cache writes). This is
likely due to disabling cleanup (and missing explicit cleanup) of all RPC
objects.
* Large RPC overhead: This might be due to the evictions happening in the
critical path and all 144 workers waiting with their RPC requests. However, in
addition we should also double check that the number of RPC handler threads is
correct, if we could get the serialization and communication out of the
critical (i.e., synchronized) path of model updates, and address unnecessary
serialization/deserialization overheads.
[~Guobao] I'll help reducing the serialization/deserialization overheads, but
it would be great if you could have a look into the other issues.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SYSTEMML-2466) Distributed paramserv fails on newer Spark version > 2.1
[
https://issues.apache.org/jira/browse/SYSTEMML-2466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559124#comment-16559124
]
Matthias Boehm commented on SYSTEMML-2466:
--
Awesome - thanks so much [~Guobao] and no there is no need for a separate PR.
> Distributed paramserv fails on newer Spark version > 2.1
>
>
> Key: SYSTEMML-2466
> URL: https://issues.apache.org/jira/browse/SYSTEMML-2466
> Project: SystemML
> Issue Type: Task
>Reporter: Matthias Boehm
>Assignee: LI Guobao
>Priority: Major
>
> {code}
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/spark/network/util/SystemPropertyConfigProvider
> at
> org.apache.sysml.runtime.instructions.cp.ParamservBuiltinCPInstruction.runOnSpark(ParamservBuiltinCPInstruction.java:163)
> at
> org.apache.sysml.runtime.instructions.cp.ParamservBuiltinCPInstruction.processInstruction(ParamservBuiltinCPInstruction.java:113)
> at
> org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:252)
> at
> org.apache.sysml.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:210)
> at
> org.apache.sysml.runtime.controlprogram.ProgramBlock.execute(ProgramBlock.java:161)
> at
> org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:116)
> at
> org.apache.sysml.api.ScriptExecutorUtils.executeRuntimeProgram(ScriptExecutorUtils.java:106)
> at org.apache.sysml.api.DMLScript.execute(DMLScript.java:487)
> at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:272)
> at org.apache.sysml.api.DMLScript.main(DMLScript.java:195)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:782)
> at
> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.spark.network.util.SystemPropertyConfigProvider
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (SYSTEMML-2468) Extended matrix histograms for matrix product chains
Matthias Boehm created SYSTEMML-2468: Summary: Extended matrix histograms for matrix product chains Key: SYSTEMML-2468 URL: https://issues.apache.org/jira/browse/SYSTEMML-2468 Project: SystemML Issue Type: Sub-task Reporter: Matthias Boehm -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (SYSTEMML-2458) Add experiment on spark paramserv
[ https://issues.apache.org/jira/browse/SYSTEMML-2458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556472#comment-16556472 ] Matthias Boehm edited comment on SYSTEMML-2458 at 7/26/18 2:47 AM: --- Thanks - I just gave it a try and the script failed due to invalid name bindings on function invocations (with the recently introduced named function arguments we now check for correct name bindings). I just pushed the fix. Subsequently, it ran into SYSTEMML-2466 - maybe you could have a look [~Guobao]? was (Author: mboehm7): Thanks - I just gave it a try and the script failed due to invalid name bindings on function invocations. I just pushed the fix. Subsequently, it ran into SYSTEMML-2466 - maybe you could have a look [~Guobao]? > Add experiment on spark paramserv > - > > Key: SYSTEMML-2458 > URL: https://issues.apache.org/jira/browse/SYSTEMML-2458 > Project: SystemML > Issue Type: Sub-task >Reporter: LI Guobao >Assignee: LI Guobao >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SYSTEMML-2467) Wrong IPA size propagation w/ permuted named function arguments
Matthias Boehm created SYSTEMML-2467: Summary: Wrong IPA size propagation w/ permuted named function arguments Key: SYSTEMML-2467 URL: https://issues.apache.org/jira/browse/SYSTEMML-2467 Project: SystemML Issue Type: Bug Reporter: Matthias Boehm -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SYSTEMML-2458) Add experiment on spark paramserv
[ https://issues.apache.org/jira/browse/SYSTEMML-2458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556472#comment-16556472 ] Matthias Boehm commented on SYSTEMML-2458: -- Thanks - I just gave it a try and the script failed due to invalid name bindings on function invocations. I just pushed the fix. Subsequently, it ran into SYSTEMML-2466 - maybe you could have a look [~Guobao]? > Add experiment on spark paramserv > - > > Key: SYSTEMML-2458 > URL: https://issues.apache.org/jira/browse/SYSTEMML-2458 > Project: SystemML > Issue Type: Sub-task >Reporter: LI Guobao >Assignee: LI Guobao >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SYSTEMML-2466) Distributed paramserv fails on newer Spark version > 2.1
Matthias Boehm created SYSTEMML-2466:
Summary: Distributed paramserv fails on newer Spark version > 2.1
Key: SYSTEMML-2466
URL: https://issues.apache.org/jira/browse/SYSTEMML-2466
Project: SystemML
Issue Type: Task
Reporter: Matthias Boehm
{code}
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/spark/network/util/SystemPropertyConfigProvider
at
org.apache.sysml.runtime.instructions.cp.ParamservBuiltinCPInstruction.runOnSpark(ParamservBuiltinCPInstruction.java:163)
at
org.apache.sysml.runtime.instructions.cp.ParamservBuiltinCPInstruction.processInstruction(ParamservBuiltinCPInstruction.java:113)
at
org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:252)
at
org.apache.sysml.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:210)
at
org.apache.sysml.runtime.controlprogram.ProgramBlock.execute(ProgramBlock.java:161)
at
org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:116)
at
org.apache.sysml.api.ScriptExecutorUtils.executeRuntimeProgram(ScriptExecutorUtils.java:106)
at org.apache.sysml.api.DMLScript.execute(DMLScript.java:487)
at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:272)
at org.apache.sysml.api.DMLScript.main(DMLScript.java:195)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:782)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException:
org.apache.spark.network.util.SystemPropertyConfigProvider
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (SYSTEMML-2464) New builtin function trunc() for conditional floor/ceil
Matthias Boehm created SYSTEMML-2464: Summary: New builtin function trunc() for conditional floor/ceil Key: SYSTEMML-2464 URL: https://issues.apache.org/jira/browse/SYSTEMML-2464 Project: SystemML Issue Type: Task Reporter: Matthias Boehm -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (SYSTEMML-2463) Function calls with named function arguments
[ https://issues.apache.org/jira/browse/SYSTEMML-2463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Boehm closed SYSTEMML-2463. Resolution: Fixed Assignee: Matthias Boehm Fix Version/s: SystemML 1.2 > Function calls with named function arguments > > > Key: SYSTEMML-2463 > URL: https://issues.apache.org/jira/browse/SYSTEMML-2463 > Project: SystemML > Issue Type: Task >Reporter: Matthias Boehm >Assignee: Matthias Boehm >Priority: Major > Fix For: SystemML 1.2 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SYSTEMML-2463) Function calls with named function arguments
Matthias Boehm created SYSTEMML-2463: Summary: Function calls with named function arguments Key: SYSTEMML-2463 URL: https://issues.apache.org/jira/browse/SYSTEMML-2463 Project: SystemML Issue Type: Task Reporter: Matthias Boehm -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (SYSTEMML-2109) Add support for maxpooling operations (forward)
[ https://issues.apache.org/jira/browse/SYSTEMML-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Boehm closed SYSTEMML-2109. Resolution: Fixed Assignee: Matthias Boehm (was: Janardhan) Fix Version/s: SystemML 1.2 > Add support for maxpooling operations (forward) > --- > > Key: SYSTEMML-2109 > URL: https://issues.apache.org/jira/browse/SYSTEMML-2109 > Project: SystemML > Issue Type: Sub-task >Reporter: Matthias Boehm >Assignee: Matthias Boehm >Priority: Major > Fix For: SystemML 1.2 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SYSTEMML-2109) Add support for maxpooling operations (forward)
[ https://issues.apache.org/jira/browse/SYSTEMML-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Boehm updated SYSTEMML-2109: - Summary: Add support for maxpooling operations (forward) (was: Add support for maxpooling operations (forward/backward)) > Add support for maxpooling operations (forward) > --- > > Key: SYSTEMML-2109 > URL: https://issues.apache.org/jira/browse/SYSTEMML-2109 > Project: SystemML > Issue Type: Sub-task >Reporter: Matthias Boehm >Assignee: Janardhan >Priority: Major > Fix For: SystemML 1.2 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (SYSTEMML-2462) Failing inplace binary sparse operations w/ CSR lhs
[
https://issues.apache.org/jira/browse/SYSTEMML-2462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Boehm closed SYSTEMML-2462.
Resolution: Fixed
Assignee: Matthias Boehm
Fix Version/s: SystemML 1.2
> Failing inplace binary sparse operations w/ CSR lhs
> ---
>
> Key: SYSTEMML-2462
> URL: https://issues.apache.org/jira/browse/SYSTEMML-2462
> Project: SystemML
> Issue Type: Sub-task
>Reporter: Matthias Boehm
>Assignee: Matthias Boehm
>Priority: Major
> Fix For: SystemML 1.2
>
>
> {code}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 111
> at
> org.apache.sysml.runtime.matrix.data.SparseBlockCSR.set(SparseBlockCSR.java:498)
> at
> org.apache.sysml.runtime.matrix.data.LibMatrixBincell.safeBinaryInPlaceSparse(LibMatrixBincell.java:1144)
> at
> org.apache.sysml.runtime.matrix.data.LibMatrixBincell.safeBinaryInPlace(LibMatrixBincell.java:1107)
> at
> org.apache.sysml.runtime.matrix.data.LibMatrixBincell.bincellOpInPlace(LibMatrixBincell.java:137)
> at
> org.apache.sysml.runtime.matrix.data.MatrixBlock.binaryOperationsInPlace(MatrixBlock.java:2802)
> at
> org.apache.sysml.runtime.controlprogram.paramserv.ParamservUtils.lambda$9(ParamservUtils.java:430)
> at
> java.util.stream.Streams$RangeIntSpliterator.forEachRemaining(Streams.java:110)
> at java.util.stream.IntPipeline$Head.forEach(IntPipeline.java:557)
> at
> org.apache.sysml.runtime.controlprogram.paramserv.ParamservUtils.accrueGradients(ParamservUtils.java:427)
> at
> org.apache.sysml.runtime.controlprogram.paramserv.ParamservUtils.accrueGradients(ParamservUtils.java:420)
> at
> org.apache.sysml.runtime.controlprogram.paramserv.LocalPSWorker.computeEpoch(LocalPSWorker.java:91)
> at
> org.apache.sysml.runtime.controlprogram.paramserv.LocalPSWorker.call(LocalPSWorker.java:66)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
