[
https://issues.apache.org/jira/browse/SYSTEMML-1349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Berthold Reinwald updated SYSTEMML-1349:
----------------------------------------
Fix Version/s: (was: SystemML 1.0)
SystemML 1.1
> Parfor optimizer support for shared reads (lower memory req)
> ------------------------------------------------------------
>
> Key: SYSTEMML-1349
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1349
> Project: SystemML
> Issue Type: Sub-task
> Components: Compiler
> Reporter: Matthias Boehm
> Assignee: Matthias Boehm
> Fix For: SystemML 1.1
>
>
> Let's assume the following example script and focus on local parfor with a
> memory budget of 10GB and k=16 threads (but this issue similarly applies to
> parfor spark).
> {code}
> X = rand(rows=1000000, cols=1000); #8GB
> parfor(i in 1:ncol(X))
> print(sum(X[,i]));
> {code}
> The memory estimate of the right indexing operation is 8GB and the parfor
> optimizer tries to set k=min(floor(10GB/8GB), 16) and hence falls back to a
> single-threaded plan with k=1. However, this is completely unnecessary
> because X is a shared-read (i.e., variable that is never updated in the
> parfor body). Accordingly, we should determine the degree of parallelism as
> k=min(floor((10GB-8GB)/8MB), 16).
> The challenge is to do this for arbitrary control flow and multiple shared
> reads of different dimension and sparsity.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
