[
https://issues.apache.org/jira/browse/SYSTEMML-1349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Boehm reassigned SYSTEMML-1349:
----------------------------------------
Assignee: Matthias Boehm
Description:
Let's assume the following example script and focus on local parfor with a
memory budget of 10GB and k=16 threads (but this issue similarly applies to
parfor spark).
{code}
X = rand(rows=1000000, cols=1000); #8GB
parfor(i in 1:ncol(X))
print(sum(X[,i]));
{code}
The memory estimate of the right indexing operation is 8GB and the parfor
optimizer tries to set k=min(floor(10GB/8GB), 16) and hence falls back to a
single-threaded plan with k=1. However, this is completely unnecessary because
X is a shared-read (i.e., variable that is never updated in the parfor body).
Accordingly, we should determine the degree of parallelism as
k=min(floor((10GB-8GB)/8MB), 16).
> Parfor optimizer support for shared reads (lower memory req)
> ------------------------------------------------------------
>
> Key: SYSTEMML-1349
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1349
> Project: SystemML
> Issue Type: Sub-task
> Components: Compiler
> Reporter: Matthias Boehm
> Assignee: Matthias Boehm
> Fix For: SystemML 1.0
>
>
> Let's assume the following example script and focus on local parfor with a
> memory budget of 10GB and k=16 threads (but this issue similarly applies to
> parfor spark).
> {code}
> X = rand(rows=1000000, cols=1000); #8GB
> parfor(i in 1:ncol(X))
> print(sum(X[,i]));
> {code}
> The memory estimate of the right indexing operation is 8GB and the parfor
> optimizer tries to set k=min(floor(10GB/8GB), 16) and hence falls back to a
> single-threaded plan with k=1. However, this is completely unnecessary
> because X is a shared-read (i.e., variable that is never updated in the
> parfor body). Accordingly, we should determine the degree of parallelism as
> k=min(floor((10GB-8GB)/8MB), 16).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
