[
https://issues.apache.org/jira/browse/SYSTEMML-1388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Glenn Weidner updated SYSTEMML-1388:
------------------------------------
Fix Version/s: (was: SystemML 1.0)
SystemML 0.14
> Load imbalance parfor spark w/ factoring task partitioning
> ----------------------------------------------------------
>
> Key: SYSTEMML-1388
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1388
> Project: SystemML
> Issue Type: Bug
> Reporter: Matthias Boehm
> Assignee: Matthias Boehm
> Fix For: SystemML 0.14
>
>
> For large numbers of iterations, the parfor optmizer uses factoring for task
> partitioning, i.e., tasks w/ exponentially decreasing sizes, which is
> supposed to achieve good load balance while keeping the number of tasks low.
> However, our current spark runtime parallelizes theses parfor tasks to a
> number of partitions equal to the number of cores. This leads to multiple
> parfor tasks falling into one physical partition and thus spark task. Due to
> exponentially decreasing task sizes, we see huge imbalance.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
