[
https://issues.apache.org/jira/browse/SYSTEMML-1392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Boehm closed SYSTEMML-1392.
------------------------------------
> Redundant parfor spark dpe result var export
> --------------------------------------------
>
> Key: SYSTEMML-1392
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1392
> Project: SystemML
> Issue Type: Bug
> Reporter: Matthias Boehm
> Assignee: Matthias Boehm
> Fix For: SystemML 1.0
>
>
> The parfor spark datapartition-execute job current writes result variables
> per parfor input partition. However, since a reduce task likely has multiple
> parfor partitions and outputs are guaranteed to have no conflicts, this leads
> to unnecessary write overhead.
> To fix this issues, we should only write result variables once per physical
> partition. Similarly, since accumulators are only reported for finished
> tasks, we should also maintain these task/iteration accumulators just once
> per task.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
