[
https://issues.apache.org/jira/browse/SYSTEMML-1879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Boehm updated SYSTEMML-1879:
-------------------------------------
Description:
Currently, we read shared inputs redundantly in each parfor worker. This causes
redundant read and is unnecessarily memory-inefficient.
This task aims to read shared inputs once per process and reuse them across
threads. The most elegant way of handling this is to reuse initially parsed
symbol table entries (instances of matrix objects), except for result
variables. Then the sharing happens automatically (similar to local parfor)
over the shared per-process buffer pool.
was:
Currently, we read shared inputs redundantly in each parfor worker. This causes
redundant read and is unnecessarily memory-inefficient.
This task aims to read shared inputs once per process and reuse them across
threads. The most elegant way of handling this is to reuse initially parsed
symbol table entries (instances of matrix objects), except for result
variables. Then the result happens automatically over the shared per-process
buffer pool.
> Parfor remote spark w/ reuse of shared inputs
> ---------------------------------------------
>
> Key: SYSTEMML-1879
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1879
> Project: SystemML
> Issue Type: Sub-task
> Components: APIs, Runtime
> Reporter: Matthias Boehm
> Fix For: SystemML 1.0
>
>
> Currently, we read shared inputs redundantly in each parfor worker. This
> causes redundant read and is unnecessarily memory-inefficient.
> This task aims to read shared inputs once per process and reuse them across
> threads. The most elegant way of handling this is to reuse initially parsed
> symbol table entries (instances of matrix objects), except for result
> variables. Then the sharing happens automatically (similar to local parfor)
> over the shared per-process buffer pool.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
