[ https://issues.apache.org/jira/browse/SYSTEMML-1309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matthias Boehm resolved SYSTEMML-1309. -------------------------------------- Resolution: Done Fix Version/s: SystemML 1.0 > Parfor spark buffer pool handling > --------------------------------- > > Key: SYSTEMML-1309 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1309 > Project: SystemML > Issue Type: Sub-task > Components: APIs, Runtime > Reporter: Matthias Boehm > Assignee: Matthias Boehm > Fix For: SystemML 1.0 > > > In contrast to parfor mr jobs, where every task has its own, process-local > buffer pool, on spark with multi-threaded executors, multiple tasks share a > common buffer pool. This is advantageous because common inputs are just read > once. However, it also requires a synchronized buffer pool initialization and > cleanup per executor. Especially the cleanup (e.g., of created cache > directories) is tricky because spark does not provide an executor close call. > Hence, our approach is to use a robust version of deleteOnExit that is > independent of the exit code and also removes remaining files that are > unknown during delete registration. -- This message was sent by Atlassian JIRA (v6.3.15#6346)