[ https://issues.apache.org/jira/browse/SYSTEMML-2419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16535629#comment-16535629 ]
Matthias Boehm commented on SYSTEMML-2419: ------------------------------------------ Yes, this can happen if the input sizes to functions are unknown. However, there is a simple fix. Whenever we force parfor into {{MR}} or {{SPARK}} execution modes, we use {{ParForProgramBlock.checkMRAndRecompileToCP}} to compile all operations to CP. Maybe you could move this function to a central place (e.g., {{Recompiler}}) and call it both from the paramserv and parfor functionality. > Setup and cleanup of remote workers > ----------------------------------- > > Key: SYSTEMML-2419 > URL: https://issues.apache.org/jira/browse/SYSTEMML-2419 > Project: SystemML > Issue Type: Sub-task > Reporter: LI Guobao > Assignee: LI Guobao > Priority: Major > > In the context of distributed spark env, we need to firstly ship the > necessary functions and variables to the remote workers and then to > initialize and register the cleanup of buffer pool for each remote worker. > All these are inspired by the parfor implementation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)