phaniarnab opened a new pull request, #1736: URL: https://github.com/apache/systemds/pull/1736
This patch introduces a new heuristic-based operator linearization order, which aims to maximize inter-operator parallelism among Spark and local operators. We first traverse the LOP DAGs to collect the roots of the Spark operator chains and the number of Spark instructions in all subDAGs. We then first place the Spark operator chains followed by the CP lanes. Finally, we place the appropriate asynchronous operators to trigger the Spark operator chains in parallel. This change along with the future-based execution of Spark actions and a manual reuse of partitioned broadcast variables improve lmDS by 2x. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
