phaniarnab opened a new pull request, #1751: URL: https://github.com/apache/systemds/pull/1751
This patch extends lineage cache to store the RDD objects which are checkpointed. This addition allows the compiler to place a chkpoint after a potentially redundant operator. During runtime, we then persist the RDD, save the RDD in the lineage cache, and reuse if the instruction repeats. It is a bit tricky to cache a RDD for a lineage trace of a previous instruction. A better way would be to be able to mark any instruction to persist the result RDD and skip the chkpoint instruction. Hyperparameter tuning of LmDS with 2.5k columns improves by 4x by caching the cpmm results in the executors. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
