I am working on an experimental research into memory use and profiling of memory use and allocation by machine learning functions across number of popular libraries.
Is there a facility within Spark, and MLlib specifically to track the allocation and use of data frames/memory by MLlib? Please advise. I will acknowledge any contributions in a paper, or add you as a co-author if you have any significant contribution (and if interested). Thank you, Edmon
