Github user akopich commented on the issue:
https://github.com/apache/spark/pull/19565
I am sure that caching may by avoided here. Hence, it should not be used.
@srowen, maybe I don't get something, but I'm afraid, that currently
lineage for a single mini-batch submission looks like this
`docs.filter(nonEmpty).sample(minibatchFraction).treeAggregate(...)`.
And I'm afraid that for each mini-batch `filter` will be performed all over
again. But if we have smth like
`docs.sample(minibatchFraction).filter(nonEmpty).treeAggregate(...)`,
this will be avoided. And no caching is needed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
