Github user WeichenXu123 commented on the issue:
https://github.com/apache/spark/pull/14333
@srowen The KMeans.initKMeansParallel already implements the pattern
"persist current step RDD, and unpersist previous one", but I think an RDD
persisted can also break down because of disk error or something? If we want to
reach the goal "the broadcast can safely be disposed(`broadcast.destroy`) at
each iteration too, rather than only at the end", I think it need to use
`RDD.checkpoint` instead of `RDD.persist` ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
