[ https://issues.apache.org/jira/browse/SPARK-22832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen resolved SPARK-22832. ------------------------------- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 20017 [https://github.com/apache/spark/pull/20017] > BisectingKMeans unpersist unused datasets > ----------------------------------------- > > Key: SPARK-22832 > URL: https://issues.apache.org/jira/browse/SPARK-22832 > Project: Spark > Issue Type: Improvement > Components: ML > Affects Versions: 2.3.0 > Reporter: zhengruifeng > Priority: Trivial > Fix For: 3.0.0 > > > {code} > scala> import org.apache.spark.ml.feature._ > import org.apache.spark.ml.feature._ > scala> import org.apache.spark.ml.linalg.{Vector, Vectors} > import org.apache.spark.ml.linalg.{Vector, Vectors} > scala> import org.apache.spark.ml.clustering._ > import org.apache.spark.ml.clustering._ > scala> > scala> val df = > spark.read.format("libsvm").load("/Users/zrf/Dev/OpenSource/spark/data/mllib/sample_libsvm_data.txt") > df: org.apache.spark.sql.DataFrame = [label: double, features: vector] > scala> > scala> val bkm = new > BisectingKMeans().setK(5).setMinDivisibleClusterSize(4).setMaxIter(4).setSeed(123) > bkm: org.apache.spark.ml.clustering.BisectingKMeans = > bisecting-kmeans_74183547126a > scala> bkm.fit(df) > res0: org.apache.spark.ml.clustering.BisectingKMeansModel = > bisecting-kmeans_74183547126a > scala> sc.getPersistentRDDs > res1: scala.collection.Map[Int,org.apache.spark.rdd.RDD[_]] = Map(12 -> > MapPartitionsRDD[12] at map at BisectingKMeans.scala:151, 54 -> > MapPartitionsRDD[54] at keys at BisectingKMeans.scala:202) > {code} > The {{norms}} and the last {{indices}} but one are not unpersisted after > training. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org