[
https://issues.apache.org/jira/browse/SPARK-18731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15726277#comment-15726277
]
Xiaoye Sun commented on SPARK-18731:
------------------------------------
Hi Sean,
Here is a part of the output I collected at the Driver. I found broadcast
related things in the log. Do you think the centroids are included in the task
closure?
2016/12/06 11:31:28.101 DEBUG ClosureCleaner: +++ Cleaning closure <function2>
(org.apache.spark.rdd.RDD$$anonfun$zip$1$$anonfun$apply$31) +++
2016/12/06 11:31:28.102 DEBUG ClosureCleaner: + declared fields: 1
2016/12/06 11:31:28.102 DEBUG ClosureCleaner: public static final long
org.apache.spark.rdd.RDD$$anonfun$zip$1$$anonfun$apply$31.serialVersionUID
2016/12/06 11:31:28.102 DEBUG ClosureCleaner: + declared methods: 2
2016/12/06 11:31:28.102 DEBUG ClosureCleaner: public final
java.lang.Object
org.apache.spark.rdd.RDD$$anonfun$zip$1$$anonfun$apply$31.apply(java.lang.Object,java.lang.Object)
2016/12/06 11:31:28.102 DEBUG ClosureCleaner: public final
scala.collection.Iterator
org.apache.spark.rdd.RDD$$anonfun$zip$1$$anonfun$apply$31.apply(scala.collection.Iterator,scala.collection.Iterator)
2016/12/06 11:31:28.102 DEBUG ClosureCleaner: + inner classes: 1
2016/12/06 11:31:28.102 DEBUG ClosureCleaner:
org.apache.spark.rdd.RDD$$anonfun$zip$1$$anonfun$apply$31$$anon$1
2016/12/06 11:31:28.102 DEBUG ClosureCleaner: + outer classes: 0
2016/12/06 11:31:28.102 DEBUG ClosureCleaner: + outer objects: 0
2016/12/06 11:31:28.103 DEBUG ClosureCleaner: + populating accessed fields
because this is the starting closure
2016/12/06 11:31:28.104 DEBUG ClosureCleaner: + fields accessed by starting
closure: 0
2016/12/06 11:31:28.104 DEBUG ClosureCleaner: + there are no enclosing objects!
2016/12/06 11:31:28.104 DEBUG ClosureCleaner: +++ closure <function2>
(org.apache.spark.rdd.RDD$$anonfun$zip$1$$anonfun$apply$31) is now cleaned +++
2016/12/06 11:31:28.104 DEBUG ClosureCleaner: +++ Cleaning closure <function1>
(org.apache.spark.mllib.clustering.KMeans$$anonfun$initKMeansParallel$1) +++
2016/12/06 11:31:28.105 DEBUG ClosureCleaner: + declared fields: 3
2016/12/06 11:31:28.105 DEBUG ClosureCleaner: public static final long
org.apache.spark.mllib.clustering.KMeans$$anonfun$initKMeansParallel$1.serialVersionUID
2016/12/06 11:31:28.105 DEBUG ClosureCleaner: private final
org.apache.spark.mllib.clustering.KMeans
org.apache.spark.mllib.clustering.KMeans$$anonfun$initKMeansParallel$1.$outer
2016/12/06 11:31:28.105 DEBUG ClosureCleaner: public final
org.apache.spark.broadcast.Broadcast
org.apache.spark.mllib.clustering.KMeans$$anonfun$initKMeansParallel$1.bcNewCenters$1
2016/12/06 11:31:28.105 DEBUG ClosureCleaner: + declared methods: 2
2016/12/06 11:31:28.105 DEBUG ClosureCleaner: public final
java.lang.Object
org.apache.spark.mllib.clustering.KMeans$$anonfun$initKMeansParallel$1.apply(java.lang.Object)
2016/12/06 11:31:28.105 DEBUG ClosureCleaner: public final double[]
org.apache.spark.mllib.clustering.KMeans$$anonfun$initKMeansParallel$1.apply(scala.Tuple2)
2016/12/06 11:31:28.105 DEBUG ClosureCleaner: + inner classes: 1
2016/12/06 11:31:28.105 DEBUG ClosureCleaner:
org.apache.spark.mllib.clustering.KMeans$$anonfun$initKMeansParallel$1$$anonfun$apply$3
2016/12/06 11:31:28.105 DEBUG ClosureCleaner: + outer classes: 1
2016/12/06 11:31:28.105 DEBUG ClosureCleaner:
org.apache.spark.mllib.clustering.KMeans
2016/12/06 11:31:28.105 DEBUG ClosureCleaner: + outer objects: 1
2016/12/06 11:31:28.105 DEBUG ClosureCleaner:
org.apache.spark.mllib.clustering.KMeans@41711125
2016/12/06 11:31:28.105 DEBUG ClosureCleaner: + populating accessed fields
because this is the starting closure
2016/12/06 11:31:28.106 DEBUG ClosureCleaner: + fields accessed by starting
closure: 1
2016/12/06 11:31:28.106 DEBUG ClosureCleaner: (class
org.apache.spark.mllib.clustering.KMeans,Set(org$apache$spark$mllib$clustering$KMeans$$runs))
2016/12/06 11:31:28.106 DEBUG ClosureCleaner: + outermost object is not a
closure, so do not clone it: (class
org.apache.spark.mllib.clustering.KMeans,org.apache.spark.mllib.clustering.KMeans@41711125)
2016/12/06 11:31:28.106 DEBUG ClosureCleaner: +++ closure <function1>
(org.apache.spark.mllib.clustering.KMeans$$anonfun$initKMeansParallel$1) is now
cleaned +++
2016/12/06 11:31:28.107 DEBUG ClosureCleaner: +++ Cleaning closure <function2>
(org.apache.spark.mllib.clustering.KMeans$$anonfun$20) +++
2016/12/06 11:31:28.107 DEBUG ClosureCleaner: + declared fields: 2
2016/12/06 11:31:28.107 DEBUG ClosureCleaner: public static final long
org.apache.spark.mllib.clustering.KMeans$$anonfun$20.serialVersionUID
2016/12/06 11:31:28.107 DEBUG ClosureCleaner: private final
org.apache.spark.mllib.clustering.KMeans
org.apache.spark.mllib.clustering.KMeans$$anonfun$20.$outer
2016/12/06 11:31:28.107 DEBUG ClosureCleaner: + declared methods: 2
2016/12/06 11:31:28.107 DEBUG ClosureCleaner: public final
java.lang.Object
org.apache.spark.mllib.clustering.KMeans$$anonfun$20.apply(java.lang.Object,java.lang.Object)
2016/12/06 11:31:28.107 DEBUG ClosureCleaner: public final double[]
org.apache.spark.mllib.clustering.KMeans$$anonfun$20.apply(double[],double[])
2016/12/06 11:31:28.107 DEBUG ClosureCleaner: + inner classes: 0
2016/12/06 11:31:28.107 DEBUG ClosureCleaner: + outer classes: 1
2016/12/06 11:31:28.107 DEBUG ClosureCleaner:
org.apache.spark.mllib.clustering.KMeans
2016/12/06 11:31:28.107 DEBUG ClosureCleaner: + outer objects: 1
2016/12/06 11:31:28.107 DEBUG ClosureCleaner:
org.apache.spark.mllib.clustering.KMeans@41711125
2016/12/06 11:31:28.107 DEBUG ClosureCleaner: + populating accessed fields
because this is the starting closure
2016/12/06 11:31:28.108 DEBUG ClosureCleaner: + fields accessed by starting
closure: 1
2016/12/06 11:31:28.108 DEBUG ClosureCleaner: (class
org.apache.spark.mllib.clustering.KMeans,Set(org$apache$spark$mllib$clustering$KMeans$$runs))
2016/12/06 11:31:28.108 DEBUG ClosureCleaner: + outermost object is not a
closure, so do not clone it: (class
org.apache.spark.mllib.clustering.KMeans,org.apache.spark.mllib.clustering.KMeans@41711125)
2016/12/06 11:31:28.108 DEBUG ClosureCleaner: +++ closure <function2>
(org.apache.spark.mllib.clustering.KMeans$$anonfun$20) is now cleaned +++
2016/12/06 11:31:28.108 DEBUG ClosureCleaner: +++ Cleaning closure <function2>
(org.apache.spark.mllib.clustering.KMeans$$anonfun$21) +++
2016/12/06 11:31:28.108 DEBUG ClosureCleaner: + declared fields: 2
2016/12/06 11:31:28.109 DEBUG ClosureCleaner: public static final long
org.apache.spark.mllib.clustering.KMeans$$anonfun$21.serialVersionUID
2016/12/06 11:31:28.109 DEBUG ClosureCleaner: private final
org.apache.spark.mllib.clustering.KMeans
org.apache.spark.mllib.clustering.KMeans$$anonfun$21.$outer
2016/12/06 11:31:28.109 DEBUG ClosureCleaner: + declared methods: 2
2016/12/06 11:31:28.109 DEBUG ClosureCleaner: public final
java.lang.Object
org.apache.spark.mllib.clustering.KMeans$$anonfun$21.apply(java.lang.Object,java.lang.Object)
2016/12/06 11:31:28.109 DEBUG ClosureCleaner: public final double[]
org.apache.spark.mllib.clustering.KMeans$$anonfun$21.apply(double[],double[])
2016/12/06 11:31:28.109 DEBUG ClosureCleaner: + inner classes: 0
2016/12/06 11:31:28.109 DEBUG ClosureCleaner: + outer classes: 1
2016/12/06 11:31:28.109 DEBUG ClosureCleaner:
org.apache.spark.mllib.clustering.KMeans
2016/12/06 11:31:28.109 DEBUG ClosureCleaner: + outer objects: 1
2016/12/06 11:31:28.109 DEBUG ClosureCleaner:
org.apache.spark.mllib.clustering.KMeans@41711125
2016/12/06 11:31:28.109 DEBUG ClosureCleaner: + populating accessed fields
because this is the starting closure
2016/12/06 11:31:28.110 DEBUG ClosureCleaner: + fields accessed by starting
closure: 1
2016/12/06 11:31:28.110 DEBUG ClosureCleaner: (class
org.apache.spark.mllib.clustering.KMeans,Set(org$apache$spark$mllib$clustering$KMeans$$runs))
2016/12/06 11:31:28.110 DEBUG ClosureCleaner: + outermost object is not a
closure, so do not clone it: (class
org.apache.spark.mllib.clustering.KMeans,org.apache.spark.mllib.clustering.KMeans@41711125)
2016/12/06 11:31:28.110 DEBUG ClosureCleaner: +++ closure <function2>
(org.apache.spark.mllib.clustering.KMeans$$anonfun$21) is now cleaned +++
> Task size in K-means is so large
> --------------------------------
>
> Key: SPARK-18731
> URL: https://issues.apache.org/jira/browse/SPARK-18731
> Project: Spark
> Issue Type: Improvement
> Components: MLlib
> Affects Versions: 1.6.1
> Reporter: Xiaoye Sun
> Priority: Minor
> Original Estimate: 5h
> Remaining Estimate: 5h
>
> When run the KMeans algorithm for a large model (e.g. 100k features and 100
> centers), there will be warning shown for many of the stages saying that the
> task size is very large. Here is an example warning.
> WARN TaskSetManager: Stage 23 contains a task of very large size (56256 KB).
> The maximum recommended task size is 100 KB.
> This could happen at (sum at KMeansModel.scala:88), (takeSample at
> KMeans.scala:378), (aggregate at KMeans.scala:404) and (collect at
> KMeans.scala:436).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
