Hi Burak, In Kmeans, I used k_value = 100, num_iteration = 2, and num_run = 1.
In the current test, I increase num-executors = 200. In the storage info 2 (as shown below), 11 executors are used (I think the data is kind of balanced) and others have zero memory usage. <http://apache-spark-user-list.1001560.n3.nabble.com/file/n16438/spark_storage.png> Currently, there is no active stage running, just as the first image I posted in the first place. You mentioned "It seems that it is hanging, but there is a lot of calculation going on." I thought if some calculation is going on, there would be an active stage with an incomplete progress bar in the UI. Am I wrong? Thanks, Burak! Ray -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-KMeans-hangs-at-reduceByKey-collectAsMap-tp16413p16438.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
