[
https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15333753#comment-15333753
]
Alessio commented on SPARK-15904:
---------------------------------
If you're so absolutely sure that I'm missing something, I'm ready to hear your
expert opinion.
You've got 16GB of RAM, dataset size 400MB, driver with 4 cores. K=9120. I've
already told you that 4GB and 8GB will result in an Out-of-memory error. 9GB
will result in this unexpected behaviour. How would you tune the driver memory?
Answers like "your code might be the problem" (which is not, of course) and
"there's something wrong with the memory setup" (how enlightening!) are way too
easy.
> High Memory Pressure using MLlib K-means
> ----------------------------------------
>
> Key: SPARK-15904
> URL: https://issues.apache.org/jira/browse/SPARK-15904
> Project: Spark
> Issue Type: Improvement
> Components: MLlib
> Affects Versions: 1.6.1
> Environment: Mac OS X 10.11.6beta on Macbook Pro 13" mid-2012. 16GB
> of RAM.
> Reporter: Alessio
> Priority: Minor
>
> *Please Note*: even though the issue has been marked as "not a problem" and
> "resolved", this is actually a problem and wasn't resolved at all. Several
> people encountered memory issues using MLlib for large and complex problems
> (see
> http://stackoverflow.com/questions/32621267/spark-1-4-0-hangs-running-randomforest
> and
> http://stackoverflow.com/questions/27367804/how-do-i-get-spark-submit-to-close)
> Running MLlib K-Means on a ~400MB dataset (12 partitions), persisted on
> Memory and Disk.
> Everything's fine, although at the end of K-Means, after the number of
> iterations, the cost function value and the running time there's a nice
> "Removing RDD <idx> from persistent list" stage. However, during this stage
> there's a high memory pressure. Weird, since RDDs are about to be removed.
> Full log of this stage:
> 16/06/12 20:37:33 INFO clustering.KMeans: Run 0 finished in 14 iterations
> 16/06/12 20:37:33 INFO clustering.KMeans: Iterations took 694.544 seconds.
> 16/06/12 20:37:33 INFO clustering.KMeans: KMeans converged in 14 iterations.
> 16/06/12 20:37:33 INFO clustering.KMeans: The cost for the best run is
> 49784.87126751288.
> 16/06/12 20:37:33 INFO rdd.MapPartitionsRDD: Removing RDD 781 from
> persistence list
> 16/06/12 20:37:33 INFO storage.BlockManager: Removing RDD 781
> 16/06/12 20:37:33 INFO rdd.MapPartitionsRDD: Removing RDD 780 from
> persistence list
> 16/06/12 20:37:33 INFO storage.BlockManager: Removing RDD 780
> I'm running this K-Means on a 16GB machine, with Spark Context as local[*].
> My machine has an i5 hyperthreaded dual-core, thus [*] means 4.
> I'm launching this application though spark-submit with --driver-memory 9G.
> _Further test #1:_ the problem appears also without persisting/caching on
> memory (i.e. persist on disk only or no caching/persisting at all).
> _Further test #2:_ changing "spark.storage.memoryFraction" doesn't help as
> well.
> _Further test #3:_ lowering the driver memory will result in an Out-of-memory
> error.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
