[jira] [Commented] (SPARK-15904) High Memory Pressure using MLlib K-means
[ https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15333753#comment-15333753 ] Alessio commented on SPARK-15904: - If you're so absolutely sure that I'm missing something, I'm ready to hear your expert opinion. You've got 16GB of RAM, dataset size 400MB, driver with 4 cores. K=9120. I've already told you that 4GB and 8GB will result in an Out-of-memory error. 9GB will result in this unexpected behaviour. How would you tune the driver memory? Answers like "your code might be the problem" (which is not, of course) and "there's something wrong with the memory setup" (how enlightening!) are way too easy. > High Memory Pressure using MLlib K-means > > > Key: SPARK-15904 > URL: https://issues.apache.org/jira/browse/SPARK-15904 > Project: Spark > Issue Type: Improvement > Components: MLlib >Affects Versions: 1.6.1 > Environment: Mac OS X 10.11.6beta on Macbook Pro 13" mid-2012. 16GB > of RAM. >Reporter: Alessio >Priority: Minor > > *Please Note*: even though the issue has been marked as "not a problem" and > "resolved", this is actually a problem and wasn't resolved at all. Several > people encountered memory issues using MLlib for large and complex problems > (see > http://stackoverflow.com/questions/32621267/spark-1-4-0-hangs-running-randomforest > and > http://stackoverflow.com/questions/27367804/how-do-i-get-spark-submit-to-close) > Running MLlib K-Means on a ~400MB dataset (12 partitions), persisted on > Memory and Disk. > Everything's fine, although at the end of K-Means, after the number of > iterations, the cost function value and the running time there's a nice > "Removing RDD from persistent list" stage. However, during this stage > there's a high memory pressure. Weird, since RDDs are about to be removed. > Full log of this stage: > 16/06/12 20:37:33 INFO clustering.KMeans: Run 0 finished in 14 iterations > 16/06/12 20:37:33 INFO clustering.KMeans: Iterations took 694.544 seconds. > 16/06/12 20:37:33 INFO clustering.KMeans: KMeans converged in 14 iterations. > 16/06/12 20:37:33 INFO clustering.KMeans: The cost for the best run is > 49784.87126751288. > 16/06/12 20:37:33 INFO rdd.MapPartitionsRDD: Removing RDD 781 from > persistence list > 16/06/12 20:37:33 INFO storage.BlockManager: Removing RDD 781 > 16/06/12 20:37:33 INFO rdd.MapPartitionsRDD: Removing RDD 780 from > persistence list > 16/06/12 20:37:33 INFO storage.BlockManager: Removing RDD 780 > I'm running this K-Means on a 16GB machine, with Spark Context as local[*]. > My machine has an i5 hyperthreaded dual-core, thus [*] means 4. > I'm launching this application though spark-submit with --driver-memory 9G. > _Further test #1:_ the problem appears also without persisting/caching on > memory (i.e. persist on disk only or no caching/persisting at all). > _Further test #2:_ changing "spark.storage.memoryFraction" doesn't help as > well. > _Further test #3:_ lowering the driver memory will result in an Out-of-memory > error. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15904) High Memory Pressure using MLlib K-means
[ https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15333719#comment-15333719 ] Sean Owen commented on SPARK-15904: --- [~Purple] I don't understand why you reopened this. Please don't unless information has meaningfully changed the discussion. What you present here clearly shows you have memory settings that aren't consistent with your physical memory or problem size. Unsurprisingly it doesn't work. The links you added have nothing to do with this problem. I am reclosing this because we did explain and resolve your problem. > High Memory Pressure using MLlib K-means > > > Key: SPARK-15904 > URL: https://issues.apache.org/jira/browse/SPARK-15904 > Project: Spark > Issue Type: Improvement > Components: MLlib >Affects Versions: 1.6.1 > Environment: Mac OS X 10.11.6beta on Macbook Pro 13" mid-2012. 16GB > of RAM. >Reporter: Alessio >Priority: Minor > > *Please Note*: even though the issue has been marked as "not a problem" and > "resolved", this is actually a problem and wasn't resolved at all. Though I > reopened it. Several people encountered memory issues using MLlib for large > and complex problems (see > http://stackoverflow.com/questions/32621267/spark-1-4-0-hangs-running-randomforest > and > http://stackoverflow.com/questions/27367804/how-do-i-get-spark-submit-to-close) > Running MLlib K-Means on a ~400MB dataset (12 partitions), persisted on > Memory and Disk. > Everything's fine, although at the end of K-Means, after the number of > iterations, the cost function value and the running time there's a nice > "Removing RDD from persistent list" stage. However, during this stage > there's a high memory pressure. Weird, since RDDs are about to be removed. > Full log of this stage: > 16/06/12 20:37:33 INFO clustering.KMeans: Run 0 finished in 14 iterations > 16/06/12 20:37:33 INFO clustering.KMeans: Iterations took 694.544 seconds. > 16/06/12 20:37:33 INFO clustering.KMeans: KMeans converged in 14 iterations. > 16/06/12 20:37:33 INFO clustering.KMeans: The cost for the best run is > 49784.87126751288. > 16/06/12 20:37:33 INFO rdd.MapPartitionsRDD: Removing RDD 781 from > persistence list > 16/06/12 20:37:33 INFO storage.BlockManager: Removing RDD 781 > 16/06/12 20:37:33 INFO rdd.MapPartitionsRDD: Removing RDD 780 from > persistence list > 16/06/12 20:37:33 INFO storage.BlockManager: Removing RDD 780 > I'm running this K-Means on a 16GB machine, with Spark Context as local[*]. > My machine has an i5 hyperthreaded dual-core, thus [*] means 4. > I'm launching this application though spark-submit with --driver-memory 9G. > _Further tests:_ the problem appears also without persisting/caching on > memory (i.e. persist on disk only or no caching/persisting at all) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15904) High Memory Pressure using MLlib K-means
[ https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15330480#comment-15330480 ] Sean Owen commented on SPARK-15904: --- I don't think the problem is your code. You're allocating on the one hand too little memory (OOME), and on the other hand too much (swapping). > High Memory Pressure using MLlib K-means > > > Key: SPARK-15904 > URL: https://issues.apache.org/jira/browse/SPARK-15904 > Project: Spark > Issue Type: Improvement > Components: MLlib >Affects Versions: 1.6.1 > Environment: Mac OS X 10.11.6beta on Macbook Pro 13" mid-2012. 16GB > of RAM. >Reporter: Alessio >Priority: Minor > > Running MLlib K-Means on a ~400MB dataset (12 partitions), persisted on > Memory and Disk. > Everything's fine, although at the end of K-Means, after the number of > iterations, the cost function value and the running time there's a nice > "Removing RDD from persistent list" stage. However, during this stage > there's a high memory pressure. Weird, since RDDs are about to be removed. > Full log of this stage: > 16/06/12 20:37:33 INFO clustering.KMeans: Run 0 finished in 14 iterations > 16/06/12 20:37:33 INFO clustering.KMeans: Iterations took 694.544 seconds. > 16/06/12 20:37:33 INFO clustering.KMeans: KMeans converged in 14 iterations. > 16/06/12 20:37:33 INFO clustering.KMeans: The cost for the best run is > 49784.87126751288. > 16/06/12 20:37:33 INFO rdd.MapPartitionsRDD: Removing RDD 781 from > persistence list > 16/06/12 20:37:33 INFO storage.BlockManager: Removing RDD 781 > 16/06/12 20:37:33 INFO rdd.MapPartitionsRDD: Removing RDD 780 from > persistence list > 16/06/12 20:37:33 INFO storage.BlockManager: Removing RDD 780 > I'm running this K-Means on a 16GB machine, with Spark Context as local[*]. > My machine has an i5 hyperthreaded dual-core, thus [*] means 4. > I'm launching this application though spark-submit with --driver-memory 9G -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15904) High Memory Pressure using MLlib K-means
[ https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15330054#comment-15330054 ] Alessio commented on SPARK-15904: - For the records, just confirming that the problem is not my code, I've been running K-means in an interactive fashion, also with its default centroid initialization (kmeans||). Same dataset, K=9120. Out of memory error. > High Memory Pressure using MLlib K-means > > > Key: SPARK-15904 > URL: https://issues.apache.org/jira/browse/SPARK-15904 > Project: Spark > Issue Type: Improvement > Components: MLlib >Affects Versions: 1.6.1 > Environment: Mac OS X 10.11.6beta on Macbook Pro 13" mid-2012. 16GB > of RAM. >Reporter: Alessio >Priority: Minor > > Running MLlib K-Means on a ~400MB dataset (12 partitions), persisted on > Memory and Disk. > Everything's fine, although at the end of K-Means, after the number of > iterations, the cost function value and the running time there's a nice > "Removing RDD from persistent list" stage. However, during this stage > there's a high memory pressure. Weird, since RDDs are about to be removed. > Full log of this stage: > 16/06/12 20:37:33 INFO clustering.KMeans: Run 0 finished in 14 iterations > 16/06/12 20:37:33 INFO clustering.KMeans: Iterations took 694.544 seconds. > 16/06/12 20:37:33 INFO clustering.KMeans: KMeans converged in 14 iterations. > 16/06/12 20:37:33 INFO clustering.KMeans: The cost for the best run is > 49784.87126751288. > 16/06/12 20:37:33 INFO rdd.MapPartitionsRDD: Removing RDD 781 from > persistence list > 16/06/12 20:37:33 INFO storage.BlockManager: Removing RDD 781 > 16/06/12 20:37:33 INFO rdd.MapPartitionsRDD: Removing RDD 780 from > persistence list > 16/06/12 20:37:33 INFO storage.BlockManager: Removing RDD 780 > I'm running this K-Means on a 16GB machine, with Spark Context as local[*]. > My machine has an i5 hyperthreaded dual-core, thus [*] means 4. > I'm launching this application though spark-submit with --driver-memory 9G -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15904) High Memory Pressure using MLlib K-means
[ https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327542#comment-15327542 ] Alessio commented on SPARK-15904: - With the --driver-memory 4G switch I've tried both. With no luck. At first I changed the storage level to serialized, then I also increased the number of partitions (from 12 - default - to 20). Still "out of memory". I guess I'll wait for 2.0 > High Memory Pressure using MLlib K-means > > > Key: SPARK-15904 > URL: https://issues.apache.org/jira/browse/SPARK-15904 > Project: Spark > Issue Type: Improvement > Components: MLlib >Affects Versions: 1.6.1 > Environment: Mac OS X 10.11.6beta on Macbook Pro 13" mid-2012. 16GB > of RAM. >Reporter: Alessio >Priority: Minor > > Running MLlib K-Means on a ~400MB dataset (12 partitions), persisted on > Memory and Disk. > Everything's fine, although at the end of K-Means, after the number of > iterations, the cost function value and the running time there's a nice > "Removing RDD from persistent list" stage. However, during this stage > there's a high memory pressure. Weird, since RDDs are about to be removed. > Full log of this stage: > 16/06/12 20:37:33 INFO clustering.KMeans: Run 0 finished in 14 iterations > 16/06/12 20:37:33 INFO clustering.KMeans: Iterations took 694.544 seconds. > 16/06/12 20:37:33 INFO clustering.KMeans: KMeans converged in 14 iterations. > 16/06/12 20:37:33 INFO clustering.KMeans: The cost for the best run is > 49784.87126751288. > 16/06/12 20:37:33 INFO rdd.MapPartitionsRDD: Removing RDD 781 from > persistence list > 16/06/12 20:37:33 INFO storage.BlockManager: Removing RDD 781 > 16/06/12 20:37:33 INFO rdd.MapPartitionsRDD: Removing RDD 780 from > persistence list > 16/06/12 20:37:33 INFO storage.BlockManager: Removing RDD 780 > I'm running this K-Means on a 16GB machine, with Spark Context as local[*]. > My machine has an i5 hyperthreaded dual-core, thus [*] means 4. > I'm launching this application though spark-submit with --driver-memory 9G -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15904) High Memory Pressure using MLlib K-means
[ https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327510#comment-15327510 ] Sean Owen commented on SPARK-15904: --- Yes, that just means "out of memory". The question is whether this is unusual or not. You might try storing the serialized representation in memory, not the 'raw' object form, which is often bigger. You almost certainly need more partitions in the source data, since I expect it's just 1 or 2 partitions according to the block size, but, you probably want the problem to be broken down into smaller chunks rather than process big chunks at once in memory. It's the second arg to textFile. Finally you may get better results with 2.0, or, by using the ML + Dataset APIs. Those are bigger changes though. > High Memory Pressure using MLlib K-means > > > Key: SPARK-15904 > URL: https://issues.apache.org/jira/browse/SPARK-15904 > Project: Spark > Issue Type: Improvement > Components: MLlib >Affects Versions: 1.6.1 > Environment: Mac OS X 10.11.6beta on Macbook Pro 13" mid-2012. 16GB > of RAM. >Reporter: Alessio >Priority: Minor > > Running MLlib K-Means on a ~400MB dataset (12 partitions), persisted on > Memory and Disk. > Everything's fine, although at the end of K-Means, after the number of > iterations, the cost function value and the running time there's a nice > "Removing RDD from persistent list" stage. However, during this stage > there's a high memory pressure. Weird, since RDDs are about to be removed. > Full log of this stage: > 16/06/12 20:37:33 INFO clustering.KMeans: Run 0 finished in 14 iterations > 16/06/12 20:37:33 INFO clustering.KMeans: Iterations took 694.544 seconds. > 16/06/12 20:37:33 INFO clustering.KMeans: KMeans converged in 14 iterations. > 16/06/12 20:37:33 INFO clustering.KMeans: The cost for the best run is > 49784.87126751288. > 16/06/12 20:37:33 INFO rdd.MapPartitionsRDD: Removing RDD 781 from > persistence list > 16/06/12 20:37:33 INFO storage.BlockManager: Removing RDD 781 > 16/06/12 20:37:33 INFO rdd.MapPartitionsRDD: Removing RDD 780 from > persistence list > 16/06/12 20:37:33 INFO storage.BlockManager: Removing RDD 780 > I'm running this K-Means on a 16GB machine, with Spark Context as local[*]. > My machine has an i5 hyperthreaded dual-core, thus [*] means 4. > I'm launching this application though spark-submit with --driver-memory 9G -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15904) High Memory Pressure using MLlib K-means
[
https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327476#comment-15327476
]
Alessio commented on SPARK-15904:
-
If anyone's interested, the dataset I'm working on is freely available from UCI
ML Repository
(http://archive.ics.uci.edu/ml/datasets/Daily+and+Sports+Activities).
I tried just now running the above K-Means for K=9120, with --driver-memory 4G.
The full traceback can be found here (https://ghostbin.com/paste/9pu9k).
The code is absolutely simple, I don't think there's nothing wrong with it:
sc = SparkContext("local[*]", "Spark K-Means")
data = sc.textFile()
parsedData = data.map(lambda line: array([float(x) for x in line.split(',')]))
parsedDataNOID=parsedData.map(lambda pattern: pattern[1:])
parsedDataNOID.persist(StorageLevel.MEMORY_AND_DISK)
K_CANDIDATES=
initCentroids=scipy.io.loadmat(<.mat file with initial seeds>)
datatmp=numpy.genfromtxt(,delimiter=",")
for K in K_CANDIDATES:
clusters = KMeans.train(parsedDataNOID, K, maxIterations=2000, runs=1,
epsilon=0.0, initialModel =
KMeansModel(datatmp[initCentroids['initSeedsA'][0][k_tmp][0]-1,:]))
> High Memory Pressure using MLlib K-means
>
>
> Key: SPARK-15904
> URL: https://issues.apache.org/jira/browse/SPARK-15904
> Project: Spark
> Issue Type: Improvement
> Components: MLlib
>Affects Versions: 1.6.1
> Environment: Mac OS X 10.11.6beta on Macbook Pro 13" mid-2012. 16GB
> of RAM.
>Reporter: Alessio
>Priority: Minor
>
> Running MLlib K-Means on a ~400MB dataset (12 partitions), persisted on
> Memory and Disk.
> Everything's fine, although at the end of K-Means, after the number of
> iterations, the cost function value and the running time there's a nice
> "Removing RDD from persistent list" stage. However, during this stage
> there's a high memory pressure. Weird, since RDDs are about to be removed.
> Full log of this stage:
> 16/06/12 20:37:33 INFO clustering.KMeans: Run 0 finished in 14 iterations
> 16/06/12 20:37:33 INFO clustering.KMeans: Iterations took 694.544 seconds.
> 16/06/12 20:37:33 INFO clustering.KMeans: KMeans converged in 14 iterations.
> 16/06/12 20:37:33 INFO clustering.KMeans: The cost for the best run is
> 49784.87126751288.
> 16/06/12 20:37:33 INFO rdd.MapPartitionsRDD: Removing RDD 781 from
> persistence list
> 16/06/12 20:37:33 INFO storage.BlockManager: Removing RDD 781
> 16/06/12 20:37:33 INFO rdd.MapPartitionsRDD: Removing RDD 780 from
> persistence list
> 16/06/12 20:37:33 INFO storage.BlockManager: Removing RDD 780
> I'm running this K-Means on a 16GB machine, with Spark Context as local[*].
> My machine has an i5 hyperthreaded dual-core, thus [*] means 4.
> I'm launching this application though spark-submit with --driver-memory 9G
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15904) High Memory Pressure using MLlib K-means
[ https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327449#comment-15327449 ] Sean Owen commented on SPARK-15904: --- It's not your 400MB data set that is the only thing in memory or using memory. OK, that's new information, but, you're also just saying that large k needs more memory. At the moment it's not clear whether it's unreasonably high, or due to Spark or your code. What ran out of memory? > High Memory Pressure using MLlib K-means > > > Key: SPARK-15904 > URL: https://issues.apache.org/jira/browse/SPARK-15904 > Project: Spark > Issue Type: Improvement > Components: MLlib >Affects Versions: 1.6.1 > Environment: Mac OS X 10.11.6beta on Macbook Pro 13" mid-2012. 16GB > of RAM. >Reporter: Alessio >Priority: Minor > > Running MLlib K-Means on a ~400MB dataset (12 partitions), persisted on > Memory and Disk. > Everything's fine, although at the end of K-Means, after the number of > iterations, the cost function value and the running time there's a nice > "Removing RDD from persistent list" stage. However, during this stage > there's a high memory pressure. Weird, since RDDs are about to be removed. > Full log of this stage: > 16/06/12 20:37:33 INFO clustering.KMeans: Run 0 finished in 14 iterations > 16/06/12 20:37:33 INFO clustering.KMeans: Iterations took 694.544 seconds. > 16/06/12 20:37:33 INFO clustering.KMeans: KMeans converged in 14 iterations. > 16/06/12 20:37:33 INFO clustering.KMeans: The cost for the best run is > 49784.87126751288. > 16/06/12 20:37:33 INFO rdd.MapPartitionsRDD: Removing RDD 781 from > persistence list > 16/06/12 20:37:33 INFO storage.BlockManager: Removing RDD 781 > 16/06/12 20:37:33 INFO rdd.MapPartitionsRDD: Removing RDD 780 from > persistence list > 16/06/12 20:37:33 INFO storage.BlockManager: Removing RDD 780 > I'm running this K-Means on a 16GB machine, with Spark Context as local[*]. > My machine has an i5 hyperthreaded dual-core, thus [*] means 4. > I'm launching this application though spark-submit with --driver-memory 9G -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15904) High Memory Pressure using MLlib K-means
[ https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327443#comment-15327443 ] Alessio commented on SPARK-15904: - Correct. Memory and Disk gives priority to Memory...but my dataset is 400MB so it shouldn't be a problem. If I give Spark less RAM (I tried with 4GB and 8GB) Java throws the Out-of-memory error for K>3000. > High Memory Pressure using MLlib K-means > > > Key: SPARK-15904 > URL: https://issues.apache.org/jira/browse/SPARK-15904 > Project: Spark > Issue Type: Improvement > Components: MLlib >Affects Versions: 1.6.1 > Environment: Mac OS X 10.11.6beta on Macbook Pro 13" mid-2012. 16GB > of RAM. >Reporter: Alessio >Priority: Minor > > Running MLlib K-Means on a ~400MB dataset (12 partitions), persisted on > Memory and Disk. > Everything's fine, although at the end of K-Means, after the number of > iterations, the cost function value and the running time there's a nice > "Removing RDD from persistent list" stage. However, during this stage > there's a high memory pressure. Weird, since RDDs are about to be removed. > Full log of this stage: > 16/06/12 20:37:33 INFO clustering.KMeans: Run 0 finished in 14 iterations > 16/06/12 20:37:33 INFO clustering.KMeans: Iterations took 694.544 seconds. > 16/06/12 20:37:33 INFO clustering.KMeans: KMeans converged in 14 iterations. > 16/06/12 20:37:33 INFO clustering.KMeans: The cost for the best run is > 49784.87126751288. > 16/06/12 20:37:33 INFO rdd.MapPartitionsRDD: Removing RDD 781 from > persistence list > 16/06/12 20:37:33 INFO storage.BlockManager: Removing RDD 781 > 16/06/12 20:37:33 INFO rdd.MapPartitionsRDD: Removing RDD 780 from > persistence list > 16/06/12 20:37:33 INFO storage.BlockManager: Removing RDD 780 > I'm running this K-Means on a 16GB machine, with Spark Context as local[*]. > My machine has an i5 hyperthreaded dual-core, thus [*] means 4. > I'm launching this application though spark-submit with --driver-memory 9G -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15904) High Memory Pressure using MLlib K-means
[ https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327438#comment-15327438 ] Alessio commented on SPARK-15904: - My machine has 16GB of RAM. I also tried closing all the other apps, leaving just the Terminal with Spark running. Still no luck. > High Memory Pressure using MLlib K-means > > > Key: SPARK-15904 > URL: https://issues.apache.org/jira/browse/SPARK-15904 > Project: Spark > Issue Type: Improvement > Components: MLlib >Affects Versions: 1.6.1 > Environment: Mac OS X 10.11.6beta on Macbook Pro 13" mid-2012. 16GB > of RAM. >Reporter: Alessio >Priority: Minor > > Running MLlib K-Means on a ~400MB dataset (12 partitions), persisted on > Memory and Disk. > Everything's fine, although at the end of K-Means, after the number of > iterations, the cost function value and the running time there's a nice > "Removing RDD from persistent list" stage. However, during this stage > there's a high memory pressure. Weird, since RDDs are about to be removed. > Full log of this stage: > 16/06/12 20:37:33 INFO clustering.KMeans: Run 0 finished in 14 iterations > 16/06/12 20:37:33 INFO clustering.KMeans: Iterations took 694.544 seconds. > 16/06/12 20:37:33 INFO clustering.KMeans: KMeans converged in 14 iterations. > 16/06/12 20:37:33 INFO clustering.KMeans: The cost for the best run is > 49784.87126751288. > 16/06/12 20:37:33 INFO rdd.MapPartitionsRDD: Removing RDD 781 from > persistence list > 16/06/12 20:37:33 INFO storage.BlockManager: Removing RDD 781 > 16/06/12 20:37:33 INFO rdd.MapPartitionsRDD: Removing RDD 780 from > persistence list > 16/06/12 20:37:33 INFO storage.BlockManager: Removing RDD 780 > I'm running this K-Means on a 16GB machine, with Spark Context as local[*]. > My machine has an i5 hyperthreaded dual-core, thus [*] means 4. > I'm launching this application though spark-submit with --driver-memory 9G -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15904) High Memory Pressure using MLlib K-means
[ https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327430#comment-15327430 ] Sean Owen commented on SPARK-15904: --- How much RAM does your machine have? 10GB heap means much more than 10GB physical memory in the JVM. Not to mention what the OS needs and all other apps that are running. If 9GB works OK, this pretty much demonstrates Spark is fine, and you overcommitting physical RAM is the problem. > High Memory Pressure using MLlib K-means > > > Key: SPARK-15904 > URL: https://issues.apache.org/jira/browse/SPARK-15904 > Project: Spark > Issue Type: Improvement > Components: MLlib >Affects Versions: 1.6.1 > Environment: Mac OS X 10.11.6beta on Macbook Pro 13" mid-2012. 16GB > of RAM. >Reporter: Alessio >Priority: Minor > > Running MLlib K-Means on a ~400MB dataset (12 partitions), persisted on > Memory and Disk. > Everything's fine, although at the end of K-Means, after the number of > iterations, the cost function value and the running time there's a nice > "Removing RDD from persistent list" stage. However, during this stage > there's a high memory pressure. Weird, since RDDs are about to be removed. > Full log of this stage: > 16/06/12 20:37:33 INFO clustering.KMeans: Run 0 finished in 14 iterations > 16/06/12 20:37:33 INFO clustering.KMeans: Iterations took 694.544 seconds. > 16/06/12 20:37:33 INFO clustering.KMeans: KMeans converged in 14 iterations. > 16/06/12 20:37:33 INFO clustering.KMeans: The cost for the best run is > 49784.87126751288. > 16/06/12 20:37:33 INFO rdd.MapPartitionsRDD: Removing RDD 781 from > persistence list > 16/06/12 20:37:33 INFO storage.BlockManager: Removing RDD 781 > 16/06/12 20:37:33 INFO rdd.MapPartitionsRDD: Removing RDD 780 from > persistence list > 16/06/12 20:37:33 INFO storage.BlockManager: Removing RDD 780 > I'm running this K-Means on a 16GB machine, with Spark Context as local[*]. > My machine has an i5 hyperthreaded dual-core, thus [*] means 4. > I'm launching this application though spark-submit with --driver-memory 9G -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15904) High Memory Pressure using MLlib K-means
[ https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327411#comment-15327411 ] Alessio commented on SPARK-15904: - This is absolutely weird to me. I gave Spark 9GB and during the K-Means execution, if I monitor the memory stat I can see that Spark/Java has 9GB (nice) and no Swap whatsoever. After K-means has reached convergence, during this last, cleaning stage everything goes wild. > High Memory Pressure using MLlib K-means > > > Key: SPARK-15904 > URL: https://issues.apache.org/jira/browse/SPARK-15904 > Project: Spark > Issue Type: Improvement > Components: MLlib >Affects Versions: 1.6.1 > Environment: Mac OS X 10.11.6beta on Macbook Pro 13" mid-2012. 16GB > of RAM. >Reporter: Alessio >Priority: Minor > > Running MLlib K-Means on a ~400MB dataset (12 partitions), persisted on > Memory and Disk. > Everything's fine, although at the end of K-Means, after the number of > iterations, the cost function value and the running time there's a nice > "Removing RDD from persistent list" stage. However, during this stage > there's a high memory pressure. Weird, since RDDs are about to be removed. > Full log of this stage: > 16/06/12 20:37:33 INFO clustering.KMeans: Run 0 finished in 14 iterations > 16/06/12 20:37:33 INFO clustering.KMeans: Iterations took 694.544 seconds. > 16/06/12 20:37:33 INFO clustering.KMeans: KMeans converged in 14 iterations. > 16/06/12 20:37:33 INFO clustering.KMeans: The cost for the best run is > 49784.87126751288. > 16/06/12 20:37:33 INFO rdd.MapPartitionsRDD: Removing RDD 781 from > persistence list > 16/06/12 20:37:33 INFO storage.BlockManager: Removing RDD 781 > 16/06/12 20:37:33 INFO rdd.MapPartitionsRDD: Removing RDD 780 from > persistence list > 16/06/12 20:37:33 INFO storage.BlockManager: Removing RDD 780 > I'm running this K-Means on a 16GB machine, with Spark Context as local[*]. > My machine has an i5 hyperthreaded dual-core, thus [*] means 4. > I'm launching this application though spark-submit with --driver-memory 9G -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15904) High Memory Pressure using MLlib K-means
[ https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327405#comment-15327405 ] Sean Owen commented on SPARK-15904: --- Hm, but that only means Spark used a lot of memory, and you gave it permission to use a lot of memory -- too much, if you're swapping. That sounds like the problem to me. It's happily consuming memory you've told it is there, but it's really not. Swapping makes things go very slowly of course. > High Memory Pressure using MLlib K-means > > > Key: SPARK-15904 > URL: https://issues.apache.org/jira/browse/SPARK-15904 > Project: Spark > Issue Type: Improvement > Components: MLlib >Affects Versions: 1.6.1 > Environment: Mac OS X 10.11.6beta on Macbook Pro 13" mid-2012. 16GB > of RAM. >Reporter: Alessio >Priority: Minor > > Running MLlib K-Means on a ~400MB dataset (12 partitions), persisted on > Memory and Disk. > Everything's fine, although at the end of K-Means, after the number of > iterations, the cost function value and the running time there's a nice > "Removing RDD from persistent list" stage. However, during this stage > there's a high memory pressure. Weird, since RDDs are about to be removed. > Full log of this stage: > 16/06/12 20:37:33 INFO clustering.KMeans: Run 0 finished in 14 iterations > 16/06/12 20:37:33 INFO clustering.KMeans: Iterations took 694.544 seconds. > 16/06/12 20:37:33 INFO clustering.KMeans: KMeans converged in 14 iterations. > 16/06/12 20:37:33 INFO clustering.KMeans: The cost for the best run is > 49784.87126751288. > 16/06/12 20:37:33 INFO rdd.MapPartitionsRDD: Removing RDD 781 from > persistence list > 16/06/12 20:37:33 INFO storage.BlockManager: Removing RDD 781 > 16/06/12 20:37:33 INFO rdd.MapPartitionsRDD: Removing RDD 780 from > persistence list > 16/06/12 20:37:33 INFO storage.BlockManager: Removing RDD 780 > I'm running this K-Means on a 16GB machine, with Spark Context as local[*]. > My machine has an i5 hyperthreaded dual-core, thus [*] means 4. > I'm launching this application though spark-submit with --driver-memory 9G -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15904) High Memory Pressure using MLlib K-means
[ https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327397#comment-15327397 ] Alessio commented on SPARK-15904: - Dear [~srowen], at the beginning I noticed that "Cleaning RDD” phase (as in the original post) took a lot of time (10~15 minutes). So I was curious and I opened the Activity Monitor on Mac OS X. That’s when I noticed the Memory Pressure indicator going crazy. The swap memory increases up to 10GB (when K=9120). And after this Cleaning RDD stage…everything’s back to normal. Swap memory will be reduced to 1GB or 2GBs. No more memory pressure and ready for the next K. Moreover, Spark does not stop the execution. I do not receive any “Out-of-memory” errors from either Java, Python or Spark. > High Memory Pressure using MLlib K-means > > > Key: SPARK-15904 > URL: https://issues.apache.org/jira/browse/SPARK-15904 > Project: Spark > Issue Type: Improvement > Components: MLlib >Affects Versions: 1.6.1 > Environment: Mac OS X 10.11.6beta on Macbook Pro 13" mid-2012. 16GB > of RAM. >Reporter: Alessio >Priority: Minor > > Running MLlib K-Means on a ~400MB dataset (12 partitions), persisted on > Memory and Disk. > Everything's fine, although at the end of K-Means, after the number of > iterations, the cost function value and the running time there's a nice > "Removing RDD from persistent list" stage. However, during this stage > there's a high memory pressure. Weird, since RDDs are about to be removed. > Full log of this stage: > 16/06/12 20:37:33 INFO clustering.KMeans: Run 0 finished in 14 iterations > 16/06/12 20:37:33 INFO clustering.KMeans: Iterations took 694.544 seconds. > 16/06/12 20:37:33 INFO clustering.KMeans: KMeans converged in 14 iterations. > 16/06/12 20:37:33 INFO clustering.KMeans: The cost for the best run is > 49784.87126751288. > 16/06/12 20:37:33 INFO rdd.MapPartitionsRDD: Removing RDD 781 from > persistence list > 16/06/12 20:37:33 INFO storage.BlockManager: Removing RDD 781 > 16/06/12 20:37:33 INFO rdd.MapPartitionsRDD: Removing RDD 780 from > persistence list > 16/06/12 20:37:33 INFO storage.BlockManager: Removing RDD 780 > I'm running this K-Means on a 16GB machine, with Spark Context as local[*]. > My machine has an i5 hyperthreaded dual-core, thus [*] means 4. > I'm launching this application though spark-submit with --driver-memory 9G -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15904) High Memory Pressure using MLlib K-means
[ https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327369#comment-15327369 ] Sean Owen commented on SPARK-15904: --- -verbose:gc is a JVM option and should write to stderr. You'd definitely see it; it's pretty verbose. But, are you saying things are running out of memory or just referring to the RDDs being unpersisted? the latter is not necessarily a sign of memory shortage. What does memory pressure mean here? > High Memory Pressure using MLlib K-means > > > Key: SPARK-15904 > URL: https://issues.apache.org/jira/browse/SPARK-15904 > Project: Spark > Issue Type: Improvement > Components: MLlib >Affects Versions: 1.6.1 > Environment: Mac OS X 10.11.6beta on Macbook Pro 13" mid-2012. 16GB > of RAM. >Reporter: Alessio >Priority: Minor > > Running MLlib K-Means on a ~400MB dataset (12 partitions), persisted on > Memory and Disk. > Everything's fine, although at the end of K-Means, after the number of > iterations, the cost function value and the running time there's a nice > "Removing RDD from persistent list" stage. However, during this stage > there's a high memory pressure. Weird, since RDDs are about to be removed. > Full log of this stage: > 16/06/12 20:37:33 INFO clustering.KMeans: Run 0 finished in 14 iterations > 16/06/12 20:37:33 INFO clustering.KMeans: Iterations took 694.544 seconds. > 16/06/12 20:37:33 INFO clustering.KMeans: KMeans converged in 14 iterations. > 16/06/12 20:37:33 INFO clustering.KMeans: The cost for the best run is > 49784.87126751288. > 16/06/12 20:37:33 INFO rdd.MapPartitionsRDD: Removing RDD 781 from > persistence list > 16/06/12 20:37:33 INFO storage.BlockManager: Removing RDD 781 > 16/06/12 20:37:33 INFO rdd.MapPartitionsRDD: Removing RDD 780 from > persistence list > 16/06/12 20:37:33 INFO storage.BlockManager: Removing RDD 780 > I'm running this K-Means on a 16GB machine, with Spark Context as local[*]. > My machine has an i5 hyperthreaded dual-core, thus [*] means 4. > I'm launching this application though spark-submit with --driver-memory 9G -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15904) High Memory Pressure using MLlib K-means
[ https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327272#comment-15327272 ] Alessio commented on SPARK-15904: - Dear Sean, I must certainly agree with you on k< High Memory Pressure using MLlib K-means > > > Key: SPARK-15904 > URL: https://issues.apache.org/jira/browse/SPARK-15904 > Project: Spark > Issue Type: Improvement > Components: MLlib >Affects Versions: 1.6.1 > Environment: Mac OS X 10.11.6beta on Macbook Pro 13" mid-2012. 16GB > of RAM. >Reporter: Alessio >Priority: Minor > > Running MLlib K-Means on a ~400MB dataset (12 partitions), persisted on > Memory and Disk. > Everything's fine, although at the end of K-Means, after the number of > iterations, the cost function value and the running time there's a nice > "Removing RDD from persistent list" stage. However, during this stage > there's a high memory pressure. Weird, since RDDs are about to be removed. > Full log of this stage: > 16/06/12 20:37:33 INFO clustering.KMeans: Run 0 finished in 14 iterations > 16/06/12 20:37:33 INFO clustering.KMeans: Iterations took 694.544 seconds. > 16/06/12 20:37:33 INFO clustering.KMeans: KMeans converged in 14 iterations. > 16/06/12 20:37:33 INFO clustering.KMeans: The cost for the best run is > 49784.87126751288. > 16/06/12 20:37:33 INFO rdd.MapPartitionsRDD: Removing RDD 781 from > persistence list > 16/06/12 20:37:33 INFO storage.BlockManager: Removing RDD 781 > 16/06/12 20:37:33 INFO rdd.MapPartitionsRDD: Removing RDD 780 from > persistence list > 16/06/12 20:37:33 INFO storage.BlockManager: Removing RDD 780 > I'm running this K-Means on a 16GB machine, with Spark Context as local[*]. > My machine has an i5 hyperthreaded dual-core, thus [*] means 4. > I'm launching this application though spark-submit with --driver-memory 9G -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15904) High Memory Pressure using MLlib K-means
[ https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327255#comment-15327255 ] Sean Owen commented on SPARK-15904: --- Yeah it's coherent, though typically k << number of points. It would help to know more about how you're running, what slows down, what -verbose:gc says during this time, etc. It may be a problem with memory settings rather than some particular problem with this value of k. > High Memory Pressure using MLlib K-means > > > Key: SPARK-15904 > URL: https://issues.apache.org/jira/browse/SPARK-15904 > Project: Spark > Issue Type: Improvement > Components: MLlib >Affects Versions: 1.6.1 > Environment: Mac OS X 10.11.6beta on Macbook Pro 13" mid-2012. 16GB > of RAM. >Reporter: Alessio >Priority: Minor > > Running MLlib K-Means on a ~400MB dataset (12 partitions), persisted on > Memory and Disk. > Everything's fine, although at the end of K-Means, after the number of > iterations, the cost function value and the running time there's a nice > "Removing RDD from persistent list" stage. However, during this stage > there's a high memory pressure. Weird, since RDDs are about to be removed. > Full log of this stage: > 16/06/12 20:37:33 INFO clustering.KMeans: Run 0 finished in 14 iterations > 16/06/12 20:37:33 INFO clustering.KMeans: Iterations took 694.544 seconds. > 16/06/12 20:37:33 INFO clustering.KMeans: KMeans converged in 14 iterations. > 16/06/12 20:37:33 INFO clustering.KMeans: The cost for the best run is > 49784.87126751288. > 16/06/12 20:37:33 INFO rdd.MapPartitionsRDD: Removing RDD 781 from > persistence list > 16/06/12 20:37:33 INFO storage.BlockManager: Removing RDD 781 > 16/06/12 20:37:33 INFO rdd.MapPartitionsRDD: Removing RDD 780 from > persistence list > 16/06/12 20:37:33 INFO storage.BlockManager: Removing RDD 780 > I'm running this K-Means on a 16GB machine, with Spark Context as local[*]. > My machine has an i5 hyperthreaded dual-core, thus [*] means 4. > I'm launching this application though spark-submit with --driver-memory 10G -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15904) High Memory Pressure using MLlib K-means
[ https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327234#comment-15327234 ] Alessio commented on SPARK-15904: - My dataset has 9000+ patterns, each of which has 2000+ attributes. Thus it's perfectly legal to search for K>3000 and (of course) smaller than or equal to the number of patterns (9120) > High Memory Pressure using MLlib K-means > > > Key: SPARK-15904 > URL: https://issues.apache.org/jira/browse/SPARK-15904 > Project: Spark > Issue Type: Improvement > Components: MLlib >Affects Versions: 1.6.1 > Environment: Mac OS X 10.11.6beta on Macbook Pro 13" mid-2012. 16GB > of RAM. >Reporter: Alessio >Priority: Minor > > Running MLlib K-Means on a ~400MB dataset (12 partitions), persisted on > Memory and Disk. > Everything's fine, although at the end of K-Means, after the number of > iterations, the cost function value and the running time there's a nice > "Removing RDD from persistent list" stage. However, during this stage > there's a high memory pressure. Weird, since RDDs are about to be removed. > Full log of this stage: > 16/06/12 20:37:33 INFO clustering.KMeans: Run 0 finished in 14 iterations > 16/06/12 20:37:33 INFO clustering.KMeans: Iterations took 694.544 seconds. > 16/06/12 20:37:33 INFO clustering.KMeans: KMeans converged in 14 iterations. > 16/06/12 20:37:33 INFO clustering.KMeans: The cost for the best run is > 49784.87126751288. > 16/06/12 20:37:33 INFO rdd.MapPartitionsRDD: Removing RDD 781 from > persistence list > 16/06/12 20:37:33 INFO storage.BlockManager: Removing RDD 781 > 16/06/12 20:37:33 INFO rdd.MapPartitionsRDD: Removing RDD 780 from > persistence list > 16/06/12 20:37:33 INFO storage.BlockManager: Removing RDD 780 > I'm running this K-Means on a 16GB machine, with Spark Context as local[*]. > My machine has an i5 hyperthreaded dual-core, thus [*] means 4. > I'm launching this application though spark-submit with --driver-memory 10G -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15904) High Memory Pressure using MLlib K-means
[ https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327220#comment-15327220 ] Nick Pentreath commented on SPARK-15904: Could you explain why you're using K>3000 when your dataset has dimension ~2000? > High Memory Pressure using MLlib K-means > > > Key: SPARK-15904 > URL: https://issues.apache.org/jira/browse/SPARK-15904 > Project: Spark > Issue Type: Improvement > Components: MLlib >Affects Versions: 1.6.1 > Environment: Mac OS X 10.11.6beta on Macbook Pro 13" mid-2012. 16GB > of RAM. >Reporter: Alessio >Priority: Minor > > Running MLlib K-Means on a ~400MB dataset (12 partitions), persisted on > Memory and Disk. > Everything's fine, although at the end of K-Means, after the number of > iterations, the cost function value and the running time there's a nice > "Removing RDD from persistent list" stage. However, during this stage > there's a high memory pressure. Weird, since RDDs are about to be removed. > Full log of this stage: > 16/06/12 20:37:33 INFO clustering.KMeans: Run 0 finished in 14 iterations > 16/06/12 20:37:33 INFO clustering.KMeans: Iterations took 694.544 seconds. > 16/06/12 20:37:33 INFO clustering.KMeans: KMeans converged in 14 iterations. > 16/06/12 20:37:33 INFO clustering.KMeans: The cost for the best run is > 49784.87126751288. > 16/06/12 20:37:33 INFO rdd.MapPartitionsRDD: Removing RDD 781 from > persistence list > 16/06/12 20:37:33 INFO storage.BlockManager: Removing RDD 781 > 16/06/12 20:37:33 INFO rdd.MapPartitionsRDD: Removing RDD 780 from > persistence list > 16/06/12 20:37:33 INFO storage.BlockManager: Removing RDD 780 > I'm running this K-Means on a 16GB machine, with Spark Context as local[*]. > My machine has an i5 hyperthreaded dual-core, thus [*] means 4. > I'm launching this application though spark-submit with --driver-memory 10G -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15904) High Memory Pressure using MLlib K-means
[ https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327108#comment-15327108 ] yuhao yang commented on SPARK-15904: Thanks for reporting it. I'm not sure if the issue is valid for now. Maybe Type -> Improvement, Priority -> minor as a start. > High Memory Pressure using MLlib K-means > > > Key: SPARK-15904 > URL: https://issues.apache.org/jira/browse/SPARK-15904 > Project: Spark > Issue Type: Bug > Components: MLlib >Affects Versions: 1.6.1 > Environment: Mac OS X 10.11.6beta on Macbook Pro 13" mid-2012. 16GB > of RAM. >Reporter: Alessio > > Running MLlib K-Means on a ~400MB dataset (12 partitions), persisted on > Memory and Disk. > Everything's fine, although at the end of K-Means, after the number of > iterations, the cost function value and the running time there's a nice > "Removing RDD from persistent list" stage. However, during this stage > there's a high memory pressure. Weird, since RDDs are about to be removed. > Full log of this stage: > 16/06/12 20:37:33 INFO clustering.KMeans: Run 0 finished in 14 iterations > 16/06/12 20:37:33 INFO clustering.KMeans: Iterations took 694.544 seconds. > 16/06/12 20:37:33 INFO clustering.KMeans: KMeans converged in 14 iterations. > 16/06/12 20:37:33 INFO clustering.KMeans: The cost for the best run is > 49784.87126751288. > 16/06/12 20:37:33 INFO rdd.MapPartitionsRDD: Removing RDD 781 from > persistence list > 16/06/12 20:37:33 INFO storage.BlockManager: Removing RDD 781 > 16/06/12 20:37:33 INFO rdd.MapPartitionsRDD: Removing RDD 780 from > persistence list > 16/06/12 20:37:33 INFO storage.BlockManager: Removing RDD 780 > I'm running this K-Means on a 16GB machine, with Spark Context as local[*]. > My machine has an i5 hyperthreaded dual-core, thus [*] means 4. > I'm launching this application though spark-submit with --driver-memory 10G -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15904) High Memory Pressure using MLlib K-means
[ https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327090#comment-15327090 ] Alessio commented on SPARK-15904: - Hi [~yuhaoyan]], the dataset size is 9120 rows and 2125 columns. This problem appears when K>3000. What do you suggest as priority label? I'm sorry if "major" is not appropriate, this is my first post on JIRA > High Memory Pressure using MLlib K-means > > > Key: SPARK-15904 > URL: https://issues.apache.org/jira/browse/SPARK-15904 > Project: Spark > Issue Type: Bug > Components: MLlib >Affects Versions: 1.6.1 > Environment: Mac OS X 10.11.6beta on Macbook Pro 13" mid-2012. 16GB > of RAM. >Reporter: Alessio > > Running MLlib K-Means on a ~400MB dataset (12 partitions), persisted on > Memory and Disk. > Everything's fine, although at the end of K-Means, after the number of > iterations, the cost function value and the running time there's a nice > "Removing RDD from persistent list" stage. However, during this stage > there's a high memory pressure. Weird, since RDDs are about to be removed. > Full log of this stage: > 16/06/12 20:37:33 INFO clustering.KMeans: Run 0 finished in 14 iterations > 16/06/12 20:37:33 INFO clustering.KMeans: Iterations took 694.544 seconds. > 16/06/12 20:37:33 INFO clustering.KMeans: KMeans converged in 14 iterations. > 16/06/12 20:37:33 INFO clustering.KMeans: The cost for the best run is > 49784.87126751288. > 16/06/12 20:37:33 INFO rdd.MapPartitionsRDD: Removing RDD 781 from > persistence list > 16/06/12 20:37:33 INFO storage.BlockManager: Removing RDD 781 > 16/06/12 20:37:33 INFO rdd.MapPartitionsRDD: Removing RDD 780 from > persistence list > 16/06/12 20:37:33 INFO storage.BlockManager: Removing RDD 780 > I'm running this K-Means on a 16GB machine, with Spark Context as local[*]. > My machine has an i5 hyperthreaded dual-core, thus [*] means 4. > I'm launching this application though spark-submit with --driver-memory 10G -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15904) High Memory Pressure using MLlib K-means
[ https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327082#comment-15327082 ] yuhao yang commented on SPARK-15904: Hi [~Purple]] What's your k and vector size? Btw, this should not be a major bug. > High Memory Pressure using MLlib K-means > > > Key: SPARK-15904 > URL: https://issues.apache.org/jira/browse/SPARK-15904 > Project: Spark > Issue Type: Bug > Components: MLlib >Affects Versions: 1.6.1 > Environment: Mac OS X 10.11.6beta on Macbook Pro 13" mid-2012. 16GB > of RAM. >Reporter: Alessio > > Running MLlib K-Means on a ~400MB dataset (12 partitions), persisted on > Memory and Disk. > Everything's fine, although at the end of K-Means, after the number of > iterations, the cost function value and the running time there's a nice > "Removing RDD from persistent list" stage. However, during this stage > there's a high memory pressure. Weird, since RDDs are about to be removed. > Full log of this stage: > 16/06/12 20:37:33 INFO clustering.KMeans: Run 0 finished in 14 iterations > 16/06/12 20:37:33 INFO clustering.KMeans: Iterations took 694.544 seconds. > 16/06/12 20:37:33 INFO clustering.KMeans: KMeans converged in 14 iterations. > 16/06/12 20:37:33 INFO clustering.KMeans: The cost for the best run is > 49784.87126751288. > 16/06/12 20:37:33 INFO rdd.MapPartitionsRDD: Removing RDD 781 from > persistence list > 16/06/12 20:37:33 INFO storage.BlockManager: Removing RDD 781 > 16/06/12 20:37:33 INFO rdd.MapPartitionsRDD: Removing RDD 780 from > persistence list > 16/06/12 20:37:33 INFO storage.BlockManager: Removing RDD 780 > I'm running this K-Means on a 16GB machine, with Spark Context as local[*]. > My machine has an i5 hyperthreaded dual-core, thus [*] means 4. > I'm launching this application though spark-submit with --driver-memory 10G -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
