[
https://issues.apache.org/jira/browse/SPARK-15904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327476#comment-15327476
]
Alessio edited comment on SPARK-15904 at 6/13/16 2:48 PM:
----------------------------------------------------------
If anyone's interested, the dataset I'm working on is freely available from UCI
ML Repository
(http://archive.ics.uci.edu/ml/datasets/Daily+and+Sports+Activities).
I tried just now running the above K-Means for K=9120, with --driver-memory 4G.
The full traceback can be found here (https://ghostbin.com/paste/9pu9k).
The code is absolutely simple, I don't think there's something wrong with it:
sc = SparkContext("local[*]", "Spark K-Means")
data = sc.textFile(<my csv dataset path>)
parsedData = data.map(lambda line: array([float(x) for x in line.split(',')]))
parsedDataNOID=parsedData.map(lambda pattern: pattern[1:])
parsedDataNOID.persist(StorageLevel.MEMORY_AND_DISK)
K_CANDIDATES=<python list with 100 values for K>
initCentroids=scipy.io.loadmat(<.mat file with initial seeds>)
datatmp=numpy.genfromtxt(<my csv dataset path>,delimiter=",")
for K in K_CANDIDATES:
clusters = KMeans.train(parsedDataNOID, K, maxIterations=2000, runs=1,
epsilon=0.0, initialModel =
KMeansModel(datatmp[initCentroids['initSeedsA'][0][k_tmp][0]-1,:]))
was (Author: purple):
If anyone's interested, the dataset I'm working on is freely available from UCI
ML Repository
(http://archive.ics.uci.edu/ml/datasets/Daily+and+Sports+Activities).
I tried just now running the above K-Means for K=9120, with --driver-memory 4G.
The full traceback can be found here (https://ghostbin.com/paste/9pu9k).
The code is absolutely simple, I don't think there's nothing wrong with it:
sc = SparkContext("local[*]", "Spark K-Means")
data = sc.textFile(<my csv dataset path>)
parsedData = data.map(lambda line: array([float(x) for x in line.split(',')]))
parsedDataNOID=parsedData.map(lambda pattern: pattern[1:])
parsedDataNOID.persist(StorageLevel.MEMORY_AND_DISK)
K_CANDIDATES=<python list with 100 values for K>
initCentroids=scipy.io.loadmat(<.mat file with initial seeds>)
datatmp=numpy.genfromtxt(<my csv dataset path>,delimiter=",")
for K in K_CANDIDATES:
clusters = KMeans.train(parsedDataNOID, K, maxIterations=2000, runs=1,
epsilon=0.0, initialModel =
KMeansModel(datatmp[initCentroids['initSeedsA'][0][k_tmp][0]-1,:]))
> High Memory Pressure using MLlib K-means
> ----------------------------------------
>
> Key: SPARK-15904
> URL: https://issues.apache.org/jira/browse/SPARK-15904
> Project: Spark
> Issue Type: Improvement
> Components: MLlib
> Affects Versions: 1.6.1
> Environment: Mac OS X 10.11.6beta on Macbook Pro 13" mid-2012. 16GB
> of RAM.
> Reporter: Alessio
> Priority: Minor
>
> Running MLlib K-Means on a ~400MB dataset (12 partitions), persisted on
> Memory and Disk.
> Everything's fine, although at the end of K-Means, after the number of
> iterations, the cost function value and the running time there's a nice
> "Removing RDD <idx> from persistent list" stage. However, during this stage
> there's a high memory pressure. Weird, since RDDs are about to be removed.
> Full log of this stage:
> 16/06/12 20:37:33 INFO clustering.KMeans: Run 0 finished in 14 iterations
> 16/06/12 20:37:33 INFO clustering.KMeans: Iterations took 694.544 seconds.
> 16/06/12 20:37:33 INFO clustering.KMeans: KMeans converged in 14 iterations.
> 16/06/12 20:37:33 INFO clustering.KMeans: The cost for the best run is
> 49784.87126751288.
> 16/06/12 20:37:33 INFO rdd.MapPartitionsRDD: Removing RDD 781 from
> persistence list
> 16/06/12 20:37:33 INFO storage.BlockManager: Removing RDD 781
> 16/06/12 20:37:33 INFO rdd.MapPartitionsRDD: Removing RDD 780 from
> persistence list
> 16/06/12 20:37:33 INFO storage.BlockManager: Removing RDD 780
> I'm running this K-Means on a 16GB machine, with Spark Context as local[*].
> My machine has an i5 hyperthreaded dual-core, thus [*] means 4.
> I'm launching this application though spark-submit with --driver-memory 9G
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
