Littlestar created SPARK-6240: --------------------------------- Summary: Spark MLlib fpm#FPGrowth genFreqItems use Array[Item] may outOfMemory for Large Sets Key: SPARK-6240 URL: https://issues.apache.org/jira/browse/SPARK-6240 Project: Spark Issue Type: Improvement Components: MLlib Affects Versions: 1.3.0 Reporter: Littlestar Priority: Minor
Spark MLlib fpm#FPGrowth genFreqItems use Array[Item] may outOfMemory for Large Sets {noformat} private def genFreqItems[Item: ClassTag]( data: RDD[Array[Item]], minCount: Long, partitioner: Partitioner): Array[Item] = { data.flatMap { t => val uniq = t.toSet if (t.size != uniq.size) { throw new SparkException(s"Items in a transaction must be unique but got ${t.toSeq}.") } t }.map(v => (v, 1L)) .reduceByKey(partitioner, _ + _) .filter(_._2 >= minCount) .collect() .sortBy(-_._2) .map(_._1) } {noformat} I use 10*10000*10000 records for test, for output all simultaneously pair. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org