[
https://issues.apache.org/jira/browse/SPARK-1405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14157605#comment-14157605
]
Guoqiang Li edited comment on SPARK-1405 at 10/22/14 3:57 PM:
--------------------------------------------------------------
Hi everyone
This is the latest performance test results
All tests were run on precisely the same 4 node cluster.
36 executors(a total of36 cores, 216g memory).
Training data: 253064 document, 29696335 words, 75496 distinct words.Training
iteration 150 times.
The spark configuration:
{noformat}
spark.akka.frameSize 20
spark.executor.instances 36
spark.rdd.compress true
spark.executor.memory 6g
spark.default.parallelism 72
spark.broadcast.blockSize 8192
spark.storage.memoryFraction 0.2
spark.serializer org.apache.spark.serializer.KryoSerializer
spark.kryo.registrator
org.apache.spark.mllib.feature.TopicModelingKryoRegistrator
{noformat}
Time-consuming in the following table:
||The number of topics||Time(minutes)
|2000 |42.26
|10000|49.47
|100000|58.20
|1000000|125.43
was (Author: gq):
Hi everyone
This is the latest performance test results
All tests were run on precisely the same 4 node cluster.
36 executors(a total of36 cores, 216g memory).
Training iteration 150 times.
The spark configuration:
{noformat}
spark.akka.frameSize 20
spark.executor.instances 36
spark.rdd.compress true
spark.executor.memory 6g
spark.default.parallelism 72
spark.broadcast.blockSize 8192
spark.storage.memoryFraction 0.2
spark.serializer org.apache.spark.serializer.KryoSerializer
spark.kryo.registrator
org.apache.spark.mllib.feature.TopicModelingKryoRegistrator
{noformat}
Time-consuming in the following table:
||The number of topics||Time(minutes)
|2000 |42.26
|10000|49.47
|100000|58.20
|1000000|125.43
> parallel Latent Dirichlet Allocation (LDA) atop of spark in MLlib
> -----------------------------------------------------------------
>
> Key: SPARK-1405
> URL: https://issues.apache.org/jira/browse/SPARK-1405
> Project: Spark
> Issue Type: New Feature
> Components: MLlib
> Reporter: Xusen Yin
> Assignee: Guoqiang Li
> Labels: features
> Attachments: performance_comparison.png
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Latent Dirichlet Allocation (a.k.a. LDA) is a topic model which extracts
> topics from text corpus. Different with current machine learning algorithms
> in MLlib, instead of using optimization algorithms such as gradient desent,
> LDA uses expectation algorithms such as Gibbs sampling.
> In this PR, I prepare a LDA implementation based on Gibbs sampling, with a
> wholeTextFiles API (solved yet), a word segmentation (import from Lucene),
> and a Gibbs sampling core.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
