[
https://issues.apache.org/jira/browse/SPARK-1405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14222105#comment-14222105
]
Evan Sparks commented on SPARK-1405:
------------------------------------
[~gq] - Those are great numbers for a very high number of topics - it's a
little tough to follow what's leading to the super-linear scaling in #topics in
your code, though. Are you using FastLDA or something similar to speed up
sampling? (http://www.ics.uci.edu/~newman/pubs/fastlda.pdf)
Pedro has been testing on a wikipedia dump on s3 which I provided. It's XML
formatted, one document per line, so it's easy to parse. I will copy this to a
requester-pays bucket (which will be free if you run your experiments on ec2)
now so that everyone working on this can use it for testing.
NIPS dataset seems fine for small-scale testing, but I think it's important
that we test this implementation across a range of values for documents, words,
topics, and tokens - hence, I think the data generator that Pedro is working on
is a really good idea (and follows the convention of the existing data
generators in MLlib). We'll have to be a little careful here, because some of
the methods for making LDA fast rely on the fact that it tends to converge
fast, and I expect that data generated by the model will be much easier to fit
than real data.
Also, can we try and be consistent in our terminology - getting the # of unique
words confused with all the words in a corpus is easy. I propose "words" and
"tokens" for these two things.
> parallel Latent Dirichlet Allocation (LDA) atop of spark in MLlib
> -----------------------------------------------------------------
>
> Key: SPARK-1405
> URL: https://issues.apache.org/jira/browse/SPARK-1405
> Project: Spark
> Issue Type: New Feature
> Components: MLlib
> Reporter: Xusen Yin
> Assignee: Guoqiang Li
> Priority: Critical
> Labels: features
> Attachments: performance_comparison.png
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Latent Dirichlet Allocation (a.k.a. LDA) is a topic model which extracts
> topics from text corpus. Different with current machine learning algorithms
> in MLlib, instead of using optimization algorithms such as gradient desent,
> LDA uses expectation algorithms such as Gibbs sampling.
> In this PR, I prepare a LDA implementation based on Gibbs sampling, with a
> wholeTextFiles API (solved yet), a word segmentation (import from Lucene),
> and a Gibbs sampling core.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
