Github user hhbyyh commented on the pull request:
https://github.com/apache/spark/pull/4807#issuecomment-93962573
Hi @jkbradley .
I've finished the correctness test against Blei's implementation, more
details are in [#4419](https://github.com/apache/spark/pull/4419).
I tried to refactor LDA as proposed and find it will be a big change (ut
and examples included).
First let's confirm on the change:
1. All Optimizers will be public, as users would be able to specify
parameters.
2. return value for `LDA.run` will become LDAModel (now it's
DistributedLDAModel)
3. users can first create optimizer and then pass it to LDA.setOptimizer
4. All the parameters specific to one algorithm should go into optimizer.
A few questions:
Based on previous discussion, users can specify algorithms by
1. passing String parameter to `LDA.run`. (hard to specify parameter)
2. or through setOptimizer of LDA.
I think it's better to leave only one place for determining which algorithm
to use, since otherwise it can be confusing and conflicted.
Another question is about existing parameters in LDA:
Except K, all other parameters (Alpha, Beta, Maxiteration, seed,
checkPointInterval) is useless or have different default values for Online LDA.
And for now I didn't use them in OnlineLDA.
Actually I find LDA and OnlineLDA share quite few things and it's kind of
difficult to merge them together. Maybe for OnlineLDA, I can only merge an
optimizer and an example code. (Later I'll probably provide a interface for
stream).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
