Github user akopich commented on the issue:
https://github.com/apache/spark/pull/18924
@jkbradley, no problem.
@jkbradley, @WeichenXu123, @hhbyyh, thank you all guys!
---
-
To unsubscribe, e-mail:
Github user jkbradley commented on the issue:
https://github.com/apache/spark/pull/18924
I'll update JIRA later; it seems like Apache JIRA is having problems right
now.
---
-
To unsubscribe, e-mail:
Github user jkbradley commented on the issue:
https://github.com/apache/spark/pull/18924
@akopich I'm afraid pings on Git don't work for me; I just have too many to
keep up with. Again, sorry for the delays; I have very limited bandwidth
nowadays.
---
Github user jkbradley commented on the issue:
https://github.com/apache/spark/pull/18924
Merging with master
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user akopich commented on the issue:
https://github.com/apache/spark/pull/18924
ping @jkbradley. Anyway, tests are passed now.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18924
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82880/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18924
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18924
**[Test build #82880 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82880/testReport)**
for PR 18924 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18924
**[Test build #82880 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82880/testReport)**
for PR 18924 at commit
Github user akopich commented on the issue:
https://github.com/apache/spark/pull/18924
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user akopich commented on the issue:
https://github.com/apache/spark/pull/18924
@jkbradley, no problem. The test build seems to be aborted. What's wrong?
---
-
To unsubscribe, e-mail:
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18924
**[Test build #3951 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3951/testReport)**
for PR 18924 at commit
Github user jkbradley commented on the issue:
https://github.com/apache/spark/pull/18924
LGTM
Sorry for the delay!
I'll merge it after re-running tests
---
-
To unsubscribe, e-mail:
Github user akopich commented on the issue:
https://github.com/apache/spark/pull/18924
@WeichenXu123, no problem! Thank you.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user WeichenXu123 commented on the issue:
https://github.com/apache/spark/pull/18924
@akopich Yes you can wait this to be merged first. I think @jkbradley will
have time to check this next week. Don't worry!
---
Github user akopich commented on the issue:
https://github.com/apache/spark/pull/18924
@WeichenXu123, yes sure. But can this wait until this PR is merged?
---
-
To unsubscribe, e-mail:
Github user WeichenXu123 commented on the issue:
https://github.com/apache/spark/pull/18924
@akopich LGTM. and do you have time to create a PR to resolve random seed
not working issue mentioned by @hhbyyh ? Thanks!
---
Github user akopich commented on the issue:
https://github.com/apache/spark/pull/18924
@WeichenXu123, could you please notify @jkbradley once again?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18924
**[Test build #82506 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82506/testReport)**
for PR 18924 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18924
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18924
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82506/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18924
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18924
**[Test build #82505 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82505/testReport)**
for PR 18924 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18924
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82505/
Test PASSed.
---
Github user akopich commented on the issue:
https://github.com/apache/spark/pull/18924
So shall we ping @jkbradley, shan't we?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18924
**[Test build #82506 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82506/testReport)**
for PR 18924 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18924
**[Test build #82505 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82505/testReport)**
for PR 18924 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18924
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82487/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18924
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18924
**[Test build #82487 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82487/testReport)**
for PR 18924 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18924
**[Test build #82487 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82487/testReport)**
for PR 18924 at commit
Github user akopich commented on the issue:
https://github.com/apache/spark/pull/18924
I have conducted some performance testing with random data.
The new implementation turns out to be notably faster.
```
OLD with hyper-parameter optimization : 237 sec
OLD
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18924
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18924
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82482/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18924
**[Test build #82482 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82482/testReport)**
for PR 18924 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18924
**[Test build #82482 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82482/testReport)**
for PR 18924 at commit
Github user akopich commented on the issue:
https://github.com/apache/spark/pull/18924
Thank you, @hhbyyh.
I have augmented the example a bit: explicitly set random seed a nd chosen
online optimizer:
`val lda = new
Github user hhbyyh commented on the issue:
https://github.com/apache/spark/pull/18924
Yes, I think local test is enough for both correctness and performance.
For consistency with old LDA, just some manual local test would be
sufficient. You may well just use the LDA example
Github user akopich commented on the issue:
https://github.com/apache/spark/pull/18924
@jkbradley, thank you!
- Correctness: in order to test the equivalence of two versions of
`submitMiniBatch` I have to bring both of them into the scope... One solution
would be to derive a
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18924
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82448/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18924
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18924
**[Test build #82448 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82448/testReport)**
for PR 18924 at commit
Github user akopich commented on the issue:
https://github.com/apache/spark/pull/18924
BTW. Seems like `updateLambda` method relies (in older version as well) on
`batchSize` only because this is `an optimization to avoid batch.count`.
Shouldn't we rather use `nonEmptyDocsN` instead
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18924
**[Test build #82448 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82448/testReport)**
for PR 18924 at commit
Github user akopich commented on the issue:
https://github.com/apache/spark/pull/18924
@hhbyyh, this change does not target performance but scalability, and I am
afraid, the change is beneficial only for huge datasets and the tests would
require massive computational resources.
Github user akopich commented on the issue:
https://github.com/apache/spark/pull/18924
@WeichenXu123. thank you
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user WeichenXu123 commented on the issue:
https://github.com/apache/spark/pull/18924
Oh, sorry for that, it should waiting @jkbradley to merge it. Don't worry,
I will contact him!
---
-
To unsubscribe,
Github user akopich commented on the issue:
https://github.com/apache/spark/pull/18924
@WeichenXu123, the PR seems to receive no attention for 10 days now... What
should I do?
---
-
To unsubscribe, e-mail:
Github user WeichenXu123 commented on the issue:
https://github.com/apache/spark/pull/18924
LGTM. Thanks! ping @jkbradley
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user akopich commented on the issue:
https://github.com/apache/spark/pull/18924
@WeichenXu123, @jkbradley, talking of merging. Is there anything else I
should improve in this PR in order for it to be mergeable?
---
Github user akopich commented on the issue:
https://github.com/apache/spark/pull/18924
@WeichenXu123, thanks for creating Jira. Yes, sure I will work on it.
---
-
To unsubscribe, e-mail:
Github user WeichenXu123 commented on the issue:
https://github.com/apache/spark/pull/18924
@akopich follow-up JIRA created here
https://issues.apache.org/jira/browse/SPARK-22111
Can you create follow up PR after this PR being merged ?
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18924
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82111/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18924
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18924
**[Test build #82111 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82111/testReport)**
for PR 18924 at commit
Github user akopich commented on the issue:
https://github.com/apache/spark/pull/18924
@jkbradley, thanks for the comments. Who is supposed to create the followup
jira?
---
-
To unsubscribe, e-mail:
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18924
**[Test build #82111 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82111/testReport)**
for PR 18924 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18924
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18924
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82030/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18924
**[Test build #82030 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82030/testReport)**
for PR 18924 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18924
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82027/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18924
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18924
**[Test build #82027 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82027/testReport)**
for PR 18924 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18924
**[Test build #82030 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82030/testReport)**
for PR 18924 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18924
**[Test build #82027 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82027/testReport)**
for PR 18924 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18924
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18924
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82007/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18924
**[Test build #82007 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82007/testReport)**
for PR 18924 at commit
Github user akopich commented on the issue:
https://github.com/apache/spark/pull/18924
@jkbradley, thank you for your comments! Please, check out the commit
adding the necessary docs.
Regarding tests: I believe, `OnlineLDAOptimizer alpha hyperparameter
optimization` from
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18924
**[Test build #82007 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82007/testReport)**
for PR 18924 at commit
Github user jkbradley commented on the issue:
https://github.com/apache/spark/pull/18924
Taking a look
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18924
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18924
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81893/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18924
**[Test build #81893 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81893/testReport)**
for PR 18924 at commit
Github user akopich commented on the issue:
https://github.com/apache/spark/pull/18924
@WeichenXu123, thank you for your prompt reply!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18924
**[Test build #81893 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81893/testReport)**
for PR 18924 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18924
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81885/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18924
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18924
**[Test build #81885 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81885/testReport)**
for PR 18924 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18924
**[Test build #81885 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81885/testReport)**
for PR 18924 at commit
Github user akopich commented on the issue:
https://github.com/apache/spark/pull/18924
Ping @jkbradley .
Thank you @WeichenXu123 one again for the comment! Please, have a look.
---
-
To unsubscribe, e-mail:
Github user akopich commented on the issue:
https://github.com/apache/spark/pull/18924
Yes, sure. Thank you for the valuable comment. Hopefully, I'll update the
code this week.
---
-
To unsubscribe, e-mail:
Github user WeichenXu123 commented on the issue:
https://github.com/apache/spark/pull/18924
ping @akopich This is an very useful improvement. Can you update the code
while you're at it ?
---
-
To unsubscribe,
Github user WeichenXu123 commented on the issue:
https://github.com/apache/spark/pull/18924
Thanks! I will take a look later.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user akopich commented on the issue:
https://github.com/apache/spark/pull/18924
@feynmanliang , @hhbyyh, @WeichenXu123, could you please review the PR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18924
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80524/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18924
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18924
**[Test build #80524 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80524/testReport)**
for PR 18924 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18924
**[Test build #80524 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80524/testReport)**
for PR 18924 at commit
89 matches
Mail list logo
