[GitHub] spark pull request: [WIP][SPARK-1405][MLLIB]Collapsed Gibbs sampli...

2014-09-14 Thread allwefantasy
Github user allwefantasy commented on the pull request: https://github.com/apache/spark/pull/1983#issuecomment-55549348 @witgo i have saw you new performance test configuration。 I will try your new code and test in my data today --- If your project is set up for it, you can reply

[GitHub] spark pull request: [WIP][SPARK-1405][MLLIB]Collapsed Gibbs sampli...

2014-09-14 Thread allwefantasy
Github user allwefantasy commented on the pull request: https://github.com/apache/spark/pull/1983#issuecomment-1073 @witgo i have try ur latest code in my corpus 。 it will not Stuck in broadcasting . However ,some exception are throw。 ![qq20140915-1](https

[GitHub] spark pull request: [WIP][SPARK-1405][MLLIB]Collapsed Gibbs sampli...

2014-09-11 Thread allwefantasy
Github user allwefantasy commented on the pull request: https://github.com/apache/spark/pull/1983#issuecomment-55343319 @witgo 好的。如果有更新后请通知我。我这里也可以第一时间进行测试。 --- If your project is set up for it, you can reply to this email

[GitHub] spark pull request: [WIP][SPARK-1405][MLLIB]Collapsed Gibbs sampli...

2014-09-11 Thread allwefantasy
Github user allwefantasy commented on the pull request: https://github.com/apache/spark/pull/1983#issuecomment-55238978 @witgo 感谢这个技巧的分享。 我目前还遇到一个问题。昨天你 问我这边24w文档的词数是多少,我统计了下,是 2400w 计算方å

[GitHub] spark pull request: [WIP][SPARK-1405][MLLIB]Collapsed Gibbs sampli...

2014-09-10 Thread allwefantasy
Github user allwefantasy commented on the pull request: https://github.com/apache/spark/pull/1983#issuecomment-55089256 @witgo 看了你的性能测试 ä½  里面没有提到迭代次数。是多少次迭代呢?一个小时就完成了。 我这里也重新测试了一份æ

[GitHub] spark pull request: [WIP][SPARK-1405][MLLIB]Collapsed Gibbs sampli...

2014-09-10 Thread allwefantasy
Github user allwefantasy commented on the pull request: https://github.com/apache/spark/pull/1983#issuecomment-55129263 @witgo 那就是我犯了错误,对Document 中content 理解错了。我以为content 是一个固定维度的向量,每个位置代表一个词,每个位ç

[GitHub] spark pull request: [WIP][SPARK-1405][MLLIB]Collapsed Gibbs sampli...

2014-09-10 Thread allwefantasy
Github user allwefantasy commented on the pull request: https://github.com/apache/spark/pull/1983#issuecomment-55221324 @witgo 西面这一段代码可以多线程化么? for (i - 0 until content.length) { val term = content(i) val