[GitHub] spark pull request: [WIP][SPARK-1405][MLLIB]Collapsed Gibbs sampli...

[allwefantasy]

Github user allwefantasy commented on the pull request:

    https://github.com/apache/spark/pull/1983#issuecomment-55221324
  
    @witgo   西面这一段代码可以多线程化么？
    
    
    for (i <- 0 until content.length) {
                val term = content(i)
                val topic = topics(i)
                val chosenTopic = topicModel.dropOneDistSampler(topicsDist, 
topicThisTerm,
                  rand, term, topic)
                if (topic != chosenTopic) {
                  topics(i) = chosenTopic
                  topicsDist(topic) += -1
                  topicsDist(chosenTopic) += 1
                  topicModel.update(term, topic, -1)
                  topicModel.update(term, chosenTopic, 1)
                }
              }
    
    
    将此代码改成
    
    content.zipWithIndex.map(f=>f._2).toList.par.foreach{i=>
                val term = content(i)
                val topic = topics(i)
                val chosenTopic = topicModel.dropOneDistSampler(topicsDist, 
topicThisTerm,
                  rand, term, topic)
                if (topic != chosenTopic) {
                  topics(i) = chosenTopic
                  topicsDist(topic) += -1
                  topicsDist(chosenTopic) += 1
                  topicModel.update(term, topic, -1)
                  topicModel.update(term, chosenTopic, 1)
                }
              }
    
    我目前的情况是单机CPU核多，24核。但内
存有限，所以希望多线程话一部分代码。


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org