Github user WeichenXu123 commented on a diff in the pull request: https://github.com/apache/spark/pull/19565#discussion_r146810442 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDAOptimizer.scala --- @@ -497,40 +495,38 @@ final class OnlineLDAOptimizer extends LDAOptimizer with Logging { (u._1, u._2, u._3 + v._3) } - val (statsSum: BDM[Double], logphatOption: Option[BDV[Double]], nonEmptyDocsN: Long) = stats + val (statsSum: BDM[Double], logphatOption: Option[BDV[Double]], batchSize: Long) = stats .treeAggregate((BDM.zeros[Double](k, vocabSize), logphatPartOptionBase(), 0L))( elementWiseSum, elementWiseSum ) --- End diff -- hmm... I think it only need to move the code block in `mapPartition` into the aggregate `seqOp` function. Will it make code more complicated ? I think it can be fix in this PR, it is only a simple modification and do not affect the logic.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org