large scheduler delay in OnlineLDAOptimizer, (MLlib and LDA)

[Xiaoye Sun]

Hi,

I am running some experiments with OnlineLDAOptimizer in Spark 1.6.1. My
Spark cluster has 30 machines.

However, I found that the Scheduler delay at job/stage "reduce at
LDAOptimizer.scala:452" is extremely large when the LDA model is large. The
delay could be tens of seconds.

Does anyone know the reason for that?

Best,
Xiaoye