Github user MLnick commented on the pull request: https://github.com/apache/spark/pull/10152#issuecomment-165093398 @ygcao thanks for making all the changes, this PR is a lot clearer now focusing in on the core issue. I think you can also link https://issues.apache.org/jira/browse/SPARK-7617 to this as your latest changes also fixes that. Could you also perhaps update your JIRA description to reflect the new scope of changes in this PR? Pinging @jkbradley @mengxr @MechCoder - can you just double check that you're happy with this - i.e. the core issue as I mentioned previously is that the current implementation is incorrect and discards all sentence boundaries, instead simply clipping "sentences" every 1000 words. This does not match the original Google impl (as well as others e.g. Gensim and DL4J). These changes now respect sentence boundaries in the input RDD, and imposes a max sentence length (default the previous hard-coded `1000`). Can we also whitelist to run the tests - I'm interested to see if this impacts the Word2Vec tests in any way.
--- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org