[ https://issues.apache.org/jira/browse/SPARK-12153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen updated SPARK-12153: ------------------------------ Assignee: YongGang Cao > Word2Vec uses a fixed length for sentences which is not reasonable for > reality, and similarity functions and fields are not accessible > -------------------------------------------------------------------------------------------------------------------------------------- > > Key: SPARK-12153 > URL: https://issues.apache.org/jira/browse/SPARK-12153 > Project: Spark > Issue Type: Bug > Components: MLlib > Affects Versions: 1.5.2 > Reporter: YongGang Cao > Assignee: YongGang Cao > Priority: Minor > > sentence boundary matters for sliding window, we shouldn't train model from a > window across sentences. > the current 1000 word as a hard split for sentences doesn't really make sense > which is not consistent with both original c version or other implementation > like deeplearning4j etc. > the max sentence length is fixed and not tunable. Made it tunable as well. > I made changes to address above issues. > here is the pull request: https://github.com/apache/spark/pull/10152 -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org