Anthony Truchet created SPARK-16440: ---------------------------------------
Summary: Undeleted broadcast variables in Word2Vec causing OoM for long runs Key: SPARK-16440 URL: https://issues.apache.org/jira/browse/SPARK-16440 Project: Spark Issue Type: Bug Components: MLlib Affects Versions: 1.6.2, 1.6.1, 1.6.0, 2.0.0 Reporter: Anthony Truchet Three broadcast variables created at the beginning of {{Word2Vec.fit()}} are never deleted nor unpersisted. This seems to cause excessive memory consumption on the driver for a job running hundreds of successive training. They are {code} val expTable = sc.broadcast(createExpTable()) val bcVocab = sc.broadcast(vocab) val bcVocabHash = sc.broadcast(vocabHash) {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org