Hello

I started using spark. I am working with Word2VecModel. However I am not
able to save the trained model. Here is what I am doing:

inp = sc.textFile("/Users/mediratta/code/word2vec/trunk-d/sub-5").map(lambda
row: row.split(" "))
word2vec = Word2Vec()
model = word2vec.fit(inp)
out = open('abc.bin', 'wb')
pickle.dump(model, out, pickle.HIGHEST_PROTOCOL)

But I get error:

"It appears that you are attempting to reference SparkContext from a
broadcast "
Exception: It appears that you are attempting to reference SparkContext from
a broadcast variable, action, or transforamtion. SparkContext can only be
used on the driver, not in code that it run on workers. For more
information, see SPARK-5063.

However when I run pickle.dump of argument of type list instead of
Word2VecModel, then pickle.dumps works fine.

So seems the error is coming because of the type of the first argument
(Word2VecModel in this case). However the error message seems misleading.

Any clue what I am doing wrong?



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/how-to-save-Word2VecModel-tp21900.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to