Hi all,
I'm using pyspark and Spark-ml to train and use Word2Vect model, here is
the logic of my program:
model = Word2VecModel.load("save path")
result_list = model.findSynonymsArray(target, top_N)
Then I use the graphframe and result_list to create graph and do some
computing. However the program failed due to the out of memory error: xxx
is running beyond physical memory limits. As a result, I want to delete the
word2vec model to free memory, since I don't need to use it after getting
the result_list.
I tried using del function in Python, and
spark.sparkContext._gateway.detach(model._java_obj)
as
https://stackoverflow.com/questions/58759929/how-to-free-the-memory-taken-by-a-pyspark-model-javamodel
suggested.
But neither two worked.
Is there anyway to unload or delete the loaded w2v model in Spark or
Pyspark?
Really appreciate for any reply and help.
Best,
Zhefu
