Hi I have data frames created every 5 minutes. I use a dict to keep the recent 1 hour data frames. So only 12 data frame can be kept in the dict. New data frame come in, old data frame pop out.
My question is when I pop out the old data frame, do I have to call dataframe.unpersist to release the memory? For example If currentTime == fiveMinutes: myDict[currentTime] = dataframe oldestDataFrame = myDict.pop(oldest) Now do I have to call oldestDataFrame.unpresist? Because I think python will automatically release unused variable -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-release-data-frame-to-avoid-memory-leak-tp26656.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org