[ https://issues.apache.org/jira/browse/SPARK-7116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14512908#comment-14512908 ]
Joseph K. Bradley commented on SPARK-7116: ------------------------------------------ [~davies] Is there any good way to fix this? It looks like no action has been performed on the persisted RDD's children before the method exits, so unpersisting might not be a good idea. > Intermediate RDD cached but never unpersisted > --------------------------------------------- > > Key: SPARK-7116 > URL: https://issues.apache.org/jira/browse/SPARK-7116 > Project: Spark > Issue Type: Improvement > Components: PySpark, SQL > Affects Versions: 1.3.1 > Reporter: Kalle Jepsen > > In > https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/pythonUdfs.scala#L233 > an intermediate RDD is cached, but never unpersisted. It shows up in the > 'Storage' section of the Web UI, but cannot be removed. There's already a > comment in the source, suggesting to 'clean up'. If that cleanup is more > involved than simply calling `unpersist`, it probably exceeds my current > Scala skills. > Why that is a problem: > I'm adding a constant column to a DataFrame of about 20M records resulting > from an inner join with {{df.withColumn(colname, ud_func())}} , where > {{ud_func}} is simply a wrapped {{lambda: 1}}. Before and after applying the > UDF the DataFrame takes up ~430MB in the cache. The cached intermediate RDD > however takes up ~10GB(!) of storage, and I know of no way to uncache it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org