[ https://issues.apache.org/jira/browse/SPARK-16921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413935#comment-15413935 ]
holdenk commented on SPARK-16921: --------------------------------- [~nchammas] are you planning on working on this yourself or would it be cool if I took a crack at it? :) > RDD/DataFrame persist() and cache() should return Python context managers > ------------------------------------------------------------------------- > > Key: SPARK-16921 > URL: https://issues.apache.org/jira/browse/SPARK-16921 > Project: Spark > Issue Type: New Feature > Components: PySpark, Spark Core, SQL > Reporter: Nicholas Chammas > Priority: Minor > > [Context > managers|https://docs.python.org/3/reference/datamodel.html#context-managers] > are a natural way to capture closely related setup and teardown code in > Python. > For example, they are commonly used when doing file I/O: > {code} > with open('/path/to/file') as f: > contents = f.read() > ... > {code} > Once the program exits the with block, {{f}} is automatically closed. > I think it makes sense to apply this pattern to persisting and unpersisting > DataFrames and RDDs. There are many cases when you want to persist a > DataFrame for a specific set of operations and then unpersist it immediately > afterwards. > For example, take model training. Today, you might do something like this: > {code} > labeled_data.persist() > model = pipeline.fit(labeled_data) > labeled_data.unpersist() > {code} > If {{persist()}} returned a context manager, you could rewrite this as > follows: > {code} > with labeled_data.persist(): > model = pipeline.fit(labeled_data) > {code} > Upon exiting the {{with}} block, {{labeled_data}} would automatically be > unpersisted. > This can be done in a backwards-compatible way since {{persist()}} would > still return the parent DataFrame or RDD as it does today, but add two > methods to the object: {{\_\_enter\_\_()}} and {{\_\_exit\_\_()}} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org