[ https://issues.apache.org/jira/browse/SPARK-20568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Marcelo Masiero Vanzin resolved SPARK-20568. -------------------------------------------- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 22952 [https://github.com/apache/spark/pull/22952] > Delete files after processing in structured streaming > ----------------------------------------------------- > > Key: SPARK-20568 > URL: https://issues.apache.org/jira/browse/SPARK-20568 > Project: Spark > Issue Type: New Feature > Components: Structured Streaming > Affects Versions: 2.1.0, 2.2.1 > Reporter: Saul Shanabrook > Assignee: Jungtaek Lim > Priority: Major > Fix For: 3.0.0 > > > It would be great to be able to delete files after processing them with > structured streaming. > For example, I am reading in a bunch of JSON files and converting them into > Parquet. If the JSON files are not deleted after they are processed, it > quickly fills up my hard drive. I originally [posted this on Stack > Overflow|http://stackoverflow.com/q/43671757/907060] and was recommended to > make a feature request for it. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org