[ https://issues.apache.org/jira/browse/SPARK-17604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Armbrust updated SPARK-17604: ------------------------------------- Issue Type: New Feature (was: Sub-task) Parent: (was: SPARK-17267) > Support purging aged file entry for FileStreamSource metadata log > ----------------------------------------------------------------- > > Key: SPARK-17604 > URL: https://issues.apache.org/jira/browse/SPARK-17604 > Project: Spark > Issue Type: New Feature > Components: SQL, Streaming > Reporter: Saisai Shao > Priority: Minor > > Currently with SPARK-15698, FileStreamSource metadata log will be compacted > periodically (10 batches by default), this means compacted batch file will > contain whole file entries been processed. With the time passed, the > compacted batch file will be accumulated to a relative large file. > With SPARK-17165, now {{FileStreamSource}} doesn't track the aged file entry, > but in the log we still keep the full records, this is not necessary and > quite time-consuming during recovery. So here propose to also add file entry > purging ability to {{FileStreamSource}} metadata log. > This is pending on SPARK-15698. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org