[ https://issues.apache.org/jira/browse/SPARK-29995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16979888#comment-16979888 ]
zhang liming commented on SPARK-29995: -------------------------------------- Ok,thanks for your reply. > Structured Streaming file-sink log grow indefinitely > ---------------------------------------------------- > > Key: SPARK-29995 > URL: https://issues.apache.org/jira/browse/SPARK-29995 > Project: Spark > Issue Type: Improvement > Components: Structured Streaming > Affects Versions: 2.4.0 > Reporter: zhang liming > Priority: Major > Attachments: file.png, task.png > > > When i use structured streaming parquet sink, I've noticed that the > File-Sink-Log files keep getting bigger, they are in > \{$checkpoint/_spark_metadata/}, i don't think this is reasonable. > And when they merge files,task batches take longer to run, just like the > screenshot below -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org