[ 
https://issues.apache.org/jira/browse/SPARK-24295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777197#comment-16777197
 ] 

Alfredo Gimenez commented on SPARK-24295:
-----------------------------------------

We've run into the exact same issue, I uploaded a minimal reproducible example 
showing the continuously growing metadata compaction files. This is especially 
an issue in streaming jobs that rely on checkpointing, as we cannot purge 
metadata files and restart–the checkpointing mechanism depends on the metadata. 
A current workaround we have is to manually grab the last checkpoint offsets, 
purge both checkpoints and metadata, and set the "startingOffsets" to the 
latest offsets that we grabbed. This is obviously not ideal, as it relies on 
the current serialized data structure for the checkpoints, which can change 
with spark versions. It also introduces the possibility of losing checkpoint 
data if a spark job fails before creating a new checkpoint file.

[~kabhwan] can you point us to your PR? 

Is there another reliable workaround for this setup?

> Purge Structured streaming FileStreamSinkLog metadata compact file data.
> ------------------------------------------------------------------------
>
>                 Key: SPARK-24295
>                 URL: https://issues.apache.org/jira/browse/SPARK-24295
>             Project: Spark
>          Issue Type: Bug
>          Components: Structured Streaming
>    Affects Versions: 2.3.0
>            Reporter: Iqbal Singh
>            Priority: Major
>         Attachments: spark_metadatalog_compaction_perfbug_repro.tar.gz
>
>
> FileStreamSinkLog metadata logs are concatenated to a single compact file 
> after defined compact interval.
> For long running jobs, compact file size can grow up to 10's of GB's, Causing 
> slowness  while reading the data from FileStreamSinkLog dir as spark is 
> defaulting to the "__spark__metadata" dir for the read.
> We need a functionality to purge the compact file size.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to