[
https://issues.apache.org/jira/browse/SPARK-55058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jerry Zheng updated SPARK-55058:
--------------------------------
Description: The {{metadata}} file holds the streaming query ID, and should
be existent if the commit and offset files are non-empty. This file not
existing will result in duplicates and incorrectness downstream if using
DeltaSink which uses the streaming query ID to dedup commits for the same
batch. If the metadata file isn’t there, but the commit and offset files are
there, we should throw an error as the checkpoint is in an inconsistent state.
(was: The {{metadata}} file holds the streaming query ID, and should be
existent if the commit and offset files are non-empty. This file not existing
will result in duplicates and incorrectness downstream if using exactly-once
sinks like DeltaSink which uses the streaming query ID to dedup commits for the
same batch. If the metadata file isn’t there, but the commit and offset files
are there, we should throw an error as the checkpoint is in an inconsistent
state.)
> Throw an error if the /metadata file is not present, but offset or commit
> directories are non-empty
> ---------------------------------------------------------------------------------------------------
>
> Key: SPARK-55058
> URL: https://issues.apache.org/jira/browse/SPARK-55058
> Project: Spark
> Issue Type: Task
> Components: Structured Streaming
> Affects Versions: 4.2.0
> Reporter: Jerry Zheng
> Priority: Major
>
> The {{metadata}} file holds the streaming query ID, and should be existent if
> the commit and offset files are non-empty. This file not existing will result
> in duplicates and incorrectness downstream if using DeltaSink which uses the
> streaming query ID to dedup commits for the same batch. If the metadata file
> isn’t there, but the commit and offset files are there, we should throw an
> error as the checkpoint is in an inconsistent state.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
