Anish Mahto created SPARK-59343:
-----------------------------------
Summary: Warn or prevent users from changing ignore-null selection
between microbatch retries
Key: SPARK-59343
URL: https://issues.apache.org/jira/browse/SPARK-59343
Project: Spark
Issue Type: Sub-task
Components: Declarative Pipelines
Affects Versions: 4.3.0
Reporter: Anish Mahto
When an AutoCDC flow partially fails, specifically between merging to the
auxiliary table and merging to the target table, some of the partially merged
rows will be pulled again for reconciliation on the next run of the AutoCDC
flow (which re-runs the same microbatch).
When that happens, there can be rows with duplicate rows pulled in for
reconciliation, with sequence numbers. When the rows are truly duplicates, the
behavior is effectively stable; either the already persisted row wins or the
incoming row in the re-processed microbatch wins, but both resolve to the same
values.
Ignore-null changes how a row in the microbatch is interpreted, and can be
changed between flow runs. If the ignore-null selection is changed between when
the microbatch was partially processed and when it is retried, then rows can be
pulled in for reconciliation with the same sequence as rows in the microbatch,
but they are not true duplicates - they were ingested with different
ignore-null selections, and persist a different version map.
Changing the ignore-null selection between retries for the same microbatch is a
very narrow edge case, but we should decide between:
- Documenting that changing ignore-null selection between retries can lead to
undefined behavior
- On microbatch retries, implement some kind of check to make sure the
ignore-null selection is not changing. This will likely require persisting
additional information
- On AutoCDC execution failures, re-wrap the exception with a generic AutoCDC
execution exception, which documents the risk of changing ignore-null selection
before the current microbatch is succesfully replayed
- Make retries deterministic when ignore-null selection is changing between
runs. Ex. stamp existing version maps/rows with a timestamp, and let incoming
rows always win
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
