AnishMahto commented on code in PR #59311:
URL: https://github.com/apache/spark/pull/59311#discussion_r4266382248
##########
sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/autocdc/Scd1BatchProcessor.scala:
##########
@@ -226,29 +227,45 @@ object Scd1BatchProcessor {
/**
* Schema of the CDC metadata struct column for SCD1.
+ *
+ * The CDC metadata of every microbatch, target, and auxiliary-table row
represents either a
+ * delete or an upsert, never both: exactly one of its delete and upsert
sequences is non-null,
+ * and only an upsert may carry a version map.
*/
private[pipelines] def cdcMetadataColSchema(sequencingType: DataType):
StructType =
StructType(
Seq(
// The sequencing of the event if it represents a delete, null
otherwise.
StructField(cdcDeleteSequenceFieldName, sequencingType, nullable =
true),
// The sequencing of the event if it represents an upsert, null
otherwise.
- StructField(cdcUpsertSequenceFieldName, sequencingType, nullable =
true)
+ StructField(cdcUpsertSequenceFieldName, sequencingType, nullable =
true),
+ // On target rows, a null version map indicates row-level
reconciliation; a non-null map
+ // indicates per-leaf reconciliation. Delete rows, including
auxiliary-table tombstones,
+ // always use a null map: a delete's sequence applies to every leaf,
so a map would only
+ // repeat it.
+ StructField(
Review Comment:
A couple things:
1. The addition of `__VERSION_MAP` in `__spark_autocdc_metadata` is done in
the previous PR in the stack. We should address this issue there, if at all
2. SPARK-58118 is actively being worked on, there's an open PR/review.
3. If these changes land before SPARK-58118, a
`USER_SPECIFIED_AND_INFERRED_SCHEMA_NOT_COMPATIBLE` exception will be thrown
that clearly lists the declared and inferred schemas. While not an ideal CUJ,
affected users will be able to easily identify the new `__VERSION_MAP` subfield
and its exact schema type, which they can add to their declared schema.
Given (2) is actively being worked on and the consequence of (3) is an
actionable error message, I'm inclined to not do anything further.
WDYT? If you'd still prefer I add to the programming guide then I'll do it
in the earlier PR that introduces the schema change:
https://github.com/apache/spark/pull/58973
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
