anew commented on code in PR #57192:
URL: https://github.com/apache/spark/pull/57192#discussion_r3592146351
##########
sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/autocdc/Scd2BatchProcessor.scala:
##########
@@ -439,9 +439,9 @@ case class Scd2BatchProcessor(
* a dataframe with the same schema as the input. Every closed
non-tombstone row that
* was bisected has been replaced by its head + tail pair; every other row
is carried
* through as-is. Each output row can be classified as one of:
{decomposition head,
- * decomposition tail, instantaneous delete, open upsert,
closed-and-unbisected row}. It's
- * possible that some of the returned decomposition tails are logically
redundant, as
- * deletion markers that are immediately overtaken by a succeeding row.
+ * decomposition tail, tombstone, open upsert, closed-and-unbisected row}.
It's possible
Review Comment:
Yes, but the aux table can contain tombstones (representing early-arriving
deletes).
##########
sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/autocdc/Scd2BatchProcessor.scala:
##########
@@ -556,7 +556,7 @@ case class Scd2BatchProcessor(
val row = Scd2IntervalColumns(recordStartAtField, startAtCol, endAtCol)
val isWellFormedRow =
RowClassifier.isDecompositionTail(row) ||
- RowClassifier.isDeleteRepresentingRow(row) ||
+ RowClassifier.isTombstone(row) ||
Review Comment:
see previous comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
