anew commented on code in PR #57192:
URL: https://github.com/apache/spark/pull/57192#discussion_r3592134277
##########
sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/autocdc/ChangeArgs.scala:
##########
@@ -153,21 +153,28 @@ object ScdType {
/**
* Configuration for an AutoCDC flow.
*
- * @param keys The column(s) that uniquely identify a row in the
source data.
- * @param sequencing Expression ordering CDC events to correctly resolve
out-of-order
- * arrivals. Must be a sortable type.
- * @param deleteCondition Expression that marks a source row as a DELETE. When
None, all
- * rows are treated as upserts.
- * @param storedAsScdType The SCD strategy these args should be applied to.
- * @param columnSelection Which source columns to select in the target table.
None means
- * all columns.
+ * @param keys The column(s) that uniquely identify a row in
the source data.
+ * @param sequencing Expression ordering CDC events to correctly
resolve out-of-order
+ * arrivals. Must be a sortable type.
+ * @param deleteCondition Expression that marks a source row as a
DELETE. When None, all
+ * rows are treated as upserts.
+ * @param storedAsScdType The SCD strategy these args should be applied
to.
+ * @param columnSelection Which source columns to select in the target
table. None means
+ * all columns.
+ * @param trackHistorySelection SCD2 only. Selects the selected user-data
columns whose values
+ * define a run: two consecutive upsert events
for the same key are
+ * coalesced into the same run iff they agree on
every selected
+ * tracking column. None means every eligible
selected user column
+ * (i.e. every selected source column that is
neither a key nor a
+ * framework column) is considered tracked.
Ignored under SCD1.
Review Comment:
I think tis is just for the internal case class here. Validation that SCD1
never speficies tracking columns should happen at the API (GraphRegistration)
layer. But just to be safe, we can also throw an internal error here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
