szehon-ho commented on code in PR #57444:
URL: https://github.com/apache/spark/pull/57444#discussion_r3642194697
##########
sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/autocdc/Scd2BatchProcessor.scala:
##########
@@ -1166,6 +1477,27 @@ object Scd2BatchProcessor {
*/
private val anchorSequenceColName: String =
s"${AutoCdcReservedNames.prefix}anchor_sequence"
+ /**
+ * Name of the temporary column projected by
[[Scd2BatchProcessor.identifyAndTagAuxRows]] to
+ * mark rows destined for the auxiliary table (tombstones and hidden no-op
upserts) rather than
+ * the target table.
+ *
+ * Temporary in that the column has no observable side effect or persistence
across microbatches.
+ */
+ private[autocdc] val shouldRouteToAuxTableColName: String =
+ s"${AutoCdcReservedNames.prefix}should_route_to_aux_table"
Review Comment:
These temporary names are not covered by , and input validation does not
reject user columns in the AutoCDC internal namespace. A user column named is
overwritten here and later dropped; the sibling is also overwritten while
constructing the target MERGE source. This can lose user data or make the merge
source ambiguous. Please reserve/reject these names (ideally the full
AutoCDC-owned namespace) and add collision tests.
##########
sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/autocdc/Scd2BatchProcessor.scala:
##########
@@ -1166,6 +1477,27 @@ object Scd2BatchProcessor {
*/
private val anchorSequenceColName: String =
s"${AutoCdcReservedNames.prefix}anchor_sequence"
+ /**
+ * Name of the temporary column projected by
[[Scd2BatchProcessor.identifyAndTagAuxRows]] to
+ * mark rows destined for the auxiliary table (tombstones and hidden no-op
upserts) rather than
+ * the target table.
+ *
+ * Temporary in that the column has no observable side effect or persistence
across microbatches.
+ */
+ private[autocdc] val shouldRouteToAuxTableColName: String =
+ s"${AutoCdcReservedNames.prefix}should_route_to_aux_table"
Review Comment:
These temporary names are not covered by `reservedFrameworkColNames`, and
input validation does not reject user columns in the AutoCDC internal
namespace. A user column named `__spark_autocdc_should_route_to_aux_table` is
overwritten here and later dropped; the sibling
`__spark_autocdc_should_delete_target_row` is also overwritten while
constructing the target MERGE source. This can lose user data or make the merge
source ambiguous. Please reserve/reject these names (ideally the full
AutoCDC-owned namespace) and add collision tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
