viirya commented on PR #55278: URL: https://github.com/apache/spark/pull/55278#issuecomment-4532362588
Thanks for the patch. I have a design question about the transparency of transactional semantics. With interfaces like `SupportsDelete` or `SupportsRowLevelOperations`, the capability is tied to a specific SQL command — if the connector doesn't support it, Spark rejects the command. The user gets an explicit signal. `TransactionalCatalogPlugin` works differently. The same `INSERT INTO` (or `MERGE`, `UPDATE`, etc.) silently runs with or without transaction protection depending on whether the underlying catalog implements the interface. There is no user-visible difference — no error, no warning, no query plan annotation. A user who writes a pipeline assuming transactional semantics (e.g., skipping idempotency logic) would not know if they switch to a catalog that doesn't implement `TransactionalCatalogPlugin`. This is somewhat analogous to `StagingTableCatalog`, which is also optional — but that one is scoped to CTAS/RTAS (commands with naturally "all-or-nothing" expectations), and its Javadoc explicitly documents the non-atomic fallback path. Here, the affected commands are general-purpose write operations where users may not think to question whether atomicity is present. Should there be at least a mechanism for users to assert their expectation — e.g., a session config like `spark.sql.requireTransactionalWrites` that fails the query if the catalog doesn't support `TransactionalCatalogPlugin`? Or a note in `EXPLAIN` output indicating whether a transaction will be opened? Even just a `WARN`-level log at query start would give users a way to detect the difference. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
