nsivabalan commented on code in PR #20454:
URL: https://github.com/apache/hudi/pull/20454#discussion_r4405572160
##########
website/docs/writing_tables_streaming_writes.md:
##########
@@ -93,9 +93,54 @@ df.writeStream.format("hudi")
</Tabs
>
+### Session settings and upgrade compatibility
+
+In the upcoming release, Structured Streaming sinks honor the active session's
Review Comment:
Could we name the version ("Starting with 1.3.0") instead of "In the
upcoming release"? `docs/` is snapshotted per release, so "upcoming" goes stale
in the versioned copy.
##########
website/docs/writing_tables_streaming_writes.md:
##########
@@ -93,9 +93,54 @@ df.writeStream.format("hudi")
</Tabs
>
+### Session settings and upgrade compatibility
+
+In the upcoming release, Structured Streaming sinks honor the active session's
+`spark.hoodie.*` settings. Previously, the streaming sink ignored these
settings,
+even though batch DataFrame writes supported them. Review ambient session
configuration
+when upgrading an existing `writeStream` application.
+
+For example, this setting selects inserts for a subsequently started streaming
query:
+
+```python
+spark.conf.set("spark.hoodie.datasource.write.operation", "insert")
+```
+
+Explicit writer options take precedence. To retain upsert behavior despite
that session setting:
+
+```python
+query = (df.writeStream.format("hudi")
+ .options(**hudi_streaming_options)
+ .option("hoodie.datasource.write.operation", "upsert")
+ .option("path", baseStreamingPath)
+ .option("checkpointLocation", checkpointLocation)
+ .start())
+```
+
+The precedence is explicit writer/storage options, then session
`spark.hoodie.*`,
+then Hudi defaults. The sink strips the `spark.` prefix before validation and
writing.
+Explicit options may use either prefix; if both forms are supplied explicitly,
+the canonical `hoodie.*` option wins. Bare `hoodie.*` session settings are not
forwarded.
+
+Settings are captured when the sink is created. Changing the session afterward
does not
+change an existing query; restart it to capture new settings. The same
captured options
+control multi-writer validation, checkpoint deduplication, and micro-batch
writes.
+Table identity options must still agree with the persisted Hudi table. An
incompatible
+record key, ordering field, partition field, key generator, or record-merge
setting
+fails validation when opening an existing table, including during checkpoint
replay.
+
+For flow-based integrations such as the proposed Spark Declarative Pipelines
integration
Review Comment:
This paragraph gives configuration guidance for an integration that is still
a proposal (RFC-112), and "legacy table storage options" is not a term users
will know from these docs. I'd drop it here and document flow-scoped `SET`
together with the SDP integration once it lands. If we keep it, please explain
what the table's storage options are and how a user would find and remove one.
##########
website/docs/writing_tables_streaming_writes.md:
##########
@@ -1,7 +1,7 @@
---
title: Streaming Writes
keywords: [hudi, spark, flink, streaming, processing]
-last_modified_at: 2024-03-13T15:59:57-04:00
+last_modified_at: 2026-10-28T00:00:00+08:00
Review Comment:
Nit: the diff adds an extra trailing blank line at the end of the file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
