cloud-fan commented on code in PR #58673:
URL: https://github.com/apache/spark/pull/58673#discussion_r3978945883
##########
docs/streaming/real-time-mode.md:
##########
@@ -602,8 +605,15 @@ spark
</div>
-This keeps every distinct key it has seen in the state store.
`dropDuplicatesWithinWatermark`, which
-bounds how long keys are retained, is not yet supported in Real-time Mode.
+This example uses `dropDuplicates`, which keeps every distinct key it has seen
in the state store.
+To bound state, define an event-time watermark and use
`dropDuplicatesWithinWatermark`. For example,
+Scala and Java use
+`withWatermark("eventTime", "10
minutes").dropDuplicatesWithinWatermark("id")`; in PySpark, pass the
Review Comment:
**Non-blocking (P2):** This bounded example cannot be applied to any of the
snippets above: every selectExpr removes Kafka's timestamp and leaves only id
and value, so eventTime does not exist when withWatermark resolves. Please
retain and alias timestamp as eventTime in each Python, Scala, and Java tab,
and show the bounded operation in a complete runnable chain.
##########
docs/streaming/real-time-mode.md:
##########
@@ -34,10 +34,11 @@ personalization.
Real-time Mode in Apache Spark supports **stateless queries** -- projections,
filters and other
map-like operations, unions, and stream-static joins -- and, starting in Spark
4.3.0, a first set of
-**stateful queries**: streaming **deduplication** (`dropDuplicates`) and
streaming **aggregations**
-(`groupBy(...).agg(...)`), and the JVM (Scala/Java) **`transformWithState`**
operator. These stateful
-operations require a shuffle, which Real-time Mode runs as a *pipelined
shuffle* so that records
-still stream through without waiting for a batch boundary; see
+**stateful queries**: streaming **deduplication** (`dropDuplicates`, plus
+`dropDuplicatesWithinWatermark` starting in Spark 5.0.0), streaming
**aggregations**
Review Comment:
**Non-blocking (P2):** These two labels should use 4.4.0 rather than 5.0.0.
The repository policy uses the latest branch-4.x version for normally
backported master changes, and this enablement is neither breaking nor a
dependency upgrade. Otherwise the 4.4 guide will say the feature starts in a
later release. If the project explicitly decides this PR is master-only, that
decision should be established instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
