HyukjinKwon commented on PR #57990:
URL: https://github.com/apache/spark/pull/57990#issuecomment-5289117594
Thanks — all three addressed in `98d3dcb`:
1. **Process-global registry / cross-session collision.** The classic
`ObservedAccumulatorRegistry` is now keyed by `(session UUID, name)` rather
than bare name, so `spark.accumulator("bad")` and
`spark.newSession().accumulator("bad")` write to disjoint slots. The harvest
listener captures its *registering* session's UUID at construction (rather than
reading `qe.sparkSession`), so a Structured Streaming micro-batch clone — which
shares the listener instance but has a fresh UUID — still attributes to the
creating session. Added tests on both Scala (`same-named accumulators in
different sessions do not collide`) and Python classic.
2. **Arg order.** Python is now `accumulator(name, zero=0, merge=None)`,
matching Scala's name-first signature, so `spark.accumulator("bad")` means the
same in both. Updated all call sites, docstrings, and the PR examples.
3. **Scala UDF in a HOF lambda.** Now fails fast: after the rewrite,
`checkNoUnrewritten` flags any accumulator UDF left in an unobservable position
(HOF lambda, `Aggregate`/`Join`/`Window`/`Sort`, or nested as a UDF argument)
and raises. The rewrite no longer descends into `LambdaFunction` bodies. Added
a Scala test.
Also in this round: the cross-session guard now raises a proper
`PySparkRuntimeError` (error class `OBSERVED_ACCUMULATOR_DIFFERENT_SESSION`),
the accumulator UDF is observed in `filter` conditions and composed expressions
(not just top-level projected columns), and `observed_accumulator.py` is fully
type-annotated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
