HyukjinKwon commented on PR #57990: URL: https://github.com/apache/spark/pull/57990#issuecomment-5335706713
Addressed the review findings in `1a447c0`: 1. **Duplicate metric collision** -- the rule now combines the deltas of one accumulator referenced by multiple UDFs in a single Project/Filter into a **single** `__oa_metric_<name>` (numeric: added then summed; typed: one `collect_list` spanning all partials), instead of emitting duplicate-named metric columns that the harvest listener read only the first of. Regression test added (`ObservedAccumulatorSuite`: two UDFs / one accumulator / one `select` -> counts both). 2. **Fractional add to an integer accumulator** -- `add()` now rejects a non-integer term on an integer-typed accumulator with a classed `OBSERVED_ACCUMULATOR_NON_INTEGER_ADD` error, consistently across all UDF flavors (previously silent truncation in a row UDF, hard crash in a vectorized one). Whole-valued terms (Python/NumPy ints, or `2.0`) are still accepted; float accumulators are unchanged. Test added. 3. **Scala Long exactness** -- documented that the Scala numeric accumulator is `Double`-backed, so `value: Long` is exact only within a `Double`'s 53-bit integer range (the PySpark integer accumulator keeps an exact `Long`). Verified locally: `ObservedAccumulatorSuite` 18/18 (incl. the new regression), and the `add()` guard by unit test. The full PySpark suite + Connect parity run in CI. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
