spark-test commented on code in PR #59615: URL: https://github.com/apache/spark/pull/59615#discussion_r4461491186
########## docs/sql-migration-guide.md: ########## @@ -28,6 +28,7 @@ license: | ## Upgrading from Spark SQL 4.3 to 4.4 +- Since Spark 4.4, `vector_avg` improves overflow and non-finite value handling for `ARRAY<FLOAT>` inputs. Extreme finite values that previously produced infinity may now produce a finite average. Infinity combined with finite values or infinity of the same sign now remains infinity instead of producing `NaN`. Opposite infinities still produce `NaN`, and `NaN` inputs continue to propagate. Review Comment: **Nit (P3):** The bullet's second sentence names infinity as the only earlier result for extreme finite inputs. On the base branch, though, that infinity became `NaN` as soon as the group received another row, or when partial averages that had overflowed in opposite directions were merged. Rows `[Float.MaxValue]`, `[-Float.MaxValue]`, `[0.0f]` returned `NaN` and now return `0.0f`, and rows `[Float.MaxValue]`, `[-Float.MaxValue]`, `[1.0f]`, `[2.0f]` returned `NaN` and now return `0.75f`. Every `NaN` case the bullet mentions involves an infinite or `NaN` input, so users whose `NaN` results from finite-only data become finite after upgrading find nothing here that explains it. ```suggestion - Since Spark 4.4, `vector_avg` improves overflow and non-finite value handling for `ARRAY<FLOAT>` inputs. Extreme finite values that previously produced infinity or `NaN` may now produce a finite average. Infinity combined with finite values or infinity of the same sign now remains infinity instead of producing `NaN`. Opposite infinities still produce `NaN`, and `NaN` inputs continue to propagate. ``` This extends an [existing discussion](https://github.com/apache/spark/pull/59615#pullrequestreview-5630506828). **Verification:** - **Inspection:** Check that the vector_avg migration-guide bullet lists NaN alongside infinity as a previous result for extreme finite inputs. On the base branch, rows of [Float.MaxValue], [-Float.MaxValue], [0.0f] returned NaN and now return 0.0f, and the bullet must not imply that NaN results from finite-only vectors are unchanged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
