[
https://issues.apache.org/jira/browse/SPARK-60421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uroš Bojanić resolved SPARK-60421.
----------------------------------
Fix Version/s: 4.4.0
Resolution: Fixed
Issue resolved by pull request 59615
[https://github.com/apache/spark/pull/59615]
> Fix behaviour of vector avg function for special float values
> -------------------------------------------------------------
>
> Key: SPARK-60421
> URL: https://issues.apache.org/jira/browse/SPARK-60421
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: Uros Stankovic
> Assignee: Uros Stankovic
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.4.0
>
>
> The existing update formula of vector_avg implementation can overflow its
> intermediate subtraction or turn an infinite average into NaN after a finite
> input. This makes updates disagree with merging singleton buffers and makes
> the result depend on input order.
> The following cases show two successive inputs for one vector element:
> | Input sequence | Current behaviour | Correct behaviour |
> |---|---|---|
> | `Float.MaxValue`, then `-Float.MaxValue` | `-Infinity` | `0.0f` |
> | `-Float.MaxValue`, then `Float.MaxValue` | `Infinity` | `0.0f` |
> | `Infinity`, then `Infinity` | `NaN` | `Infinity` |
> | `-Infinity`, then `-Infinity` | `NaN` | `-Infinity` |
> | `Infinity`, then `-Infinity` (either order) | `NaN` | `NaN` |
> | `Infinity`, then a finite value | `NaN` | `Infinity` |
> | `-Infinity`, then a finite value | `NaN` | `-Infinity` |
> | A finite value, then `Infinity` | `Infinity` | `Infinity` |
> | A finite value, then `-Infinity` | `-Infinity` | `-Infinity` |
> | `NaN` and any value (either order) | `NaN` | `NaN` |
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
