hansva opened a new issue, #9299:
URL: https://github.com/apache/hop/issues/9299
### Apache Hop version?
2.20.0
### Java version?
21
### Operating system
Other
### What happened?
In Memory group by, an aggregate of type Standard deviation fails the
transform as soon as its field contains a null value:
java.lang.NullPointerException: Cannot invoke
"java.lang.Double.doubleValue()" because the return value of
"org.apache.hop.core.row.IValueMeta.getNumber(Object)" is null
at
org.apache.hop.pipeline.transforms.memgroupby.MemoryGroupBy.addToAggregate(MemoryGroupBy.java:285)
`MemoryGroupBy.addToAggregate()` reads the value with `double x =
subjMeta.getNumber(subj)` without checking for null first. For a null value
`getNumber` returns `null`, which can't be unboxed into a `double`. The other
aggregates in the same method (Sum, Average, Median, Percentile, Count
distinct) skip null values with `if (!subjMeta.isNull(subj))`; Standard
deviation is the only one without that check.
Even if the check were added, a group whose values are all null would fail
at the end: `getAggregateResult()` divides the running sum with `(Double) ag /
counts`, and `ag` is still null.
The Group by transform already handles both cases: it skips null values in
the standard deviation, and returns null when every value in the group is null.
### Steps to reproduce
1. Input rows with a Number field `x` containing the values `2`, null and
`4`.
2. Memory group by with an aggregate of type Standard deviation on `x`.
3. Run the pipeline: the transform fails with the `NullPointerException`
above. Group by on the same data returns 1.0.
### Proposed fix
Handle nulls the way Group by does:
- In `addToAggregate()`, only update the count, the mean and the running sum
when the value is not null.
- In `getAggregateResult()`, leave the result null when no non-null value
was seen.
A new `MemoryGroupByStandardDeviationTest` covers both cases: `2`, null, `4`
gives 1.0 (the population standard deviation of 2 and 4), and a group with only
nulls gives null. Both tests fail with the `NullPointerException` on main.
### Issue Priority
Priority: 2
### Issue Component
Component: Transforms
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]