hansva opened a new issue, #9299:
URL: https://github.com/apache/hop/issues/9299

   ### Apache Hop version?
   
   2.20.0
   
   ### Java version?
   
   21
   
   ### Operating system
   
   Other
   
   ### What happened?
   
   In Memory group by, an aggregate of type Standard deviation fails the 
transform as soon as its field contains a null value:
   
       java.lang.NullPointerException: Cannot invoke 
"java.lang.Double.doubleValue()" because the return value of 
"org.apache.hop.core.row.IValueMeta.getNumber(Object)" is null
           at 
org.apache.hop.pipeline.transforms.memgroupby.MemoryGroupBy.addToAggregate(MemoryGroupBy.java:285)
   
   `MemoryGroupBy.addToAggregate()` reads the value with `double x = 
subjMeta.getNumber(subj)` without checking for null first. For a null value 
`getNumber` returns `null`, which can't be unboxed into a `double`. The other 
aggregates in the same method (Sum, Average, Median, Percentile, Count 
distinct) skip null values with `if (!subjMeta.isNull(subj))`; Standard 
deviation is the only one without that check.
   
   Even if the check were added, a group whose values are all null would fail 
at the end: `getAggregateResult()` divides the running sum with `(Double) ag / 
counts`, and `ag` is still null.
   
   The Group by transform already handles both cases: it skips null values in 
the standard deviation, and returns null when every value in the group is null.
   
   ### Steps to reproduce
   
   1. Input rows with a Number field `x` containing the values `2`, null and 
`4`.
   2. Memory group by with an aggregate of type Standard deviation on `x`.
   3. Run the pipeline: the transform fails with the `NullPointerException` 
above. Group by on the same data returns 1.0.
   
   ### Proposed fix
   
   Handle nulls the way Group by does:
   
   - In `addToAggregate()`, only update the count, the mean and the running sum 
when the value is not null.
   - In `getAggregateResult()`, leave the result null when no non-null value 
was seen.
   
   A new `MemoryGroupByStandardDeviationTest` covers both cases: `2`, null, `4` 
gives 1.0 (the population standard deviation of 2 and 4), and a group with only 
nulls gives null. Both tests fail with the `NullPointerException` on main.
   
   ### Issue Priority
   
   Priority: 2
   
   ### Issue Component
   
   Component: Transforms


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to