amalsaidov opened a new pull request, #20233:
URL: https://github.com/apache/pinot/pull/20233
## Description
Adds a `CORR(x, y)` aggregation function that returns the Pearson
correlation coefficient of two numeric expressions. Resolves #9277.
The implementation mirrors the existing `covarPop` / `covarSamp` functions.
A `CorrelationTuple` accumulates `count`, `sum(x)`, `sum(y)`, `sum(x*y)`,
`sum(x^2)` and `sum(y^2)`, and the final result is:
```
corr(x, y) = (n*sum(xy) - sum(x)*sum(y))
/ sqrt((n*sum(x^2) - sum(x)^2) * (n*sum(y^2) - sum(y)^2))
```
Notes on the behavior:
- The sample vs population distinction cancels out in the ratio, so only the
population form is computed.
- The result is undefined when fewer than one row contributes or when either
column is constant (zero variance). In those cases it returns `NULL` when null
handling is enabled and the same sentinel the other statistical functions use
otherwise. This is the case that produced `NaN` in earlier attempts, so both
the zero denominator and a variance term driven slightly negative by rounding
are handled explicitly.
- Rounding error is clamped to the mathematical `[-1, 1]` range.
Unlike `COVAR_POP` / `COVAR_SAMP`, `CORR` is not a Calcite built-in in the
version Pinot uses, so it is registered in `AggregationFunctionType` with an
explicit `DOUBLE` return type and a two numeric operand signature so the
multi-stage engine can resolve it. It works in both the single-stage and
multi-stage engines.
## Testing
- `CorrelationAggregationFunctionTest`: null handling (row skipped when
either column is null, nothing aggregated when all null, folding with the
option off), perfect positive and negative correlation, a constant column being
undefined, agreement with Apache Commons Math `PearsonsCorrelation` on
arbitrary data, the empty-accumulator contract, and the intermediate tuple
serialize/deserialize round trip.
- `QueryCompilationTest#testCorrelationAggregateResolves`: confirms `CORR`
resolves and plans in the multi-stage engine as a plain aggregate and in a
GROUP BY.
- `AggregationFunctionNullContractTest` updated to account for the new
function; the full class passes.
- `StatisticalQueriesTest` and `AggregationFunctionTypeTest` pass unchanged.
Build and tests run with `./mvnw -pl pinot-core -am -Dtest=...` and the
equivalent for the planner module. Spotless, checkstyle and license checks pass
on the touched modules.
## Labels
`feature`, `release-notes` (new aggregation function)
---
AI-assisted; directed, reviewed and tested by the author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
