mrk-andreev opened a new pull request, #58634:
URL: https://github.com/apache/spark/pull/58634
### What changes were proposed in this pull request?
Add SQL support for `GROUPS` window frames. Offsets count peer groups: rows
with equal values for all window `ORDER BY` expressions within a partition.
`CURRENT ROW` includes the whole current peer group.
The change adds parsing and validation, implements group boundaries in the
existing window execution paths, and supports eligible aggregates through the
segment-tree path. Group tracking retains only the current ordering keys as it
advances through a partition. The SQL reference and keyword documentation are
updated.
### Why are the changes needed?
Some moving calculations need to include complete groups of tied rows. For
example, a total across the current and previous batch should include every row
in both batches, even when batch IDs have gaps. `GROUPS` expresses this
directly.
### Does this PR introduce _any_ user-facing change?
Yes. Previously, Spark rejected `GROUPS` frame syntax. This query now sums
each batch with the previous batch:
```sql
SELECT batch_id, amount,
SUM(amount) OVER (
ORDER BY batch_id GROUPS BETWEEN 1 PRECEDING AND CURRENT ROW
) AS moving_sum
FROM VALUES (1, 10), (1, 15), (2, 20), (3, 25), (3, 30), (9, 40)
AS batches(batch_id, amount)
ORDER BY batch_id, amount;
```
```text
batch_id amount moving_sum
1 10 25
1 15 25
2 20 45
3 25 75
3 30 75
9 40 95
```
For batch 9, the previous group is batch 3, so the total is `25 + 30 + 40 =
95`.
`GROUPS` requires `ORDER BY`, supports multiple ordering expressions, and
accepts constant, non-null, non-negative integer offsets. `GROUPS` remains
usable as an identifier.
### How was this patch tested?
The commit adds parser and validation tests, SQL golden coverage, and
execution tests for tied values, multiple ordering keys, nulls, descending
order, partition boundaries, and spilling. It also enables previously disabled
PostgreSQL `GROUPS` test cases.
Execution coverage includes randomized comparisons against `DENSE_RANK` plus
`RANGE`, segment-tree enabled/disabled comparisons, fallback and metrics
checks, and a cluster-mode test in `HiveSparkSubmitSuite`.
### Was this patch authored or co-authored using generative AI tooling?
Yes. The code was co-authored using the following tools.
Generated-by:
- Codex (GPT-6 Astra)
- Claude Code Opus 5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
