[
https://issues.apache.org/jira/browse/SPARK-58928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod KC updated SPARK-58928:
-----------------------------
Description:
Several window and aggregation operators detect group or partition boundaries
by comparing grouping keys byte-for-byte. Byte comparison is wrong for collated
strings: under UTF8_LCASE, 'foo' and 'FOO' are equal but have different bytes,
so a single group gets split into several.
Affected sites:
- WindowGroupLimitExec (rank-filter queries like WHERE rank <= N): the final
result is still correct (the downstream WindowExec recomputes the ranking), but
the limit no longer prunes collated partitions and forwards many more rows than
needed.
- ArrowWindowPythonEvaluatorFactory.fetchNextPartition (pandas/Arrow window
UDFs): no downstream re-grouping, so the UDF output itself is wrong — a
grouped-agg count over ('foo', 1), ('FOO', 2), ('foo', 3) returns 1 for every
row instead of 3.
- MergingSessionsIterator (batch session-window aggregation, plain
aggregates): sessions merge per byte-spelling instead of per collation key —
('foo', 10:00), ('FOO', 10:05), ('foo', 10:08) with a 10-minute gap produce
three one-row sessions instead of one 10:00–10:18 session with count 3.
- UpdatingSessionsIterator (session-window DISTINCT aggregates, also used by
pandas grouped aggregation over session windows): fails with the internal error
"The iterator must be sorted by key and session start!" once a spelling recurs
after the first session of its byte-group.
Repro:
{code:java}
CREATE TABLE t (s STRING COLLATE UTF8_LCASE, i INT) USING PARQUET;
INSERT INTO t VALUES ('foo', 1), ('FOO', 2), ('bar', 3), ('BAR', 4);
SELECT s, i, r FROM (
SELECT s, i, RANK() OVER (PARTITION BY s ORDER BY i) AS r FROM t
) WHERE r <= 1; -- 'foo'/'FOO' and 'bar'/'BAR' are one partition each
{code}
Fix: compare keys the way WindowExec already does — byte comparison only when
all keys are binary-stable, collation-aware (ordering-based) comparison
otherwise — via a shared UnsafeRowUtils.rowEqualityChecker helper.
SortBasedAggregationIterator and WindowExec's evaluator, which already compared
collated keys with the interpreted ordering, share the helper too.
was:
For a query like _RANK() OVER (PARTITION BY s ...)_ with {_}WHERE rank <= N{_},
the optimizer adds a WindowGroupLimitExec to drop rows early. It finds
partition boundaries by comparing keys byte-for-byte (nextGroup ==
currentGroup).
Byte comparison is wrong for collated strings: under UTF8_LCASE, 'foo' and
'FOO' are equal but have different bytes, so one partition gets split into
several.
The final result is still correct (the later WindowExec recomputes the
ranking), but the limit no longer prunes collated partitions, so it forwards
many more rows than needed and the optimization stops helping.
Repro:
{code:java}
CREATE TABLE t (s STRING COLLATE UTF8_LCASE, i INT) USING PARQUET;
INSERT INTO t VALUES ('foo', 1), ('FOO', 2), ('bar', 3), ('BAR', 4);
SELECT s, i, r FROM (
SELECT s, i, RANK() OVER (PARTITION BY s ORDER BY i) AS r FROM t
) WHERE r <= 1; -- 'foo'/'FOO' and 'bar'/'BAR' are one partition each
{code}
Need to compare keys the same way WindowExec already does, use byte comparison
only when all keys are binary-stable, and fall back to collation-aware
comparison when any key is not.
> Fix collation-aware grouping in window and aggregation operators
> ----------------------------------------------------------------
>
> Key: SPARK-58928
> URL: https://issues.apache.org/jira/browse/SPARK-58928
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: Vinod KC
> Priority: Major
> Labels: pull-request-available
>
> Several window and aggregation operators detect group or partition boundaries
> by comparing grouping keys byte-for-byte. Byte comparison is wrong for
> collated strings: under UTF8_LCASE, 'foo' and 'FOO' are equal but have
> different bytes, so a single group gets split into several.
> Affected sites:
> - WindowGroupLimitExec (rank-filter queries like WHERE rank <= N): the final
> result is still correct (the downstream WindowExec recomputes the ranking),
> but the limit no longer prunes collated partitions and forwards many more
> rows than needed.
> - ArrowWindowPythonEvaluatorFactory.fetchNextPartition (pandas/Arrow window
> UDFs): no downstream re-grouping, so the UDF output itself is wrong — a
> grouped-agg count over ('foo', 1), ('FOO', 2), ('foo', 3) returns 1 for every
> row instead of 3.
> - MergingSessionsIterator (batch session-window aggregation, plain
> aggregates): sessions merge per byte-spelling instead of per collation key —
> ('foo', 10:00), ('FOO', 10:05), ('foo', 10:08) with a 10-minute gap produce
> three one-row sessions instead of one 10:00–10:18 session with count 3.
> - UpdatingSessionsIterator (session-window DISTINCT aggregates, also used by
> pandas grouped aggregation over session windows): fails with the internal
> error "The iterator must be sorted by key and session start!" once a spelling
> recurs after the first session of its byte-group.
> Repro:
> {code:java}
> CREATE TABLE t (s STRING COLLATE UTF8_LCASE, i INT) USING PARQUET;
> INSERT INTO t VALUES ('foo', 1), ('FOO', 2), ('bar', 3), ('BAR', 4);
> SELECT s, i, r FROM (
> SELECT s, i, RANK() OVER (PARTITION BY s ORDER BY i) AS r FROM t
> ) WHERE r <= 1; -- 'foo'/'FOO' and 'bar'/'BAR' are one partition each
>
> {code}
>
> Fix: compare keys the way WindowExec already does — byte comparison only when
> all keys are binary-stable, collation-aware (ordering-based) comparison
> otherwise — via a shared UnsafeRowUtils.rowEqualityChecker helper.
> SortBasedAggregationIterator and WindowExec's evaluator, which already
> compared collated keys with the interpreted ordering, share the helper too.
>
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]