ulysses-you commented on PR #58339:
URL: https://github.com/apache/spark/pull/58339#issuecomment-5439199372
## Reproduction
Only `spark.sql.sources.v2.bucketing.shuffle.enabled` is needed (the other
SPJ toggles are at their defaults). The keyed tables need a catalog that
reports partitioning (here `testcat` is the in-memory test catalog).
```sql
-- 1) config
SET spark.sql.sources.v2.bucketing.shuffle.enabled = true;
-- 2) tables + data
-- a: keyed on id, keys {1, 2}
CREATE TABLE testcat.ns.a (id BIGINT, data STRING) PARTITIONED BY (id);
INSERT INTO testcat.ns.a VALUES (1, 'a1'), (2, 'a2');
-- t: v1 parquet, keys {1, 2, 3} (contains a key `a` does not have)
CREATE TABLE t (id BIGINT, data STRING) USING parquet;
INSERT INTO t VALUES (1, 't1'), (2, 't2'), (3, 't3');
-- u: keyed on id, keys {1, 2, 3}
CREATE TABLE testcat.ns.u (id BIGINT, data STRING) PARTITIONED BY (id);
INSERT INTO testcat.ns.u VALUES (1, 'u1'), (2, 'u2'), (3, 'u3');
-- 3) query: the RIGHT OUTER join preserves t's id=3 row; the following
-- storage-partitioned join must still match it against u.
SELECT r.id, u.data
FROM (
SELECT t.id AS id
FROM testcat.ns.a a RIGHT OUTER JOIN t ON a.id = t.id
) r
JOIN testcat.ns.u u ON r.id = u.id;
```
**Result**
| | id |
|---|---|
| Expected | 1, 2, 3 |
| Actual (before this PR) | 1, 2 ← **id=3's match silently lost** |
Turning `spark.sql.sources.v2.bucketing.shuffle.enabled` off (or disabling
SPJ entirely) returns all three rows, confirming this is an SPJ-path bug rather
than a query semantics issue.
**Mechanism**
1. The first join one-side-shuffles `t` onto `a`'s declared keys `{1, 2}`.
`t`'s id=3 is not among the declared keys, so `KeyGroupedPartitioner` silently
routes it to an arbitrary partition, while the shuffle still declares the `{1,
2}` layout.
2. `a RIGHT OUTER JOIN t` preserves that misplaced id=3 row, so the join
output's declared partitioning (`{1, 2}`) no longer matches its data.
3. The second join trusts the declared layout when planning its
storage-partitioned join against `u` (`{1, 2, 3}`) and never co-locates the
id=3 rows, silently dropping the match.
The fix marks such one-side-shuffled partitionings with
`mayContainUnknownPartitionKeys` and restricts `areKeysCompatible` to only
co-partition them with a side whose keys are a subset of the declared keys (in
the same order when both sides are flagged), so the second join falls back to a
shuffle and returns correct results.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
