Luka Zdravic created SPARK-60744:
------------------------------------
Summary: Read the right side once with a cursor in the sort-merge
ASOF join
Key: SPARK-60744
URL: https://issues.apache.org/jira/browse/SPARK-60744
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.4.0
Reporter: Luka Zdravic
The sort-merge ASOF join buffers each right-side group. Then, for every left
row, it scans that buffer again from the start. So a group with N left rows and
M right rows can cost up to N x M condition checks, and a large group spills to
disk.
{code:sql}
SELECT * FROM trades t LEFT ASOF JOIN quotes q
MATCH_CONDITION (t.ts >= q.ts) ON t.sym = q.sym
{code}
Both sides are already sorted by the equi-keys and then by the as-of key. So in
a backward or forward join, the matching right row only moves forward as the
left key grows. The scan can walk the right side once with a cursor and keep
only the best row so far. That costs about N + M checks and buffers nothing.
The cursor scan is safe only when:
* The join is backward or forward, not nearest.
* No residual join condition remains.
* The as-of condition compares the two sort keys, one on each side, of the
same type.
* The key type is scalar, not STRUCT or ARRAY.
Every other join keeps the buffered scan. An internal flag turns the cursor
scan off, and a new metric counts the tasks that use it.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
