[
https://issues.apache.org/jira/browse/SPARK-60744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenchen Fan updated SPARK-60744:
--------------------------------
Fix Version/s: 4.3.0
(was: 5.0.0)
> Read the right side once with a cursor in the sort-merge ASOF join
> ------------------------------------------------------------------
>
> Key: SPARK-60744
> URL: https://issues.apache.org/jira/browse/SPARK-60744
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: Luka Zdravic
> Assignee: Luka Zdravic
> Priority: Minor
> Labels: pull-request-available
> Fix For: 4.3.0
>
>
> The sort-merge ASOF join buffers each right-side group. Then, for every left
> row, it scans that buffer again from the start. So a group with N left rows
> and M right rows can cost up to N x M condition checks, and a large group
> spills to disk.
> {code:sql}
> SELECT * FROM trades t LEFT ASOF JOIN quotes q
> MATCH_CONDITION (t.ts >= q.ts) ON t.sym = q.sym
> {code}
> Both sides are already sorted by the equi-keys and then by the as-of key. So
> in a backward or forward join, the matching right row only moves forward as
> the left key grows. The scan can walk the right side once with a cursor and
> keep only the best row so far. That costs about N + M checks and buffers
> nothing.
> The cursor scan is safe only when:
> * The join is backward or forward, not nearest.
> * No residual join condition remains.
> * The as-of condition compares the two sort keys, one on each side, of the
> same type.
> * The key type is scalar, not STRUCT or ARRAY.
> Every other join keeps the buffered scan. An internal flag turns the cursor
> scan off, and a new metric counts the tasks that use it.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
