[ 
https://issues.apache.org/jira/browse/SPARK-60691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jungtaek Lim resolved SPARK-60691.
----------------------------------
    Fix Version/s: 4.4.0
       Resolution: Fixed

Issue resolved by pull request 60019
[https://github.com/apache/spark/pull/60019]

> Add the EXTERNAL_LOOKUP join hint for external lookup join
> ----------------------------------------------------------
>
>                 Key: SPARK-60691
>                 URL: https://issues.apache.org/jira/browse/SPARK-60691
>             Project: Spark
>          Issue Type: Task
>          Components: SQL, Structured Streaming
>    Affects Versions: 4.4.0
>            Reporter: Jungtaek Lim
>            Assignee: Jungtaek Lim
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.4.0
>
>
> Part of SPARK-60031.
> In scope:
> - The `EXTERNAL_LOOKUP` hint in DataFrame and SQL forms. Without parameters 
> it marks the plan it annotates, as join strategy hints do; with relation 
> names it marks those relations.
> - A dual-capability table (`EXTERNAL_LOOKUP` and `BATCH_READ`) becomes a 
> lookup join only with the hint; without it, it keeps the ordinary join.
> - The lookup side is found through hint markers, aliases and views, so SQL 
> and catalog table references run.
> - A join that must be a lookup join but reaches its table through any other 
> operator, such as a projection or a filter, fails with `LOOKUP_PLAN` when the 
> DataFrame is built.
> - A user join hint is never silently overridden:
>   - a physical strategy hint on a join over a lookup-only table fails with 
> `CONFLICTING_JOIN_STRATEGY`;
>   - a physical strategy hint together with the `EXTERNAL_LOOKUP` hint on a 
> dual-capability table fails with 
> `INVALID_EXTERNAL_LOOKUP_HINT.CONFLICTING_JOIN_STRATEGY`;
>   - a hinted join must also be stream-static (`STATIC_LEFT_SIDE`).
> - Error reporting:
>   - ineligible hinted joins fail with `JOIN_TYPE`, `MISSING_JOIN_CONDITION` 
> or `STREAMING_LOOKUP_SIDE`, and a hinted table without the `EXTERNAL_LOOKUP` 
> capability fails with `TABLE`;
>   - misplaced or invalid hints fail with `INVALID_EXTERNAL_LOOKUP_HINT` 
> (`NOT_ON_LOOKUP_SIDE`, `MULTIPLE_HINTS`, `RELATION_NOT_FOUND`, 
> `DUPLICATE_RELATION`, `UNSUPPORTED_PARAMETER`);
>   - a hint attached to no join is dropped with a warning.
> - USING joins (`df.join(lookup.hint("external_lookup"), "id")`) run.
> Out of scope:
> - Lookup sides with projections or filters, including ones that optimization 
> would fold away (they fail with `LOOKUP_PLAN`).
> - Left semi/anti joins, `<=>` and range predicates.
> - Hint options, such as cache options.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to