[
https://issues.apache.org/jira/browse/SPARK-60031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18134011#comment-18134011
]
Jungtaek Lim commented on SPARK-60031:
--------------------------------------
Now the vote has passed. I'm going to file JIRA tickets and working on them.
> SPIP: External Lookup Join
> --------------------------
>
> Key: SPARK-60031
> URL: https://issues.apache.org/jira/browse/SPARK-60031
> Project: Spark
> Issue Type: Epic
> Components: SQL, Structured Streaming
> Affects Versions: 4.4.0
> Reporter: Jungtaek Lim
> Assignee: Jungtaek Lim
> Priority: Major
> Labels: SPIP
>
> We want to make stream data to be joined with external storages which are
> (more) efficient to the access pattern of “lookup” based on the key
> (retrieving specific data on demand) instead of loading the data in prior
> before joining.
> For example, suppose we want to join the employee data with department data,
> to show the department name they belong to. (This is called “enrichment”.)
> Instead of loading the whole (or pruned) department data from the external
> storage at once, we read the employee data, and retrieve the department data
> by associated data (e.g. department ID) the employee belongs to, and join the
> department name with employee data.
> This will enable us to achieve low latency on join operation since we do not
> wait for loading the data to process but process the stream side immediately
> with querying the static side on demand. This also helps to achieve better
> throughput for the case where Spark cannot derive the efficient filter to
> prune the static side outstandingly, since the new join would only require
> the data “as needed” rather than scanning partial/whole data.
>
> SPIP doc:
> [https://docs.google.com/document/d/1fBPM5cku8UJXacG2fw9df1zqALJ-WFm8E9-b9oiFP5M/edit?tab=t.0]
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
