Hi devs, I'm coming up with a proposal for a new join strategy, named "External Lookup Join".
tl;dr: All existing join strategies require the static side (DataFrame/Relation) to be materialized before the join operator. This doesn't allow us to achieve low latency join where the storage of the static side is expected to produce low latency response over on-demand request. (e.g. OLTP optimized like RDBMS, in-memory storage, NoSQL, etc.) The new join strategy will issue the query to the external storage "on demand", when processing the data on the left side. The most benefit will come with real-time mode, but it is not limited to real-time mode. This join strategy would be useful as long as the external storage of the static side is better suited for on-demand requests (rather than bulk load with pruning). Both static-static and stream-static can be benefited by this, though static-static join needs to be careful on doing this. More details are included in the SPIP doc. Issue: https://issues.apache.org/jira/browse/SPARK-60031 SPIP doc: https://docs.google.com/document/d/1fBPM5cku8UJXacG2fw9df1zqALJ-WFm8E9-b9oiFP5M/edit?tab=t.0 Looking forward to hearing from you. Thanks! Jungtaek Lim (HeartSaVioR)
