[
https://issues.apache.org/jira/browse/FLINK-41068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gustavo de Morais reassigned FLINK-41068:
-----------------------------------------
Assignee: Rion Williams
> LATERAL SNAPSHOT join stores build-side rows whose join key can never match
> ---------------------------------------------------------------------------
>
> Key: FLINK-41068
> URL: https://issues.apache.org/jira/browse/FLINK-41068
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Runtime
> Affects Versions: 2.3.0
> Reporter: Rion Williams
> Assignee: Rion Williams
> Priority: Major
> Labels: pull-request-available
>
> {{LateralSnapshotJoinOperator}} materializes the whole build side.
> {{processElement2}} buffers every build-side change in {{buildChangeBuffer}},
> and {{applyBuildChanges}} then adds it to {{buildTableState}}, a multiset
> that is only bounded by the state TTL. That includes rows with a NULL in a
> join key field that isn't null-safe. Such rows can never match: the NULL
> check only runs in {{JoinConditionWithNullFilters}} when a probe row is
> joined.
> The state is keyed by the join key, so all NULL-key build rows end up in the
> state of the same subtask and stay there for as long as the build-side table
> contains them.
> h3. Proposed change
> The build side is never preserved, so no output depends on build rows whose
> key can never match. {{processElement2}} can skip an INSERT or UPDATE_AFTER
> build change when {{JoinConditionWithNullFilters#currentKeyCannotMatch}}
> (from FLINK-41034) is true, before it is buffered.
> Retractions need care:
> * Rows stored by an earlier version must still be removed, so retractions of
> NULL-key rows should still be applied. Removing a row that was never stored
> is a no-op.
> * But {{applyBuildChanges}} counts a retraction of a row that isn't stored in
> {{numUnmatchedBuildRetractions}}. Retractions of rows whose key can never
> match should not be counted, or the metric would report every NULL-key delete.
> Optionally, probe rows whose key can never match don't need to be buffered
> during the LOAD phase for an inner join. For a left outer join, the
> null-padded row still has to be emitted in order with the other buffered
> probe rows of the same key.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
