rionmonster opened a new pull request, #29565: URL: https://github.com/apache/flink/pull/29565
## What is the purpose of the change `LateralSnapshotJoinOperator` stored build rows whose join key has a NULL in a field that isn't null-safe. Such rows can never match. Unlike in the window and interval joins, nothing bounds this state except state TTL: a row stays until the build side retracts it, or until TTL removes it if TTL is configured. All such rows share one key, so they pile up in a single subtask, and every probe with a NULL key scans all of them. This is the follow-up to FLINK-41034 for this operator. ## Brief change log - `processElement2` doesn't buffer an INSERT or UPDATE_AFTER build change when `JoinConditionWithNullFilters#currentKeyCannotMatch` is true. The build side is never preserved, so the output doesn't change. The idle-flip timer is still armed first, because any build record counts as a build-side signal. - Retractions are still buffered and applied, so a row that an earlier version stored is removed. - `applyBuildChanges` doesn't count a retraction of such a row in `numUnmatchedBuildRetractions` when it finds nothing. Otherwise the metric would count every NULL-key delete. The metric's description in the docs (English and Chinese) says so. Non-NULL keys are counted as before. - The probe side doesn't change. A NULL-key probe now scans an empty build table for its key. ## Verifying this change This change added tests: - `LateralSnapshotJoinOperatorTest#joinRespectsNullKeysFilter` now also checks that a NULL-key build row isn't stored, for INNER and LEFT joins. With a null-safe key the row is still stored, which is the positive control. - `LateralSnapshotJoinOperatorTest#retractsNullKeyBuildRowStoredByEarlierVersion` restores state with a stored NULL-key row, on the heap and RocksDB backends. It checks that the row's retraction still removes it, and that a retraction of a row that was never stored isn't counted in the metric. The new checks fail if NULL-key rows are stored, if their retractions are dropped too, or if those retractions are counted. The existing LATERAL SNAPSHOT tests pass: the runtime join tests, and the planner plan, semantic, restore and ITCase tests. No new restore test is needed, because the state layout doesn't change. NULL-key rows stored by an earlier version stay until they are retracted or TTL removes them, as before. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): yes. It adds one null check per build-side accumulation, and in return it skips buffering and storing rows that can never match. - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable. The description of `numUnmatchedBuildRetractions` is updated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
