hanyuzheng7 opened a new pull request, #29723:
URL: https://github.com/apache/flink/pull/29723
## What is the purpose of the change
With `table.optimizer.join-reorder-enabled = true`, a filter on the null
generating side of an outer join is applied a second time above that outer join
after `JoinToMultiJoinForReorderRule` merges the joins. Rows that should come
back with NULLs are removed. The rule runs before both reorder strategies
(bushy and LOPT), in batch and in streaming mode.
For example, this query loses the rows of T1 without a match in v, because
`COALESCE(b3, b2) > 3` is applied again above the outer LEFT JOIN:
```sql
SELECT * FROM T1 LEFT JOIN
(SELECT * FROM T2 LEFT JOIN T3 ON a2 = a3 WHERE COALESCE(b3, b2) > 3) v
ON a1 = a2;
```
## Brief change log
- `JoinToMultiJoinForReorderRule` pulled up the post-join filter of every
`MultiJoin` input, also of an input that it doesn't combine into the new
`MultiJoin`, e.g. the null generating input of an outer join. Such an input
stays an input of the new `MultiJoin` and keeps its post-join filter. The rule
now pulls up the post-join filter of an input only if it combines the input, as
it already does for the inputs and join filters.
## Verifying this change
- Added `testFilterOnNullGeneratingSideOfLeftOuterJoin` to
`JoinReorderTestBase`: in the batch and stream plans, the filter is applied
only below the outer join.
- Added `testJoinReorderWithFilterOnNullGeneratingSideOfLeftOuterJoin` to
`JoinReorderITCaseBase`, for batch and streaming. Without the fix, the row of
T1 without a match is missing.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no. Only the plans of
queries with this bug change.
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
