[
https://issues.apache.org/jira/browse/FLINK-10474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16638101#comment-16638101
]
ASF GitHub Bot commented on FLINK-10474:
----------------------------------------
hequn8128 opened a new pull request #6792: [FLINK-10474][table] Don't translate
IN/NOT_IN to JOIN with VALUES
URL: https://github.com/apache/flink/pull/6792
## What is the purpose of the change
IN clauses are translated to JOIN with VALUES if the number of elements in
the IN clause exceeds a certain threshold. This should not be done, because a
streaming join is very heavy and materializes both inputs (which is fine for
the VALUES) input but not for the other.
This pull request force usage of a cascade of predicates in all cases, both
for streaming and batch. Also a rule has been added to convert these predicates
to IN and NOT_IN.
Currently Flink code has already use HashSet when generate code for
IN/NOT_IN.
## Brief change log
- Add a configuration for SqlToRelConverter to force usage of a cascade of
predicates in all cases, both for streaming and batch.
- Add some rules to convert predicates to IN and NOT_IN.
- Add test cases.
## Verifying this change
This change added tests and can be verified as follows:
- Add integration tests for IN/NOT_IN in CalcITCase.
- Add unit tests to test plan in CalcTest.
- Added test that validates replace SqlToRelConverter Config
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Don't translate IN to JOIN with VALUES for streaming queries
> ------------------------------------------------------------
>
> Key: FLINK-10474
> URL: https://issues.apache.org/jira/browse/FLINK-10474
> Project: Flink
> Issue Type: Improvement
> Components: Table API & SQL
> Affects Versions: 1.6.1, 1.7.0
> Reporter: Fabian Hueske
> Assignee: Hequn Cheng
> Priority: Major
> Labels: pull-request-available
>
> IN clauses are translated to JOIN with VALUES if the number of elements in
> the IN clause exceeds a certain threshold. This should not be done, because a
> streaming join is very heavy and materializes both inputs (which is fine for
> the VALUES) input but not for the other.
> There are two ways to solve this:
> # don't translate IN to a JOIN at all
> # translate it to a JOIN but have a special join strategy if one input is
> bound and final (non-updating)
> Option 1. should be easy to do, option 2. requires much more effort.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
