[jira] [Commented] (FLINK-35687) JSON_QUERY should return a proper JSON for ARRAY
[
https://issues.apache.org/jira/browse/FLINK-35687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17859674#comment-17859674
]
david radley commented on FLINK-35687:
--
[~dwysakowicz] Thanks for clarifying.
* are you saying that this is not an issue before the RETURNING keyword. I had
thought that the default would be a return a string. Does this string have the
_=_ s in , the response in the example looks like a String (i.e. what would
have been returned before the RETURNING keyword was added) .
* the example given has ARRAY. Shouldn't this return an array of
Strings in this case and not a JSON Object?
* If this behaviour is new at 1.19 then I agree that correcting in 1.19.1 is
not an issue. If this has been around since 1.15, there could be more of a
migration impact.
WDYT?
> JSON_QUERY should return a proper JSON for ARRAY
>
>
> Key: FLINK-35687
> URL: https://issues.apache.org/jira/browse/FLINK-35687
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.19.1
>Reporter: Dawid Wysakowicz
>Assignee: Dawid Wysakowicz
>Priority: Major
> Fix For: 1.20.0
>
>
> {code}
> SELECT JSON_QUERY('{"items": [{"itemId":1234, "count":10}]}', '$.items'
> RETURNING ARRAY)
> {code}
> returns
> {code}
> {itemId=1234, count=10}
> {code}
> but it should:
> {code}
> {"itemId":1234, "count":10}
> {code}
> We should call jsonize for Collection types here:
> https://github.com/apache/flink/blob/f6f88135b3a5fa5616fe905346e5ab6dce084555/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/SqlJsonUtils.java#L268
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-35687) JSON_QUERY should return a proper JSON for ARRAY
[
https://issues.apache.org/jira/browse/FLINK-35687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17859664#comment-17859664
]
david radley edited comment on FLINK-35687 at 6/24/24 1:07 PM:
---
[~dwysakowicz] I agree your proposal does look better and matches the
documentation. I am concerned that there is a migration issue, as you have
targeted 1.20 and 1.19 and this is an SQL API change (even if it is a
correction). Existing applications since Flink 1.15 (when this came in) will
expect the existing response - so are likely to break and need changing. If we
want to target 1.19 and 1.20, I suggest having a config switch that would
enable this behaviour, or wait for Flink 2.0 - where more breaking changes are
appropriate.
was (Author: JIRAUSER300523):
[~dwysakowicz] I agree your proposal does look better. I am concerned that
there is a migration issue, as you have targeted 1.20 and 1.19 and this is an
SQL API change. Existing applications since Flink 1.15 (when this came in) will
expect the existing response - so are likely to break and need changing. If we
want to target 1.19 and 1.20, I suggest having a config switch that would
enable this behaviour, or wait for Flink 2.0 - where more breaking changes are
appropriate.
> JSON_QUERY should return a proper JSON for ARRAY
>
>
> Key: FLINK-35687
> URL: https://issues.apache.org/jira/browse/FLINK-35687
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.19.1
>Reporter: Dawid Wysakowicz
>Assignee: Dawid Wysakowicz
>Priority: Major
> Fix For: 1.20.0, 1.19.2
>
>
> {code}
> SELECT JSON_QUERY('{"items": [{"itemId":1234, "count":10}]}', '$.items'
> RETURNING ARRAY)
> {code}
> returns
> {code}
> {itemId=1234, count=10}
> {code}
> but it should:
> {code}
> {"itemId":1234, "count":10}
> {code}
> We should call jsonize for Collection types here:
> https://github.com/apache/flink/blob/f6f88135b3a5fa5616fe905346e5ab6dce084555/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/SqlJsonUtils.java#L268
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-35687) JSON_QUERY should return a proper JSON for ARRAY
[
https://issues.apache.org/jira/browse/FLINK-35687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17859664#comment-17859664
]
david radley commented on FLINK-35687:
--
[~dwysakowicz] I agree your proposal does look better. I am concerned that
there is a migration issue, as you have targeted 1.20 and 1.19 and this is an
SQL API change. Existing applications since Flink 1.15 (when this came in) will
expect the existing response - so are likely to break and need changing. If we
want to target 1.19 and 1.20, I suggest having a config switch that would
enable this behaviour, or wait for Flink 2.0 - where more breaking changes are
appropriate.
> JSON_QUERY should return a proper JSON for ARRAY
>
>
> Key: FLINK-35687
> URL: https://issues.apache.org/jira/browse/FLINK-35687
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.19.1
>Reporter: Dawid Wysakowicz
>Assignee: Dawid Wysakowicz
>Priority: Major
> Fix For: 1.20.0, 1.19.2
>
>
> {code}
> SELECT JSON_QUERY('{"items": [{"itemId":1234, "count":10}]}', '$.items'
> RETURNING ARRAY)
> {code}
> returns
> {code}
> {itemId=1234, count=10}
> {code}
> but it should:
> {code}
> {"itemId":1234, "count":10}
> {code}
> We should call jsonize for Collection types here:
> https://github.com/apache/flink/blob/f6f88135b3a5fa5616fe905346e5ab6dce084555/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/SqlJsonUtils.java#L268
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-20110) Support 'merge' method for first_value and last_value UDAF
[ https://issues.apache.org/jira/browse/FLINK-20110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17853139#comment-17853139 ] david radley commented on FLINK-20110: -- [~adrianalexvasiliu] I have had a look it looks like the PR was closed due to inactivity so it is not in master or any other branches. [@wangxlong|https://github.com/wangxlong] , [@leonardBang|https://github.com/leonardBang] Are you in a position to finish this PR? I can help if required. > Support 'merge' method for first_value and last_value UDAF > -- > > Key: FLINK-20110 > URL: https://issues.apache.org/jira/browse/FLINK-20110 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Runtime >Affects Versions: 1.12.0 >Reporter: hailong wang >Priority: Major > Labels: auto-unassigned, pull-request-available > Fix For: 1.20.0 > > > From the user-zh email, when use first_value function in hop window, It > throws the exception because first_vaue does not implement the merge method. > We can support 'merge' method for first_value and last_value UDAF -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35408) Add 30 min tolerance value when validating the time-zone setting
[
https://issues.apache.org/jira/browse/FLINK-35408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849944#comment-17849944
]
david radley commented on FLINK-35408:
--
[~leonard] [~ShawnHx] thanks
> Add 30 min tolerance value when validating the time-zone setting
>
>
> Key: FLINK-35408
> URL: https://issues.apache.org/jira/browse/FLINK-35408
> Project: Flink
> Issue Type: Improvement
> Components: Flink CDC
>Affects Versions: cdc-3.1.0
>Reporter: Xiao Huang
>Assignee: Xiao Huang
>Priority: Minor
> Labels: pull-request-available
> Fix For: cdc-3.2.0
>
>
> Now, MySQL CDC connector will retrieve the offset seconds between the
> configured timezone and UTC by executing the SQL statement below, and then
> compare it with the configured timezone.
> {code:java}
> SELECT TIME_TO_SEC(TIMEDIFF(NOW(), UTC_TIMESTAMP())) {code}
> For some MySQL instances, the validating for time-zone is too strict. We can
> add 30min tolerance value.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-35408) Add 30 min tolerance value when validating the time-zone setting
[ https://issues.apache.org/jira/browse/FLINK-35408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17848161#comment-17848161 ] david radley commented on FLINK-35408: -- some feedback: * I wonder why you need a 30 minute tolerance for mySQL time zone setting for CDC? * you have no unit tests in your pr. * I notice [Australia|https://en.wikipedia.org/wiki/Australia]: [Lord Howe Island|https://en.wikipedia.org/wiki/Lord_Howe_Island] has 30 minute DSL - will this work? * as this is an improvement - a Flip is the usual process - is there a reason you are not raising a Flip for this. > Add 30 min tolerance value when validating the time-zone setting > > > Key: FLINK-35408 > URL: https://issues.apache.org/jira/browse/FLINK-35408 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Reporter: Xiao Huang >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35311) FLIP-454: New Apicurio Avro format
david radley created FLINK-35311: Summary: FLIP-454: New Apicurio Avro format Key: FLINK-35311 URL: https://issues.apache.org/jira/browse/FLINK-35311 Project: Flink Issue Type: Improvement Components: Connectors / Kafka Affects Versions: 1.18.1, 1.19.0, 1.17.2 Reporter: david radley Fix For: 2.0.0, 1.20.0 This Jira is for the accepted [FLIP-454|https://cwiki.apache.org/confluence/display/FLINK/FLIP-454%3A+New+Apicurio+Avro+format]. It involves changes to 2 repositories, core Flink and the Flink Kafka connector -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31223) sql-client.sh fails to start with ssl enabled
[
https://issues.apache.org/jira/browse/FLINK-31223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841957#comment-17841957
]
david radley commented on FLINK-31223:
--
I have raised backport prs:
#24741 for 1.18
[#24742|https://github.com/apache/flink/pull/24742] for 1.19
> sql-client.sh fails to start with ssl enabled
> -
>
> Key: FLINK-31223
> URL: https://issues.apache.org/jira/browse/FLINK-31223
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.17.0
>Reporter: macdoor615
>Assignee: Weijie Guo
>Priority: Major
> Labels: pull-request-available, stale-assigned
> Fix For: 1.20.0
>
>
> *Version:* 1.17-SNAPSHOT *Commit:* c66ef25
> 1. ssl disabled
> sql-client.sh works properly
> 2. ssl enabled
> web ui can access with [https://url|https://url/]
> The task can be submitted correctly through sql-gateway. I can confirm that
> sql-gateway exposes the http protocol, not https.
> But sql-client.sh fails to start with the following exceptions. It seems that
> sql-client.sh expect https protocol
>
> {code:java}
> 2023-02-25 14:43:19,317 INFO org.apache.flink.configuration.Configuration
> [] - Config uses fallback configuration key 'rest.port' instead
> of key 'rest.bind-port'
> 2023-02-25 14:43:19,343 INFO
> org.apache.flink.table.gateway.rest.SqlGatewayRestEndpoint [] - Starting
> rest endpoint.
> 2023-02-25 14:43:19,713 INFO
> org.apache.flink.table.gateway.rest.SqlGatewayRestEndpoint [] - Rest
> endpoint listening at localhost:44922
> 2023-02-25 14:43:19,715 INFO org.apache.flink.table.client.SqlClient
> [] - Start embedded gateway on port 44922
> 2023-02-25 14:43:20,040 INFO
> org.apache.flink.table.gateway.rest.SqlGatewayRestEndpoint [] - Shutting
> down rest endpoint.
> 2023-02-25 14:43:20,088 INFO
> org.apache.flink.table.gateway.rest.SqlGatewayRestEndpoint [] - Shut down
> complete.
> 2023-02-25 14:43:20,089 ERROR org.apache.flink.table.client.SqlClient
> [] - SQL Client must stop.
> org.apache.flink.table.client.SqlClientException: Failed to create the
> executor.
> at
> org.apache.flink.table.client.gateway.ExecutorImpl.(ExecutorImpl.java:170)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.client.gateway.ExecutorImpl.(ExecutorImpl.java:113)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.client.gateway.Executor.create(Executor.java:34)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at org.apache.flink.table.client.SqlClient.start(SqlClient.java:110)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.client.SqlClient.startClient(SqlClient.java:228)
> [flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at org.apache.flink.table.client.SqlClient.main(SqlClient.java:179)
> [flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> Caused by: org.apache.flink.table.client.gateway.SqlExecutionException:
> Failed to get response.
> at
> org.apache.flink.table.client.gateway.ExecutorImpl.getResponse(ExecutorImpl.java:427)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.client.gateway.ExecutorImpl.getResponse(ExecutorImpl.java:416)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.client.gateway.ExecutorImpl.negotiateVersion(ExecutorImpl.java:447)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.client.gateway.ExecutorImpl.(ExecutorImpl.java:132)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> ... 5 more
> Caused by:
> org.apache.flink.shaded.netty4.io.netty.handler.codec.DecoderException:
> org.apache.flink.shaded.netty4.io.netty.handler.ssl.NotSslRecordException:
> not an SSL/TLS record:
> 485454502f312e3120343034204e6f7420466f756e640d0a636f6e74656e742d747970653a206170706c69636174696f6e2f6a736f6e3b20636861727365743d5554462d380d0a6163636573732d636f6e74726f6c2d616c6c6f772d6f726967696e3a202a0d0a636f6e74656e742d6c656e6774683a2033380d0a0d0a7b226572726f7273223a5b224e6f7420666f756e643a202f6261642d72657175657374225d7d
> at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:489)
> ~[flink-dist-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:280)
> ~[flink-dist-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
>
[jira] (FLINK-31223) sql-client.sh fails to start with ssl enabled
[ https://issues.apache.org/jira/browse/FLINK-31223 ]
david radley deleted comment on FLINK-31223:
--
was (Author: JIRAUSER300523):
[~Weijie Guo] I have created pr [https://github.com/apache/flink/pull/24729]
for the 1.18 backport
> sql-client.sh fails to start with ssl enabled
> -
>
> Key: FLINK-31223
> URL: https://issues.apache.org/jira/browse/FLINK-31223
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.17.0
>Reporter: macdoor615
>Assignee: Weijie Guo
>Priority: Major
> Labels: pull-request-available, stale-assigned
> Fix For: 1.20.0
>
>
> *Version:* 1.17-SNAPSHOT *Commit:* c66ef25
> 1. ssl disabled
> sql-client.sh works properly
> 2. ssl enabled
> web ui can access with [https://url|https://url/]
> The task can be submitted correctly through sql-gateway. I can confirm that
> sql-gateway exposes the http protocol, not https.
> But sql-client.sh fails to start with the following exceptions. It seems that
> sql-client.sh expect https protocol
>
> {code:java}
> 2023-02-25 14:43:19,317 INFO org.apache.flink.configuration.Configuration
> [] - Config uses fallback configuration key 'rest.port' instead
> of key 'rest.bind-port'
> 2023-02-25 14:43:19,343 INFO
> org.apache.flink.table.gateway.rest.SqlGatewayRestEndpoint [] - Starting
> rest endpoint.
> 2023-02-25 14:43:19,713 INFO
> org.apache.flink.table.gateway.rest.SqlGatewayRestEndpoint [] - Rest
> endpoint listening at localhost:44922
> 2023-02-25 14:43:19,715 INFO org.apache.flink.table.client.SqlClient
> [] - Start embedded gateway on port 44922
> 2023-02-25 14:43:20,040 INFO
> org.apache.flink.table.gateway.rest.SqlGatewayRestEndpoint [] - Shutting
> down rest endpoint.
> 2023-02-25 14:43:20,088 INFO
> org.apache.flink.table.gateway.rest.SqlGatewayRestEndpoint [] - Shut down
> complete.
> 2023-02-25 14:43:20,089 ERROR org.apache.flink.table.client.SqlClient
> [] - SQL Client must stop.
> org.apache.flink.table.client.SqlClientException: Failed to create the
> executor.
> at
> org.apache.flink.table.client.gateway.ExecutorImpl.(ExecutorImpl.java:170)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.client.gateway.ExecutorImpl.(ExecutorImpl.java:113)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.client.gateway.Executor.create(Executor.java:34)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at org.apache.flink.table.client.SqlClient.start(SqlClient.java:110)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.client.SqlClient.startClient(SqlClient.java:228)
> [flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at org.apache.flink.table.client.SqlClient.main(SqlClient.java:179)
> [flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> Caused by: org.apache.flink.table.client.gateway.SqlExecutionException:
> Failed to get response.
> at
> org.apache.flink.table.client.gateway.ExecutorImpl.getResponse(ExecutorImpl.java:427)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.client.gateway.ExecutorImpl.getResponse(ExecutorImpl.java:416)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.client.gateway.ExecutorImpl.negotiateVersion(ExecutorImpl.java:447)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.client.gateway.ExecutorImpl.(ExecutorImpl.java:132)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> ... 5 more
> Caused by:
> org.apache.flink.shaded.netty4.io.netty.handler.codec.DecoderException:
> org.apache.flink.shaded.netty4.io.netty.handler.ssl.NotSslRecordException:
> not an SSL/TLS record:
> 485454502f312e3120343034204e6f7420466f756e640d0a636f6e74656e742d747970653a206170706c69636174696f6e2f6a736f6e3b20636861727365743d5554462d380d0a6163636573732d636f6e74726f6c2d616c6c6f772d6f726967696e3a202a0d0a636f6e74656e742d6c656e6774683a2033380d0a0d0a7b226572726f7273223a5b224e6f7420666f756e643a202f6261642d72657175657374225d7d
> at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:489)
> ~[flink-dist-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:280)
> ~[flink-dist-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> ~[flink-dist-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
>
[jira] [Comment Edited] (FLINK-31223) sql-client.sh fails to start with ssl enabled
[
https://issues.apache.org/jira/browse/FLINK-31223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841279#comment-17841279
]
david radley edited comment on FLINK-31223 at 4/26/24 3:47 PM:
---
[~Weijie Guo] I have created pr [https://github.com/apache/flink/pull/24729]
for the 1.18 backport
was (Author: JIRAUSER300523):
[~Weijie Guo] I have created pr [https://github.com/apache/flink/pull/24729]
for the 118 backport
> sql-client.sh fails to start with ssl enabled
> -
>
> Key: FLINK-31223
> URL: https://issues.apache.org/jira/browse/FLINK-31223
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.17.0
>Reporter: macdoor615
>Assignee: Weijie Guo
>Priority: Major
> Labels: pull-request-available, stale-assigned
> Fix For: 1.20.0
>
>
> *Version:* 1.17-SNAPSHOT *Commit:* c66ef25
> 1. ssl disabled
> sql-client.sh works properly
> 2. ssl enabled
> web ui can access with [https://url|https://url/]
> The task can be submitted correctly through sql-gateway. I can confirm that
> sql-gateway exposes the http protocol, not https.
> But sql-client.sh fails to start with the following exceptions. It seems that
> sql-client.sh expect https protocol
>
> {code:java}
> 2023-02-25 14:43:19,317 INFO org.apache.flink.configuration.Configuration
> [] - Config uses fallback configuration key 'rest.port' instead
> of key 'rest.bind-port'
> 2023-02-25 14:43:19,343 INFO
> org.apache.flink.table.gateway.rest.SqlGatewayRestEndpoint [] - Starting
> rest endpoint.
> 2023-02-25 14:43:19,713 INFO
> org.apache.flink.table.gateway.rest.SqlGatewayRestEndpoint [] - Rest
> endpoint listening at localhost:44922
> 2023-02-25 14:43:19,715 INFO org.apache.flink.table.client.SqlClient
> [] - Start embedded gateway on port 44922
> 2023-02-25 14:43:20,040 INFO
> org.apache.flink.table.gateway.rest.SqlGatewayRestEndpoint [] - Shutting
> down rest endpoint.
> 2023-02-25 14:43:20,088 INFO
> org.apache.flink.table.gateway.rest.SqlGatewayRestEndpoint [] - Shut down
> complete.
> 2023-02-25 14:43:20,089 ERROR org.apache.flink.table.client.SqlClient
> [] - SQL Client must stop.
> org.apache.flink.table.client.SqlClientException: Failed to create the
> executor.
> at
> org.apache.flink.table.client.gateway.ExecutorImpl.(ExecutorImpl.java:170)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.client.gateway.ExecutorImpl.(ExecutorImpl.java:113)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.client.gateway.Executor.create(Executor.java:34)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at org.apache.flink.table.client.SqlClient.start(SqlClient.java:110)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.client.SqlClient.startClient(SqlClient.java:228)
> [flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at org.apache.flink.table.client.SqlClient.main(SqlClient.java:179)
> [flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> Caused by: org.apache.flink.table.client.gateway.SqlExecutionException:
> Failed to get response.
> at
> org.apache.flink.table.client.gateway.ExecutorImpl.getResponse(ExecutorImpl.java:427)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.client.gateway.ExecutorImpl.getResponse(ExecutorImpl.java:416)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.client.gateway.ExecutorImpl.negotiateVersion(ExecutorImpl.java:447)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.client.gateway.ExecutorImpl.(ExecutorImpl.java:132)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> ... 5 more
> Caused by:
> org.apache.flink.shaded.netty4.io.netty.handler.codec.DecoderException:
> org.apache.flink.shaded.netty4.io.netty.handler.ssl.NotSslRecordException:
> not an SSL/TLS record:
> 485454502f312e3120343034204e6f7420466f756e640d0a636f6e74656e742d747970653a206170706c69636174696f6e2f6a736f6e3b20636861727365743d5554462d380d0a6163636573732d636f6e74726f6c2d616c6c6f772d6f726967696e3a202a0d0a636f6e74656e742d6c656e6774683a2033380d0a0d0a7b226572726f7273223a5b224e6f7420666f756e643a202f6261642d72657175657374225d7d
> at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:489)
> ~[flink-dist-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:280)
>
[jira] [Commented] (FLINK-31223) sql-client.sh fails to start with ssl enabled
[
https://issues.apache.org/jira/browse/FLINK-31223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841279#comment-17841279
]
david radley commented on FLINK-31223:
--
[~Weijie Guo] I have created pr [https://github.com/apache/flink/pull/24729]
for the 118 backport
> sql-client.sh fails to start with ssl enabled
> -
>
> Key: FLINK-31223
> URL: https://issues.apache.org/jira/browse/FLINK-31223
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.17.0
>Reporter: macdoor615
>Assignee: Weijie Guo
>Priority: Major
> Labels: pull-request-available, stale-assigned
> Fix For: 1.20.0
>
>
> *Version:* 1.17-SNAPSHOT *Commit:* c66ef25
> 1. ssl disabled
> sql-client.sh works properly
> 2. ssl enabled
> web ui can access with [https://url|https://url/]
> The task can be submitted correctly through sql-gateway. I can confirm that
> sql-gateway exposes the http protocol, not https.
> But sql-client.sh fails to start with the following exceptions. It seems that
> sql-client.sh expect https protocol
>
> {code:java}
> 2023-02-25 14:43:19,317 INFO org.apache.flink.configuration.Configuration
> [] - Config uses fallback configuration key 'rest.port' instead
> of key 'rest.bind-port'
> 2023-02-25 14:43:19,343 INFO
> org.apache.flink.table.gateway.rest.SqlGatewayRestEndpoint [] - Starting
> rest endpoint.
> 2023-02-25 14:43:19,713 INFO
> org.apache.flink.table.gateway.rest.SqlGatewayRestEndpoint [] - Rest
> endpoint listening at localhost:44922
> 2023-02-25 14:43:19,715 INFO org.apache.flink.table.client.SqlClient
> [] - Start embedded gateway on port 44922
> 2023-02-25 14:43:20,040 INFO
> org.apache.flink.table.gateway.rest.SqlGatewayRestEndpoint [] - Shutting
> down rest endpoint.
> 2023-02-25 14:43:20,088 INFO
> org.apache.flink.table.gateway.rest.SqlGatewayRestEndpoint [] - Shut down
> complete.
> 2023-02-25 14:43:20,089 ERROR org.apache.flink.table.client.SqlClient
> [] - SQL Client must stop.
> org.apache.flink.table.client.SqlClientException: Failed to create the
> executor.
> at
> org.apache.flink.table.client.gateway.ExecutorImpl.(ExecutorImpl.java:170)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.client.gateway.ExecutorImpl.(ExecutorImpl.java:113)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.client.gateway.Executor.create(Executor.java:34)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at org.apache.flink.table.client.SqlClient.start(SqlClient.java:110)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.client.SqlClient.startClient(SqlClient.java:228)
> [flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at org.apache.flink.table.client.SqlClient.main(SqlClient.java:179)
> [flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> Caused by: org.apache.flink.table.client.gateway.SqlExecutionException:
> Failed to get response.
> at
> org.apache.flink.table.client.gateway.ExecutorImpl.getResponse(ExecutorImpl.java:427)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.client.gateway.ExecutorImpl.getResponse(ExecutorImpl.java:416)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.client.gateway.ExecutorImpl.negotiateVersion(ExecutorImpl.java:447)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.client.gateway.ExecutorImpl.(ExecutorImpl.java:132)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> ... 5 more
> Caused by:
> org.apache.flink.shaded.netty4.io.netty.handler.codec.DecoderException:
> org.apache.flink.shaded.netty4.io.netty.handler.ssl.NotSslRecordException:
> not an SSL/TLS record:
> 485454502f312e3120343034204e6f7420466f756e640d0a636f6e74656e742d747970653a206170706c69636174696f6e2f6a736f6e3b20636861727365743d5554462d380d0a6163636573732d636f6e74726f6c2d616c6c6f772d6f726967696e3a202a0d0a636f6e74656e742d6c656e6774683a2033380d0a0d0a7b226572726f7273223a5b224e6f7420666f756e643a202f6261642d72657175657374225d7d
> at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:489)
> ~[flink-dist-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:280)
> ~[flink-dist-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
>
[jira] [Commented] (FLINK-31223) sql-client.sh fails to start with ssl enabled
[
https://issues.apache.org/jira/browse/FLINK-31223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840422#comment-17840422
]
david radley commented on FLINK-31223:
--
[~Weijie Guo] we urgently need this fixed. What is you current outlook on this?
Many thanks. We are looking at testing your pr locally to confirm it resolves
the problem for us.
> sql-client.sh fails to start with ssl enabled
> -
>
> Key: FLINK-31223
> URL: https://issues.apache.org/jira/browse/FLINK-31223
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.17.0
>Reporter: macdoor615
>Assignee: Weijie Guo
>Priority: Major
> Labels: pull-request-available, stale-assigned
>
> *Version:* 1.17-SNAPSHOT *Commit:* c66ef25
> 1. ssl disabled
> sql-client.sh works properly
> 2. ssl enabled
> web ui can access with [https://url|https://url/]
> The task can be submitted correctly through sql-gateway. I can confirm that
> sql-gateway exposes the http protocol, not https.
> But sql-client.sh fails to start with the following exceptions. It seems that
> sql-client.sh expect https protocol
>
> {code:java}
> 2023-02-25 14:43:19,317 INFO org.apache.flink.configuration.Configuration
> [] - Config uses fallback configuration key 'rest.port' instead
> of key 'rest.bind-port'
> 2023-02-25 14:43:19,343 INFO
> org.apache.flink.table.gateway.rest.SqlGatewayRestEndpoint [] - Starting
> rest endpoint.
> 2023-02-25 14:43:19,713 INFO
> org.apache.flink.table.gateway.rest.SqlGatewayRestEndpoint [] - Rest
> endpoint listening at localhost:44922
> 2023-02-25 14:43:19,715 INFO org.apache.flink.table.client.SqlClient
> [] - Start embedded gateway on port 44922
> 2023-02-25 14:43:20,040 INFO
> org.apache.flink.table.gateway.rest.SqlGatewayRestEndpoint [] - Shutting
> down rest endpoint.
> 2023-02-25 14:43:20,088 INFO
> org.apache.flink.table.gateway.rest.SqlGatewayRestEndpoint [] - Shut down
> complete.
> 2023-02-25 14:43:20,089 ERROR org.apache.flink.table.client.SqlClient
> [] - SQL Client must stop.
> org.apache.flink.table.client.SqlClientException: Failed to create the
> executor.
> at
> org.apache.flink.table.client.gateway.ExecutorImpl.(ExecutorImpl.java:170)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.client.gateway.ExecutorImpl.(ExecutorImpl.java:113)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.client.gateway.Executor.create(Executor.java:34)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at org.apache.flink.table.client.SqlClient.start(SqlClient.java:110)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.client.SqlClient.startClient(SqlClient.java:228)
> [flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at org.apache.flink.table.client.SqlClient.main(SqlClient.java:179)
> [flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> Caused by: org.apache.flink.table.client.gateway.SqlExecutionException:
> Failed to get response.
> at
> org.apache.flink.table.client.gateway.ExecutorImpl.getResponse(ExecutorImpl.java:427)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.client.gateway.ExecutorImpl.getResponse(ExecutorImpl.java:416)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.client.gateway.ExecutorImpl.negotiateVersion(ExecutorImpl.java:447)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.client.gateway.ExecutorImpl.(ExecutorImpl.java:132)
> ~[flink-sql-client-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> ... 5 more
> Caused by:
> org.apache.flink.shaded.netty4.io.netty.handler.codec.DecoderException:
> org.apache.flink.shaded.netty4.io.netty.handler.ssl.NotSslRecordException:
> not an SSL/TLS record:
> 485454502f312e3120343034204e6f7420466f756e640d0a636f6e74656e742d747970653a206170706c69636174696f6e2f6a736f6e3b20636861727365743d5554462d380d0a6163636573732d636f6e74726f6c2d616c6c6f772d6f726967696e3a202a0d0a636f6e74656e742d6c656e6774683a2033380d0a0d0a7b226572726f7273223a5b224e6f7420666f756e643a202f6261642d72657175657374225d7d
> at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:489)
> ~[flink-dist-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:280)
> ~[flink-dist-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
>
[jira] [Commented] (FLINK-32497) IF FUNCTION is FALSE and the false_value parameter is a function, then an exception will be thrown
[
https://issues.apache.org/jira/browse/FLINK-32497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17823172#comment-17823172
]
david radley commented on FLINK-32497:
--
I notice this is a critical issue. But from reading [~Sergey Nuyanzin]'s
comments, it looks like this issue is a duplicate of FLINK-30966 which is fixed
in 1.16.3 and 1.17.2.
[~jarieshan] Could you confirm your problem is fixed and close out this issue
please.
> IF FUNCTION is FALSE and the false_value parameter is a function, then an
> exception will be thrown
> --
>
> Key: FLINK-32497
> URL: https://issues.apache.org/jira/browse/FLINK-32497
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API, Table SQL / Client
>Affects Versions: 1.17.1
> Environment: {color:#172b4d}- Flink Version -{color}
> {color:#172b4d}V{color}{color:#172b4d}ersion: 1.17.1, Commit ID:
> 2750d5c{color}
>
> {color:#172b4d}- Java Version -{color}
> {color:#172b4d}java version "1.8.0_202"{color}
> {color:#172b4d}Java(TM) SE Runtime Environment (build 1.8.0_202-b08){color}
> {color:#172b4d}Java HotSpot(TM) 64-Bit Server VM (build 25.202-b08, mixed
> mode){color}
>
>Reporter: Han Zhuo
>Priority: Critical
> Attachments: flink-1.17.1_logs_jarieshan_20230630.tgz,
> image-2023-06-30-15-02-13-197.png, image-2023-06-30-15-02-26-099.png,
> image-2023-06-30-15-02-57-082.png, image-2023-06-30-15-07-08-588.png,
> image-2023-06-30-15-09-44-623.png, image-2023-06-30-15-10-08-619.png,
> image-2023-06-30-15-13-56-625.png, image-2023-06-30-15-14-21-038.png
>
>
> It is successful to execute certain functions individually.
> {code:java}
> SELECT SPLIT_INDEX('TEST:ABC', ':', 0); {code}
> !image-2023-06-30-15-02-57-082.png|width=189,height=36!
>
> And it is also successful for these functions to be located in the true_value
> parameter of the {color:#172b4d}+IF function+{color}.
> {code:java}
> SELECT IF(2>1, SPLIT_INDEX('TEST:ABC', ':', 1), 'FALSE'); {code}
> !image-2023-06-30-15-02-13-197.png|width=185,height=36!
>
> Only when these functions are located in the false_value parameter of the
> {+}IF function{+}, an exception will be thrown.
> func1.
> {code:java}
> SELECT IF(2>1, 'TRUE', SPLIT_INDEX('TEST:ABC', ':', 0)); {code}
> {color:#172b4d}!image-2023-06-30-15-09-44-623.png|width=385,height=42!{color}
> func2.
> {code:java}
> SELECT IF(2>1, 'TRUE', LOWER('TEST')); {code}
> !image-2023-06-30-15-14-21-038.png|width=337,height=246!
>
> {color:#172b4d}And it is also successful for{color} +CASE function+
> {code:java}
> SELECT CASE WHEN 2=1 THEN 'TRUE' ELSE SPLIT_INDEX('TEST:ABC', ':', 0) END;
> {code}
> {color:#172b4d}!image-2023-06-30-15-10-08-619.png|width=188,height=41!{color}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-34211) Filtering on Column names with ?s fails for JDBC lookup join.
[ https://issues.apache.org/jira/browse/FLINK-34211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17821216#comment-17821216 ] david radley commented on FLINK-34211: -- Hi [~xuyangzhong] , I have changed this to an improvement. Are you ok to assign this to me and merge the pr (with unit tests)? As the fix seems reasonably obvious; or do you think there are other considerations around this that warrant deeper discussion? As part of this I can add a tests for ? for scan queries. > Filtering on Column names with ?s fails for JDBC lookup join. > -- > > Key: FLINK-34211 > URL: https://issues.apache.org/jira/browse/FLINK-34211 > Project: Flink > Issue Type: Improvement > Components: Connectors / JDBC, Table SQL / JDBC >Reporter: david radley >Priority: Minor > > There is a check for ? character in > [https://github.com/apache/flink-connector-jdbc/blob/e3dd84160cd665ae17672da8b6e742e61a72a32d/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/statement/FieldNamedPreparedStatementImpl.java#L186 > |FieldNamedPreparedStatementImpl.java] > Removing this check allows column names containing _?_ -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34211) Filtering on Column names with ?s fails for JDBC lookup join.
[ https://issues.apache.org/jira/browse/FLINK-34211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] david radley updated FLINK-34211: - Issue Type: Improvement (was: Bug) > Filtering on Column names with ?s fails for JDBC lookup join. > -- > > Key: FLINK-34211 > URL: https://issues.apache.org/jira/browse/FLINK-34211 > Project: Flink > Issue Type: Improvement > Components: Connectors / JDBC, Table SQL / JDBC >Reporter: david radley >Priority: Minor > > There is a check for ? character in > [https://github.com/apache/flink-connector-jdbc/blob/e3dd84160cd665ae17672da8b6e742e61a72a32d/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/statement/FieldNamedPreparedStatementImpl.java#L186 > |FieldNamedPreparedStatementImpl.java] > Removing this check allows column names containing _?_ -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-21672) End to end tests (streaming) aren't Java vendor neutral (sun.management bean used)
[ https://issues.apache.org/jira/browse/FLINK-21672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17818854#comment-17818854 ] david radley commented on FLINK-21672: -- It seems that for Sun usages 2 and 3. We can use _--add-modules=jdk.unsupported ._ > End to end tests (streaming) aren't Java vendor neutral (sun.management bean > used) > -- > > Key: FLINK-21672 > URL: https://issues.apache.org/jira/browse/FLINK-21672 > Project: Flink > Issue Type: Improvement > Components: Tests >Reporter: Adam Roberts >Assignee: david radley >Priority: Minor > > Hi everyone, have been looking to run the tests for Flink using an 11 > AdoptOpenJDK 11 distribution (so the latest for Linux, x86-64 specifically) > and I see > > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile > (default-compile) on project flink-local-recovery-and-allocation-test: > Compilation failure: Compilation failure: > [ERROR] > /var/home/core/flink/flink-end-to-end-tests/flink-local-recovery-and-allocation-test/src/main/java/org/apache/flink/streaming/tests/StickyAllocationAndLocalRecoveryTestJob.java:[416,23] > cannot find symbol > [ERROR] symbol: class VMManagement > [ERROR] location: package sun.management > [ERROR] > /var/home/core/flink/flink-end-to-end-tests/flink-local-recovery-and-allocation-test/src/main/java/org/apache/flink/streaming/tests/StickyAllocationAndLocalRecoveryTestJob.java:[416,59] > cannot find symbol > [ERROR] symbol: class VMManagement > [ERROR] location: package sun.management > [ERROR] -> [Help 1] > > my guess is that AdoptOpenJDK's class-library simply doesn't have this > package and we should use a more neutral one if that's available - I went > with > [https://adoptopenjdk.net/releases.html?variant=openjdk11=openj9] > personally (OpenJ9 being the name for IBM's open-sourced J9 JVM), but I > wonder if that has its own platform specific bean as well; I haven't worked > on IBM's distribution of Java for almost seven years* but hopefully someone > may have more insight so you don't need to be using OpenJDK backed by HotSpot > to run said tests. It would be helpful if we didn't need an (if vendor == IBM > do this || vendor == Oracle do this ... etc) statement, so you can run the > tests no matter where you're getting Java from. > > *full disclaimer, I work at IBM and helped create AdoptOpenJDK, and used to > work in its Java team...I've honestly forgot if we have a vendor-neutral bean > available and now work on something totally different! > Cheers! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-21672) End to end tests (streaming) aren't Java vendor neutral (sun.management bean used)
[
https://issues.apache.org/jira/browse/FLINK-21672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17818182#comment-17818182
]
david radley edited comment on FLINK-21672 at 2/19/24 9:55 AM:
---
Hi [~martijnvisser]
Here is a list of sun. classes I have found in core Flink.
# The one that I reported in StickyAllocationAndLocalRecoveryTestJob. This
test code uses the sun management class to get the current pid. We have some
options
## we change the constructor so it is driven reflectively so there is no
compile error. This change would allow the tst to compile.
## To have this test run in Semeru we could use logic like Datadog and try to
reflectively load other classes. Something like
[https://github.com/DataDog/dd-trace-java/blob/aee3ca59c6a05233f4295552f2ede80bc4fc[…]agent/src/main/java/datadog/trace/bootstrap/AgentBootstrap.java|https://github.com/DataDog/dd-trace-java/blob/aee3ca59c6a05233f4295552f2ede80bc4fc%5B%E2%80%A6%5Dagent/src/main/java/datadog/trace/bootstrap/AgentBootstrap.java]
. I see we could reflectively drive
com.ibm.lang.management.RuntimeMXBean.getProcessID() if the sun class is not
there.
## We fix this properly at Flink v2, using
### methods introduced at java 10:{_}java.lang.management.RuntimeMXBean
runtime={_}{_}java.lang.management.ManagementFactory.getRuntimeMXBean();{_}
_runtime.getPid();_
_2. Or use a java 9 approach_ _long pid =
ProcessHandle.current().pid();_
2. core Flink MemorySegment.java uses sun.misc.Unsafe. I see
[chrome-extension://efaidnbmnnnibpcajpcglclefindmkaj/https://cr.openjdk.org/~psandoz/dv14-uk-paul-sandoz-unsafe-the-situation.pdf]
I wonder if we can remove unsafe at Flink v2; I am not sure how used _off-heap
unsafe_ memory is (also Flink v2 is changing how memory is being handled). I am
not seeing an alternative. It has
@SuppressWarnings("restriction") so we do not see failures around this.
3. I see _SignalHandler and_ TestSignalHandler uses sun.misc.Signal . This
seems to have come from Hadoop implementations that have been inherited.
4. There are 2 imports of import sun.security.krb5.KrbException; that can
be produced when calling sun.security.krb5.Config.refresh()
I would like to implement 1.2 or if this is not acceptable 1.1. This would
really help us short term as we could at least build with skipTests on Semeru.
Sun usages 2 and 3 would need so some consensus in the community - as is seems
we would be removing capability unless we can find an alternative. The
sun.security references are used when testing Hadoop with Kerberos, I have not
looked into it.
was (Author: JIRAUSER300523):
Hi [~martijnvisser]
Here is a list of sun. classes I have found in core Flink.
# The one that I reported in StickyAllocationAndLocalRecoveryTestJob. This
test code uses the sun management class to get the current pid. We have some
options
## we change the constructor so it is driven reflectively so there is no
compile error. This change would allow the tst to compile.
## To have this test run in Semeru we could use logic like Datadog and try to
reflectively load other classes. Something like
[https://github.com/DataDog/dd-trace-java/blob/aee3ca59c6a05233f4295552f2ede80bc4fc[…]agent/src/main/java/datadog/trace/bootstrap/AgentBootstrap.java|https://github.com/DataDog/dd-trace-java/blob/aee3ca59c6a05233f4295552f2ede80bc4fc%5B%E2%80%A6%5Dagent/src/main/java/datadog/trace/bootstrap/AgentBootstrap.java]
. I see we could reflectively drive
com.ibm.lang.management.RuntimeMXBean.getProcessID() if the sun class is not
there.
## We fix this properly at Flink v2, using
### methods introduced at java 10:{_}java.lang.management.RuntimeMXBean
runtime={_}{_}java.lang.management.ManagementFactory.getRuntimeMXBean();{_}
_runtime.getPid();_
_2. Or use a java 9 approach_ _long pid =
ProcessHandle.current().pid();_
2. core Flink MemorySegment.java uses sun.misc.Unsafe. I see
[chrome-extension://efaidnbmnnnibpcajpcglclefindmkaj/https://cr.openjdk.org/~psandoz/dv14-uk-paul-sandoz-unsafe-the-situation.pdf]
I wonder if we can remove unsafe at Flink v2; I am not sure how used _off-heap
unsafe_ memory is (also Flink v2 is changing how memory is being handled). I am
not seeing an alternative.
3. I see _SignalHandler and_ TestSignalHandler uses sun.misc.Signal . This
seems to have come from Hadoop implementations that have been inherited.
4. There are 2 imports of import sun.security.krb5.KrbException; that can
be produced when calling sun.security.krb5.Config.refresh()
I would like to implement 1.2 or if this is not acceptable 1.1. This would
really help us short term as we could at least build with skipTests on Semeru.
Sun usages 2 and 3 would need so some consensus in
[jira] [Comment Edited] (FLINK-21672) End to end tests (streaming) aren't Java vendor neutral (sun.management bean used)
[
https://issues.apache.org/jira/browse/FLINK-21672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17818182#comment-17818182
]
david radley edited comment on FLINK-21672 at 2/19/24 9:51 AM:
---
Hi [~martijnvisser]
Here is a list of sun. classes I have found in core Flink.
# The one that I reported in StickyAllocationAndLocalRecoveryTestJob. This
test code uses the sun management class to get the current pid. We have some
options
## we change the constructor so it is driven reflectively so there is no
compile error. This change would allow the tst to compile.
## To have this test run in Semeru we could use logic like Datadog and try to
reflectively load other classes. Something like
[https://github.com/DataDog/dd-trace-java/blob/aee3ca59c6a05233f4295552f2ede80bc4fc[…]agent/src/main/java/datadog/trace/bootstrap/AgentBootstrap.java|https://github.com/DataDog/dd-trace-java/blob/aee3ca59c6a05233f4295552f2ede80bc4fc%5B%E2%80%A6%5Dagent/src/main/java/datadog/trace/bootstrap/AgentBootstrap.java]
. I see we could reflectively drive
com.ibm.lang.management.RuntimeMXBean.getProcessID() if the sun class is not
there.
## We fix this properly at Flink v2, using
### methods introduced at java 10:{_}java.lang.management.RuntimeMXBean
runtime={_}{_}java.lang.management.ManagementFactory.getRuntimeMXBean();{_}
_runtime.getPid();_
_2. Or use a java 9 approach_ _long pid =
ProcessHandle.current().pid();_
2. core Flink MemorySegment.java uses sun.misc.Unsafe. I see
[chrome-extension://efaidnbmnnnibpcajpcglclefindmkaj/https://cr.openjdk.org/~psandoz/dv14-uk-paul-sandoz-unsafe-the-situation.pdf]
I wonder if we can remove unsafe at Flink v2; I am not sure how used _off-heap
unsafe_ memory is (also Flink v2 is changing how memory is being handled). I am
not seeing an alternative.
3. I see _SignalHandler and_ TestSignalHandler uses sun.misc.Signal . This
seems to have come from Hadoop implementations that have been inherited.
4. There are 2 imports of import sun.security.krb5.KrbException; that can
be produced when calling sun.security.krb5.Config.refresh()
I would like to implement 1.2 or if this is not acceptable 1.1. This would
really help us short term as we could at least build with skipTests on Semeru.
Sun usages 2 and 3 would need so some consensus in the community - as is seems
we would be removing capability unless we can find an alternative. The
sun.security references are used when testing Hadoop with Kerberos, I have not
looked into it.
was (Author: JIRAUSER300523):
Hi [~martijnvisser]
Here is a list of sun. classes I have found in core Flink.
# The one that I reported in StickyAllocationAndLocalRecoveryTestJob. This
test code uses the sun management class to get the current pid. We have some
options
## we change the constructor so it is driven reflectively so there is no
compile error. This change would allow the tst to compile.
## To have this test run in Semeru we could use logic like Datadog and try to
reflectively load other classes. Something like
[https://github.com/DataDog/dd-trace-java/blob/aee3ca59c6a05233f4295552f2ede80bc4fc[…]agent/src/main/java/datadog/trace/bootstrap/AgentBootstrap.java|https://github.com/DataDog/dd-trace-java/blob/aee3ca59c6a05233f4295552f2ede80bc4fc%5B%E2%80%A6%5Dagent/src/main/java/datadog/trace/bootstrap/AgentBootstrap.java]
. I see we could reflectively drive
com.ibm.lang.management.RuntimeMXBean.getProcessID() if the sun class is not
there.
## We fix this properly at Flink v2, using methods introduced at java 10:
## _java.lang.management.RuntimeMXBean runtime =_
_java.lang.management.ManagementFactory.getRuntimeMXBean();_
_runtime.getPid();_
_2. Or use a java 9 approach_ _long pid =
ProcessHandle.current().pid();_
2. core Flink MemorySegment.java uses sun.misc.Unsafe. I see
[chrome-extension://efaidnbmnnnibpcajpcglclefindmkaj/https://cr.openjdk.org/~psandoz/dv14-uk-paul-sandoz-unsafe-the-situation.pdf]
I wonder if we can remove unsafe at Flink v2; I am not sure how used _off-heap
unsafe_ memory is. I am not seeing an alternative.
3. I see _SignalHandler and_ TestSignalHandler uses sun.misc.Signal . This
seems to have come from Hadoop implementations that have been inherited.
4. There are 2 imports of import sun.security.krb5.KrbException; that can
be produced when calling sun.security.krb5.Config.refresh()
I would like to implement 1.2 or if this is not acceptable 1.1. This would
really help us short term as we could at least build with skipTests on Semeru.
Sun usages 2 and 3 would need so some consensus in the community - as is seems
we would be removing capability unless we can find an alternative. The
sun.security
[jira] [Comment Edited] (FLINK-21672) End to end tests (streaming) aren't Java vendor neutral (sun.management bean used)
[ https://issues.apache.org/jira/browse/FLINK-21672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17818182#comment-17818182 ] david radley edited comment on FLINK-21672 at 2/19/24 9:48 AM: --- Hi [~martijnvisser] Here is a list of sun. classes I have found in core Flink. # The one that I reported in StickyAllocationAndLocalRecoveryTestJob. This test code uses the sun management class to get the current pid. We have some options ## we change the constructor so it is driven reflectively so there is no compile error. This change would allow the tst to compile. ## To have this test run in Semeru we could use logic like Datadog and try to reflectively load other classes. Something like [https://github.com/DataDog/dd-trace-java/blob/aee3ca59c6a05233f4295552f2ede80bc4fc[…]agent/src/main/java/datadog/trace/bootstrap/AgentBootstrap.java|https://github.com/DataDog/dd-trace-java/blob/aee3ca59c6a05233f4295552f2ede80bc4fc%5B%E2%80%A6%5Dagent/src/main/java/datadog/trace/bootstrap/AgentBootstrap.java] . I see we could reflectively drive com.ibm.lang.management.RuntimeMXBean.getProcessID() if the sun class is not there. ## We fix this properly at Flink v2, using methods introduced at java 10: ## _java.lang.management.RuntimeMXBean runtime =_ _java.lang.management.ManagementFactory.getRuntimeMXBean();_ _runtime.getPid();_ _2. Or use a java 9 approach_ _long pid = ProcessHandle.current().pid();_ 2. core Flink MemorySegment.java uses sun.misc.Unsafe. I see [chrome-extension://efaidnbmnnnibpcajpcglclefindmkaj/https://cr.openjdk.org/~psandoz/dv14-uk-paul-sandoz-unsafe-the-situation.pdf] I wonder if we can remove unsafe at Flink v2; I am not sure how used _off-heap unsafe_ memory is. I am not seeing an alternative. 3. I see _SignalHandler and_ TestSignalHandler uses sun.misc.Signal . This seems to have come from Hadoop implementations that have been inherited. 4. There are 2 imports of import sun.security.krb5.KrbException; that can be produced when calling sun.security.krb5.Config.refresh() I would like to implement 1.2 or if this is not acceptable 1.1. This would really help us short term as we could at least build with skipTests on Semeru. Sun usages 2 and 3 would need so some consensus in the community - as is seems we would be removing capability unless we can find an alternative. The sun.security references are used when testing Hadoop with Kerberos, I have not looked into it. was (Author: JIRAUSER300523): Hi [~martijnvisser] Here is a list of sun. classes I have found in core Flink. # The one that I reported in StickyAllocationAndLocalRecoveryTestJob. This test code uses the sun management class to get the current pid. We have some options # we change the constructor so it is driven reflectively so there is no compile error. This change would allow the tst to compile. # To have this test run in Semeru we could use logic like Datadog and try to reflectively load other classes. Something like [https://github.com/DataDog/dd-trace-java/blob/aee3ca59c6a05233f4295552f2ede80bc4fc[…]agent/src/main/java/datadog/trace/bootstrap/AgentBootstrap.java|https://github.com/DataDog/dd-trace-java/blob/aee3ca59c6a05233f4295552f2ede80bc4fc%5B%E2%80%A6%5Dagent/src/main/java/datadog/trace/bootstrap/AgentBootstrap.java] . I see we could reflectively drive com.ibm.lang.management.RuntimeMXBean.getProcessID() if the sun class is not there. # We fix this properly at Flink v2, using methods introduced at java 10: # _java.lang.management.RuntimeMXBean runtime =_ _java.lang.management.ManagementFactory.getRuntimeMXBean();_ _runtime.getPid();_ _ 2. Or use a java 9 approach_ _long pid = ProcessHandle.current().pid();_ 2. core Flink MemorySegment.java uses sun.misc.Unsafe. I see [chrome-extension://efaidnbmnnnibpcajpcglclefindmkaj/https://cr.openjdk.org/~psandoz/dv14-uk-paul-sandoz-unsafe-the-situation.pdf] I wonder if we can remove unsafe at Flink v2; I am not sure how used _off-heap unsafe_ memory is. I am not seeing an alternative. 3. I see _SignalHandler and_ TestSignalHandler uses sun.misc.Signal . This seems to have come from Hadoop implementations that have been inherited. 4. There are 2 imports of import sun.security.krb5.KrbException; that can be produced when calling sun.security.krb5.Config.refresh() I would like to implement 1.2 or if this is not acceptable 1.1. This would really help us short term as we could at least build with skipTests on Semeru. Sun usages 2 and 3 would need so some consensus in the community - as is seems we would be removing capability unless we can find an alternative. The sun.security references are used when testing Hadoop with
[jira] [Commented] (FLINK-21672) End to end tests (streaming) aren't Java vendor neutral (sun.management bean used)
[ https://issues.apache.org/jira/browse/FLINK-21672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17818182#comment-17818182 ] david radley commented on FLINK-21672: -- Hi [~martijnvisser] Here is a list of sun. classes I have found in core Flink. # The one that I reported in StickyAllocationAndLocalRecoveryTestJob. This test code uses the sun management class to get the current pid. We have some options # we change the constructor so it is driven reflectively so there is no compile error. This change would allow the tst to compile. # To have this test run in Semeru we could use logic like Datadog and try to reflectively load other classes. Something like [https://github.com/DataDog/dd-trace-java/blob/aee3ca59c6a05233f4295552f2ede80bc4fc[…]agent/src/main/java/datadog/trace/bootstrap/AgentBootstrap.java|https://github.com/DataDog/dd-trace-java/blob/aee3ca59c6a05233f4295552f2ede80bc4fc%5B%E2%80%A6%5Dagent/src/main/java/datadog/trace/bootstrap/AgentBootstrap.java] . I see we could reflectively drive com.ibm.lang.management.RuntimeMXBean.getProcessID() if the sun class is not there. # We fix this properly at Flink v2, using methods introduced at java 10: # _java.lang.management.RuntimeMXBean runtime =_ _java.lang.management.ManagementFactory.getRuntimeMXBean();_ _runtime.getPid();_ _ 2. Or use a java 9 approach_ _long pid = ProcessHandle.current().pid();_ 2. core Flink MemorySegment.java uses sun.misc.Unsafe. I see [chrome-extension://efaidnbmnnnibpcajpcglclefindmkaj/https://cr.openjdk.org/~psandoz/dv14-uk-paul-sandoz-unsafe-the-situation.pdf] I wonder if we can remove unsafe at Flink v2; I am not sure how used _off-heap unsafe_ memory is. I am not seeing an alternative. 3. I see _SignalHandler and_ TestSignalHandler uses sun.misc.Signal . This seems to have come from Hadoop implementations that have been inherited. 4. There are 2 imports of import sun.security.krb5.KrbException; that can be produced when calling sun.security.krb5.Config.refresh() I would like to implement 1.2 or if this is not acceptable 1.1. This would really help us short term as we could at least build with skipTests on Semeru. Sun usages 2 and 3 would need so some consensus in the community - as is seems we would be removing capability unless we can find an alternative. The sun.security references are used when testing Hadoop with Kerberos, I have not looked into it. > End to end tests (streaming) aren't Java vendor neutral (sun.management bean > used) > -- > > Key: FLINK-21672 > URL: https://issues.apache.org/jira/browse/FLINK-21672 > Project: Flink > Issue Type: Improvement > Components: Tests >Reporter: Adam Roberts >Assignee: david radley >Priority: Minor > > Hi everyone, have been looking to run the tests for Flink using an 11 > AdoptOpenJDK 11 distribution (so the latest for Linux, x86-64 specifically) > and I see > > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile > (default-compile) on project flink-local-recovery-and-allocation-test: > Compilation failure: Compilation failure: > [ERROR] > /var/home/core/flink/flink-end-to-end-tests/flink-local-recovery-and-allocation-test/src/main/java/org/apache/flink/streaming/tests/StickyAllocationAndLocalRecoveryTestJob.java:[416,23] > cannot find symbol > [ERROR] symbol: class VMManagement > [ERROR] location: package sun.management > [ERROR] > /var/home/core/flink/flink-end-to-end-tests/flink-local-recovery-and-allocation-test/src/main/java/org/apache/flink/streaming/tests/StickyAllocationAndLocalRecoveryTestJob.java:[416,59] > cannot find symbol > [ERROR] symbol: class VMManagement > [ERROR] location: package sun.management > [ERROR] -> [Help 1] > > my guess is that AdoptOpenJDK's class-library simply doesn't have this > package and we should use a more neutral one if that's available - I went > with > [https://adoptopenjdk.net/releases.html?variant=openjdk11=openj9] > personally (OpenJ9 being the name for IBM's open-sourced J9 JVM), but I > wonder if that has its own platform specific bean as well; I haven't worked > on IBM's distribution of Java for almost seven years* but hopefully someone > may have more insight so you don't need to be using OpenJDK backed by HotSpot > to run said tests. It would be helpful if we didn't need an (if vendor == IBM > do this || vendor == Oracle do this ... etc) statement, so you can run the > tests no matter where you're getting Java from. > > *full disclaimer, I work at IBM and helped create AdoptOpenJDK, and used to > work in its Java team...I've honestly forgot if we have a
[jira] [Commented] (FLINK-21672) End to end tests (streaming) aren't Java vendor neutral (sun.management bean used)
[ https://issues.apache.org/jira/browse/FLINK-21672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17818042#comment-17818042 ] david radley commented on FLINK-21672: -- Thanks [~martijnvisser] - It looks like there is only this reference to sun.management in the code. I do see that there are references to _com.sun.management_ classes, for example [https://github.com/apache/flink/blob/620e5975985944a02886b82362a2bc1774c733e3/flink-runtime/src/main/java/org/apache/flink/runtime/util/EnvironmentInformation.java#L369] It would be great if we could remove these, but my worry is that there may not be a vendor neutral way of exercising this capability. > End to end tests (streaming) aren't Java vendor neutral (sun.management bean > used) > -- > > Key: FLINK-21672 > URL: https://issues.apache.org/jira/browse/FLINK-21672 > Project: Flink > Issue Type: Improvement > Components: Tests >Reporter: Adam Roberts >Assignee: david radley >Priority: Minor > > Hi everyone, have been looking to run the tests for Flink using an 11 > AdoptOpenJDK 11 distribution (so the latest for Linux, x86-64 specifically) > and I see > > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile > (default-compile) on project flink-local-recovery-and-allocation-test: > Compilation failure: Compilation failure: > [ERROR] > /var/home/core/flink/flink-end-to-end-tests/flink-local-recovery-and-allocation-test/src/main/java/org/apache/flink/streaming/tests/StickyAllocationAndLocalRecoveryTestJob.java:[416,23] > cannot find symbol > [ERROR] symbol: class VMManagement > [ERROR] location: package sun.management > [ERROR] > /var/home/core/flink/flink-end-to-end-tests/flink-local-recovery-and-allocation-test/src/main/java/org/apache/flink/streaming/tests/StickyAllocationAndLocalRecoveryTestJob.java:[416,59] > cannot find symbol > [ERROR] symbol: class VMManagement > [ERROR] location: package sun.management > [ERROR] -> [Help 1] > > my guess is that AdoptOpenJDK's class-library simply doesn't have this > package and we should use a more neutral one if that's available - I went > with > [https://adoptopenjdk.net/releases.html?variant=openjdk11=openj9] > personally (OpenJ9 being the name for IBM's open-sourced J9 JVM), but I > wonder if that has its own platform specific bean as well; I haven't worked > on IBM's distribution of Java for almost seven years* but hopefully someone > may have more insight so you don't need to be using OpenJDK backed by HotSpot > to run said tests. It would be helpful if we didn't need an (if vendor == IBM > do this || vendor == Oracle do this ... etc) statement, so you can run the > tests no matter where you're getting Java from. > > *full disclaimer, I work at IBM and helped create AdoptOpenJDK, and used to > work in its Java team...I've honestly forgot if we have a vendor-neutral bean > available and now work on something totally different! > Cheers! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-21672) End to end tests (streaming) aren't Java vendor neutral (sun.management bean used)
[ https://issues.apache.org/jira/browse/FLINK-21672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17817997#comment-17817997 ] david radley commented on FLINK-21672: -- I see [https://forums.oracle.com/ords/apexds/post/sun-package-is-forbidden-then-what-about-com-package-in-fut-9023] we should not be using packages starting with _sun._ > End to end tests (streaming) aren't Java vendor neutral (sun.management bean > used) > -- > > Key: FLINK-21672 > URL: https://issues.apache.org/jira/browse/FLINK-21672 > Project: Flink > Issue Type: Improvement > Components: Tests >Reporter: Adam Roberts >Priority: Minor > > Hi everyone, have been looking to run the tests for Flink using an 11 > AdoptOpenJDK 11 distribution (so the latest for Linux, x86-64 specifically) > and I see > > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile > (default-compile) on project flink-local-recovery-and-allocation-test: > Compilation failure: Compilation failure: > [ERROR] > /var/home/core/flink/flink-end-to-end-tests/flink-local-recovery-and-allocation-test/src/main/java/org/apache/flink/streaming/tests/StickyAllocationAndLocalRecoveryTestJob.java:[416,23] > cannot find symbol > [ERROR] symbol: class VMManagement > [ERROR] location: package sun.management > [ERROR] > /var/home/core/flink/flink-end-to-end-tests/flink-local-recovery-and-allocation-test/src/main/java/org/apache/flink/streaming/tests/StickyAllocationAndLocalRecoveryTestJob.java:[416,59] > cannot find symbol > [ERROR] symbol: class VMManagement > [ERROR] location: package sun.management > [ERROR] -> [Help 1] > > my guess is that AdoptOpenJDK's class-library simply doesn't have this > package and we should use a more neutral one if that's available - I went > with > [https://adoptopenjdk.net/releases.html?variant=openjdk11=openj9] > personally (OpenJ9 being the name for IBM's open-sourced J9 JVM), but I > wonder if that has its own platform specific bean as well; I haven't worked > on IBM's distribution of Java for almost seven years* but hopefully someone > may have more insight so you don't need to be using OpenJDK backed by HotSpot > to run said tests. It would be helpful if we didn't need an (if vendor == IBM > do this || vendor == Oracle do this ... etc) statement, so you can run the > tests no matter where you're getting Java from. > > *full disclaimer, I work at IBM and helped create AdoptOpenJDK, and used to > work in its Java team...I've honestly forgot if we have a vendor-neutral bean > available and now work on something totally different! > Cheers! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-21672) End to end tests (streaming) aren't Java vendor neutral (sun.management bean used)
[ https://issues.apache.org/jira/browse/FLINK-21672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17817944#comment-17817944 ] david radley commented on FLINK-21672: -- I notice [https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/master/src/java.management/share/classes/sun/management/VMManagement.java|http://example.com/] This implies that the AdoptOpenJDK does have this class - at least in the source. > End to end tests (streaming) aren't Java vendor neutral (sun.management bean > used) > -- > > Key: FLINK-21672 > URL: https://issues.apache.org/jira/browse/FLINK-21672 > Project: Flink > Issue Type: Improvement > Components: Tests >Reporter: Adam Roberts >Priority: Minor > > Hi everyone, have been looking to run the tests for Flink using an 11 > AdoptOpenJDK 11 distribution (so the latest for Linux, x86-64 specifically) > and I see > > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile > (default-compile) on project flink-local-recovery-and-allocation-test: > Compilation failure: Compilation failure: > [ERROR] > /var/home/core/flink/flink-end-to-end-tests/flink-local-recovery-and-allocation-test/src/main/java/org/apache/flink/streaming/tests/StickyAllocationAndLocalRecoveryTestJob.java:[416,23] > cannot find symbol > [ERROR] symbol: class VMManagement > [ERROR] location: package sun.management > [ERROR] > /var/home/core/flink/flink-end-to-end-tests/flink-local-recovery-and-allocation-test/src/main/java/org/apache/flink/streaming/tests/StickyAllocationAndLocalRecoveryTestJob.java:[416,59] > cannot find symbol > [ERROR] symbol: class VMManagement > [ERROR] location: package sun.management > [ERROR] -> [Help 1] > > my guess is that AdoptOpenJDK's class-library simply doesn't have this > package and we should use a more neutral one if that's available - I went > with > [https://adoptopenjdk.net/releases.html?variant=openjdk11=openj9] > personally (OpenJ9 being the name for IBM's open-sourced J9 JVM), but I > wonder if that has its own platform specific bean as well; I haven't worked > on IBM's distribution of Java for almost seven years* but hopefully someone > may have more insight so you don't need to be using OpenJDK backed by HotSpot > to run said tests. It would be helpful if we didn't need an (if vendor == IBM > do this || vendor == Oracle do this ... etc) statement, so you can run the > tests no matter where you're getting Java from. > > *full disclaimer, I work at IBM and helped create AdoptOpenJDK, and used to > work in its Java team...I've honestly forgot if we have a vendor-neutral bean > available and now work on something totally different! > Cheers! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-34211) Filtering on Column names with ?s fails for JDBC lookup join.
[ https://issues.apache.org/jira/browse/FLINK-34211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17809857#comment-17809857 ] david radley commented on FLINK-34211: -- fyi JDBC lookup Filters are ignored until the fix for [https://issues.apache.org/jira/projects/FLINK/issues/FLINK-33365] is merged > Filtering on Column names with ?s fails for JDBC lookup join. > -- > > Key: FLINK-34211 > URL: https://issues.apache.org/jira/browse/FLINK-34211 > Project: Flink > Issue Type: Bug > Components: Connectors / JDBC, Table SQL / JDBC >Reporter: david radley >Priority: Minor > > There is a check for ? character in > [https://github.com/apache/flink-connector-jdbc/blob/e3dd84160cd665ae17672da8b6e742e61a72a32d/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/statement/FieldNamedPreparedStatementImpl.java#L186 > |FieldNamedPreparedStatementImpl.java] > Removing this check allows column names containing _?_ -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34211) Filtering on Column names with ?s fails for JDBC lookup join.
david radley created FLINK-34211: Summary: Filtering on Column names with ?s fails for JDBC lookup join. Key: FLINK-34211 URL: https://issues.apache.org/jira/browse/FLINK-34211 Project: Flink Issue Type: Bug Components: Connectors / JDBC, Table SQL / JDBC Reporter: david radley There is a check for ? character in [https://github.com/apache/flink-connector-jdbc/blob/e3dd84160cd665ae17672da8b6e742e61a72a32d/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/statement/FieldNamedPreparedStatementImpl.java#L186 |FieldNamedPreparedStatementImpl.java] Removing this check allows column names containing _?_ -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34170) Include the look up join conditions in the optimised plan.
david radley created FLINK-34170: Summary: Include the look up join conditions in the optimised plan. Key: FLINK-34170 URL: https://issues.apache.org/jira/browse/FLINK-34170 Project: Flink Issue Type: Improvement Components: Table SQL / Planner Reporter: david radley As per [https://github.com/apache/flink-connector-jdbc/pull/79#discussion_r1458664773] [~libenchao] asked that I raise this issue to Include the look up join conditions in the optimised plan; in lime with the scan conditions. The JDBC and other lookup sources could then be updated to pick up the conditions from the plan. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34146) JDBC lookup joins fail with RDB column names containing colons
david radley created FLINK-34146:
Summary: JDBC lookup joins fail with RDB column names containing
colons
Key: FLINK-34146
URL: https://issues.apache.org/jira/browse/FLINK-34146
Project: Flink
Issue Type: Bug
Components: Connectors / JDBC, Table SQL / JDBC
Affects Versions: 1.18.1
Reporter: david radley
[https://github.com/apache/flink-connector-jdbc/pull/79] adds filter support
for lookup joins. This was implemented using FieldNamedPreparedStatements in
line with the way that the join key was implemented. The
[FieldNamedPreparedStatementImpl
logic|https://github.com/apache/flink-connector-jdbc/blob/e3dd84160cd665ae17672da8b6e742e61a72a32d/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/statement/FieldNamedPreparedStatementImpl.java#L221]
explicitly tests for the colon key and can incorrectly pickup column names.
So JDBC lookup joins fail with RDB column names containing colons when used in
filters and lookup keys.
It looks like we have used the approach from
[https://stackoverflow.com/questions/2309970/named-parameters-in-jdbc]. It says
{{Please note that the above simple example does not handle using named
parameter twice. Nor does it handle using the : sign inside quotes.}} It looks
like we could play with some Regex Patterns to see if we can get one that works
well for us.
A junit that shows the issue can be added to
FieldNamedPreparedStatementImplTest
...
private final String[] fieldNames2 =
new String[] \{"id?:", "name:?", "email", "ts", "field1", "field_2",
"__field_3__"};
private final String[] keyFields2 = new String[] \{"id?:", "__field_3__"};
...
@Test
void testSelectStatementWithWeirdCharacters() {
String selectStmt = dialect.getSelectFromStatement(tableName, fieldNames2,
keyFields2);
assertThat(selectStmt)
.isEqualTo(
"SELECT `id?:`, `name:?`, `email`, `ts`, `field1`, `field_2`, `__field_3__`
FROM `tbl` "
+ "WHERE `id?:` = :id?: AND `__field_3__` = :__field_3__");
NamedStatementMatcher.parsedSql(
"SELECT `id?:`, `name:?`, `email`, `ts`, `field1`, `field_2`, `__field_3__`
FROM `tbl` "
+ "WHERE `id?:` = ? AND `__field_3__` = ?")
.parameter("id", singletonList(1))
.parameter("__field_3__", singletonList(2))
.matches(selectStmt);
}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17798719#comment-17798719
]
david radley edited comment on FLINK-33365 at 12/19/23 7:36 PM:
[~Sergey Nuyanzin] I have pushed up my latest changes including the fix for the
OR case. The tests are more simplified but they are not data driven - with one
method looking of test case data. I did not find a way with the existing data
to drive the null case. I have tested that and the where locally but there is
no automated test for it. Please review and see what more needs to be changed?
I appreciate your time on this. I added debug - as I think this will speed
future diagnostics in this area. Scan logic has a similar debug.
was (Author: JIRAUSER300523):
[~Sergey Nuyanzin] I have pushed up my latest changes including the fix for the
OR case. The tests are more simplified but they are not data driven - with one
method looking of test case data. I did not find a way with the existing data
to drive the null case. I have tested that and the where locally but there is
no automated test for it. Please review and see what more needs to be changed?
I appreciate your time on this.
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Assignee: david radley
>Priority: Critical
> Labels: pull-request-available
> Attachments: flink-connector-jdbc-3.0.0-1.16.png,
> flink-connector-jdbc-3.1.1-1.17.png
>
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[ip=ip], select=[ip, proctime, ip, CAST(0 AS INTEGER) AS type, CAST(ip
> AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]){code}
>
> excute same sql in sql-client with Flink 1.17.1/ 1.18.0 and
> *flink-connector-jdbc-3.0.0-1.16.jar*
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[type=0, ip=ip], where=[(type = 0)], select=[ip, proctime, ip, CAST(0
> AS INTEGER) AS type, CAST(ip AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]) {code}
> with flink-connector-jdbc-3.1.1-1.17.jar, the condition is
> *lookup=[ip=ip]*
> with flink-connector-jdbc-3.0.0-1.16.jar , the condition is
> *lookup=[type=0, ip=ip], where=[(type = 0)]*
>
> In out real world production environment, this lead incorrect data output
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17798719#comment-17798719
]
david radley commented on FLINK-33365:
--
[~Sergey Nuyanzin] I have pushed up my latest changes including the fix for the
OR case. The tests are more simplified but they are not data driven - with one
method looking of test case data. I did not find a way with the existing data
to drive the null case. I have tested that and the where locally but there is
no automated test for it. Please review and see what more needs to be changed?
I appreciate your time on this.
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Assignee: david radley
>Priority: Critical
> Labels: pull-request-available
> Attachments: flink-connector-jdbc-3.0.0-1.16.png,
> flink-connector-jdbc-3.1.1-1.17.png
>
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[ip=ip], select=[ip, proctime, ip, CAST(0 AS INTEGER) AS type, CAST(ip
> AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]){code}
>
> excute same sql in sql-client with Flink 1.17.1/ 1.18.0 and
> *flink-connector-jdbc-3.0.0-1.16.jar*
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[type=0, ip=ip], where=[(type = 0)], select=[ip, proctime, ip, CAST(0
> AS INTEGER) AS type, CAST(ip AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]) {code}
> with flink-connector-jdbc-3.1.1-1.17.jar, the condition is
> *lookup=[ip=ip]*
> with flink-connector-jdbc-3.0.0-1.16.jar , the condition is
> *lookup=[type=0, ip=ip], where=[(type = 0)]*
>
> In out real world production environment, this lead incorrect data output
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17798552#comment-17798552
]
david radley commented on FLINK-33365:
--
[~Sergey Nuyanzin] sounds great! Thank you for your support. I realise that I
need to add tests that exercise the 2 types of filters, the one on the _join_
which can get nulls and the one in the _where_ that does not. I will push up
new code for you to review, when I have completed the tests.
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Assignee: david radley
>Priority: Critical
> Labels: pull-request-available
> Attachments: flink-connector-jdbc-3.0.0-1.16.png,
> flink-connector-jdbc-3.1.1-1.17.png
>
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[ip=ip], select=[ip, proctime, ip, CAST(0 AS INTEGER) AS type, CAST(ip
> AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]){code}
>
> excute same sql in sql-client with Flink 1.17.1/ 1.18.0 and
> *flink-connector-jdbc-3.0.0-1.16.jar*
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[type=0, ip=ip], where=[(type = 0)], select=[ip, proctime, ip, CAST(0
> AS INTEGER) AS type, CAST(ip AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]) {code}
> with flink-connector-jdbc-3.1.1-1.17.jar, the condition is
> *lookup=[ip=ip]*
> with flink-connector-jdbc-3.0.0-1.16.jar , the condition is
> *lookup=[type=0, ip=ip], where=[(type = 0)]*
>
> In out real world production environment, this lead incorrect data output
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17798178#comment-17798178
]
david radley edited comment on FLINK-33365 at 12/18/23 4:12 PM:
[~Sergey Nuyanzin] from the discussion above. I am thinking the s issue
was not really an issue, this was my misunderstanding. The options I see are:
1) Do as you say, but wouldn't this be regression for the jdbc driver without
filters, as there is likely to be a performance impact without filter pushdown
2) We could complete the fix I have. I have it working locally with the OR
case. I am looking at implementing more test cases.I am struggling to make the
unit test cases, to be defined in data with a for loop to execute them. If you
are ok with the style of unit tests I have, I could have this up today or
tomorrow.
3) If we do not view this as a blocker - we could release without reverting the
commit and continue investigating.
My preference would be option 2 - we are looking to have this fix in and would
probably need to patch out code to pick it up. But if this takes too long to
get merged, option 3 would be an option.
Strangely the config option to disable predicate pushdowns works for me. I was
not expecting it to work as it should only effect the legacy style of predicate
pushdown.
WDYT?
was (Author: JIRAUSER300523):
[~Sergey Nuyanzin] from the discussion above. I am thinking the s issue
was not really an issue, this was my misunderstanding. The options I see are:
1) Do as you say, but this wouldn't this be regression for the jdbc driver
without filters, as there is likely to be a performance impact without filter
pushdown
2) We could complete the fix I have. I have it working locally with the OR
case. I am looking at implementing more test cases.I am struggling to make the
unit test cases, to be defined in data with a for loop to execute them. If you
are ok with the style of unit tests I have, I could have this up today or
tomorrow.
3) If we do not view this as a blocker - we could release without reverting the
commit and continue investigating.
My preference would be option 2 - we are looking to have this fix in and would
probably need to patch out code to pick it up. But if this takes too long to
get merged, option 3 would be an option.
Strangely the config option to disable predicate pushdowns works for me. I was
not expecting it to work as it should only effect the legacy style of predicate
pushdown.
WDYT?
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Assignee: david radley
>Priority: Critical
> Labels: pull-request-available
> Attachments: flink-connector-jdbc-3.0.0-1.16.png,
> flink-connector-jdbc-3.1.1-1.17.png
>
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[ip=ip], select=[ip, proctime, ip, CAST(0 AS INTEGER) AS type, CAST(ip
> AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]){code}
>
> excute same sql in sql-client with Flink 1.17.1/ 1.18.0 and
> *flink-connector-jdbc-3.0.0-1.16.jar*
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[type=0, ip=ip], where=[(type = 0)], select=[ip, proctime, ip, CAST(0
> AS INTEGER) AS type, CAST(ip AS VARCHAR(2147483647)) AS ip0])
> +-
[jira] [Comment Edited] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17798178#comment-17798178
]
david radley edited comment on FLINK-33365 at 12/18/23 4:10 PM:
[~Sergey Nuyanzin] from the discussion above. I am thinking the s issue
was not really an issue, this was my misunderstanding. The options I see are:
1) Do as you say, but this wouldn't this be regression for the jdbc driver
without filters, as there is likely to be a performance impact without filter
pushdown
2) We could complete the fix I have. I have it working locally with the OR
case. I am looking at implementing more test cases.I am struggling to make the
unit test cases, to be defined in data with a for loop to execute them. If you
are ok with the style of unit tests I have, I could have this up today or
tomorrow.
3) If we do not view this as a blocker - we could release without reverting the
commit and continue investigating.
My preference would be option 2 - we are looking to have this fix in and would
probably need to patch out code to pick it up. But if this takes too long to
get merged, option 3 would be an option.
Strangely the config option to disable predicate pushdowns works for me. I was
not expecting it to work as it should only effect the legacy style of predicate
pushdown.
WDYT?
was (Author: JIRAUSER300523):
[~Sergey Nuyanzin] from the discussion above. I am thinking the s issue
was not really an issue, this was my misunderstanding. The options I see are:
1) Do as you say, but this wouldn't this be regression for the jdbc driver
without filters, as there is likely to be a performance impact without filter
pushdown
2) We could complete the fix I have. I have it working locally with the OR
case. I am looking at the implementing more test cases.I am struggling to make
the unit test cases, to be defined in data with a for loop to execute them. If
you are ok with the style of unit tests I have, I could have this up today or
tomorrow.
3) If we do not view this as a blocker - we could release without reverting the
commit and continue investigating.
My preference would be option 2 - we are looking to have this fix in and would
probably need to patch out code to pick it up. But if this takes too long to
get merged, option 3 would be an option.
Strangely the config option to disable predicate pushdowns works for me. I was
not expecting it to work as it should only effect the legacy style of predicate
pushdown.
WDYT?
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Assignee: david radley
>Priority: Critical
> Labels: pull-request-available
> Attachments: flink-connector-jdbc-3.0.0-1.16.png,
> flink-connector-jdbc-3.1.1-1.17.png
>
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[ip=ip], select=[ip, proctime, ip, CAST(0 AS INTEGER) AS type, CAST(ip
> AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]){code}
>
> excute same sql in sql-client with Flink 1.17.1/ 1.18.0 and
> *flink-connector-jdbc-3.0.0-1.16.jar*
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[type=0, ip=ip], where=[(type = 0)], select=[ip, proctime, ip, CAST(0
> AS INTEGER) AS type, CAST(ip AS VARCHAR(2147483647)) AS ip0])
> +-
[jira] [Comment Edited] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17798178#comment-17798178
]
david radley edited comment on FLINK-33365 at 12/18/23 1:28 PM:
[~Sergey Nuyanzin] from the discussion above. I am thinking the s issue
was not really an issue, this was my misunderstanding. The options I see are:
1) Do as you say, but this wouldn't this be regression for the jdbc driver
without filters, as there is likely to be a performance impact without filter
pushdown
2) We could complete the fix I have. I have it working locally with the OR
case. I am looking at the implementing more test cases.I am struggling to make
the unit test cases, to be defined in data with a for loop to execute them. If
you are ok with the style of unit tests I have, I could have this up today or
tomorrow.
3) If we do not view this as a blocker - we could release without reverting the
commit and continue investigating.
My preference would be option 2 - we are looking to have this fix in and would
probably need to patch out code to pick it up. But if this takes too long to
get merged, option 3 would be an option.
Strangely the config option to disable predicate pushdowns works for me. I was
not expecting it to work as it should only effect the legacy style of predicate
pushdown.
WDYT?
was (Author: JIRAUSER300523):
[~Sergey Nuyanzin] from the discussion above. I am thinking the s issue
was not really an issue, this was my misunderstanding. The options I see are:
1) Do as you say, but this wouldn't this be regression for the jdbc driver
without filters, as there is likely to be a performance impact without filter
pushdown
2) We could complete the fix I have. I have it working locally with the OR
case. I am looking at the implementing more test cases.I am struggling to make
the unit test cases, to be defined in data with a for loop to execute them. If
you are ok with the style of unit tests I have, I could have this up today or
tomorrow.
3) If we do not view this as a blocker - we could release without reverting the
commit and continue investigating.
My preference would be option 2 - we are looking to have this fix in and would
probably need to patch out code to pick it up. But if this takes too long to
get merged, option 3 would be an option.
Strangely the config option to disable predicate pushdowns works for me. I was
not expecting it to work as it should only effect the legacy style of predicate
pushdown.
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Assignee: david radley
>Priority: Critical
> Labels: pull-request-available
> Attachments: flink-connector-jdbc-3.0.0-1.16.png,
> flink-connector-jdbc-3.1.1-1.17.png
>
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[ip=ip], select=[ip, proctime, ip, CAST(0 AS INTEGER) AS type, CAST(ip
> AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]){code}
>
> excute same sql in sql-client with Flink 1.17.1/ 1.18.0 and
> *flink-connector-jdbc-3.0.0-1.16.jar*
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[type=0, ip=ip], where=[(type = 0)], select=[ip, proctime, ip, CAST(0
> AS INTEGER) AS type, CAST(ip AS VARCHAR(2147483647)) AS ip0])
> +-
[jira] [Commented] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17798178#comment-17798178
]
david radley commented on FLINK-33365:
--
[~Sergey Nuyanzin] from the discussion above. I am thinking the s issue
was not really an issue, this was my misunderstanding. The options I see are:
1) Do as you say, but this wouldn't this be regression for the jdbc driver
without filters, as there is likely to be a performance impact without filter
pushdown
2) We could complete the fix I have. I have it working locally with the OR
case. I am looking at the implementing more test cases.I am struggling to make
the unit test cases, to be defined in data with a for loop to execute them. If
you are ok with the style of unit tests I have, I could have this up today or
tomorrow.
3) If we do not view this as a blocker - we could release without reverting the
commit and continue investigating.
My preference would be option 2 - we are looking to have this fix in and would
probably need to patch out code to pick it up. But if this takes too long to
get merged, option 3 would be an option.
Strangely the config option to disable predicate pushdowns works for me. I was
not expecting it to work as it should only effect the legacy style of predicate
pushdown.
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Assignee: david radley
>Priority: Critical
> Labels: pull-request-available
> Attachments: flink-connector-jdbc-3.0.0-1.16.png,
> flink-connector-jdbc-3.1.1-1.17.png
>
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[ip=ip], select=[ip, proctime, ip, CAST(0 AS INTEGER) AS type, CAST(ip
> AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]){code}
>
> excute same sql in sql-client with Flink 1.17.1/ 1.18.0 and
> *flink-connector-jdbc-3.0.0-1.16.jar*
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[type=0, ip=ip], where=[(type = 0)], select=[ip, proctime, ip, CAST(0
> AS INTEGER) AS type, CAST(ip AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]) {code}
> with flink-connector-jdbc-3.1.1-1.17.jar, the condition is
> *lookup=[ip=ip]*
> with flink-connector-jdbc-3.0.0-1.16.jar , the condition is
> *lookup=[type=0, ip=ip], where=[(type = 0)]*
>
> In out real world production environment, this lead incorrect data output
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17798165#comment-17798165
]
david radley commented on FLINK-33365:
--
[~libenchao] thanks for the quick feedback - I guess it is where the filter is
applied - I was expecting the filter to be applied to the table then we join.
You are saying we join and show non null values where the filter matches. I
have done some testing on sqiflddle, it looks like you are right. If I wanted
the bevaviour, I was expecting I should use:
SELECT * FROM a left join mariadb_catalog.menagerie.e FOR SYSTEM_TIME AS OF
a.proctime on a.ip = e.ip _where_ e.age = 50 ;
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Assignee: david radley
>Priority: Critical
> Labels: pull-request-available
> Attachments: flink-connector-jdbc-3.0.0-1.16.png,
> flink-connector-jdbc-3.1.1-1.17.png
>
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[ip=ip], select=[ip, proctime, ip, CAST(0 AS INTEGER) AS type, CAST(ip
> AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]){code}
>
> excute same sql in sql-client with Flink 1.17.1/ 1.18.0 and
> *flink-connector-jdbc-3.0.0-1.16.jar*
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[type=0, ip=ip], where=[(type = 0)], select=[ip, proctime, ip, CAST(0
> AS INTEGER) AS type, CAST(ip AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]) {code}
> with flink-connector-jdbc-3.1.1-1.17.jar, the condition is
> *lookup=[ip=ip]*
> with flink-connector-jdbc-3.0.0-1.16.jar , the condition is
> *lookup=[type=0, ip=ip], where=[(type = 0)]*
>
> In out real world production environment, this lead incorrect data output
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17798091#comment-17798091
]
david radley commented on FLINK-33365:
--
[~libenchao] [~Sergey Nuyanzin] Bringing back a discussion onto the issue. I
have many of the scenarios working including the Or scenario. There are
scenarios
like the one [https://lists.apache.org/thread/7r49c5ffohz1oqm3oz2g60g5lnovq0l2]
if the jdbc lookup source has rows that do not match the filter, then nulls
come out in the lookup join resultset. The same behaviour occurs with predicate
pushdown disabled and with my current patch (when I push it up).
On the discussion thread [~ruanhang1993] agreed that the nulls are not correct.
The way I am reading this is that, unless we have a reason to see these nulls,
there is an issue in lookup joins in certain scenarios. It does not appear to
relate to JDBC. Are we ok if I make this fix work the same as when predicate
pushdown is not enabled. And raise a separate issue for the lookup join which
appears not to be JDBC related.
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Assignee: david radley
>Priority: Critical
> Labels: pull-request-available
> Attachments: flink-connector-jdbc-3.0.0-1.16.png,
> flink-connector-jdbc-3.1.1-1.17.png
>
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[ip=ip], select=[ip, proctime, ip, CAST(0 AS INTEGER) AS type, CAST(ip
> AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]){code}
>
> excute same sql in sql-client with Flink 1.17.1/ 1.18.0 and
> *flink-connector-jdbc-3.0.0-1.16.jar*
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[type=0, ip=ip], where=[(type = 0)], select=[ip, proctime, ip, CAST(0
> AS INTEGER) AS type, CAST(ip AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]) {code}
> with flink-connector-jdbc-3.1.1-1.17.jar, the condition is
> *lookup=[ip=ip]*
> with flink-connector-jdbc-3.0.0-1.16.jar , the condition is
> *lookup=[type=0, ip=ip], where=[(type = 0)]*
>
> In out real world production environment, this lead incorrect data output
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33462) Sort out the document page about the new Jdbc source.
[ https://issues.apache.org/jira/browse/FLINK-33462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17787258#comment-17787258 ] david radley commented on FLINK-33462: -- I am curious what we intend to document here. the main class is labelled with the internal tag in the open pr [https://github.com/apache/flink-connector-jdbc/pull/78/files#diff-1cb96d83a1a96840413da2630840a47bea8aeb07e1ec4cab6111cf2a3a013fa0] . I would not expect us to document internal classes - or should it not be tagged internal? > Sort out the document page about the new Jdbc source. > - > > Key: FLINK-33462 > URL: https://issues.apache.org/jira/browse/FLINK-33462 > Project: Flink > Issue Type: Sub-task > Components: Connectors / JDBC >Reporter: RocMarshal >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17787123#comment-17787123
]
david radley edited comment on FLINK-33365 at 11/17/23 10:53 AM:
-
[~libenchao]
I fixed a bug in the fix that was stopping multiple lookup keys being processed
properly.
was (Author: JIRAUSER300523):
[~libenchao]
fyi this fix asis appears to have regressed when there are 2 lookup key
conditions and no filters. I am investigating.
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Assignee: david radley
>Priority: Critical
> Labels: pull-request-available
> Attachments: flink-connector-jdbc-3.0.0-1.16.png,
> flink-connector-jdbc-3.1.1-1.17.png
>
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[ip=ip], select=[ip, proctime, ip, CAST(0 AS INTEGER) AS type, CAST(ip
> AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]){code}
>
> excute same sql in sql-client with Flink 1.17.1/ 1.18.0 and
> *flink-connector-jdbc-3.0.0-1.16.jar*
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[type=0, ip=ip], where=[(type = 0)], select=[ip, proctime, ip, CAST(0
> AS INTEGER) AS type, CAST(ip AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]) {code}
> with flink-connector-jdbc-3.1.1-1.17.jar, the condition is
> *lookup=[ip=ip]*
> with flink-connector-jdbc-3.0.0-1.16.jar , the condition is
> *lookup=[type=0, ip=ip], where=[(type = 0)]*
>
> In out real world production environment, this lead incorrect data output
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17787123#comment-17787123
]
david radley commented on FLINK-33365:
--
[~libenchao]
fyi this fix asis appears to have regressed when there are 2 lookup key
conditions and no filters. I am investigating.
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Assignee: david radley
>Priority: Critical
> Labels: pull-request-available
> Attachments: flink-connector-jdbc-3.0.0-1.16.png,
> flink-connector-jdbc-3.1.1-1.17.png
>
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[ip=ip], select=[ip, proctime, ip, CAST(0 AS INTEGER) AS type, CAST(ip
> AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]){code}
>
> excute same sql in sql-client with Flink 1.17.1/ 1.18.0 and
> *flink-connector-jdbc-3.0.0-1.16.jar*
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[type=0, ip=ip], where=[(type = 0)], select=[ip, proctime, ip, CAST(0
> AS INTEGER) AS type, CAST(ip AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]) {code}
> with flink-connector-jdbc-3.1.1-1.17.jar, the condition is
> *lookup=[ip=ip]*
> with flink-connector-jdbc-3.0.0-1.16.jar , the condition is
> *lookup=[type=0, ip=ip], where=[(type = 0)]*
>
> In out real world production environment, this lead incorrect data output
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786427#comment-17786427
]
david radley edited comment on FLINK-33365 at 11/15/23 4:34 PM:
[~libenchao]
I have moved the code from the AbstractDialog and pushed up the change. I could
not see how to get a _PreparedStatement_ to be able to set the condition
on{_}.{_} Please could you give me some pointers.
I have successfully tested using the supplied test tables:
* other simple predicates work
* multiple simple predicates work
I added extra tests to JdbcDynamicTableSourceITCase but cannot test those
changes as I get errors locally when running the tests without my changes. Any
pointers would be great I am running on a Mac.
I wanted to test 2 look up keys, so I created a new table d (on MariaDB) and e
(on paimon) and set up the following tables and did the some joins with
filters and a join with multiple keys. The results do not look right to me (but
I may be misunderstanding) - WDYT?
select * from mariadb_catalog.menagerie.d ;
{+}---{-}{-}{+}--{-}++{-}---{-}{-}+
|op| ip| type| age|
{+}---{-}{-}{+}--{-}++{-}---{-}{-}+
|+I| 10.10.10.10| 1| 30|
|+I| 10.10.10.10| 2| 40|
|+I| 10.10.10.10| 2| 50|
|+I| 10.10.10.10| 3| 50|
{+}---{-}{-}{+}--{-}++{-}---{-}{-}+
Received a total of 4 rows
Flink SQL> select * from e;
{+}---{-}{-}{+}--{-}++{-}---{-}{-}+
|op| ip| age| proctime|
{+}---{-}{-}{+}--{-}++{-}---{-}{-}+
|+I| 10.10.10.10| 40|2023-11-15 16:12:57.553|
|+I| 10.10.10.10| 50|2023-11-15 16:12:57.554|
Flink SQL> SELECT * FROM e left join mariadb_catalog.menagerie.d FOR
SYSTEM_TIME AS OF e.proctime on d.type = 2 and d.age = 50 and e.ip = d.ip;
{+}---{-}{-}{+}--{-}++{-}---{-}{-}---{-}++{-}--{-}{-}---{-}++{-}
|op| ip| age| proctime|
ip0| type| age0|
{+}---{-}{-}{+}--{-}++{-}---{-}{-}---{-}++{-}--{-}{-}---{-}++{-}
|+I| 10.10.10.10| 40|2023-11-15 16:08:40.973|
10.10.10.10| 2| 50|
|+I| 10.10.10.10| 50|2023-11-15 16:08:40.974|
10.10.10.10| 2| 50|
Is this what you would expect?
Also I get
SELECT * FROM e left join mariadb_catalog.menagerie.d FOR SYSTEM_TIME AS OF
e.proctime on e.age = d.age;
{+}---{-}{-}{+}--{-}++{-}---{-}{-}---{-}++{-}--{-}{-}---{-}++{-}
|op| ip| age| proctime|
ip0| type| age0|
{+}---{-}{-}{+}--{-}++{-}---{-}{-}---{-}++{-}--{-}{-}---{-}++{-}
^CQuery terminated, received a total of 0 row
and
SELECT * FROM e left join mariadb_catalog.menagerie.d FOR SYSTEM_TIME AS OF
e.proctime on e.ip = d.ip;
+++-+-++-+-+
| op | ip | age | proctime |
ip0 | type | age0 |
+++-+-++-+-+
^CQuery terminated, received a total of 0 row
and
SELECT * FROM e left join mariadb_catalog.menagerie.d FOR SYSTEM_TIME AS OF
e.proctime on e.age = d.age and d.ip = e.ip;
{+}---{-}{-}{+}--{-}++{-}---{-}{-}---{-}++{-}--{-}{-}---{-}++{-}
|op| ip| age| proctime|
ip0| type| age0|
{+}---{-}{-}{+}--{-}++{-}---{-}{-}---{-}++{-}--{-}{-}---{-}++{-}
^CQuery terminated, received a total of 0
[jira] [Commented] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786427#comment-17786427
]
david radley commented on FLINK-33365:
--
[~libenchao]
I have moved the code from the AbstractDialog and pushed up the change. I could
not see how to get a _PreparedStatement_ to be able to set the condition
on{_}.{_} Please could you give me some pointers.
I have successfully tested using the supplied test tables:
* other simple predicates work
* multiple simple predicates work
I added extra tests to JdbcDynamicTableSourceITCase but cannot test those
changes as I get errors locally when running the tests without my changes. Any
pointers would be great I am running on a Mac.
I wanted to test 2 look up keys, so I created a new table d (on MariaDB) and e
(on paimon) and set up the following tables and did the some joins with
filters and a join with multiple keys. The results do not look right to me (but
I may be misunderstanding) - but want to check with you:
select * from mariadb_catalog.menagerie.d ;
+++-+-+
| op | ip | type | age |
+++-+-+
| +I | 10.10.10.10 | 1 | 30 |
| +I | 10.10.10.10 | 2 | 40 |
| +I | 10.10.10.10 | 2 | 50 |
| +I | 10.10.10.10 | 3 | 50 |
+++-+-+
Received a total of 4 rows
Flink SQL> select * from e;
+++-+-+
| op | ip | age | proctime |
+++-+-+
| +I | 10.10.10.10 | 40 | 2023-11-15 16:12:57.553 |
| +I | 10.10.10.10 | 50 | 2023-11-15 16:12:57.554 |
Flink SQL> SELECT * FROM e left join mariadb_catalog.menagerie.d FOR
SYSTEM_TIME AS OF e.proctime on d.type = 2 and d.age = 50 and e.ip = d.ip;
+++-+-++-+-+
| op | ip | age | proctime |
ip0 | type | age0 |
+++-+-++-+-+
| +I | 10.10.10.10 | 40 | 2023-11-15 16:08:40.973 |
10.10.10.10 | 2 | 50 |
| +I | 10.10.10.10 | 50 | 2023-11-15 16:08:40.974 |
10.10.10.10 | 2 | 50 |
Is this what you would expect?
Also I get
SELECT * FROM e left join mariadb_catalog.menagerie.d FOR SYSTEM_TIME AS OF
e.proctime on e.age = d.age;
+++-+-++-+-+
| op | ip | age | proctime |
ip0 | type | age0 |
+++-+-++-+-+
^CQuery terminated, received a total of 0 row
and
SELECT * FROM e left join mariadb_catalog.menagerie.d FOR SYSTEM_TIME AS OF
e.proctime on e.age = d.age and d.ip = e.ip;
+++-+-++-+-+
| op | ip | age | proctime |
ip0 | type | age0 |
+++-+-++-+-+
^CQuery terminated, received a total of 0 row
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Assignee: david radley
>Priority: Critical
> Attachments: flink-connector-jdbc-3.0.0-1.16.png,
>
[jira] [Commented] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786355#comment-17786355
]
david radley commented on FLINK-33365:
--
[~libenchao] Thanks for the quick feedback . I agree, I was thinking the second
point you make will make it more in line with the scan logic, where it appends
to the select statement with the where clauses. I was not aware of the
_PreparedStatement_ thanks for pointing me to it :)
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Assignee: david radley
>Priority: Critical
> Attachments: flink-connector-jdbc-3.0.0-1.16.png,
> flink-connector-jdbc-3.1.1-1.17.png
>
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[ip=ip], select=[ip, proctime, ip, CAST(0 AS INTEGER) AS type, CAST(ip
> AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]){code}
>
> excute same sql in sql-client with Flink 1.17.1/ 1.18.0 and
> *flink-connector-jdbc-3.0.0-1.16.jar*
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[type=0, ip=ip], where=[(type = 0)], select=[ip, proctime, ip, CAST(0
> AS INTEGER) AS type, CAST(ip AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]) {code}
> with flink-connector-jdbc-3.1.1-1.17.jar, the condition is
> *lookup=[ip=ip]*
> with flink-connector-jdbc-3.0.0-1.16.jar , the condition is
> *lookup=[type=0, ip=ip], where=[(type = 0)]*
>
> In out real world production environment, this lead incorrect data output
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17785967#comment-17785967
]
david radley edited comment on FLINK-33365 at 11/14/23 4:50 PM:
[~libenchao] and [~qingwei91] I have had a play and have some code that is
working. I have put it in a draft pr
[https://github.com/apache/flink-connector-jdbc/pull/79]
Could you feedback on it please?
My thinking behind the change:
The issue here is that the filters are not honoured in the loop up join when
there are push down predicates.
The lookup processing has access to the lookup keys as well as the predicate
pushdowns, the predicate push down information is parameterized.
The existing getSelectStatement takes as it last parameter conditions, it takes
the lookupkeys and creates an equality conditions in AbstractDialect like this
_String fieldExpressions =_
_Arrays.stream(conditionFields)_
_.map(f -> format("%s = :%s", quoteIdentifier(f), f))_
_.collect(Collectors.joining(" AND "));_
This is not appropriate for a condition like a.type = 0 containing a literal
and potentially not being an equals condition.
The predicate pushdown is currently only is implemented for simple
expressions.
The idea of the change is to resolve the conditions completely in the lookup
code and pass an array of Strings to a new overloaded implementation of
getSelectStatement, so we can add the appropriate conditions as a new
parameter. Note that the conditions for non-predicate pushdown would end up as
filters in the existing select conditions.
I do not think the pr is ready yet, but I would love to hear your feedback. I
intend to:
* to do more testing
** multiple conditions including non equalities
** multiple lookup keys
* ideally add test cases along the lines that have been suggested by [~Sergey
Nuyanzin]
* look into whether I can use existing Flink classes to resolve the condition
insert (I have used a replace); I see there is a
ResolvedExpression class I might be able to use.
* Any ideas on other tests I could try to break this?
* We could do the resolution of the String inserts in the getSelectStatement -
but this would involved passing both the predicate and the parameter in rather
than resolving in the caller. I think resolving it in the caller makes for a
simpler call.
* Currently there is no specific logic to account for partitions (which the
scan logic does), we may need to do something for partitions here.
was (Author: JIRAUSER300523):
[~libenchao] and [~qingwei91] I have had a play and have some code that is
working. I have put it in a draft pr
[https://github.com/apache/flink-connector-jdbc/pull/79|https://github.com/apache/flink-connector-jdbc/pull/79.]
Could you feedback on it please?
My thinking behind the change:
The issue here is that the filters are not honoured in the loop up join when
there are push down predicates.
The lookup processing has access to the lookup keys as well as the predicate
pushdowns, the predicate push down information is parameterized.
The existing getSelectStatement takes as it last parameter conditions, it takes
the lookupkeys and creates an equality conditions in AbstractDialect like this
_String fieldExpressions =_
_Arrays.stream(conditionFields)_
_.map(f -> format("%s = :%s", quoteIdentifier(f), f))_
_.collect(Collectors.joining(" AND "));_
This is not appropriate for a condition like a.type = 0 containing a literal
and potentially not being an equals condition.
The predicate pushdown is currently only is implemented for simple
expressions.
The idea of the change is to resolve the conditions completely in the lookup
code and pass an array of Strings to a new overloaded implementation of
getSelectStatement, so we can add the appropriate conditions as a new
parameter. Note that the conditions for non-predicate pushdown would end up as
filters in the existing select conditions.
I do not think the pr is ready yet, but I would love to hear your feedback. I
intend to:
* to do more testing
** multiple conditions including non equalities
** multiple lookup keys
* ideally add test cases along the lines that have been suggested by [~Sergey
Nuyanzin]
* look into whether I can use existing Flink classes to resolve the condition
insert (I have used a replace); I see there is a
ResolvedExpression class I might be able to use.
* Any ideas on other tests I could try to break this?
* We could do the resolution of the String inserts in the getSelectStatement -
but this would involved passing both the predicate and the parameter in rather
than resolving in the caller. I think resolving it in the caller makes for a
simpler call.
* Currently there is no specific logic to account for partitions (which the
scan logic does), we may need to do something for partitions here.
> Missing filter
[jira] [Comment Edited] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17785967#comment-17785967
]
david radley edited comment on FLINK-33365 at 11/14/23 4:49 PM:
[~libenchao] and [~qingwei91] I have had a play and have some code that is
working. I have put it in a draft pr
[https://github.com/apache/flink-connector-jdbc/pull/79|https://github.com/apache/flink-connector-jdbc/pull/79.]
Could you feedback on it please?
My thinking behind the change:
The issue here is that the filters are not honoured in the loop up join when
there are push down predicates.
The lookup processing has access to the lookup keys as well as the predicate
pushdowns, the predicate push down information is parameterized.
The existing getSelectStatement takes as it last parameter conditions, it takes
the lookupkeys and creates an equality conditions in AbstractDialect like this
_String fieldExpressions =_
_Arrays.stream(conditionFields)_
_.map(f -> format("%s = :%s", quoteIdentifier(f), f))_
_.collect(Collectors.joining(" AND "));_
This is not appropriate for a condition like a.type = 0 containing a literal
and potentially not being an equals condition.
The predicate pushdown is currently only is implemented for simple
expressions.
The idea of the change is to resolve the conditions completely in the lookup
code and pass an array of Strings to a new overloaded implementation of
getSelectStatement, so we can add the appropriate conditions as a new
parameter. Note that the conditions for non-predicate pushdown would end up as
filters in the existing select conditions.
I do not think the pr is ready yet, but I would love to hear your feedback. I
intend to:
* to do more testing
** multiple conditions including non equalities
** multiple lookup keys
* ideally add test cases along the lines that have been suggested by [~Sergey
Nuyanzin]
* look into whether I can use existing Flink classes to resolve the condition
insert (I have used a replace); I see there is a
ResolvedExpression class I might be able to use.
* Any ideas on other tests I could try to break this?
* We could do the resolution of the String inserts in the getSelectStatement -
but this would involved passing both the predicate and the parameter in rather
than resolving in the caller. I think resolving it in the caller makes for a
simpler call.
* Currently there is no specific logic to account for partitions (which the
scan logic does), we may need to do something for partitions here.
was (Author: JIRAUSER300523):
[~libenchao] and [~qingwei91] I have had a play and have some code that is
working. I have put it in a draft pr
[https://github.com/apache/flink-connector-jdbc/pull/79.]
Could you feedback on it please?
My thinking behind the change:
The issue here is that the filters are not honoured in the loop up join when
there are push down predicates.
The lookup processing has access to the lookup keys as well as the predicate
pushdowns, the predicate push down information is parameterized.
The existing getSelectStatement takes as it last parameter conditions, it takes
the lookupkeys and creates an equality conditions in AbstractDialect like this
_String fieldExpressions =_
_Arrays.stream(conditionFields)_
_.map(f -> format("%s = :%s", quoteIdentifier(f), f))_
_.collect(Collectors.joining(" AND "));_
This is not appropriate for a condition like a.type = 0 containing a literal
and potentially not being an equals condition.
The predicate pushdown is currently only is implemented for simple
expressions.
The idea of the change is to resolve the conditions completely in the lookup
code and pass an array of Strings to a new overloaded implementation of
getSelectStatement, so we can add the appropriate conditions as a new
parameter. Note that the conditions for non-predicate pushdown would end up as
filters in the existing select conditions.
I do not think the pr is ready yet, but I would love to hear your feedback. I
intend to:
* to do more testing
** multiple conditions including non equalities
** multiple lookup keys
* ideally add test cases along the lines that have been suggested by [~Sergey
Nuyanzin]
* look into whether I can use existing Flink classes to resolve the condition
insert (I have used a replace); I see there is a
ResolvedExpression class I might be able to use.
* Any ideas on other tests I could try to break this?
* We could do the resolution of the String inserts in the getSelectStatement -
but this would involved passing both the predicate and the parameter in rather
than resolving in the caller. I think resolving it in the caller makes for a
simpler call.
* Currently there is no specific logic to account for partitions (which the
scan logic does), we may need to do something for partitions here.
> Missing filter
[jira] [Comment Edited] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17785967#comment-17785967
]
david radley edited comment on FLINK-33365 at 11/14/23 4:33 PM:
[~libenchao] and [~qingwei91] I have had a play and have some code that is
working. I have put it in a draft pr
[https://github.com/apache/flink-connector-jdbc/pull/79.]
Could you feedback on it please?
My thinking behind the change:
The issue here is that the filters are not honoured in the loop up join when
there are push down predicates.
The lookup processing has access to the lookup keys as well as the predicate
pushdowns, the predicate push down information is parameterized.
The existing getSelectStatement takes as it last parameter conditions, it takes
the lookupkeys and creates an equality conditions in AbstractDialect like this
_String fieldExpressions =_
_Arrays.stream(conditionFields)_
_.map(f -> format("%s = :%s", quoteIdentifier(f), f))_
_.collect(Collectors.joining(" AND "));_
This is not appropriate for a condition like a.type = 0 containing a literal
and potentially not being an equals condition.
The predicate pushdown is currently only is implemented for simple
expressions.
The idea of the change is to resolve the conditions completely in the lookup
code and pass an array of Strings to a new overloaded implementation of
getSelectStatement, so we can add the appropriate conditions as a new
parameter. Note that the conditions for non-predicate pushdown would end up as
filters in the existing select conditions.
I do not think the pr is ready yet, but I would love to hear your feedback. I
intend to:
* to do more testing
** multiple conditions including non equalities
** multiple lookup keys
* ideally add test cases along the lines that have been suggested by [~Sergey
Nuyanzin]
* look into whether I can use existing Flink classes to resolve the condition
insert (I have used a replace); I see there is a
ResolvedExpression class I might be able to use.
* Any ideas on other tests I could try to break this?
* We could do the resolution of the String inserts in the getSelectStatement -
but this would involved passing both the predicate and the parameter in rather
than resolving in the caller. I think resolving it in the caller makes for a
simpler call.
* Currently there is no specific logic to account for partitions (which the
scan logic does), we may need to do something for partitions here.
was (Author: JIRAUSER300523):
[~libenchao] and [~qingwei91] I have had a play and have some code that is
working. I have put it in a draft pr
[https://github.com/apache/flink-connector-jdbc/pull/79.]
Could you feedback on it please?
My thinking behind the change:
The issue here is that the filters are not honoured in the loop up join when
there are push down predicates.
The lookup processing has access to the lookup keys as well as the predicate
pushdowns, the predicate push down information is parameterized.
The existing getSelectStatement takes as it last parameter conditions, it takes
the lookupkeys and creates an equality conditions in AbstractDialect like this
_String fieldExpressions =_
_Arrays.stream(conditionFields)_
_.map(f -> format("%s = :%s", quoteIdentifier(f), f))_
_.collect(Collectors.joining(" AND "));_
This is not appropriate for a condition like a.type = 0 containing a literal
and potentially not being an equals condition.
The predicate pushdown is currently only is implemented for simple
expressions.
The idea of the change is to resolve the conditions completely in the lookup
code and pass an array of Strings to a new overloaded implementation of
getSelectStatement, so we can add the appropriate conditions as a new
parameter. Note that the conditions for non-predicate pushdown would end up as
filters in the existing select conditions.
I do not think the pr is ready yet, but I would love to hear your feedback. I
intend to:
* to do more testing
** multiple conditions including non equalities
** multiple lookup keys
* ideally add test cases along the lines that have been suggested by [~Sergey
Nuyanzin]
* look into whether I can use existing Flink classes to resolve the condition
insert (I have used a replace); I see there is a
ResolvedExpression class I might be able to use.
* Any ideas on other tests I could try to break this?
* We could do the resolution of the String inserts in the getSelectStatement -
but this would involved passing both the predicate and the parameter in rather
than resolving in the caller. I think resolving it in the caller makes for a
simpler call.
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key:
[jira] [Comment Edited] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17785967#comment-17785967
]
david radley edited comment on FLINK-33365 at 11/14/23 4:32 PM:
[~libenchao] and [~qingwei91] I have had a play and have some code that is
working. I have put it in a draft pr
[https://github.com/apache/flink-connector-jdbc/pull/79.]
Could you feedback on it please?
My thinking behind the change:
The issue here is that the filters are not honoured in the loop up join when
there are push down predicates.
The lookup processing has access to the lookup keys as well as the predicate
pushdowns, the predicate push down information is parameterized.
The existing getSelectStatement takes as it last parameter conditions, it takes
the lookupkeys and creates an equality conditions in AbstractDialect like this
_String fieldExpressions =_
_Arrays.stream(conditionFields)_
_.map(f -> format("%s = :%s", quoteIdentifier(f), f))_
_.collect(Collectors.joining(" AND "));_
This is not appropriate for a condition like a.type = 0 containing a literal
and potentially not being an equals condition.
The predicate pushdown is currently only is implemented for simple
expressions.
The idea of the change is to resolve the conditions completely in the lookup
code and pass an array of Strings to a new overloaded implementation of
getSelectStatement, so we can add the appropriate conditions as a new
parameter. Note that the conditions for non-predicate pushdown would end up as
filters in the existing select conditions.
I do not think the pr is ready yet, but I would love to hear your feedback. I
intend to:
* to do more testing
** multiple conditions including non equalities
** multiple lookup keys
* ideally add test cases along the lines that have been suggested by [~Sergey
Nuyanzin]
* look into whether I can use existing Flink classes to resolve the condition
insert (I have used a replace); I see there is a
ResolvedExpression class I might be able to use.
* Any ideas on other tests I could try to break this?
* We could do the resolution of the String inserts in the getSelectStatement -
but this would involved passing both the predicate and the parameter in rather
than resolving in the caller. I think resolving it in the caller makes for a
simpler call.
was (Author: JIRAUSER300523):
[~libenchao] and [~qingwei91] I have had a play and have some code that is
working. I have put it in a draft pr
[https://github.com/apache/flink-connector-jdbc/pull/79.]
Could you feedback on it please?
My thinking behind the change:
The issue here is that the filters are not honoured in the loop up join when
there are push down predicates.
The lookup processing has access to the lookup keys as well as the predicate
pushdowns, the predicate push down information is parameterized.
The existing getSelectStatement takes as it last parameter conditions, it takes
the lookupkeys and creates an equality conditions in AbstractDialect like this
_String fieldExpressions =_
_Arrays.stream(conditionFields)_
_.map(f -> format("%s = :%s", quoteIdentifier(f), f))_
_.collect(Collectors.joining(" AND "));_
This is not appropriate for a condition like a.type = 0 containing a literal
and potentially not being an equals condition.
The predicate pushdown is currently only is implemented for simple
expressions.
The idea of the change is to resolve the conditions completely in the lookup
code and pass an array of Strings to a new overloaded implementation of
getSelectStatement, so we can add the appropriate conditions as a new
parameter. Note that the conditions for non-predicate pushdown would end up as
filters in the existing select conditions.
I do not think the pr is ready yet, but I would love to hear your feedback. I
intend to:
* to do more testing
** multiple conditions including non equalities
** multiple lookup keys
* ideally add test cases along the lines that have been suggested by [~Sergey
Nuyanzin]
* look into whether I can use existing Flink classes to resolve the condition
insert (I have used a replace); I see there is a
ResolvedExpression class I might be able to use.
* Any ideas on other tests I could try to break this?
* We could do the resolution of the String inserts in the getSelectStatement -
but this would involved passing both the predicate and the parameter in rather
than resolving in the caller. I think resolving it in the caller makes for a
simpler call.
I do not think this is ready a pr yet as I would like to do more testing and
hopefully add tests.
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
>
[jira] [Comment Edited] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17785967#comment-17785967
]
david radley edited comment on FLINK-33365 at 11/14/23 4:31 PM:
[~libenchao] and [~qingwei91] I have had a play and have some code that is
working. I have put it in a draft pr
[https://github.com/apache/flink-connector-jdbc/pull/79.]
Could you feedback on it please?
My thinking behind the change:
The issue here is that the filters are not honoured in the loop up join when
there are push down predicates.
The lookup processing has access to the lookup keys as well as the predicate
pushdowns, the predicate push down information is parameterized.
The existing getSelectStatement takes as it last parameter conditions, it takes
the lookupkeys and creates an equality conditions in AbstractDialect like this
_String fieldExpressions =_
_Arrays.stream(conditionFields)_
_.map(f -> format("%s = :%s", quoteIdentifier(f), f))_
_.collect(Collectors.joining(" AND "));_
This is not appropriate for a condition like a.type = 0 containing a literal
and potentially not being an equals condition.
The predicate pushdown is currently only is implemented for simple
expressions.
The idea of the change is to resolve the conditions completely in the lookup
code and pass an array of Strings to a new overloaded implementation of
getSelectStatement, so we can add the appropriate conditions as a new
parameter. Note that the conditions for non-predicate pushdown would end up as
filters in the existing select conditions.
I do not think the pr is ready yet, but I would love to hear your feedback. I
intend to:
* to do more testing
** multiple conditions including non equalities
** multiple lookup keys
* ideally add test cases along the lines that have been suggested by [~Sergey
Nuyanzin]
* look into whether I can use existing Flink classes to resolve the condition
insert (I have used a replace); I see there is a
ResolvedExpression class I might be able to use.
* Any ideas on other tests I could try to break this?
* We could do the resolution of the String inserts in the getSelectStatement -
but this would involved passing both the predicate and the parameter in rather
than resolving in the caller. I think resolving it in the caller makes for a
simpler call.
I do not think this is ready a pr yet as I would like to do more testing and
hopefully add tests.
was (Author: JIRAUSER300523):
[~qingwei91] I have had a play and have some code that is working. I have put
it in a draft pr [https://github.com/apache/flink-connector-jdbc/pull/79.]
Could you feedback on it please?
My thinking behind the change:
The issue here is that the filters are not honoured in the loop up join when
there are push down predicates.
The lookup processing has access to the lookup keys as well as the predicate
pushdowns, the predicate push down information is parameterized.
The existing getSelectStatement takes as it last parameter conditions, it takes
the lookupkeys and creates an equality conditions in AbstractDialect like this
_String fieldExpressions =_
_Arrays.stream(conditionFields)_
_.map(f -> format("%s = :%s", quoteIdentifier(f), f))_
_.collect(Collectors.joining(" AND "));_
This is not appropriate for a condition like a.type = 0 containing a literal
and potentially not being an equals condition.
The predicate pushdown is currently only is implemented for simple
expressions.
The idea of the change is to resolve the conditions completely in the lookup
code and pass an array of Strings to a new overloaded implementation of
getSelectStatement, so we can add the appropriate conditions as a new
parameter. Note that the conditions for non-predicate pushdown would end up as
filters in the existing select conditions.
I do not think the pr is ready yet, but I would love to hear your feedback. I
intend to:
* to do more testing
** multiple conditions including non equalities
** multiple lookup keys
* ideally add test cases along the lines that have been suggested by [~Sergey
Nuyanzin]
* look into whether I can use existing Flink classes to resolve the condition
insert (I have used a replace); I see there is a
ResolvedExpression class I might be able to use.
* Any ideas on other tests I could try to break this?
* We could do the resolution of the String inserts in the getSelectStatement -
but this would involved passing both the predicate and the parameter in rather
than resolving in the caller. I think resolving it in the caller makes for a
simpler call.
I do not think this is ready a pr yet as I would like to do more testing and
hopefully add tests.
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
>
[jira] [Commented] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17785967#comment-17785967
]
david radley commented on FLINK-33365:
--
[~qingwei91] I have had a play and have some code that is working. I have put
it in a draft pr [https://github.com/apache/flink-connector-jdbc/pull/79.]
Could you feedback on it please?
My thinking behind the change:
The issue here is that the filters are not honoured in the loop up join when
there are push down predicates.
The lookup processing has access to the lookup keys as well as the predicate
pushdowns, the predicate push down information is parameterized.
The existing getSelectStatement takes as it last parameter conditions, it takes
the lookupkeys and creates an equality conditions in AbstractDialect like this
_String fieldExpressions =_
_Arrays.stream(conditionFields)_
_.map(f -> format("%s = :%s", quoteIdentifier(f), f))_
_.collect(Collectors.joining(" AND "));_
This is not appropriate for a condition like a.type = 0 containing a literal
and potentially not being an equals condition.
The predicate pushdown is currently only is implemented for simple
expressions.
The idea of the change is to resolve the conditions completely in the lookup
code and pass an array of Strings to a new overloaded implementation of
getSelectStatement, so we can add the appropriate conditions as a new
parameter. Note that the conditions for non-predicate pushdown would end up as
filters in the existing select conditions.
I do not think the pr is ready yet, but I would love to hear your feedback. I
intend to:
* to do more testing
** multiple conditions including non equalities
** multiple lookup keys
* ideally add test cases along the lines that have been suggested by [~Sergey
Nuyanzin]
* look into whether I can use existing Flink classes to resolve the condition
insert (I have used a replace); I see there is a
ResolvedExpression class I might be able to use.
* Any ideas on other tests I could try to break this?
* We could do the resolution of the String inserts in the getSelectStatement -
but this would involved passing both the predicate and the parameter in rather
than resolving in the caller. I think resolving it in the caller makes for a
simpler call.
I do not think this is ready a pr yet as I would like to do more testing and
hopefully add tests.
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Assignee: david radley
>Priority: Critical
> Attachments: flink-connector-jdbc-3.0.0-1.16.png,
> flink-connector-jdbc-3.1.1-1.17.png
>
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[ip=ip], select=[ip, proctime, ip, CAST(0 AS INTEGER) AS type, CAST(ip
> AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]){code}
>
> excute same sql in sql-client with Flink 1.17.1/ 1.18.0 and
> *flink-connector-jdbc-3.0.0-1.16.jar*
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[type=0, ip=ip], where=[(type = 0)], select=[ip, proctime, ip, CAST(0
> AS INTEGER) AS type, CAST(ip AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]) {code}
> with
[jira] [Commented] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17785813#comment-17785813
]
david radley commented on FLINK-33365:
--
[~qingwei91] thank you very much for you support :) . One question I have about
the design intent.
1) Would it be true to say that the Calcite rule's should dictate the general
shape of the query, I can see that there are rules around predicate pushdown
and lookups. Then the JDBC dialect would be minimal type mapping of the graph.
2) The solution we are considering, because the predicate pushdown logic was
done in the JDBC code, we are fixing up the query in the JDBC code to not lose
the conditions.
Would it be true to say the 2nd approach is more tactical and short term as
this is a blocker and the 1st approach would be more ideal? Or am I
misunderstanding?
Either way I will look into option 2), as a change of the rules is much more
impactful outside of JDBC.
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Assignee: david radley
>Priority: Critical
> Attachments: flink-connector-jdbc-3.0.0-1.16.png,
> flink-connector-jdbc-3.1.1-1.17.png
>
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[ip=ip], select=[ip, proctime, ip, CAST(0 AS INTEGER) AS type, CAST(ip
> AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]){code}
>
> excute same sql in sql-client with Flink 1.17.1/ 1.18.0 and
> *flink-connector-jdbc-3.0.0-1.16.jar*
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[type=0, ip=ip], where=[(type = 0)], select=[ip, proctime, ip, CAST(0
> AS INTEGER) AS type, CAST(ip AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]) {code}
> with flink-connector-jdbc-3.1.1-1.17.jar, the condition is
> *lookup=[ip=ip]*
> with flink-connector-jdbc-3.0.0-1.16.jar , the condition is
> *lookup=[type=0, ip=ip], where=[(type = 0)]*
>
> In out real world production environment, this lead incorrect data output
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17785563#comment-17785563
]
david radley edited comment on FLINK-33365 at 11/13/23 6:18 PM:
[~Sergey Nuyanzin] [~qingwei91] thanks for your input. The 2 circumventions do
not work for me on master, on the reported case.
It sounds like the introduction of pushdown predicate capability for jdbc,
broke this case. At the moment the JdbcRowDataLookupFunction just adds the
keyNames to the conditions
_this.query =_
_options.getDialect()_
_.getSelectFromStatement(options.getTableName(), fieldNames, keyNames);_
I was hoping to contribute this as a fix, as I have been assigned the issue;
[~qingwei91] is this still possible- or do you plan to fix this? I see that
function getLookupRuntimeProvider does have access to this.resolvedPredicates
and pushdownParams it would be great if we could use these to create the
condition to add to the keyNames.
I would be great to hear your thoughts on this.
was (Author: JIRAUSER300523):
[~Sergey Nuyanzin] [~qingwei91] thanks for your input. The 2 circumventions do
not work for me on master, on the reported case.
It sounds like the introduction of pushdown predicate capability for jdbc,
broke this case. At the moment the JdbcRowDataLookupFunction just adds the
keyNames to the conditions
_this.query =_
_options.getDialect()_
_.getSelectFromStatement(options.getTableName(), fieldNames, keyNames);_
I was hoping to contribute this as a fix, as I have been assigned the issue;
[~qingwei91] is this still possible- or do you plan to fix this? I see that
function getLookupRuntimeProvider does have access to this.resolvedPredicates
and pushdownParams it would be great if we could use these to create the
condition to add to the keyNames.
It would be great to hear your thoughts on this.
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Assignee: david radley
>Priority: Critical
> Attachments: flink-connector-jdbc-3.0.0-1.16.png,
> flink-connector-jdbc-3.1.1-1.17.png
>
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[ip=ip], select=[ip, proctime, ip, CAST(0 AS INTEGER) AS type, CAST(ip
> AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]){code}
>
> excute same sql in sql-client with Flink 1.17.1/ 1.18.0 and
> *flink-connector-jdbc-3.0.0-1.16.jar*
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[type=0, ip=ip], where=[(type = 0)], select=[ip, proctime, ip, CAST(0
> AS INTEGER) AS type, CAST(ip AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]) {code}
> with flink-connector-jdbc-3.1.1-1.17.jar, the condition is
> *lookup=[ip=ip]*
> with flink-connector-jdbc-3.0.0-1.16.jar , the condition is
> *lookup=[type=0, ip=ip], where=[(type = 0)]*
>
> In out real world production environment, this lead incorrect data output
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17785563#comment-17785563
]
david radley edited comment on FLINK-33365 at 11/13/23 6:18 PM:
[~Sergey Nuyanzin] [~qingwei91] thanks for your input. The 2 circumventions do
not work for me on master, on the reported case.
It sounds like the introduction of pushdown predicate capability for jdbc,
broke this case. At the moment the JdbcRowDataLookupFunction just adds the
keyNames to the conditions
_this.query =_
_options.getDialect()_
_.getSelectFromStatement(options.getTableName(), fieldNames, keyNames);_
I was hoping to contribute this as a fix, as I have been assigned the issue;
[~qingwei91] is this still possible- or do you plan to fix this? I see that
function getLookupRuntimeProvider does have access to this.resolvedPredicates
and pushdownParams it would be great if we could use these to create the
condition to add to the keyNames.
It would be great to hear your thoughts on this.
was (Author: JIRAUSER300523):
[~Sergey Nuyanzin] [~qingwei91] thanks for your input. The 2 circumventions do
not work for me on master, on the reported case.
It sounds like the introduction of pushdown predicate capability for jdbc,
broke this case. At the moment the
JdbcRowDataLookupFunction just adds the keyNames to the conditions
_this.query =_
_options.getDialect()_
_.getSelectFromStatement(options.getTableName(), fieldNames, keyNames);_
It seems to me that we need to save the filter information when we do the
pushdown logic, so we can add this as a condition to the last parameter. I was
hoping to contribute this as a fix, as I have been assigned the issue;
[~qingwei91] is this still possible- or do you plan to fix this? It would be
great to hear your thoughts on this.
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Assignee: david radley
>Priority: Critical
> Attachments: flink-connector-jdbc-3.0.0-1.16.png,
> flink-connector-jdbc-3.1.1-1.17.png
>
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[ip=ip], select=[ip, proctime, ip, CAST(0 AS INTEGER) AS type, CAST(ip
> AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]){code}
>
> excute same sql in sql-client with Flink 1.17.1/ 1.18.0 and
> *flink-connector-jdbc-3.0.0-1.16.jar*
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[type=0, ip=ip], where=[(type = 0)], select=[ip, proctime, ip, CAST(0
> AS INTEGER) AS type, CAST(ip AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]) {code}
> with flink-connector-jdbc-3.1.1-1.17.jar, the condition is
> *lookup=[ip=ip]*
> with flink-connector-jdbc-3.0.0-1.16.jar , the condition is
> *lookup=[type=0, ip=ip], where=[(type = 0)]*
>
> In out real world production environment, this lead incorrect data output
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17785563#comment-17785563
]
david radley edited comment on FLINK-33365 at 11/13/23 6:13 PM:
[~Sergey Nuyanzin] [~qingwei91] thanks for your input. The 2 circumventions do
not work for me on master, on the reported case.
It sounds like the introduction of pushdown predicate capability for jdbc,
broke this case. At the moment the
JdbcRowDataLookupFunction just adds the keyNames to the conditions
_this.query =_
_options.getDialect()_
_.getSelectFromStatement(options.getTableName(), fieldNames, keyNames);_
It seems to me that we need to save the filter information when we do the
pushdown logic, so we can add this as a condition to the last parameter. I was
hoping to contribute this as a fix, as I have been assigned the issue;
[~qingwei91] is this still possible- or do you plan to fix this? It would be
great to hear your thoughts on this.
was (Author: JIRAUSER300523):
[~Sergey Nuyanzin] [~qingwei91] thanks for your input. The 2 circumventions do
not work for me on master, on the reported case.
It sounds like the introduction of pushdown predicate capability for jdbc,
broke this case. At the moment the
JdbcRowDataLookupFunction just adds the keyNames to the conditions
_this.query =_
_options.getDialect()_
_.getSelectFromStatement(options.getTableName(), fieldNames, keyNames);_
It seems to me that we need to save the filter information when we do the
pushdown logic, so we can add this as a condition to the last parameter. I was
hoping to contribute this as a fix, as I have been assigned the issue;
[~qingwei91] is this still possible- or do you plan to fix this? What do you
think of this approach?
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Assignee: david radley
>Priority: Critical
> Attachments: flink-connector-jdbc-3.0.0-1.16.png,
> flink-connector-jdbc-3.1.1-1.17.png
>
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[ip=ip], select=[ip, proctime, ip, CAST(0 AS INTEGER) AS type, CAST(ip
> AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]){code}
>
> excute same sql in sql-client with Flink 1.17.1/ 1.18.0 and
> *flink-connector-jdbc-3.0.0-1.16.jar*
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[type=0, ip=ip], where=[(type = 0)], select=[ip, proctime, ip, CAST(0
> AS INTEGER) AS type, CAST(ip AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]) {code}
> with flink-connector-jdbc-3.1.1-1.17.jar, the condition is
> *lookup=[ip=ip]*
> with flink-connector-jdbc-3.0.0-1.16.jar , the condition is
> *lookup=[type=0, ip=ip], where=[(type = 0)]*
>
> In out real world production environment, this lead incorrect data output
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17785563#comment-17785563
]
david radley edited comment on FLINK-33365 at 11/13/23 5:08 PM:
[~Sergey Nuyanzin] [~qingwei91] thanks for your input. The 2 circumventions do
not work for me on master, on the reported case.
It sounds like the introduction of pushdown predicate capability for jdbc,
broke this case. At the moment the
JdbcRowDataLookupFunction just adds the keyNames to the conditions
_this.query =_
_options.getDialect()_
_.getSelectFromStatement(options.getTableName(), fieldNames, keyNames);_
It seems to me that we need to save the filter information when we do the
pushdown logic, so we can add this as a condition to the last parameter. I was
hoping to contribute this as a fix, as I have been assigned the issue;
[~qingwei91] is this still possible- or do you plan to fix this? What do you
think of this approach?
was (Author: JIRAUSER300523):
[~Sergey Nuyanzin] [~qingwei91] thanks for your input. The 2 circumventions do
not work for me on master, on the reported case.
It sounds like the introduction of pushdown predicate capability for jdbc,
broke this case. At the moment the
JdbcRowDataLookupFunction just adds the keyNames to the conditions
_this.query =_
_options.getDialect()_
_.getSelectFromStatement(options.getTableName(), fieldNames, keyNames);_
It seems to me that we need to include the filter information into the context
when we do the pushdown logic, so we can add this as a condition to the last
parameter. I was hoping to contribute this as a fix, as I have been assigned
the issue; [~qingwei91] is this still possible- or do you plan to fix this?
What do you think of this approach?
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Assignee: david radley
>Priority: Critical
> Attachments: flink-connector-jdbc-3.0.0-1.16.png,
> flink-connector-jdbc-3.1.1-1.17.png
>
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[ip=ip], select=[ip, proctime, ip, CAST(0 AS INTEGER) AS type, CAST(ip
> AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]){code}
>
> excute same sql in sql-client with Flink 1.17.1/ 1.18.0 and
> *flink-connector-jdbc-3.0.0-1.16.jar*
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[type=0, ip=ip], where=[(type = 0)], select=[ip, proctime, ip, CAST(0
> AS INTEGER) AS type, CAST(ip AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]) {code}
> with flink-connector-jdbc-3.1.1-1.17.jar, the condition is
> *lookup=[ip=ip]*
> with flink-connector-jdbc-3.0.0-1.16.jar , the condition is
> *lookup=[type=0, ip=ip], where=[(type = 0)]*
>
> In out real world production environment, this lead incorrect data output
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17785563#comment-17785563
]
david radley edited comment on FLINK-33365 at 11/13/23 4:45 PM:
[~Sergey Nuyanzin] [~qingwei91] thanks for your input. The 2 circumventions do
not work for me on master, on the reported case.
It sounds like the introduction of pushdown predicate capability for jdbc,
broke this case. At the moment the
JdbcRowDataLookupFunction just adds the keyNames to the conditions
_this.query =_
_options.getDialect()_
_.getSelectFromStatement(options.getTableName(), fieldNames, keyNames);_
It seems to me that we need to include the filter information into the context
when we do the pushdown logic, so we can add this as a condition to the last
parameter. I was hoping to contribute this as a fix, as I have been assigned
the issue; [~qingwei91] is this still possible- or do you plan to fix this?
What do you think of this approach?
was (Author: JIRAUSER300523):
[~Sergey Nuyanzin] [~qingwei91] thanks for you input. The 2 circumventions do
not work for me on master, on the reported case.
It sounds like the introduction of pushdown predicate capability for jdbc,
broke this case. At the moment the
JdbcRowDataLookupFunction just adds the keyNames to the conditions
_this.query =_
_options.getDialect()_
_.getSelectFromStatement(options.getTableName(), fieldNames, keyNames);_
It seems to me that we need to include the filter information into the context
when we do the pushdown logic, so we can add this as a condition to the last
parameter. I was hoping to contribute this as a fix, as I have been assigned
the issue; [~qingwei91] is this still possible- or do you plan to fix this?
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Assignee: david radley
>Priority: Critical
> Attachments: flink-connector-jdbc-3.0.0-1.16.png,
> flink-connector-jdbc-3.1.1-1.17.png
>
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[ip=ip], select=[ip, proctime, ip, CAST(0 AS INTEGER) AS type, CAST(ip
> AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]){code}
>
> excute same sql in sql-client with Flink 1.17.1/ 1.18.0 and
> *flink-connector-jdbc-3.0.0-1.16.jar*
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[type=0, ip=ip], where=[(type = 0)], select=[ip, proctime, ip, CAST(0
> AS INTEGER) AS type, CAST(ip AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]) {code}
> with flink-connector-jdbc-3.1.1-1.17.jar, the condition is
> *lookup=[ip=ip]*
> with flink-connector-jdbc-3.0.0-1.16.jar , the condition is
> *lookup=[type=0, ip=ip], where=[(type = 0)]*
>
> In out real world production environment, this lead incorrect data output
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17785563#comment-17785563
]
david radley commented on FLINK-33365:
--
[~Sergey Nuyanzin] [~qingwei91] thanks for you input. The 2 circumventions do
not work for me on master, on the reported case.
It sounds like the introduction of pushdown predicate capability for jdbc,
broke this case. At the moment the
JdbcRowDataLookupFunction just adds the keyNames to the conditions
_this.query =_
_options.getDialect()_
_.getSelectFromStatement(options.getTableName(), fieldNames, keyNames);_
It seems to me that we need to include the filter information into the context
when we do the pushdown logic, so we can add this as a condition to the last
parameter. I was hoping to contribute this as a fix, as I have been assigned
the issue; [~qingwei91] is this still possible- or do you plan to fix this?
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Assignee: david radley
>Priority: Critical
> Attachments: flink-connector-jdbc-3.0.0-1.16.png,
> flink-connector-jdbc-3.1.1-1.17.png
>
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[ip=ip], select=[ip, proctime, ip, CAST(0 AS INTEGER) AS type, CAST(ip
> AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]){code}
>
> excute same sql in sql-client with Flink 1.17.1/ 1.18.0 and
> *flink-connector-jdbc-3.0.0-1.16.jar*
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[type=0, ip=ip], where=[(type = 0)], select=[ip, proctime, ip, CAST(0
> AS INTEGER) AS type, CAST(ip AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]) {code}
> with flink-connector-jdbc-3.1.1-1.17.jar, the condition is
> *lookup=[ip=ip]*
> with flink-connector-jdbc-3.0.0-1.16.jar , the condition is
> *lookup=[type=0, ip=ip], where=[(type = 0)]*
>
> In out real world production environment, this lead incorrect data output
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17785031#comment-17785031
]
david radley edited comment on FLINK-33365 at 11/10/23 6:18 PM:
I am not sure about the design intent in this area. Should the pushed down
predicate be in the lookup keys as the java doc for
([https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/nodes/physical/common/CommonPhysicalLookupJoin.scala)]
Indicates. Or should it just be honoured as a filter. If it is honoured a
filter, then does the dialect need to do anything special for the pushdown
predicates, like it does for the scan case.
I am thinking the lookup processing should honour the pushed down predicates
and they should not be in the lookup keys (unless the source does not support
predicate push down). There is no current way I can see for the dialect to add
in these predicates, which has been optimized in the Calcite rules processing.
Maybe a change to the lookup processing (probably in
[CommonPhysicalLookupJoin|https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/nodes/physical/common/CommonPhysicalLookupJoin.scala)])
to honour the table source resolvedPredicates and pushed down params, is
required.
I am new to this area and would appreciate the communities thoughts on whether
I am on the right track.
WDYT?
was (Author: JIRAUSER300523):
I am not sure about the design intent in this area. Should the pushed down
predicate be in the lookup keys as the java doc for
([https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/nodes/physical/common/CommonPhysicalLookupJoin.scala)]
Indicates. Or should it just be honoured as a filter. If it is honoured a
filter, then does the dialect need to do anything special for the pushdown
predicates, like it does for the scan case.
I think the lookup processing should honour the pushed down predicates and they
should not be in the lookup keys (unless the source does not support predicate
push down). There is no current way I can see for the dialect to add in these
predicates, which has been optimized in the Calcite rules processing.
So I think a change to the lookup processing (probably in
[CommonPhysicalLookupJoin|https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/nodes/physical/common/CommonPhysicalLookupJoin.scala)])
to honour the table source resolvedPredicates and pushed down params, is
required.
WDYT?
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Assignee: david radley
>Priority: Critical
> Attachments: flink-connector-jdbc-3.0.0-1.16.png,
> flink-connector-jdbc-3.1.1-1.17.png
>
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[ip=ip], select=[ip, proctime, ip, CAST(0 AS INTEGER) AS type, CAST(ip
> AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]){code}
>
> excute same sql in sql-client with Flink 1.17.1/ 1.18.0 and
> *flink-connector-jdbc-3.0.0-1.16.jar*
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[type=0, ip=ip],
[jira] [Commented] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17785031#comment-17785031
]
david radley commented on FLINK-33365:
--
I am not sure about the design intent in this area. Should the pushed down
predicate be in the lookup keys as the java doc for
([https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/nodes/physical/common/CommonPhysicalLookupJoin.scala)]
Indicates. Or should it just be honoured as a filter. If it is honoured a
filter, then does the dialect need to do anything special for the pushdown
predicates, like it does for the scan case.
I think the lookup processing should honour the pushed down predicates and they
should not be in the lookup keys (unless the source does not support predicate
push down). There is no current way I can see for the dialect to add in these
predicates, which has been optimized in the Calcite rules processing.
So I think a change to the lookup processing (probably in
[CommonPhysicalLookupJoin|https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/nodes/physical/common/CommonPhysicalLookupJoin.scala)])
to honour the table source resolvedPredicates and pushed down params, is
required.
WDYT?
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Assignee: david radley
>Priority: Critical
> Attachments: flink-connector-jdbc-3.0.0-1.16.png,
> flink-connector-jdbc-3.1.1-1.17.png
>
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[ip=ip], select=[ip, proctime, ip, CAST(0 AS INTEGER) AS type, CAST(ip
> AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]){code}
>
> excute same sql in sql-client with Flink 1.17.1/ 1.18.0 and
> *flink-connector-jdbc-3.0.0-1.16.jar*
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[type=0, ip=ip], where=[(type = 0)], select=[ip, proctime, ip, CAST(0
> AS INTEGER) AS type, CAST(ip AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]) {code}
> with flink-connector-jdbc-3.1.1-1.17.jar, the condition is
> *lookup=[ip=ip]*
> with flink-connector-jdbc-3.0.0-1.16.jar , the condition is
> *lookup=[type=0, ip=ip], where=[(type = 0)]*
>
> In out real world production environment, this lead incorrect data output
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[ https://issues.apache.org/jira/browse/FLINK-33365 ]
david radley deleted comment on FLINK-33365:
--
was (Author: JIRAUSER300523):
Testing against MariaDB I see:
select CAST(0 AS INT) as type from c;
+--+
|type|
+--+
| 0|
| 0|
+--+
Just confirming the cast is not a filter.
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Assignee: david radley
>Priority: Critical
> Attachments: flink-connector-jdbc-3.0.0-1.16.png,
> flink-connector-jdbc-3.1.1-1.17.png
>
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[ip=ip], select=[ip, proctime, ip, CAST(0 AS INTEGER) AS type, CAST(ip
> AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]){code}
>
> excute same sql in sql-client with Flink 1.17.1/ 1.18.0 and
> *flink-connector-jdbc-3.0.0-1.16.jar*
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[type=0, ip=ip], where=[(type = 0)], select=[ip, proctime, ip, CAST(0
> AS INTEGER) AS type, CAST(ip AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]) {code}
> with flink-connector-jdbc-3.1.1-1.17.jar, the condition is
> *lookup=[ip=ip]*
> with flink-connector-jdbc-3.0.0-1.16.jar , the condition is
> *lookup=[type=0, ip=ip], where=[(type = 0)]*
>
> In out real world production environment, this lead incorrect data output
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17784915#comment-17784915
]
david radley edited comment on FLINK-33365 at 11/10/23 4:33 PM:
an update on what I have found:
I have switched on DEBUG put out the rules that are being driven for my
recreation. I see :
org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram [] -
optimize time_indicator cost 1 ms.
optimize result:
FlinkLogicalSink(table=[*anonymous_collect$4*], fields=[ip, proctime, ip0,
type])
+- FlinkLogicalCalc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime,
ip0, type])
+- FlinkLogicalJoin(condition=[=($0, $4)], joinType=[left])
:- FlinkLogicalCalc(select=[ip, PROCTIME() AS proctime])
: +- FlinkLogicalTableSourceScan(table=[[paimon_catalog, default, a]],
fields=[ip])
+- FlinkLogicalSnapshot(period=[$cor0.proctime])
+- FlinkLogicalCalc(select=[ip, CAST(0 AS INTEGER) AS type, CAST(ip AS
VARCHAR(2147483647) CHARACTER SET "UTF-16LE") AS ip0])
+- FlinkLogicalTableSourceScan(table=[[mariadb_catalog, menagerie,
c, {*}filter=[=(type, 0)]]]{*}, fields=[ip, type])
Is removed in the next stage.
org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram [] -
optimize physical cost 3 ms.
optimize result:
Sink(table=[*anonymous_collect$4*], fields=[ip, proctime, ip0, type])
+- Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
+- LookupJoin(table=[mariadb_catalog.menagerie.c], joinType=[LeftOuterJoin],
lookup=[ip=ip], select=[ip, proctime, ip, *CAST(0 AS INTEGER)* AS type, CAST(ip
AS VARCHAR(2147483647) CHARACTER SET "UTF-16LE") AS ip0])
+- Calc(select=[ip, PROCTIME() AS proctime])
+- TableSourceScan(table=[[paimon_catalog, default, a]], fields=[ip])
The *CAST(0 AS INTEGER)* is in the final Optimized Execution Plan we see in the
explain This cast is fine as long as the filter is there.
I am not an expert at this, but the comments at the top of
[CommonPhysicalLookupJoin.scala]([https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/nodes/physical/common/CommonPhysicalLookupJoin.scala)]
are correct and this filter should actually be in the lookup keys. The
comments says
_* For a lookup join query:_
_*_
_* SELECT T.id, T.content, D.age FROM T JOIN userTable FOR SYSTEM_TIME AS
OF T.proctime AS D_
_* ON T.content = concat(D.name, '!') AND D.age = 11 AND T.id = D.id WHERE
D.name LIKE 'Jack%'_
_* _
_*_
_* The LookupJoin physical node encapsulates the following RelNode tree:_
_*_
_* Join (l.name = r.name) / \ RelNode Calc (concat(name, "!") as name,
name LIKE 'Jack%') |_
_* DimTable (lookup-keys: age=11, id=l.id) (age, id, name) _
_*_
_* The important member fields in LookupJoin: allLookupKeys: [$0=11,
$1=l.id] ($0 and $1 is_
_* the indexes of age and id in dim table) remainingCondition:
l.name=r.name _
_*_
_* The workflow of lookup join:_
_*_
_* 1) lookup records dimension table using the lookup-keys 2) project &
filter on the lookup-ed_
_* records 3) join left input record and lookup-ed records 4) only
outputs the rows which_
_* match to the remainingCondition _
I would have thought that a filter on a value on one of the sources could be
pushed down a a filter to that source, rather than being in the lookup keys.
Maybe that could be a subsequent optimization. We see this in:
FlinkLogicalTableSourceScan(table=[[mariadb_catalog, menagerie, c,
filter=[=(type, 0)]]], fields=[ip, type])
but this is then lost.
It seems the table scan with the pushed down filter is being ignored during the
the construction of the LookupJoin representation
was (Author: JIRAUSER300523):
an update on what I have found:
I have switched on DEBUG put out the rules that are being driven for my
recreation. I see :
org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram [] -
optimize time_indicator cost 1 ms.
optimize result:
FlinkLogicalSink(table=[*anonymous_collect$4*], fields=[ip, proctime, ip0,
type])
+- FlinkLogicalCalc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime,
ip0, type])
+- FlinkLogicalJoin(condition=[=($0, $4)], joinType=[left])
:- FlinkLogicalCalc(select=[ip, PROCTIME() AS proctime])
: +- FlinkLogicalTableSourceScan(table=[[paimon_catalog, default, a]],
fields=[ip])
+- FlinkLogicalSnapshot(period=[$cor0.proctime])
+- FlinkLogicalCalc(select=[ip, CAST(0 AS INTEGER) AS type, CAST(ip AS
VARCHAR(2147483647) CHARACTER SET "UTF-16LE") AS ip0])
+- FlinkLogicalTableSourceScan(table=[[mariadb_catalog, menagerie,
c, {*}filter=[=(type, 0)]]]{*}, fields=[ip, type])
Is removed in the next stage.
org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram [] -
[jira] [Comment Edited] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17784915#comment-17784915
]
david radley edited comment on FLINK-33365 at 11/10/23 3:08 PM:
an update on what I have found:
I have switched on DEBUG put out the rules that are being driven for my
recreation. I see :
org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram [] -
optimize time_indicator cost 1 ms.
optimize result:
FlinkLogicalSink(table=[*anonymous_collect$4*], fields=[ip, proctime, ip0,
type])
+- FlinkLogicalCalc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime,
ip0, type])
+- FlinkLogicalJoin(condition=[=($0, $4)], joinType=[left])
:- FlinkLogicalCalc(select=[ip, PROCTIME() AS proctime])
: +- FlinkLogicalTableSourceScan(table=[[paimon_catalog, default, a]],
fields=[ip])
+- FlinkLogicalSnapshot(period=[$cor0.proctime])
+- FlinkLogicalCalc(select=[ip, CAST(0 AS INTEGER) AS type, CAST(ip AS
VARCHAR(2147483647) CHARACTER SET "UTF-16LE") AS ip0])
+- FlinkLogicalTableSourceScan(table=[[mariadb_catalog, menagerie,
c, {*}filter=[=(type, 0)]]]{*}, fields=[ip, type])
Is removed in the next stage.
org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram [] -
optimize physical cost 3 ms.
optimize result:
Sink(table=[*anonymous_collect$4*], fields=[ip, proctime, ip0, type])
+- Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
+- LookupJoin(table=[mariadb_catalog.menagerie.c], joinType=[LeftOuterJoin],
lookup=[ip=ip], select=[ip, proctime, ip, *CAST(0 AS INTEGER)* AS type, CAST(ip
AS VARCHAR(2147483647) CHARACTER SET "UTF-16LE") AS ip0])
+- Calc(select=[ip, PROCTIME() AS proctime])
+- TableSourceScan(table=[[paimon_catalog, default, a]], fields=[ip])
The *CAST(0 AS INTEGER)* is in the final Optimized Execution Plan we see in the
explain This cast is fine as long as the filter is there.
I am not an expert at this, but the comments at the top of
[CommonPhysicalLookupJoin.scala]([https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/nodes/physical/common/CommonPhysicalLookupJoin.scala)]
are correct and this filter should actually be in the lookup keys. The
comments says
_* For a lookup join query:_
_*_
_* SELECT T.id, T.content, D.age FROM T JOIN userTable FOR SYSTEM_TIME AS
OF T.proctime AS D_
_* ON T.content = concat(D.name, '!') AND D.age = 11 AND T.id = D.id WHERE
D.name LIKE 'Jack%'_
_* _
_*_
_* The LookupJoin physical node encapsulates the following RelNode tree:_
_*_
_* Join (l.name = r.name) / \ RelNode Calc (concat(name, "!") as name,
name LIKE 'Jack%') |_
_* DimTable (lookup-keys: age=11, id=l.id) (age, id, name) _
_*_
_* The important member fields in LookupJoin: allLookupKeys: [$0=11,
$1=l.id] ($0 and $1 is_
_* the indexes of age and id in dim table) remainingCondition:
l.name=r.name _
_*_
_* The workflow of lookup join:_
_*_
_* 1) lookup records dimension table using the lookup-keys 2) project &
filter on the lookup-ed_
_* records 3) join left input record and lookup-ed records 4) only
outputs the rows which_
_* match to the remainingCondition _
I would have thought that a filter on a value on one of the sources could be
pushed down a a filter to that source, rather than being in the lookup keys.
Maybe that could be a subsequent optimization. We see this in:
FlinkLogicalTableSourceScan(table=[[mariadb_catalog, menagerie, c,
filter=[=(type, 0)]]], fields=[ip, type])
but this is then lost.
was (Author: JIRAUSER300523):
an update on what I have found:
I have switched on DEBUG put out the rules that are being driven for my
recreation. I see :
org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram [] -
optimize time_indicator cost 1 ms.
optimize result:
FlinkLogicalSink(table=[*anonymous_collect$4*], fields=[ip, proctime, ip0,
type])
+- FlinkLogicalCalc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime,
ip0, type])
+- FlinkLogicalJoin(condition=[=($0, $4)], joinType=[left])
:- FlinkLogicalCalc(select=[ip, PROCTIME() AS proctime])
: +- FlinkLogicalTableSourceScan(table=[[paimon_catalog, default, a]],
fields=[ip])
+- FlinkLogicalSnapshot(period=[$cor0.proctime])
+- FlinkLogicalCalc(select=[ip, CAST(0 AS INTEGER) AS type, CAST(ip AS
VARCHAR(2147483647) CHARACTER SET "UTF-16LE") AS ip0])
+- FlinkLogicalTableSourceScan(table=[[mariadb_catalog, menagerie,
c, {*}filter=[=(type, 0)]]]{*}, fields=[ip, type])
Is removed in the next stage.
org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram [] -
optimize physical cost 3 ms.
optimize result:
Sink(table=[*anonymous_collect$4*], fields=[ip, proctime, ip0, type])
+- Calc(select=[ip,
[jira] [Comment Edited] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17784915#comment-17784915
]
david radley edited comment on FLINK-33365 at 11/10/23 3:04 PM:
an update on what I have found:
I have switched on DEBUG put out the rules that are being driven for my
recreation. I see :
org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram [] -
optimize time_indicator cost 1 ms.
optimize result:
FlinkLogicalSink(table=[*anonymous_collect$4*], fields=[ip, proctime, ip0,
type])
+- FlinkLogicalCalc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime,
ip0, type])
+- FlinkLogicalJoin(condition=[=($0, $4)], joinType=[left])
:- FlinkLogicalCalc(select=[ip, PROCTIME() AS proctime])
: +- FlinkLogicalTableSourceScan(table=[[paimon_catalog, default, a]],
fields=[ip])
+- FlinkLogicalSnapshot(period=[$cor0.proctime])
+- FlinkLogicalCalc(select=[ip, CAST(0 AS INTEGER) AS type, CAST(ip AS
VARCHAR(2147483647) CHARACTER SET "UTF-16LE") AS ip0])
+- FlinkLogicalTableSourceScan(table=[[mariadb_catalog, menagerie,
c, {*}filter=[=(type, 0)]]]{*}, fields=[ip, type])
Is removed in the next stage.
org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram [] -
optimize physical cost 3 ms.
optimize result:
Sink(table=[*anonymous_collect$4*], fields=[ip, proctime, ip0, type])
+- Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
+- LookupJoin(table=[mariadb_catalog.menagerie.c], joinType=[LeftOuterJoin],
lookup=[ip=ip], select=[ip, proctime, ip, *CAST(0 AS INTEGER)* AS type, CAST(ip
AS VARCHAR(2147483647) CHARACTER SET "UTF-16LE") AS ip0])
+- Calc(select=[ip, PROCTIME() AS proctime])
+- TableSourceScan(table=[[paimon_catalog, default, a]], fields=[ip])
The *CAST(0 AS INTEGER)* is in the final Optimized Execution Plan we see in the
explain This cast is fine as long as the filter is there.
I am not an expert at this, but the comments at the top of
[CommonPhysicalLookupJoin.scala]([https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/nodes/physical/common/CommonPhysicalLookupJoin.scala)]
are correct and this filter should actually be in the lookup keys. The
comments says
_* For a lookup join query:_
_*_
_* SELECT T.id, T.content, D.age FROM T JOIN userTable FOR SYSTEM_TIME AS
OF T.proctime AS D_
_* ON T.content = concat(D.name, '!') AND D.age = 11 AND T.id = D.id WHERE
D.name LIKE 'Jack%'_
_* _
_*_
_* The LookupJoin physical node encapsulates the following RelNode tree:_
_*_
_* Join (l.name = r.name) / \ RelNode Calc (concat(name, "!") as name,
name LIKE 'Jack%') |_
_* DimTable (lookup-keys: age=11, id=l.id) (age, id, name) _
_*_
_* The important member fields in LookupJoin: allLookupKeys: [$0=11,
$1=l.id] ($0 and $1 is_
_* the indexes of age and id in dim table) remainingCondition:
l.name=r.name _
_*_
_* The workflow of lookup join:_
_*_
_* 1) lookup records dimension table using the lookup-keys 2) project &
filter on the lookup-ed_
_* records 3) join left input record and lookup-ed records 4) only
outputs the rows which_
_* match to the remainingCondition _
I would have thought that a filter on a value on one of the sources could be
pushed down a a filter to that source, rather than being in the lookup keys.
Maybe that could be a subsequent optimization. I could be missing something.
was (Author: JIRAUSER300523):
an update on what I have found:
I have switched on DEBUG put out the rules that are being driven for my
recreation. I see :
org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram [] -
optimize time_indicator cost 1 ms.
optimize result:
FlinkLogicalSink(table=[*anonymous_collect$4*], fields=[ip, proctime, ip0,
type])
+- FlinkLogicalCalc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime,
ip0, type])
+- FlinkLogicalJoin(condition=[=($0, $4)], joinType=[left])
:- FlinkLogicalCalc(select=[ip, PROCTIME() AS proctime])
: +- FlinkLogicalTableSourceScan(table=[[paimon_catalog, default, a]],
fields=[ip])
+- FlinkLogicalSnapshot(period=[$cor0.proctime])
+- FlinkLogicalCalc(select=[ip, CAST(0 AS INTEGER) AS type, CAST(ip AS
VARCHAR(2147483647) CHARACTER SET "UTF-16LE") AS ip0])
+- FlinkLogicalTableSourceScan(table=[[mariadb_catalog, menagerie,
c, {*}filter=[=(type, 0)]]]{*}, fields=[ip, type])
Is removed in the next stage.
org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram [] -
optimize physical cost 3 ms.
optimize result:
Sink(table=[*anonymous_collect$4*], fields=[ip, proctime, ip0, type])
+- Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
+- LookupJoin(table=[mariadb_catalog.menagerie.c],
[jira] [Comment Edited] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17784924#comment-17784924
]
david radley edited comment on FLINK-33365 at 11/10/23 3:01 PM:
Testing against MariaDB I see:
select CAST(0 AS INT) as type from c;
+--+
|type|
+--+
| 0|
| 0|
+--+
Just confirming the cast is not a filter.
was (Author: JIRAUSER300523):
Testing against MariaDB I see:
select CAST(0 AS INT) as type from c;
+--+
|type|
+--+
| 0|
| 0|
+--+
It looks like this change from a filter to a cast is not logically equivalent.
It just forces the result set to have that value but does not act as a filter.
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Assignee: david radley
>Priority: Critical
> Attachments: flink-connector-jdbc-3.0.0-1.16.png,
> flink-connector-jdbc-3.1.1-1.17.png
>
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[ip=ip], select=[ip, proctime, ip, CAST(0 AS INTEGER) AS type, CAST(ip
> AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]){code}
>
> excute same sql in sql-client with Flink 1.17.1/ 1.18.0 and
> *flink-connector-jdbc-3.0.0-1.16.jar*
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[type=0, ip=ip], where=[(type = 0)], select=[ip, proctime, ip, CAST(0
> AS INTEGER) AS type, CAST(ip AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]) {code}
> with flink-connector-jdbc-3.1.1-1.17.jar, the condition is
> *lookup=[ip=ip]*
> with flink-connector-jdbc-3.0.0-1.16.jar , the condition is
> *lookup=[type=0, ip=ip], where=[(type = 0)]*
>
> In out real world production environment, this lead incorrect data output
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17784915#comment-17784915
]
david radley edited comment on FLINK-33365 at 11/10/23 3:01 PM:
an update on what I have found:
I have switched on DEBUG put out the rules that are being driven for my
recreation. I see :
org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram [] -
optimize time_indicator cost 1 ms.
optimize result:
FlinkLogicalSink(table=[*anonymous_collect$4*], fields=[ip, proctime, ip0,
type])
+- FlinkLogicalCalc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime,
ip0, type])
+- FlinkLogicalJoin(condition=[=($0, $4)], joinType=[left])
:- FlinkLogicalCalc(select=[ip, PROCTIME() AS proctime])
: +- FlinkLogicalTableSourceScan(table=[[paimon_catalog, default, a]],
fields=[ip])
+- FlinkLogicalSnapshot(period=[$cor0.proctime])
+- FlinkLogicalCalc(select=[ip, CAST(0 AS INTEGER) AS type, CAST(ip AS
VARCHAR(2147483647) CHARACTER SET "UTF-16LE") AS ip0])
+- FlinkLogicalTableSourceScan(table=[[mariadb_catalog, menagerie,
c, {*}filter=[=(type, 0)]]]{*}, fields=[ip, type])
Is removed in the next stage.
org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram [] -
optimize physical cost 3 ms.
optimize result:
Sink(table=[*anonymous_collect$4*], fields=[ip, proctime, ip0, type])
+- Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
+- LookupJoin(table=[mariadb_catalog.menagerie.c], joinType=[LeftOuterJoin],
lookup=[ip=ip], select=[ip, proctime, ip, *CAST(0 AS INTEGER)* AS type, CAST(ip
AS VARCHAR(2147483647) CHARACTER SET "UTF-16LE") AS ip0])
+- Calc(select=[ip, PROCTIME() AS proctime])
+- TableSourceScan(table=[[paimon_catalog, default, a]], fields=[ip])
The *CAST(0 AS INTEGER)* is in the final Optimized Execution Plan we see in the
explain This cast is fine as long as the filter is there.
I am not an expert at this, but it seems to me that either 2 things are
happening:
1) This change to the graph is a valid optimization but it is not being
actioned properly when executed, such that the CAST(0 AS INTEGER) is ignored.
or
2) the comments at the top of
[CommonPhysicalLookupJoin.scala]([https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/nodes/physical/common/CommonPhysicalLookupJoin.scala)]
are correct and this filter should actually be in the lookup keys. The
comments says
_* For a lookup join query:_
_*_
_* SELECT T.id, T.content, D.age FROM T JOIN userTable FOR SYSTEM_TIME AS
OF T.proctime AS D_
_* ON T.content = concat(D.name, '!') AND D.age = 11 AND T.id = D.id WHERE
D.name LIKE 'Jack%'_
_* _
_*_
_* The LookupJoin physical node encapsulates the following RelNode tree:_
_*_
_* Join (l.name = r.name) / \ RelNode Calc (concat(name, "!") as name,
name LIKE 'Jack%') |_
_* DimTable (lookup-keys: age=11, id=l.id) (age, id, name) _
_*_
_* The important member fields in LookupJoin: allLookupKeys: [$0=11,
$1=l.id] ($0 and $1 is_
_* the indexes of age and id in dim table) remainingCondition:
l.name=r.name _
_*_
_* The workflow of lookup join:_
_*_
_* 1) lookup records dimension table using the lookup-keys 2) project &
filter on the lookup-ed_
_* records 3) join left input record and lookup-ed records 4) only
outputs the rows which_
_* match to the remainingCondition _
was (Author: JIRAUSER300523):
an update on what I have found:
I have switched on DEBUG put out the rules that are being driven for my
recreation. I see :
org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram [] -
optimize time_indicator cost 1 ms.
optimize result:
FlinkLogicalSink(table=[*anonymous_collect$4*], fields=[ip, proctime, ip0,
type])
+- FlinkLogicalCalc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime,
ip0, type])
+- FlinkLogicalJoin(condition=[=($0, $4)], joinType=[left])
:- FlinkLogicalCalc(select=[ip, PROCTIME() AS proctime])
: +- FlinkLogicalTableSourceScan(table=[[paimon_catalog, default, a]],
fields=[ip])
+- FlinkLogicalSnapshot(period=[$cor0.proctime])
+- FlinkLogicalCalc(select=[ip, CAST(0 AS INTEGER) AS type, CAST(ip AS
VARCHAR(2147483647) CHARACTER SET "UTF-16LE") AS ip0])
+- FlinkLogicalTableSourceScan(table=[[mariadb_catalog, menagerie,
c, {*}filter=[=(type, 0)]]]{*}, fields=[ip, type])
Is changed in the next rule to
org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram [] -
optimize physical cost 3 ms.
optimize result:
Sink(table=[*anonymous_collect$4*], fields=[ip, proctime, ip0, type])
+- Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
+- LookupJoin(table=[mariadb_catalog.menagerie.c], joinType=[LeftOuterJoin],
[jira] [Comment Edited] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17784924#comment-17784924
]
david radley edited comment on FLINK-33365 at 11/10/23 12:47 PM:
-
Testing against MariaDB I see:
select CAST(0 AS INT) as type from c;
+--+
|type|
+--+
| 0|
| 0|
+--+
It looks like this change from a filter to a cast is not logically equivalent.
It just forces the result set to have that value but does not act as a filter.
was (Author: JIRAUSER300523):
Testing against MariaDB I see:
select CAST(0 AS INT) as type from c;
+--+
|type|
+--+
| 0|
| 0|
+--+
It looks like this change from a filter to a cast is not logically equivalent.
It just forces the result set to have that value.
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Assignee: david radley
>Priority: Critical
> Attachments: flink-connector-jdbc-3.0.0-1.16.png,
> flink-connector-jdbc-3.1.1-1.17.png
>
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[ip=ip], select=[ip, proctime, ip, CAST(0 AS INTEGER) AS type, CAST(ip
> AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]){code}
>
> excute same sql in sql-client with Flink 1.17.1/ 1.18.0 and
> *flink-connector-jdbc-3.0.0-1.16.jar*
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[type=0, ip=ip], where=[(type = 0)], select=[ip, proctime, ip, CAST(0
> AS INTEGER) AS type, CAST(ip AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]) {code}
> with flink-connector-jdbc-3.1.1-1.17.jar, the condition is
> *lookup=[ip=ip]*
> with flink-connector-jdbc-3.0.0-1.16.jar , the condition is
> *lookup=[type=0, ip=ip], where=[(type = 0)]*
>
> In out real world production environment, this lead incorrect data output
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17784924#comment-17784924
]
david radley commented on FLINK-33365:
--
Testing against MariaDB I see:
select CAST(0 AS INT) as type from c;
+--+
| type |
+--+
| 0 |
| 0 |
+--+
It looks like this change from a filter to a cast is not logically equivalent.
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Assignee: david radley
>Priority: Critical
> Attachments: flink-connector-jdbc-3.0.0-1.16.png,
> flink-connector-jdbc-3.1.1-1.17.png
>
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[ip=ip], select=[ip, proctime, ip, CAST(0 AS INTEGER) AS type, CAST(ip
> AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]){code}
>
> excute same sql in sql-client with Flink 1.17.1/ 1.18.0 and
> *flink-connector-jdbc-3.0.0-1.16.jar*
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[type=0, ip=ip], where=[(type = 0)], select=[ip, proctime, ip, CAST(0
> AS INTEGER) AS type, CAST(ip AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]) {code}
> with flink-connector-jdbc-3.1.1-1.17.jar, the condition is
> *lookup=[ip=ip]*
> with flink-connector-jdbc-3.0.0-1.16.jar , the condition is
> *lookup=[type=0, ip=ip], where=[(type = 0)]*
>
> In out real world production environment, this lead incorrect data output
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17784924#comment-17784924
]
david radley edited comment on FLINK-33365 at 11/10/23 12:46 PM:
-
Testing against MariaDB I see:
select CAST(0 AS INT) as type from c;
+--+
|type|
+--+
| 0|
| 0|
+--+
It looks like this change from a filter to a cast is not logically equivalent.
It just forces the result set to have that value.
was (Author: JIRAUSER300523):
Testing against MariaDB I see:
select CAST(0 AS INT) as type from c;
+--+
| type |
+--+
| 0 |
| 0 |
+--+
It looks like this change from a filter to a cast is not logically equivalent.
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Assignee: david radley
>Priority: Critical
> Attachments: flink-connector-jdbc-3.0.0-1.16.png,
> flink-connector-jdbc-3.1.1-1.17.png
>
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[ip=ip], select=[ip, proctime, ip, CAST(0 AS INTEGER) AS type, CAST(ip
> AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]){code}
>
> excute same sql in sql-client with Flink 1.17.1/ 1.18.0 and
> *flink-connector-jdbc-3.0.0-1.16.jar*
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[type=0, ip=ip], where=[(type = 0)], select=[ip, proctime, ip, CAST(0
> AS INTEGER) AS type, CAST(ip AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]) {code}
> with flink-connector-jdbc-3.1.1-1.17.jar, the condition is
> *lookup=[ip=ip]*
> with flink-connector-jdbc-3.0.0-1.16.jar , the condition is
> *lookup=[type=0, ip=ip], where=[(type = 0)]*
>
> In out real world production environment, this lead incorrect data output
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17784915#comment-17784915
]
david radley edited comment on FLINK-33365 at 11/10/23 12:29 PM:
-
an update on what I have found:
I have switched on DEBUG put out the rules that are being driven for my
recreation. I see :
org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram [] -
optimize time_indicator cost 1 ms.
optimize result:
FlinkLogicalSink(table=[*anonymous_collect$4*], fields=[ip, proctime, ip0,
type])
+- FlinkLogicalCalc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime,
ip0, type])
+- FlinkLogicalJoin(condition=[=($0, $4)], joinType=[left])
:- FlinkLogicalCalc(select=[ip, PROCTIME() AS proctime])
: +- FlinkLogicalTableSourceScan(table=[[paimon_catalog, default, a]],
fields=[ip])
+- FlinkLogicalSnapshot(period=[$cor0.proctime])
+- FlinkLogicalCalc(select=[ip, CAST(0 AS INTEGER) AS type, CAST(ip AS
VARCHAR(2147483647) CHARACTER SET "UTF-16LE") AS ip0])
+- FlinkLogicalTableSourceScan(table=[[mariadb_catalog, menagerie,
c, {*}filter=[=(type, 0)]]]{*}, fields=[ip, type])
Is changed in the next rule to
org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram [] -
optimize physical cost 3 ms.
optimize result:
Sink(table=[*anonymous_collect$4*], fields=[ip, proctime, ip0, type])
+- Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
+- LookupJoin(table=[mariadb_catalog.menagerie.c], joinType=[LeftOuterJoin],
lookup=[ip=ip], select=[ip, proctime, ip, *CAST(0 AS INTEGER)* AS type, CAST(ip
AS VARCHAR(2147483647) CHARACTER SET "UTF-16LE") AS ip0])
+- Calc(select=[ip, PROCTIME() AS proctime])
+- TableSourceScan(table=[[paimon_catalog, default, a]], fields=[ip])
The *CAST(0 AS INTEGER)* is in the final Optimized Execution Plan we see in the
explain.
I am not an expert at this, but it seems to me that either 2 things are
happening:
1) This change to the graph is a valid optimization but it is not being
actioned properly when executed, such that the CAST(0 AS INTEGER) is ignored.
or
2) the comments at the top of
[CommonPhysicalLookupJoin.scala]([https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/nodes/physical/common/CommonPhysicalLookupJoin.scala)]
are correct and this filter should actually be in the lookup keys. The
comments says
_* For a lookup join query:_
_*_
_* SELECT T.id, T.content, D.age FROM T JOIN userTable FOR SYSTEM_TIME AS
OF T.proctime AS D_
_* ON T.content = concat(D.name, '!') AND D.age = 11 AND T.id = D.id WHERE
D.name LIKE 'Jack%'_
_* _
_*_
_* The LookupJoin physical node encapsulates the following RelNode tree:_
_*_
_* Join (l.name = r.name) / \ RelNode Calc (concat(name, "!") as name,
name LIKE 'Jack%') |_
_* DimTable (lookup-keys: age=11, id=l.id) (age, id, name) _
_*_
_* The important member fields in LookupJoin: allLookupKeys: [$0=11,
$1=l.id] ($0 and $1 is_
_* the indexes of age and id in dim table) remainingCondition:
l.name=r.name _
_*_
_* The workflow of lookup join:_
_*_
_* 1) lookup records dimension table using the lookup-keys 2) project &
filter on the lookup-ed_
_* records 3) join left input record and lookup-ed records 4) only
outputs the rows which_
_* match to the remainingCondition _
was (Author: JIRAUSER300523):
an update on what I have found:
I have switched on DEBUG put out the rules that are being driven for my
recreation. I see :
org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram [] -
optimize time_indicator cost 1 ms.
optimize result:
FlinkLogicalSink(table=[*anonymous_collect$4*], fields=[ip, proctime, ip0,
type])
+- FlinkLogicalCalc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime,
ip0, type])
+- FlinkLogicalJoin(condition=[=($0, $4)], joinType=[left])
:- FlinkLogicalCalc(select=[ip, PROCTIME() AS proctime])
: +- FlinkLogicalTableSourceScan(table=[[paimon_catalog, default, a]],
fields=[ip])
+- FlinkLogicalSnapshot(period=[$cor0.proctime])
+- FlinkLogicalCalc(select=[ip, CAST(0 AS INTEGER) AS type, CAST(ip AS
VARCHAR(2147483647) CHARACTER SET "UTF-16LE") AS ip0])
+- FlinkLogicalTableSourceScan(table=[[mariadb_catalog, menagerie,
c, {*}filter=[=(type, 0)]]]{*}, fields=[ip, type])
Is changed in the next rule to
org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram [] -
optimize physical cost 3 ms.
optimize result:
Sink(table=[*anonymous_collect$4*], fields=[ip, proctime, ip0, type])
+- Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
+- LookupJoin(table=[mariadb_catalog.menagerie.c], joinType=[LeftOuterJoin],
lookup=[ip=ip], select=[ip, proctime, ip, *CAST(0 AS
[jira] [Commented] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17784915#comment-17784915
]
david radley commented on FLINK-33365:
--
an update on what I have found:
I have switched on DEBUG put out the rules that are being driven for my
recreation. I see :
org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram [] -
optimize time_indicator cost 1 ms.
optimize result:
FlinkLogicalSink(table=[*anonymous_collect$4*], fields=[ip, proctime, ip0,
type])
+- FlinkLogicalCalc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime,
ip0, type])
+- FlinkLogicalJoin(condition=[=($0, $4)], joinType=[left])
:- FlinkLogicalCalc(select=[ip, PROCTIME() AS proctime])
: +- FlinkLogicalTableSourceScan(table=[[paimon_catalog, default, a]],
fields=[ip])
+- FlinkLogicalSnapshot(period=[$cor0.proctime])
+- FlinkLogicalCalc(select=[ip, CAST(0 AS INTEGER) AS type, CAST(ip AS
VARCHAR(2147483647) CHARACTER SET "UTF-16LE") AS ip0])
+- FlinkLogicalTableSourceScan(table=[[mariadb_catalog, menagerie,
c, {*}filter=[=(type, 0)]]]{*}, fields=[ip, type])
Is changed in the next rule to
org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram [] -
optimize physical cost 3 ms.
optimize result:
Sink(table=[*anonymous_collect$4*], fields=[ip, proctime, ip0, type])
+- Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
+- LookupJoin(table=[mariadb_catalog.menagerie.c], joinType=[LeftOuterJoin],
lookup=[ip=ip], select=[ip, proctime, ip, *CAST(0 AS INTEGER)* AS type, CAST(ip
AS VARCHAR(2147483647) CHARACTER SET "UTF-16LE") AS ip0])
+- Calc(select=[ip, PROCTIME() AS proctime])
+- TableSourceScan(table=[[paimon_catalog, default, a]], fields=[ip])
The *CAST(0 AS INTEGER)* is in the final Optimized Execution Plan we see in the
explain.
I am not an expert at this, but it seems to me that either 2 things are
happening:
1) This change to the graph is a valid optimization but it is not being
actioned properly when executed, such that the CAST(0 AS INTEGER) is ignored **
or
2) the comments at the top of
[CommonPhysicalLookupJoin.scala]([https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/nodes/physical/common/CommonPhysicalLookupJoin.scala)]
are correct and this filter should actually be in the lookup keys. The
comments says
_* For a lookup join query:_
_*_
_* SELECT T.id, T.content, D.age FROM T JOIN userTable FOR SYSTEM_TIME AS
OF T.proctime AS D_
_* ON T.content = concat(D.name, '!') AND D.age = 11 AND T.id = D.id WHERE
D.name LIKE 'Jack%'_
_* _
_*_
_* The LookupJoin physical node encapsulates the following RelNode tree:_
_*_
_* Join (l.name = r.name) / \ RelNode Calc (concat(name, "!") as name,
name LIKE 'Jack%') \|_
_* DimTable (lookup-keys: age=11, id=l.id) (age, id, name) _
_*_
_* The important member fields in LookupJoin: allLookupKeys: [$0=11,
$1=l.id] ($0 and $1 is_
_* the indexes of age and id in dim table) remainingCondition:
l.name=r.name _
_*_
_* The workflow of lookup join:_
_*_
_* 1) lookup records dimension table using the lookup-keys 2) project &
filter on the lookup-ed_
_* records 3) join left input record and lookup-ed records 4) only
outputs the rows which_
_* match to the remainingCondition _
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Assignee: david radley
>Priority: Critical
> Attachments: flink-connector-jdbc-3.0.0-1.16.png,
> flink-connector-jdbc-3.1.1-1.17.png
>
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +-
[jira] [Commented] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17782488#comment-17782488
]
david radley commented on FLINK-33365:
--
I did not create using the back level jdbc driver. Though I am on a Mac, using
MariaDB and Flink master; so that could be sufficiently different for it not to
recreate for me . I am focussing on the issue I can recreate.
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Assignee: david radley
>Priority: Critical
> Attachments: flink-connector-jdbc-3.0.0-1.16.png,
> flink-connector-jdbc-3.1.1-1.17.png
>
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[ip=ip], select=[ip, proctime, ip, CAST(0 AS INTEGER) AS type, CAST(ip
> AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]){code}
>
> excute same sql in sql-client with Flink 1.17.1/ 1.18.0 and
> *flink-connector-jdbc-3.0.0-1.16.jar*
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[type=0, ip=ip], where=[(type = 0)], select=[ip, proctime, ip, CAST(0
> AS INTEGER) AS type, CAST(ip AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]) {code}
> with flink-connector-jdbc-3.1.1-1.17.jar, the condition is
> *lookup=[ip=ip]*
> with flink-connector-jdbc-3.0.0-1.16.jar , the condition is
> *lookup=[type=0, ip=ip], where=[(type = 0)]*
>
> In out real world production environment, this lead incorrect data output
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33384) MySQL JDBC driver is deprecated
[ https://issues.apache.org/jira/browse/FLINK-33384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17782199#comment-17782199 ] david radley commented on FLINK-33384: -- note that I also see a deprecation message in the log as below: cat flink-davidradley-taskexecutor-0-Davids-MBP-2.out Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary. > MySQL JDBC driver is deprecated > --- > > Key: FLINK-33384 > URL: https://issues.apache.org/jira/browse/FLINK-33384 > Project: Flink > Issue Type: Improvement > Components: Table SQL / JDBC >Reporter: david radley >Assignee: david radley >Priority: Major > > I see when running tests on the JDBC connector, I get a warning > _Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver > class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered > via the SPI and manual loading of the driver class is generally unnecessary._ > > I suggest we change the class to be loaded from the old to the new non > deprecated class name. > > I am happy to implement this and do testing on it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17782128#comment-17782128
]
david radley commented on FLINK-33365:
--
[~macdoor615] thanks.
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Assignee: david radley
>Priority: Critical
> Attachments: flink-connector-jdbc-3.0.0-1.16.png,
> flink-connector-jdbc-3.1.1-1.17.png
>
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[ip=ip], select=[ip, proctime, ip, CAST(0 AS INTEGER) AS type, CAST(ip
> AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]){code}
>
> excute same sql in sql-client with Flink 1.17.1/ 1.18.0 and
> *flink-connector-jdbc-3.0.0-1.16.jar*
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[type=0, ip=ip], where=[(type = 0)], select=[ip, proctime, ip, CAST(0
> AS INTEGER) AS type, CAST(ip AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]) {code}
> with flink-connector-jdbc-3.1.1-1.17.jar, the condition is
> *lookup=[ip=ip]*
> with flink-connector-jdbc-3.0.0-1.16.jar , the condition is
> *lookup=[type=0, ip=ip], where=[(type = 0)]*
>
> In out real world production environment, this lead incorrect data output
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781843#comment-17781843
]
david radley edited comment on FLINK-33365 at 11/2/23 9:15 AM:
---
[~macdoor615] I switched out to use the 3.10 jar but got some compatibility
issues (
*RROR] Could not execute SQL statement. Reason:*
*java.lang.ClassNotFoundException:
org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableMap*
). I assume you are testing that with Flink 3.16? Could you include the output
of the explain when the select is working please?
was (Author: JIRAUSER300523):
[~macdoor615] I switched out to use the 3.10 jar but got some compatibility
issues. I assume you are testing that with Flink 3.16? Could you include the
output of the explain when the select is working please?
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Assignee: david radley
>Priority: Critical
> Attachments: flink-connector-jdbc-3.0.0-1.16.png,
> flink-connector-jdbc-3.1.1-1.17.png
>
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[ip=ip], select=[ip, proctime, ip, CAST(0 AS INTEGER) AS type, CAST(ip
> AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]){code}
>
> excute same sql in sql-client with Flink 1.17.1/ 1.18.0 and
> *flink-connector-jdbc-3.0.0-1.16.jar*
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[type=0, ip=ip], where=[(type = 0)], select=[ip, proctime, ip, CAST(0
> AS INTEGER) AS type, CAST(ip AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]) {code}
> with flink-connector-jdbc-3.1.1-1.17.jar, the condition is
> *lookup=[ip=ip]*
> with flink-connector-jdbc-3.0.0-1.16.jar , the condition is
> *lookup=[type=0, ip=ip], where=[(type = 0)]*
>
> In out real world production environment, this lead incorrect data output
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781829#comment-17781829
]
david radley edited comment on FLINK-33365 at 11/1/23 10:44 PM:
Some good news in terms of recreations :). I have managed to recreate your
paimon scenario using the latest drivers and MariaDB on the Mac. This issue is
not a missing filter (as per the issue title), it seems to be incorrect
execution. I can look into this but it does mismatch the issue title.
was (Author: JIRAUSER300523):
Some good news in terms of recreations :). I have managed to recreate your
paimon scenario using the latest drivers and MariaDB on the Mac. This issue is
not a missing filter (as per the issue), it seems to be incorrect execution. I
can look into this but it does mismatch the issue title.
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Assignee: david radley
>Priority: Critical
> Attachments: flink-connector-jdbc-3.0.0-1.16.png,
> flink-connector-jdbc-3.1.1-1.17.png
>
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[ip=ip], select=[ip, proctime, ip, CAST(0 AS INTEGER) AS type, CAST(ip
> AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]){code}
>
> excute same sql in sql-client with Flink 1.17.1/ 1.18.0 and
> *flink-connector-jdbc-3.0.0-1.16.jar*
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[type=0, ip=ip], where=[(type = 0)], select=[ip, proctime, ip, CAST(0
> AS INTEGER) AS type, CAST(ip AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]) {code}
> with flink-connector-jdbc-3.1.1-1.17.jar, the condition is
> *lookup=[ip=ip]*
> with flink-connector-jdbc-3.0.0-1.16.jar , the condition is
> *lookup=[type=0, ip=ip], where=[(type = 0)]*
>
> In out real world production environment, this lead incorrect data output
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781843#comment-17781843
]
david radley edited comment on FLINK-33365 at 11/1/23 10:43 PM:
[~macdoor615] I switched out to use the 3.10 jar but got some compatibility
issues. I assume you are testing that with Flink 3.16? Could you include the
output of the explain when the select is working please?
was (Author: JIRAUSER300523):
[~macdoor615] I switched out to use the 3.10 jar but hot some compatibility
issues. I assume you are testing that with Flink 3.16? Could you include the
output of the explain when the select is working please?
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Assignee: david radley
>Priority: Critical
> Attachments: flink-connector-jdbc-3.0.0-1.16.png,
> flink-connector-jdbc-3.1.1-1.17.png
>
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[ip=ip], select=[ip, proctime, ip, CAST(0 AS INTEGER) AS type, CAST(ip
> AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]){code}
>
> excute same sql in sql-client with Flink 1.17.1/ 1.18.0 and
> *flink-connector-jdbc-3.0.0-1.16.jar*
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[type=0, ip=ip], where=[(type = 0)], select=[ip, proctime, ip, CAST(0
> AS INTEGER) AS type, CAST(ip AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]) {code}
> with flink-connector-jdbc-3.1.1-1.17.jar, the condition is
> *lookup=[ip=ip]*
> with flink-connector-jdbc-3.0.0-1.16.jar , the condition is
> *lookup=[type=0, ip=ip], where=[(type = 0)]*
>
> In out real world production environment, this lead incorrect data output
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33384) MySQL JDBC driver is deprecated
[ https://issues.apache.org/jira/browse/FLINK-33384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781846#comment-17781846 ] david radley commented on FLINK-33384: -- It looks like Oracle now recommend using a different driver for mySQL - here is the discussion [https://blogs.oracle.com/mysql/post/mysql-connectorj-has-new-maven-coordinates.|https://blogs.oracle.com/mysql/post/mysql-connectorj-has-new-maven-coordinates] I see that the mySQL tests bring in this dependancy as well. mysql mysql-connector-java 8.0.29 test it is required to resolve an import in test code MySqlMetadata.java. I could not get these tests to run in main. > MySQL JDBC driver is deprecated > --- > > Key: FLINK-33384 > URL: https://issues.apache.org/jira/browse/FLINK-33384 > Project: Flink > Issue Type: Improvement > Components: Table SQL / JDBC >Reporter: david radley >Assignee: david radley >Priority: Major > > I see when running tests on the JDBC connector, I get a warning > _Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver > class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered > via the SPI and manual loading of the driver class is generally unnecessary._ > > I suggest we change the class to be loaded from the old to the new non > deprecated class name. > > I am happy to implement this and do testing on it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781843#comment-17781843
]
david radley commented on FLINK-33365:
--
[~macdoor615] I switched out to use the 3.10 jar but hot some compatibility
issues. I assume you are testing that with Flink 3.16? Could you include the
output of the explain when the select is working please?
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Assignee: david radley
>Priority: Critical
> Attachments: flink-connector-jdbc-3.0.0-1.16.png,
> flink-connector-jdbc-3.1.1-1.17.png
>
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[ip=ip], select=[ip, proctime, ip, CAST(0 AS INTEGER) AS type, CAST(ip
> AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]){code}
>
> excute same sql in sql-client with Flink 1.17.1/ 1.18.0 and
> *flink-connector-jdbc-3.0.0-1.16.jar*
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[type=0, ip=ip], where=[(type = 0)], select=[ip, proctime, ip, CAST(0
> AS INTEGER) AS type, CAST(ip AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]) {code}
> with flink-connector-jdbc-3.1.1-1.17.jar, the condition is
> *lookup=[ip=ip]*
> with flink-connector-jdbc-3.0.0-1.16.jar , the condition is
> *lookup=[type=0, ip=ip], where=[(type = 0)]*
>
> In out real world production environment, this lead incorrect data output
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781829#comment-17781829
]
david radley edited comment on FLINK-33365 at 11/1/23 5:18 PM:
---
Some good news in terms of recreations :). I have managed to recreate your
paimon scenario using the latest drivers and MariaDB on the Mac. This issue is
not a missing filter (as per the issue), it seems to be incorrect execution. I
can look into this but it does mismatch the issue title.
was (Author: JIRAUSER300523):
Some good news in terms of recreations :). I have managed to recreate your
paimon scenario using the latest drivers and MariaDB on the Mac. I will look
into this .
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Assignee: david radley
>Priority: Critical
> Attachments: flink-connector-jdbc-3.0.0-1.16.png,
> flink-connector-jdbc-3.1.1-1.17.png
>
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[ip=ip], select=[ip, proctime, ip, CAST(0 AS INTEGER) AS type, CAST(ip
> AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]){code}
>
> excute same sql in sql-client with Flink 1.17.1/ 1.18.0 and
> *flink-connector-jdbc-3.0.0-1.16.jar*
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[type=0, ip=ip], where=[(type = 0)], select=[ip, proctime, ip, CAST(0
> AS INTEGER) AS type, CAST(ip AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]) {code}
> with flink-connector-jdbc-3.1.1-1.17.jar, the condition is
> *lookup=[ip=ip]*
> with flink-connector-jdbc-3.0.0-1.16.jar , the condition is
> *lookup=[type=0, ip=ip], where=[(type = 0)]*
>
> In out real world production environment, this lead incorrect data output
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781829#comment-17781829
]
david radley edited comment on FLINK-33365 at 11/1/23 5:03 PM:
---
Some good news in terms of recreations :). I have managed to recreate your
paimon scenario using the latest drivers and MariaDB on the Mac. I will look
into this .
was (Author: JIRAUSER300523):
I have managed to recreate your paimon scenario using the latest drivers and
MariaDB on the Mac. I will look into this issue - as this is the only one I can
recreate.
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Assignee: david radley
>Priority: Critical
> Attachments: flink-connector-jdbc-3.0.0-1.16.png,
> flink-connector-jdbc-3.1.1-1.17.png
>
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[ip=ip], select=[ip, proctime, ip, CAST(0 AS INTEGER) AS type, CAST(ip
> AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]){code}
>
> excute same sql in sql-client with Flink 1.17.1/ 1.18.0 and
> *flink-connector-jdbc-3.0.0-1.16.jar*
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[type=0, ip=ip], where=[(type = 0)], select=[ip, proctime, ip, CAST(0
> AS INTEGER) AS type, CAST(ip AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]) {code}
> with flink-connector-jdbc-3.1.1-1.17.jar, the condition is
> *lookup=[ip=ip]*
> with flink-connector-jdbc-3.0.0-1.16.jar , the condition is
> *lookup=[type=0, ip=ip], where=[(type = 0)]*
>
> In out real world production environment, this lead incorrect data output
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781829#comment-17781829
]
david radley commented on FLINK-33365:
--
I have managed to recreate your paimon scenario using the latest drivers and
MariaDB on the Mac. I will look into this issue - as this is the only one I can
recreate.
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Assignee: david radley
>Priority: Critical
> Attachments: flink-connector-jdbc-3.0.0-1.16.png,
> flink-connector-jdbc-3.1.1-1.17.png
>
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[ip=ip], select=[ip, proctime, ip, CAST(0 AS INTEGER) AS type, CAST(ip
> AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]){code}
>
> excute same sql in sql-client with Flink 1.17.1/ 1.18.0 and
> *flink-connector-jdbc-3.0.0-1.16.jar*
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[type=0, ip=ip], where=[(type = 0)], select=[ip, proctime, ip, CAST(0
> AS INTEGER) AS type, CAST(ip AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]) {code}
> with flink-connector-jdbc-3.1.1-1.17.jar, the condition is
> *lookup=[ip=ip]*
> with flink-connector-jdbc-3.0.0-1.16.jar , the condition is
> *lookup=[type=0, ip=ip], where=[(type = 0)]*
>
> In out real world production environment, this lead incorrect data output
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781757#comment-17781757
]
david radley edited comment on FLINK-33365 at 11/1/23 3:15 PM:
---
I did a further test- I rebuilt 3.1.1 (in case there was something fixing it in
main) with the MariaDB change and it did not recreate the issue. I will test
with you paimon scenario
was (Author: JIRAUSER300523):
I did a further test- I rebuild 3.1.1 (in case there was something fixing it in
main) with the MariaDB change and it did not recreate the issue. I will test
with you paimon scenario
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Assignee: david radley
>Priority: Critical
> Attachments: flink-connector-jdbc-3.0.0-1.16.png,
> flink-connector-jdbc-3.1.1-1.17.png
>
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[ip=ip], select=[ip, proctime, ip, CAST(0 AS INTEGER) AS type, CAST(ip
> AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]){code}
>
> excute same sql in sql-client with Flink 1.17.1/ 1.18.0 and
> *flink-connector-jdbc-3.0.0-1.16.jar*
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[type=0, ip=ip], where=[(type = 0)], select=[ip, proctime, ip, CAST(0
> AS INTEGER) AS type, CAST(ip AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]) {code}
> with flink-connector-jdbc-3.1.1-1.17.jar, the condition is
> *lookup=[ip=ip]*
> with flink-connector-jdbc-3.0.0-1.16.jar , the condition is
> *lookup=[type=0, ip=ip], where=[(type = 0)]*
>
> In out real world production environment, this lead incorrect data output
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781757#comment-17781757
]
david radley edited comment on FLINK-33365 at 11/1/23 2:35 PM:
---
I did a further test- I rebuild 3.1.1 (in case there was something fixing it in
main) with the MariaDB change and it did not recreate the issue. I will test
with you paimon scenario
was (Author: JIRAUSER300523):
I did a further test- I rebuild 3.1.1 (in case there was something fixing it in
main) with the MariaDB change and it did not recreate the issue.
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Assignee: david radley
>Priority: Critical
> Attachments: flink-connector-jdbc-3.0.0-1.16.png,
> flink-connector-jdbc-3.1.1-1.17.png
>
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[ip=ip], select=[ip, proctime, ip, CAST(0 AS INTEGER) AS type, CAST(ip
> AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]){code}
>
> excute same sql in sql-client with Flink 1.17.1/ 1.18.0 and
> *flink-connector-jdbc-3.0.0-1.16.jar*
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[type=0, ip=ip], where=[(type = 0)], select=[ip, proctime, ip, CAST(0
> AS INTEGER) AS type, CAST(ip AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]) {code}
> with flink-connector-jdbc-3.1.1-1.17.jar, the condition is
> *lookup=[ip=ip]*
> with flink-connector-jdbc-3.0.0-1.16.jar , the condition is
> *lookup=[type=0, ip=ip], where=[(type = 0)]*
>
> In out real world production environment, this lead incorrect data output
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781757#comment-17781757
]
david radley commented on FLINK-33365:
--
I did a further test- I rebuild 3.1.1 (in case there was something fixing it in
main) with the MariaDB change and it did not recreate the issue.
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Assignee: david radley
>Priority: Critical
> Attachments: flink-connector-jdbc-3.0.0-1.16.png,
> flink-connector-jdbc-3.1.1-1.17.png
>
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[ip=ip], select=[ip, proctime, ip, CAST(0 AS INTEGER) AS type, CAST(ip
> AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]){code}
>
> excute same sql in sql-client with Flink 1.17.1/ 1.18.0 and
> *flink-connector-jdbc-3.0.0-1.16.jar*
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[type=0, ip=ip], where=[(type = 0)], select=[ip, proctime, ip, CAST(0
> AS INTEGER) AS type, CAST(ip AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]) {code}
> with flink-connector-jdbc-3.1.1-1.17.jar, the condition is
> *lookup=[ip=ip]*
> with flink-connector-jdbc-3.0.0-1.16.jar , the condition is
> *lookup=[type=0, ip=ip], where=[(type = 0)]*
>
> In out real world production environment, this lead incorrect data output
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781712#comment-17781712
]
david radley commented on FLINK-33365:
--
[~macdoor615] I am running on a Mac and have Mariadb (I tweaked the dialect
factory to allow mariadb in the jdbc url - see
https://issues.apache.org/jira/browse/FLINK-28433 ) and the latest Flink jdbc
connector (master 3.2-SNAPSHOT that I built from source) and I cannot recreate
the problem you are seeing (I see the filter as expected in the explain).
Can you try:
* the latest Flink JDBC connector 3.2-SNAPSHOT built from source
* the latest mySQL JDBC driver. I used the platform independent
[https://dev.mysql.com/downloads/connector/j/] . This seems to be recommended
[https://blogs.oracle.com/mysql/post/mysql-connectorj-has-new-maven-coordinates]([https://blogs.oracle.com/mysql/post/mysql-connectorj-has-new-maven-coordinates).]
I have raised issue https://issues.apache.org/jira/browse/FLINK-33384 to
change to docs and code that reference the old deprecated JDBC driver.
I hope these suggestions resoilve this for you; if not I can look to setup a
Linux image with mySQL; which JDBC driver do you see the issue on?
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Assignee: david radley
>Priority: Critical
> Attachments: flink-connector-jdbc-3.0.0-1.16.png,
> flink-connector-jdbc-3.1.1-1.17.png
>
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[ip=ip], select=[ip, proctime, ip, CAST(0 AS INTEGER) AS type, CAST(ip
> AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]){code}
>
> excute same sql in sql-client with Flink 1.17.1/ 1.18.0 and
> *flink-connector-jdbc-3.0.0-1.16.jar*
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[type=0, ip=ip], where=[(type = 0)], select=[ip, proctime, ip, CAST(0
> AS INTEGER) AS type, CAST(ip AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]) {code}
> with flink-connector-jdbc-3.1.1-1.17.jar, the condition is
> *lookup=[ip=ip]*
> with flink-connector-jdbc-3.0.0-1.16.jar , the condition is
> *lookup=[type=0, ip=ip], where=[(type = 0)]*
>
> In out real world production environment, this lead incorrect data output
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781712#comment-17781712
]
david radley edited comment on FLINK-33365 at 11/1/23 12:06 PM:
[~macdoor615] I am running on a Mac and have Mariadb (I tweaked the dialect
factory to allow mariadb in the jdbc url - see
https://issues.apache.org/jira/browse/FLINK-28433 ) and the latest Flink jdbc
connector (master 3.2-SNAPSHOT that I built from source) and I cannot recreate
the problem you are seeing (I see the filter as expected in the explain).
Can you try:
* the latest Flink JDBC connector 3.2-SNAPSHOT built from source
* the latest mySQL JDBC driver. I used the platform independent
[https://dev.mysql.com/downloads/connector/j/] . This seems to be recommended
[https://blogs.oracle.com/mysql/post/mysql-connectorj-has-new-maven-coordinates]([https://blogs.oracle.com/mysql/post/mysql-connectorj-has-new-maven-coordinates).]
I have raised issue https://issues.apache.org/jira/browse/FLINK-33384 to
change to docs and code that reference the old deprecated JDBC driver.
I hope these suggestions resolve this for you; if not I can look to setup a
Linux image with mySQL; which JDBC driver do you see the issue on?
was (Author: JIRAUSER300523):
[~macdoor615] I am running on a Mac and have Mariadb (I tweaked the dialect
factory to allow mariadb in the jdbc url - see
https://issues.apache.org/jira/browse/FLINK-28433 ) and the latest Flink jdbc
connector (master 3.2-SNAPSHOT that I built from source) and I cannot recreate
the problem you are seeing (I see the filter as expected in the explain).
Can you try:
* the latest Flink JDBC connector 3.2-SNAPSHOT built from source
* the latest mySQL JDBC driver. I used the platform independent
[https://dev.mysql.com/downloads/connector/j/] . This seems to be recommended
[https://blogs.oracle.com/mysql/post/mysql-connectorj-has-new-maven-coordinates]([https://blogs.oracle.com/mysql/post/mysql-connectorj-has-new-maven-coordinates).]
I have raised issue https://issues.apache.org/jira/browse/FLINK-33384 to
change to docs and code that reference the old deprecated JDBC driver.
I hope these suggestions resoilve this for you; if not I can look to setup a
Linux image with mySQL; which JDBC driver do you see the issue on?
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Assignee: david radley
>Priority: Critical
> Attachments: flink-connector-jdbc-3.0.0-1.16.png,
> flink-connector-jdbc-3.1.1-1.17.png
>
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[ip=ip], select=[ip, proctime, ip, CAST(0 AS INTEGER) AS type, CAST(ip
> AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]){code}
>
> excute same sql in sql-client with Flink 1.17.1/ 1.18.0 and
> *flink-connector-jdbc-3.0.0-1.16.jar*
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[type=0, ip=ip], where=[(type = 0)], select=[ip, proctime, ip, CAST(0
> AS INTEGER) AS type, CAST(ip AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]) {code}
> with flink-connector-jdbc-3.1.1-1.17.jar, the condition is
> *lookup=[ip=ip]*
> with
[jira] [Comment Edited] (FLINK-28433) Allow connection to mysql through mariadb driver
[
https://issues.apache.org/jira/browse/FLINK-28433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781681#comment-17781681
]
david radley edited comment on FLINK-28433 at 11/1/23 9:57 AM:
---
hi [~bzhaoop] ,
are you still looking at this? It looks like there was an agreed pr, but then
the JDBC connector got moved out of the core repository. The fix was not ported
over.
I wondered if the pr is basically the same as mySQL apart from the prefix (they
are supposed to be the same). If so I wonder if we could just change the mysql
dialect library to accept mariadb url prefixes as well as mysql - or is more
required?
Change MySqlDialectFactory like this:
public boolean acceptsURL(String url) {
return url.startsWith("jdbc:mysql:") || url.startsWith("jdbc:mariadb:");
}
This change works for me with my limited testing.
was (Author: JIRAUSER300523):
hi [~bzhaoop] ,
are you still looking at this?
The reason MariaDB does not work is because of the jdbc url. Order Mariadb JDBC
drivers would accept mysql, later ones require a query param permitMysqlScheme
on the jdbc url, see
[https://mariadb.com/kb/en/about-mariadb-connector-j/|http://example.com](https://mariadb.com/kb/en/about-mariadb-connector-j/).
The mysql dialect factory has
MySqlDialectFactory implements JdbcDialectFactory {
@Override
public boolean acceptsURL(String url) {
return url.startsWith("jdbc:mysql:");
}
@Override
public JdbcDialect create() {
return new MySqlDialect();
}
}
If we change the accepts URL to:
return url.startsWith("jdbc:mysql:") || url.startsWith("jdbc:mariadb:");
Then it works. Is this an acceptable change, with some associated documentation
changes? If so, please assign me the issue and I will fix it.
> Allow connection to mysql through mariadb driver
>
>
> Key: FLINK-28433
> URL: https://issues.apache.org/jira/browse/FLINK-28433
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / JDBC
>Reporter: PengLei
>Assignee: bo zhao
>Priority: Minor
> Labels: pull-request-available, stale-assigned
> Attachments: image-2022-07-07-09-29-06-834.png
>
>
> Flink connector support connection to mysql. But the url must be started with
> "jdbc:mysql".
> Some user need to use mariadb dirver to connect to mysql. It can be achieved
> by setting the driver parameter in jdbcOptions. Unfortunately， the url
> verification fails.
>
> as follows:
> !image-2022-07-07-09-29-06-834.png!
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-28433) Allow connection to mysql through mariadb driver
[
https://issues.apache.org/jira/browse/FLINK-28433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781681#comment-17781681
]
david radley commented on FLINK-28433:
--
hi [~bzhaoop] ,
are you still looking at this?
The reason MariaDB does not work is because of the jdbc url. Order Mariadb JDBC
drivers would accept mysql, later ones require a query param permitMysqlScheme
on the jdbc url, see
[https://mariadb.com/kb/en/about-mariadb-connector-j/|http://example.com](https://mariadb.com/kb/en/about-mariadb-connector-j/).
The mysql dialect factory has
MySqlDialectFactory implements JdbcDialectFactory {
@Override
public boolean acceptsURL(String url) {
return url.startsWith("jdbc:mysql:");
}
@Override
public JdbcDialect create() {
return new MySqlDialect();
}
}
If we change the accepts URL to:
return url.startsWith("jdbc:mysql:") || url.startsWith("jdbc:mariadb:");
Then it works. Is this an acceptable change, with some associated documentation
changes? If so, please assign me the issue and I will fix it.
> Allow connection to mysql through mariadb driver
>
>
> Key: FLINK-28433
> URL: https://issues.apache.org/jira/browse/FLINK-28433
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / JDBC
>Reporter: PengLei
>Assignee: bo zhao
>Priority: Minor
> Labels: pull-request-available, stale-assigned
> Attachments: image-2022-07-07-09-29-06-834.png
>
>
> Flink connector support connection to mysql. But the url must be started with
> "jdbc:mysql".
> Some user need to use mariadb dirver to connect to mysql. It can be achieved
> by setting the driver parameter in jdbcOptions. Unfortunately， the url
> verification fails.
>
> as follows:
> !image-2022-07-07-09-29-06-834.png!
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781275#comment-17781275
]
david radley commented on FLINK-33365:
--
[~macdoor615] I am going to dig into this later this week.
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Assignee: david radley
>Priority: Critical
> Attachments: flink-connector-jdbc-3.0.0-1.16.png,
> flink-connector-jdbc-3.1.1-1.17.png
>
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[ip=ip], select=[ip, proctime, ip, CAST(0 AS INTEGER) AS type, CAST(ip
> AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]){code}
>
> excute same sql in sql-client with Flink 1.17.1/ 1.18.0 and
> *flink-connector-jdbc-3.0.0-1.16.jar*
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[type=0, ip=ip], where=[(type = 0)], select=[ip, proctime, ip, CAST(0
> AS INTEGER) AS type, CAST(ip AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]) {code}
> with flink-connector-jdbc-3.1.1-1.17.jar, the condition is
> *lookup=[ip=ip]*
> with flink-connector-jdbc-3.0.0-1.16.jar , the condition is
> *lookup=[type=0, ip=ip], where=[(type = 0)]*
>
> In out real world production environment, this lead incorrect data output
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17780403#comment-17780403
]
david radley commented on FLINK-33365:
--
[~macdoor615] I notice that I can run
JdbcDynamicTableSourceITCase method , testLookupJoin in my Intellij workspace,
and then choose mySQL as the implmentation.
As is this runs correctly.
I then change the select to something like your failing case:
_List collected =_
_executeQuery(_
_"SELECT S.id, S.name, D.id, D.timestamp6_col, D.decimal_col FROM value_source"_
_+ " AS S JOIN jdbc_lookup for system_time as of S.proctime AS D ON S.id = D.id
AND S.id = 2");_
and rerun the test, it fails with error:
org.apache.flink.table.api.TableException: Temporal table join requires an
equality condition on fields of table
[default_catalog.default_database.jdbc_lookup].
There are some hits in google indicating that this is a restriction for
temporal table joins. This Exception comes from a validate
method in commonExecLookupJoin.java.
Do you think this test is representative of your case? If not we can try other
selects here, I have tried a few variants.
I am going to see if this happens at the back level version of the JDBC
connector.
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Assignee: david radley
>Priority: Critical
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[ip=ip], select=[ip, proctime, ip, CAST(0 AS INTEGER) AS type, CAST(ip
> AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]){code}
>
> excute same sql in sql-client with Flink 1.17.1/ 1.18.0 and
> *flink-connector-jdbc-3.0.0-1.16.jar*
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[type=0, ip=ip], where=[(type = 0)], select=[ip, proctime, ip, CAST(0
> AS INTEGER) AS type, CAST(ip AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]) {code}
> with flink-connector-jdbc-3.1.1-1.17.jar, the condition is
> *lookup=[ip=ip]*
> with flink-connector-jdbc-3.0.0-1.16.jar , the condition is
> *lookup=[type=0, ip=ip], where=[(type = 0)]*
>
> In out real world production environment, this lead incorrect data output
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-33384) MySQL JDBC driver is deprecated
david radley created FLINK-33384: Summary: MySQL JDBC driver is deprecated Key: FLINK-33384 URL: https://issues.apache.org/jira/browse/FLINK-33384 Project: Flink Issue Type: Improvement Components: Table SQL / JDBC Reporter: david radley I see when running tests on the JDBC connector, I get a warning _Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary._ I suggest we change the class to be loaded from the old to the new non deprecated class name. I am happy to implement this and do testing on it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17780272#comment-17780272
]
david radley commented on FLINK-33365:
--
found the discussion in the mailing list with an example. We have recreated
this situation with a getindata (lookup) connector. So I am thinking this is
not just a mySQL issue.
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Priority: Critical
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[ip=ip], select=[ip, proctime, ip, CAST(0 AS INTEGER) AS type, CAST(ip
> AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]){code}
>
> excute same sql in sql-client with Flink 1.17.1/ 1.18.0 and
> *flink-connector-jdbc-3.0.0-1.16.jar*
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[type=0, ip=ip], where=[(type = 0)], select=[ip, proctime, ip, CAST(0
> AS INTEGER) AS type, CAST(ip AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]) {code}
> with flink-connector-jdbc-3.1.1-1.17.jar, the condition is
> *lookup=[ip=ip]*
> with flink-connector-jdbc-3.0.0-1.16.jar , the condition is
> *lookup=[type=0, ip=ip], where=[(type = 0)]*
>
> In out real world production environment, this lead incorrect data output
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17780259#comment-17780259
]
david radley commented on FLINK-33365:
--
Or are the filters you are talking about part of the ON clause?
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Priority: Critical
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[ip=ip], select=[ip, proctime, ip, CAST(0 AS INTEGER) AS type, CAST(ip
> AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]){code}
>
> excute same sql in sql-client with Flink 1.17.1/ 1.18.0 and
> *flink-connector-jdbc-3.0.0-1.16.jar*
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[type=0, ip=ip], where=[(type = 0)], select=[ip, proctime, ip, CAST(0
> AS INTEGER) AS type, CAST(ip AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]) {code}
> with flink-connector-jdbc-3.1.1-1.17.jar, the condition is
> *lookup=[ip=ip]*
> with flink-connector-jdbc-3.0.0-1.16.jar , the condition is
> *lookup=[type=0, ip=ip], where=[(type = 0)]*
>
> In out real world production environment, this lead incorrect data output
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33365) Missing filter condition in execution plan containing lookup join with mysql jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-33365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17780255#comment-17780255
]
david radley commented on FLINK-33365:
--
Hi,
It would be really helpful if you could supply a test case (the tables and SQL
an expected results) that shows this issue.
I assume if we follow the example in
[[https://nightlies.apache.org/flink/flink-table-store-docs-master/docs/development/lookup-join/](https://nightlies.apache.org/flink/flink-table-store-docs-master/docs/development/lookup-join/)]
then you are using mySQL as the lookup source connector and you specify a
WHERE clause that is being ignored. Am I understanding correctly?
> Missing filter condition in execution plan containing lookup join with mysql
> jdbc connector
> ---
>
> Key: FLINK-33365
> URL: https://issues.apache.org/jira/browse/FLINK-33365
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.18.0, 1.17.1
> Environment: Flink 1.17.1 & Flink 1.18.0 with
> flink-connector-jdbc-3.1.1-1.17.jar
>Reporter: macdoor615
>Priority: Critical
>
> create table in flink with sql-client.sh
> {code:java}
> CREATE TABLE default_catalog.default_database.a (
> ip string,
> proctime as proctime()
> )
> WITH (
> 'connector' = 'datagen'
> );{code}
> create table in mysql
> {code:java}
> create table b (
> ip varchar(20),
> type int
> ); {code}
>
> Flink 1.17.1/ 1.18.0 and *flink-connector-jdbc-3.1.1-1.17.jar*
> excute in sql-client.sh
> {code:java}
> explain SELECT * FROM default_catalog.default_database.a left join
> bnpmp_mysql_test.gem_tmp.b FOR SYSTEM_TIME AS OF a.proctime on b.type = 0 and
> a.ip = b.ip; {code}
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[ip=ip], select=[ip, proctime, ip, CAST(0 AS INTEGER) AS type, CAST(ip
> AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]){code}
>
> excute same sql in sql-client with Flink 1.17.1/ 1.18.0 and
> *flink-connector-jdbc-3.0.0-1.16.jar*
> get the execution plan
> {code:java}
> ...
> == Optimized Execution Plan ==
> Calc(select=[ip, PROCTIME_MATERIALIZE(proctime) AS proctime, ip0, type])
> +- LookupJoin(table=[bnpmp_mysql_test.gem_tmp.b], joinType=[LeftOuterJoin],
> lookup=[type=0, ip=ip], where=[(type = 0)], select=[ip, proctime, ip, CAST(0
> AS INTEGER) AS type, CAST(ip AS VARCHAR(2147483647)) AS ip0])
> +- Calc(select=[ip, PROCTIME() AS proctime])
> +- TableSourceScan(table=[[default_catalog, default_database, a]],
> fields=[ip]) {code}
> with flink-connector-jdbc-3.1.1-1.17.jar, the condition is
> *lookup=[ip=ip]*
> with flink-connector-jdbc-3.0.0-1.16.jar , the condition is
> *lookup=[type=0, ip=ip], where=[(type = 0)]*
>
> In out real world production environment, this lead incorrect data output
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33269) light and dark scss files do not have apache licenses at the top.
[ https://issues.apache.org/jira/browse/FLINK-33269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17774867#comment-17774867 ] david radley commented on FLINK-33269: -- Created pr https://github.com/apache/flink-web/pull/687 > light and dark scss files do not have apache licenses at the top. > -- > > Key: FLINK-33269 > URL: https://issues.apache.org/jira/browse/FLINK-33269 > Project: Flink > Issue Type: Bug > Components: Documentation, Runtime / Web Frontend >Reporter: david radley >Priority: Major > > I notice that > [https://github.com/apache/flink-web/blob/asf-site/docs/assets/_code-dark.scss] > and > [https://github.com/apache/flink-web/blob/asf-site/docs/assets/_code-light.scss] > > recently added for https://issues.apache.org/jira/browse/FLINK-33046 do not > have the Apache license at the top like the other scss files. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33269) light and dark scss files do not have apache licenses at the top.
[ https://issues.apache.org/jira/browse/FLINK-33269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17774863#comment-17774863 ] david radley commented on FLINK-33269: -- I can work on this > light and dark scss files do not have apache licenses at the top. > -- > > Key: FLINK-33269 > URL: https://issues.apache.org/jira/browse/FLINK-33269 > Project: Flink > Issue Type: Bug > Components: Documentation, Runtime / Web Frontend >Reporter: david radley >Priority: Major > > I notice that > [https://github.com/apache/flink-web/blob/asf-site/docs/assets/_code-dark.scss] > and > [https://github.com/apache/flink-web/blob/asf-site/docs/assets/_code-light.scss] > > recently added for https://issues.apache.org/jira/browse/FLINK-33046 do not > have the Apache license at the top like the other scss files. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33269) light and dark scss files do not have apache licenses at the top.
david radley created FLINK-33269: Summary: light and dark scss files do not have apache licenses at the top. Key: FLINK-33269 URL: https://issues.apache.org/jira/browse/FLINK-33269 Project: Flink Issue Type: Bug Components: Documentation, Runtime / Web Frontend Reporter: david radley I notice that [https://github.com/apache/flink-web/blob/asf-site/docs/assets/_code-dark.scss] and [https://github.com/apache/flink-web/blob/asf-site/docs/assets/_code-light.scss] recently added for https://issues.apache.org/jira/browse/FLINK-33046 do not have the Apache license at the top like the other scss files. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31778) Casting array of rows produces incorrect result
[
https://issues.apache.org/jira/browse/FLINK-31778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17770463#comment-17770463
]
david radley commented on FLINK-31778:
--
[~soin08] It looks like this is fixed in master
[https://github.com/apache/flink/commit/6d62f9918ea2cbb8a10c705a25a4ff6deab60711]
- we should close out this issue.
> Casting array of rows produces incorrect result
> ---
>
> Key: FLINK-31778
> URL: https://issues.apache.org/jira/browse/FLINK-31778
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.16.1, 1.15.4
>Reporter: Ilya Soin
>Assignee: Ilya Soin
>Priority: Major
> Labels: pull-request-available, stale-assigned
> Attachments: image-2023-04-12-15-23-46-669.png, screenshot-2.png
>
>
> {code:java}
> select CAST(commissions AS ARRAY>) as commissions
>from (select ARRAY[ROW(123), ROW(234)] commissions){code}
> Expected output:
> {code:java}
> +++
> | op | commissions |
> +++
> | +I | [(123.0), (234.0)] |
> +++
> {code}
> Actual output:
> {code:java}
> +++
> | op | commissions |
> +++
> | +I | [(234.0), (234.0)] |
> +++ {code}
> Full working example:
> https://gist.github.com/soin08/5e0038dbefeba9192706e05a78ef3bc1
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31778) Casting array of rows produces incorrect result
[
https://issues.apache.org/jira/browse/FLINK-31778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17770414#comment-17770414
]
david radley commented on FLINK-31778:
--
[~nictownsend] I have done a test on the scenario you describe in your comment
with the fix on and it does fix this scenario as well.
> Casting array of rows produces incorrect result
> ---
>
> Key: FLINK-31778
> URL: https://issues.apache.org/jira/browse/FLINK-31778
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.16.1, 1.15.4
>Reporter: Ilya Soin
>Assignee: Ilya Soin
>Priority: Major
> Labels: pull-request-available, stale-assigned
> Attachments: image-2023-04-12-15-23-46-669.png, screenshot-2.png
>
>
> {code:java}
> select CAST(commissions AS ARRAY>) as commissions
>from (select ARRAY[ROW(123), ROW(234)] commissions){code}
> Expected output:
> {code:java}
> +++
> | op | commissions |
> +++
> | +I | [(123.0), (234.0)] |
> +++
> {code}
> Actual output:
> {code:java}
> +++
> | op | commissions |
> +++
> | +I | [(234.0), (234.0)] |
> +++ {code}
> Full working example:
> https://gist.github.com/soin08/5e0038dbefeba9192706e05a78ef3bc1
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33165) Flink UI stack trace popup continually displayed when a job is deleted
[ https://issues.apache.org/jira/browse/FLINK-33165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17769998#comment-17769998 ] david radley edited comment on FLINK-33165 at 9/28/23 11:07 AM: [~Sergey Nuyanzin] thanks - good spot. It was a mistake I have removed that version. I have added the latest version as the affect version. was (Author: JIRAUSER300523): [~Sergey Nuyanzin] thanks - good spot. It was a mistake I have removed that version. > Flink UI stack trace popup continually displayed when a job is deleted > -- > > Key: FLINK-33165 > URL: https://issues.apache.org/jira/browse/FLINK-33165 > Project: Flink > Issue Type: Bug > Components: Runtime / Web Frontend >Affects Versions: 1.17.1 > Environment: MacOS M1 . >Reporter: david radley >Priority: Major > > If you run a job and view that running job in the Flink dashboard, then > delete the job, the UI displays a pop-up with an information icon and a > message and stack trace . This is repeatedly displayed every second or so. If > we must display a popup is should be once without the stack (which implies > there is an error - the stack trace is not useful to the user). We should > look into whether we need a popup in this case. It would be better to not > display a popup, as this should be just a state change. > An example popup content is: > *Server Response Message:* > org.apache.flink.runtime.rest.NotFoundException: Job > 4e332318d164d86e76a239eabd49bf93 not found > at > org.apache.flink.runtime.rest.handler.job.AbstractExecutionGraphHandler.lambda$handleRequest$1(AbstractExecutionGraphHandler.java:99) > at java.base/java.util.concurrent.CompletableFuture.uniExceptionally(Unknown > Source) > at > java.base/java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(Unknown > Source) > at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown > Source) > at > java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown > Source) > at > org.apache.flink.runtime.rest.handler.legacy.DefaultExecutionGraphCache.lambda$getExecutionGraphInternal$0(DefaultExecutionGraphCache.java:109) > at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown > Source) > at > java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown > Source) > at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown > Source) > at > java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown > Source) > at > org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$1(AkkaInvocationHandler.java:260) > at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown > Source) > at > java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown > Source) > at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown > Source) > at > java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown > Source) > at > org.apache.flink.util.concurrent.FutureUtils.doForward(FutureUtils.java:1298) > at > org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.lambda$guardCompletionWithContextClassLoader$1(ClassLoadingUtils.java:93) > at > org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68) > at > org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.lambda$guardCompletionWithContextClassLoader$2(ClassLoadingUtils.java:92) > at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown > Source) > at > java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown > Source) > at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown > Source) > at > java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown > Source) > at > org.apache.flink.runtime.concurrent.akka.AkkaFutureUtils$1.onComplete(AkkaFutureUtils.java:45) > at akka.dispatch.OnComplete.internal(Future.scala:299) > at akka.dispatch.OnComplete.internal(Future.scala:297) > at akka.dispatch.japi$CallbackBridge.apply(Future.scala:224) > at akka.dispatch.japi$CallbackBridge.apply(Future.scala:221) > at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64) > at > org.apache.flink.runtime.concurrent.akka.AkkaFutureUtils$DirectExecutionContext.execute(AkkaFutureUtils.java:65) > at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:72) > at > scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:288) > at > scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:288) > at
[jira] [Updated] (FLINK-33165) Flink UI stack trace popup continually displayed when a job is deleted
[ https://issues.apache.org/jira/browse/FLINK-33165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] david radley updated FLINK-33165: - Affects Version/s: 1.17.1 > Flink UI stack trace popup continually displayed when a job is deleted > -- > > Key: FLINK-33165 > URL: https://issues.apache.org/jira/browse/FLINK-33165 > Project: Flink > Issue Type: Bug > Components: Runtime / Web Frontend >Affects Versions: 1.17.1 > Environment: MacOS M1 . >Reporter: david radley >Priority: Major > > If you run a job and view that running job in the Flink dashboard, then > delete the job, the UI displays a pop-up with an information icon and a > message and stack trace . This is repeatedly displayed every second or so. If > we must display a popup is should be once without the stack (which implies > there is an error - the stack trace is not useful to the user). We should > look into whether we need a popup in this case. It would be better to not > display a popup, as this should be just a state change. > An example popup content is: > *Server Response Message:* > org.apache.flink.runtime.rest.NotFoundException: Job > 4e332318d164d86e76a239eabd49bf93 not found > at > org.apache.flink.runtime.rest.handler.job.AbstractExecutionGraphHandler.lambda$handleRequest$1(AbstractExecutionGraphHandler.java:99) > at java.base/java.util.concurrent.CompletableFuture.uniExceptionally(Unknown > Source) > at > java.base/java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(Unknown > Source) > at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown > Source) > at > java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown > Source) > at > org.apache.flink.runtime.rest.handler.legacy.DefaultExecutionGraphCache.lambda$getExecutionGraphInternal$0(DefaultExecutionGraphCache.java:109) > at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown > Source) > at > java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown > Source) > at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown > Source) > at > java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown > Source) > at > org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$1(AkkaInvocationHandler.java:260) > at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown > Source) > at > java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown > Source) > at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown > Source) > at > java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown > Source) > at > org.apache.flink.util.concurrent.FutureUtils.doForward(FutureUtils.java:1298) > at > org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.lambda$guardCompletionWithContextClassLoader$1(ClassLoadingUtils.java:93) > at > org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68) > at > org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.lambda$guardCompletionWithContextClassLoader$2(ClassLoadingUtils.java:92) > at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown > Source) > at > java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown > Source) > at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown > Source) > at > java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown > Source) > at > org.apache.flink.runtime.concurrent.akka.AkkaFutureUtils$1.onComplete(AkkaFutureUtils.java:45) > at akka.dispatch.OnComplete.internal(Future.scala:299) > at akka.dispatch.OnComplete.internal(Future.scala:297) > at akka.dispatch.japi$CallbackBridge.apply(Future.scala:224) > at akka.dispatch.japi$CallbackBridge.apply(Future.scala:221) > at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64) > at > org.apache.flink.runtime.concurrent.akka.AkkaFutureUtils$DirectExecutionContext.execute(AkkaFutureUtils.java:65) > at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:72) > at > scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:288) > at > scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:288) > at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:288) > at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:622) > at > akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:25) > at > akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:23) > at
[jira] [Commented] (FLINK-33165) Flink UI stack trace popup continually displayed when a job is deleted
[ https://issues.apache.org/jira/browse/FLINK-33165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17769998#comment-17769998 ] david radley commented on FLINK-33165: -- [~Sergey Nuyanzin] thanks - good spot. It was a mistake I have removed that version. > Flink UI stack trace popup continually displayed when a job is deleted > -- > > Key: FLINK-33165 > URL: https://issues.apache.org/jira/browse/FLINK-33165 > Project: Flink > Issue Type: Bug > Components: Runtime / Web Frontend > Environment: MacOS M1 . >Reporter: david radley >Priority: Major > > If you run a job and view that running job in the Flink dashboard, then > delete the job, the UI displays a pop-up with an information icon and a > message and stack trace . This is repeatedly displayed every second or so. If > we must display a popup is should be once without the stack (which implies > there is an error - the stack trace is not useful to the user). We should > look into whether we need a popup in this case. It would be better to not > display a popup, as this should be just a state change. > An example popup content is: > *Server Response Message:* > org.apache.flink.runtime.rest.NotFoundException: Job > 4e332318d164d86e76a239eabd49bf93 not found > at > org.apache.flink.runtime.rest.handler.job.AbstractExecutionGraphHandler.lambda$handleRequest$1(AbstractExecutionGraphHandler.java:99) > at java.base/java.util.concurrent.CompletableFuture.uniExceptionally(Unknown > Source) > at > java.base/java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(Unknown > Source) > at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown > Source) > at > java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown > Source) > at > org.apache.flink.runtime.rest.handler.legacy.DefaultExecutionGraphCache.lambda$getExecutionGraphInternal$0(DefaultExecutionGraphCache.java:109) > at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown > Source) > at > java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown > Source) > at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown > Source) > at > java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown > Source) > at > org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$1(AkkaInvocationHandler.java:260) > at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown > Source) > at > java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown > Source) > at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown > Source) > at > java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown > Source) > at > org.apache.flink.util.concurrent.FutureUtils.doForward(FutureUtils.java:1298) > at > org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.lambda$guardCompletionWithContextClassLoader$1(ClassLoadingUtils.java:93) > at > org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68) > at > org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.lambda$guardCompletionWithContextClassLoader$2(ClassLoadingUtils.java:92) > at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown > Source) > at > java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown > Source) > at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown > Source) > at > java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown > Source) > at > org.apache.flink.runtime.concurrent.akka.AkkaFutureUtils$1.onComplete(AkkaFutureUtils.java:45) > at akka.dispatch.OnComplete.internal(Future.scala:299) > at akka.dispatch.OnComplete.internal(Future.scala:297) > at akka.dispatch.japi$CallbackBridge.apply(Future.scala:224) > at akka.dispatch.japi$CallbackBridge.apply(Future.scala:221) > at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64) > at > org.apache.flink.runtime.concurrent.akka.AkkaFutureUtils$DirectExecutionContext.execute(AkkaFutureUtils.java:65) > at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:72) > at > scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:288) > at > scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:288) > at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:288) > at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:622) > at > akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:25) > at >
