[jira] [Commented] (FLINK-33040) flink-connector-hive builds might be blocked (but not fail) because Maven tries to access conjars.org repository (which times out)
[
https://issues.apache.org/jira/browse/FLINK-33040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17878985#comment-17878985
]
Martijn Visser commented on FLINK-33040:
[~mapohl] I do see in the CI runs that net.minidev:json-smart is being tried to
get downloaded though.
[~luoyuxia] Do you know if we can we safely exclude this dependency from the
Hive connector?
> flink-connector-hive builds might be blocked (but not fail) because Maven
> tries to access conjars.org repository (which times out)
> --
>
> Key: FLINK-33040
> URL: https://issues.apache.org/jira/browse/FLINK-33040
> Project: Flink
> Issue Type: Bug
> Components: Build System / CI
>Affects Versions: 1.16.2, 1.18.0, 1.17.1, 1.19.0
>Reporter: Matthias Pohl
>Priority: Major
>
> We (thanks to [~mdedetrich]) experienced timeouts when building
> {{flink-connectors/flink-connector-hive}} because Maven tries to access
> {{http://conjars.org}} to retrieve meta information for
> {{net.minidev:json-smart}} which fails because the repository is gone.
> [~gunnar.morling] already described this in his blog post
> https://www.morling.dev/blog/maven-what-are-you-waiting-for/. The result is a
> 75s delay. This doesn't appear to happen in CI.
> We investigated where this {{conjar}} repository is coming from. It turns out
> that the
> [org.apache.hive:hive-exec:2.3.9|https://repo1.maven.org/maven2/org/apache/hive/hive-exec/2.3.9/hive-exec-2.3.9.pom]
> dependency derives from its parent
> [org.apache.hive:hive:2.3.9|https://repo1.maven.org/maven2/org/apache/hive/hive/2.3.9/hive-2.3.9.pom]
> which pulls in the conjar.org repository:
> {code}
>
> conjars
> Conjars
> http://conjars.org/repo
> default
>
> true
> always
> warn
>
>
> {code}
> The subsequent hive dependency
> [org.apache.hive:hive:3.0.0|https://repo1.maven.org/maven2/org/apache/hive/hive/3.0.0/hive-3.0.0.pom]
> doesn't have this reference anymore.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-36189) Documentation build is failing with NPM installation problem
[
https://issues.apache.org/jira/browse/FLINK-36189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17878239#comment-17878239
]
Martijn Visser commented on FLINK-36189:
Also backported to:
apache/flink:release-1.20 69d293bff01a1188cb4beb64c9ce8bff43752427
apache/flink:release-1.19 3d1c41662186f15febf4020b8d6783c6d7687d3a
apache/flink:release-1.18 f8178b9daeafa6f7b2387e14404df0c7ac295cda
apache/flink:release-1.17 4fc3c76e81036da3b8bb90abc32f828df0dbaaa7
This is to make sure that the backports of FLINK-36173 don't cause the problems
of this ticket to appear in the backport branches too
> Documentation build is failing with NPM installation problem
>
>
> Key: FLINK-36189
> URL: https://issues.apache.org/jira/browse/FLINK-36189
> Project: Flink
> Issue Type: Bug
> Components: Build System, Documentation
>Affects Versions: 2.0.0
>Reporter: Martijn Visser
>Assignee: Martijn Visser
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 2.0.0
>
>
> {code:java}
> [INFO] --- frontend-maven-plugin:1.12.1:npm (npm install) @ flink-runtime-web
> ---
> [INFO] Running 'npm ci --cache-max=0 --no-save ${npm.proxy}' in
> /root/flink/flink-runtime-web/web-dashboard
> [INFO] npm WARN deprecated uuid@3.4.0: Please upgrade to version 7 or
> higher. Older versions may use Math.random() in certain circumstances, which
> is known to be problematic. See https://v8.dev/blog/math-random for details.
> [INFO] npm WARN deprecated har-validator@5.1.5: this library is no longer
> supported
> [INFO] npm WARN deprecated request@2.88.2: request has been deprecated, see
> https://github.com/request/request/issues/3142
> [INFO] npm notice
> [INFO] npm notice New major version of npm available! 8.1.2 -> 10.8.3
> [INFO] npm notice Changelog:
> [INFO] npm notice Run `npm install -g npm@10.8.3` to update!
> [INFO] npm notice
> [INFO] npm ERR! code 127
> [INFO] npm ERR! path
> /root/flink/flink-runtime-web/web-dashboard/node_modules/core-js
> [INFO] npm ERR! command failed
> [INFO] npm ERR! command sh -c node -e
> "try{require('./postinstall')}catch(e){}"
> [INFO] npm ERR! sh: 1: node: Permission denied
> {code}
> https://github.com/apache/flink/actions/runs/10624373229/job/29452554522#step:5:35663
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-36116) Javadocs aren't visible anymore for Flink 1.17 and above
[ https://issues.apache.org/jira/browse/FLINK-36116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser updated FLINK-36116: --- Fix Version/s: 1.17.3 1.18.2 1.19.2 1.20.1 > Javadocs aren't visible anymore for Flink 1.17 and above > > > Key: FLINK-36116 > URL: https://issues.apache.org/jira/browse/FLINK-36116 > Project: Flink > Issue Type: Bug > Components: Documentation >Affects Versions: 2.0.0, 1.17.2, 1.18.1, 1.20.0, 1.19.1 >Reporter: Martijn Visser >Assignee: Martijn Visser >Priority: Critical > Labels: pull-request-available > Fix For: 2.0.0, 1.17.3, 1.18.2, 1.19.2, 1.20.1 > > > The JavaDocs for 1.16 correctly open: > https://nightlies.apache.org/flink/flink-docs-release-1.16/api/java/ > However, for 1.17 and later versions, they aren't visible/rendered properly: > https://nightlies.apache.org/flink/flink-docs-release-1.17/api/java/ -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-36116) Javadocs aren't visible anymore for Flink 1.17 and above
[ https://issues.apache.org/jira/browse/FLINK-36116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser closed FLINK-36116. -- Resolution: Fixed > Javadocs aren't visible anymore for Flink 1.17 and above > > > Key: FLINK-36116 > URL: https://issues.apache.org/jira/browse/FLINK-36116 > Project: Flink > Issue Type: Bug > Components: Documentation >Affects Versions: 2.0.0, 1.17.2, 1.18.1, 1.20.0, 1.19.1 >Reporter: Martijn Visser >Assignee: Martijn Visser >Priority: Critical > Labels: pull-request-available > Fix For: 2.0.0, 1.17.3, 1.18.2, 1.19.2, 1.20.1 > > > The JavaDocs for 1.16 correctly open: > https://nightlies.apache.org/flink/flink-docs-release-1.16/api/java/ > However, for 1.17 and later versions, they aren't visible/rendered properly: > https://nightlies.apache.org/flink/flink-docs-release-1.17/api/java/ -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-36116) Javadocs aren't visible anymore for Flink 1.17 and above
[ https://issues.apache.org/jira/browse/FLINK-36116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17878238#comment-17878238 ] Martijn Visser commented on FLINK-36116: Fixed in: apache/flink:release-1.20 9021ad8d37856c3c1ed888e9abb05ad489dd9108 apache/flink:release-1.19 84aaa7cdb742470f678aa89d31eccb492c25f4e8 apache/flink:release-1.18 856d08b66e2275c680c1aca4cec1ef358c01ec8d apache/flink:release-1.17 e92a749b25469cee26bbcf0238677825dabba86a > Javadocs aren't visible anymore for Flink 1.17 and above > > > Key: FLINK-36116 > URL: https://issues.apache.org/jira/browse/FLINK-36116 > Project: Flink > Issue Type: Bug > Components: Documentation >Affects Versions: 2.0.0, 1.17.2, 1.18.1, 1.20.0, 1.19.1 >Reporter: Martijn Visser >Assignee: Martijn Visser >Priority: Critical > Labels: pull-request-available > Fix For: 2.0.0 > > > The JavaDocs for 1.16 correctly open: > https://nightlies.apache.org/flink/flink-docs-release-1.16/api/java/ > However, for 1.17 and later versions, they aren't visible/rendered properly: > https://nightlies.apache.org/flink/flink-docs-release-1.17/api/java/ -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-36189) Documentation build is failing with NPM installation problem
[
https://issues.apache.org/jira/browse/FLINK-36189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser closed FLINK-36189.
--
Fix Version/s: 2.0.0
Resolution: Fixed
Fixed in apache/flink:master e8ca6be6f9c24d8a1799bebdb77522fca2e28cbd
> Documentation build is failing with NPM installation problem
>
>
> Key: FLINK-36189
> URL: https://issues.apache.org/jira/browse/FLINK-36189
> Project: Flink
> Issue Type: Bug
> Components: Build System, Documentation
>Affects Versions: 2.0.0
>Reporter: Martijn Visser
>Assignee: Martijn Visser
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 2.0.0
>
>
> {code:java}
> [INFO] --- frontend-maven-plugin:1.12.1:npm (npm install) @ flink-runtime-web
> ---
> [INFO] Running 'npm ci --cache-max=0 --no-save ${npm.proxy}' in
> /root/flink/flink-runtime-web/web-dashboard
> [INFO] npm WARN deprecated uuid@3.4.0: Please upgrade to version 7 or
> higher. Older versions may use Math.random() in certain circumstances, which
> is known to be problematic. See https://v8.dev/blog/math-random for details.
> [INFO] npm WARN deprecated har-validator@5.1.5: this library is no longer
> supported
> [INFO] npm WARN deprecated request@2.88.2: request has been deprecated, see
> https://github.com/request/request/issues/3142
> [INFO] npm notice
> [INFO] npm notice New major version of npm available! 8.1.2 -> 10.8.3
> [INFO] npm notice Changelog:
> [INFO] npm notice Run `npm install -g npm@10.8.3` to update!
> [INFO] npm notice
> [INFO] npm ERR! code 127
> [INFO] npm ERR! path
> /root/flink/flink-runtime-web/web-dashboard/node_modules/core-js
> [INFO] npm ERR! command failed
> [INFO] npm ERR! command sh -c node -e
> "try{require('./postinstall')}catch(e){}"
> [INFO] npm ERR! sh: 1: node: Permission denied
> {code}
> https://github.com/apache/flink/actions/runs/10624373229/job/29452554522#step:5:35663
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-36189) Documentation build is failing with NPM installation problem
[
https://issues.apache.org/jira/browse/FLINK-36189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser reassigned FLINK-36189:
--
Assignee: Martijn Visser
> Documentation build is failing with NPM installation problem
>
>
> Key: FLINK-36189
> URL: https://issues.apache.org/jira/browse/FLINK-36189
> Project: Flink
> Issue Type: Bug
> Components: Build System, Documentation
>Affects Versions: 2.0.0
>Reporter: Martijn Visser
>Assignee: Martijn Visser
>Priority: Blocker
>
> {code:java}
> [INFO] --- frontend-maven-plugin:1.12.1:npm (npm install) @ flink-runtime-web
> ---
> [INFO] Running 'npm ci --cache-max=0 --no-save ${npm.proxy}' in
> /root/flink/flink-runtime-web/web-dashboard
> [INFO] npm WARN deprecated uuid@3.4.0: Please upgrade to version 7 or
> higher. Older versions may use Math.random() in certain circumstances, which
> is known to be problematic. See https://v8.dev/blog/math-random for details.
> [INFO] npm WARN deprecated har-validator@5.1.5: this library is no longer
> supported
> [INFO] npm WARN deprecated request@2.88.2: request has been deprecated, see
> https://github.com/request/request/issues/3142
> [INFO] npm notice
> [INFO] npm notice New major version of npm available! 8.1.2 -> 10.8.3
> [INFO] npm notice Changelog:
> [INFO] npm notice Run `npm install -g npm@10.8.3` to update!
> [INFO] npm notice
> [INFO] npm ERR! code 127
> [INFO] npm ERR! path
> /root/flink/flink-runtime-web/web-dashboard/node_modules/core-js
> [INFO] npm ERR! command failed
> [INFO] npm ERR! command sh -c node -e
> "try{require('./postinstall')}catch(e){}"
> [INFO] npm ERR! sh: 1: node: Permission denied
> {code}
> https://github.com/apache/flink/actions/runs/10624373229/job/29452554522#step:5:35663
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-36189) Documentation build is failing with NPM installation problem
Martijn Visser created FLINK-36189:
--
Summary: Documentation build is failing with NPM installation
problem
Key: FLINK-36189
URL: https://issues.apache.org/jira/browse/FLINK-36189
Project: Flink
Issue Type: Bug
Components: Build System, Documentation
Affects Versions: 2.0.0
Reporter: Martijn Visser
{code:java}
[INFO] --- frontend-maven-plugin:1.12.1:npm (npm install) @ flink-runtime-web
---
[INFO] Running 'npm ci --cache-max=0 --no-save ${npm.proxy}' in
/root/flink/flink-runtime-web/web-dashboard
[INFO] npm WARN deprecated uuid@3.4.0: Please upgrade to version 7 or higher.
Older versions may use Math.random() in certain circumstances, which is known
to be problematic. See https://v8.dev/blog/math-random for details.
[INFO] npm WARN deprecated har-validator@5.1.5: this library is no longer
supported
[INFO] npm WARN deprecated request@2.88.2: request has been deprecated, see
https://github.com/request/request/issues/3142
[INFO] npm notice
[INFO] npm notice New major version of npm available! 8.1.2 -> 10.8.3
[INFO] npm notice Changelog:
[INFO] npm notice Run `npm install -g npm@10.8.3` to update!
[INFO] npm notice
[INFO] npm ERR! code 127
[INFO] npm ERR! path
/root/flink/flink-runtime-web/web-dashboard/node_modules/core-js
[INFO] npm ERR! command failed
[INFO] npm ERR! command sh -c node -e "try{require('./postinstall')}catch(e){}"
[INFO] npm ERR! sh: 1: node: Permission denied
{code}
https://github.com/apache/flink/actions/runs/10624373229/job/29452554522#step:5:35663
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-36181) Drop support for Java 8 and use Java 17 by default
[ https://issues.apache.org/jira/browse/FLINK-36181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17878101#comment-17878101 ] Martijn Visser commented on FLINK-36181: [~Sergey Nuyanzin] I was planning to go for the latest, since we're breaking things with Flink 2.0 anyway > Drop support for Java 8 and use Java 17 by default > -- > > Key: FLINK-36181 > URL: https://issues.apache.org/jira/browse/FLINK-36181 > Project: Flink > Issue Type: Technical Debt > Components: Build System, Build System / Azure Pipelines, Build > System / CI >Affects Versions: 2.0.0 >Reporter: Martijn Visser >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-36181) Drop support for Java 8 and use Java 17 by default
[ https://issues.apache.org/jira/browse/FLINK-36181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17878101#comment-17878101 ] Martijn Visser edited comment on FLINK-36181 at 8/30/24 12:59 PM: -- [~Sergey Nuyanzin] I was planning to go for the latest 2.12.19, since we're breaking things with Flink 2.0 anyway was (Author: martijnvisser): [~Sergey Nuyanzin] I was planning to go for the latest, since we're breaking things with Flink 2.0 anyway > Drop support for Java 8 and use Java 17 by default > -- > > Key: FLINK-36181 > URL: https://issues.apache.org/jira/browse/FLINK-36181 > Project: Flink > Issue Type: Technical Debt > Components: Build System, Build System / Azure Pipelines, Build > System / CI >Affects Versions: 2.0.0 >Reporter: Martijn Visser >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-36181) Drop support for Java 8 and use Java 17 by default
Martijn Visser created FLINK-36181: -- Summary: Drop support for Java 8 and use Java 17 by default Key: FLINK-36181 URL: https://issues.apache.org/jira/browse/FLINK-36181 Project: Flink Issue Type: Technical Debt Components: Build System, Build System / Azure Pipelines, Build System / CI Affects Versions: 2.0.0 Reporter: Martijn Visser -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-36116) Javadocs aren't visible anymore for Flink 1.17 and above
[ https://issues.apache.org/jira/browse/FLINK-36116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser updated FLINK-36116: --- Fix Version/s: 2.0.0 > Javadocs aren't visible anymore for Flink 1.17 and above > > > Key: FLINK-36116 > URL: https://issues.apache.org/jira/browse/FLINK-36116 > Project: Flink > Issue Type: Bug > Components: Documentation >Affects Versions: 2.0.0, 1.17.2, 1.18.1, 1.20.0, 1.19.1 >Reporter: Martijn Visser >Assignee: Martijn Visser >Priority: Critical > Labels: pull-request-available > Fix For: 2.0.0 > > > The JavaDocs for 1.16 correctly open: > https://nightlies.apache.org/flink/flink-docs-release-1.16/api/java/ > However, for 1.17 and later versions, they aren't visible/rendered properly: > https://nightlies.apache.org/flink/flink-docs-release-1.17/api/java/ -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-36116) Javadocs aren't visible anymore for Flink 1.17 and above
[ https://issues.apache.org/jira/browse/FLINK-36116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17877721#comment-17877721 ] Martijn Visser commented on FLINK-36116: Fixed in apache/flink:master 2ca359a140b99f8e94c192fc7775074138436dca First running a build step, to validate that the problem has indeed been resolved, before backporting this to the other Flink versions that are affected > Javadocs aren't visible anymore for Flink 1.17 and above > > > Key: FLINK-36116 > URL: https://issues.apache.org/jira/browse/FLINK-36116 > Project: Flink > Issue Type: Bug > Components: Documentation >Affects Versions: 2.0.0, 1.17.2, 1.18.1, 1.20.0, 1.19.1 >Reporter: Martijn Visser >Assignee: Martijn Visser >Priority: Critical > Labels: pull-request-available > Fix For: 2.0.0 > > > The JavaDocs for 1.16 correctly open: > https://nightlies.apache.org/flink/flink-docs-release-1.16/api/java/ > However, for 1.17 and later versions, they aren't visible/rendered properly: > https://nightlies.apache.org/flink/flink-docs-release-1.17/api/java/ -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-36116) Javadocs aren't visible anymore for Flink 1.17 and above
[ https://issues.apache.org/jira/browse/FLINK-36116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser reassigned FLINK-36116: -- Assignee: Martijn Visser > Javadocs aren't visible anymore for Flink 1.17 and above > > > Key: FLINK-36116 > URL: https://issues.apache.org/jira/browse/FLINK-36116 > Project: Flink > Issue Type: Bug > Components: Documentation >Affects Versions: 2.0.0, 1.17.2, 1.18.1, 1.20.0, 1.19.1 >Reporter: Martijn Visser >Assignee: Martijn Visser >Priority: Critical > > The JavaDocs for 1.16 correctly open: > https://nightlies.apache.org/flink/flink-docs-release-1.16/api/java/ > However, for 1.17 and later versions, they aren't visible/rendered properly: > https://nightlies.apache.org/flink/flink-docs-release-1.17/api/java/ -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-36116) Javadocs aren't visible anymore for Flink 1.17 and above
[ https://issues.apache.org/jira/browse/FLINK-36116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser updated FLINK-36116: --- Priority: Critical (was: Major) > Javadocs aren't visible anymore for Flink 1.17 and above > > > Key: FLINK-36116 > URL: https://issues.apache.org/jira/browse/FLINK-36116 > Project: Flink > Issue Type: Bug > Components: Documentation >Affects Versions: 2.0.0, 1.17.2, 1.18.1, 1.20.0, 1.19.1 >Reporter: Martijn Visser >Priority: Critical > > The JavaDocs for 1.16 correctly open: > https://nightlies.apache.org/flink/flink-docs-release-1.16/api/java/ > However, for 1.17 and later versions, they aren't visible/rendered properly: > https://nightlies.apache.org/flink/flink-docs-release-1.17/api/java/ -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-36116) Javadocs aren't visible anymore for Flink 1.17 and above
Martijn Visser created FLINK-36116: -- Summary: Javadocs aren't visible anymore for Flink 1.17 and above Key: FLINK-36116 URL: https://issues.apache.org/jira/browse/FLINK-36116 Project: Flink Issue Type: Bug Components: Documentation Affects Versions: 1.19.1, 1.20.0, 1.18.1, 1.17.2, 2.0.0 Reporter: Martijn Visser The JavaDocs for 1.16 correctly open: https://nightlies.apache.org/flink/flink-docs-release-1.16/api/java/ However, for 1.17 and later versions, they aren't visible/rendered properly: https://nightlies.apache.org/flink/flink-docs-release-1.17/api/java/ -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29991) KinesisFirehoseSinkTest#firehoseSinkFailsWhenUnableToConnectToRemoteService failed
[
https://issues.apache.org/jira/browse/FLINK-29991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17874793#comment-17874793
]
Martijn Visser commented on FLINK-29991:
[~chalixar] Sure thing
> KinesisFirehoseSinkTest#firehoseSinkFailsWhenUnableToConnectToRemoteService
> failed
> ---
>
> Key: FLINK-29991
> URL: https://issues.apache.org/jira/browse/FLINK-29991
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kinesis
>Affects Versions: 1.15.2
>Reporter: Martijn Visser
>Assignee: Ahmed Hamdy
>Priority: Major
> Labels: stale-assigned, test-stability
>
> {code:java}
> Nov 10 10:22:53 [ERROR]
> org.apache.flink.connector.firehose.sink.KinesisFirehoseSinkTest.firehoseSinkFailsWhenUnableToConnectToRemoteService
> Time elapsed: 7.394 s <<< FAILURE!
> Nov 10 10:22:53 java.lang.AssertionError:
> Nov 10 10:22:53
> Nov 10 10:22:53 Expecting throwable message:
> Nov 10 10:22:53 "An OperatorEvent from an OperatorCoordinator to a task was
> lost. Triggering task failover to ensure consistency. Event:
> '[NoMoreSplitEvent]', targetTask: Source: Sequence Source -> Map -> Map ->
> Sink: Writer (15/32) - execution #0"
> Nov 10 10:22:53 to contain:
> Nov 10 10:22:53 "Received an UnknownHostException when attempting to
> interact with a service."
> Nov 10 10:22:53 but did not.
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=43017&view=logs&j=d44f43ce-542c-597d-bf94-b0718c71e5e8&t=ed165f3f-d0f6-524b-5279-86f8ee7d0e2d&l=44513
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-29991) KinesisFirehoseSinkTest#firehoseSinkFailsWhenUnableToConnectToRemoteService failed
[
https://issues.apache.org/jira/browse/FLINK-29991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser closed FLINK-29991.
--
Resolution: Cannot Reproduce
> KinesisFirehoseSinkTest#firehoseSinkFailsWhenUnableToConnectToRemoteService
> failed
> ---
>
> Key: FLINK-29991
> URL: https://issues.apache.org/jira/browse/FLINK-29991
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kinesis
>Affects Versions: 1.15.2
>Reporter: Martijn Visser
>Assignee: Ahmed Hamdy
>Priority: Major
> Labels: stale-assigned, test-stability
>
> {code:java}
> Nov 10 10:22:53 [ERROR]
> org.apache.flink.connector.firehose.sink.KinesisFirehoseSinkTest.firehoseSinkFailsWhenUnableToConnectToRemoteService
> Time elapsed: 7.394 s <<< FAILURE!
> Nov 10 10:22:53 java.lang.AssertionError:
> Nov 10 10:22:53
> Nov 10 10:22:53 Expecting throwable message:
> Nov 10 10:22:53 "An OperatorEvent from an OperatorCoordinator to a task was
> lost. Triggering task failover to ensure consistency. Event:
> '[NoMoreSplitEvent]', targetTask: Source: Sequence Source -> Map -> Map ->
> Sink: Writer (15/32) - execution #0"
> Nov 10 10:22:53 to contain:
> Nov 10 10:22:53 "Received an UnknownHostException when attempting to
> interact with a service."
> Nov 10 10:22:53 but did not.
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=43017&view=logs&j=d44f43ce-542c-597d-bf94-b0718c71e5e8&t=ed165f3f-d0f6-524b-5279-86f8ee7d0e2d&l=44513
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-29031) FlinkKinesisConsumerTest.testSourceSynchronization failed with AssertionFailedError
[
https://issues.apache.org/jira/browse/FLINK-29031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser closed FLINK-29031.
--
Fix Version/s: (was: 2.0.0)
Resolution: Cannot Reproduce
> FlinkKinesisConsumerTest.testSourceSynchronization failed with
> AssertionFailedError
> ---
>
> Key: FLINK-29031
> URL: https://issues.apache.org/jira/browse/FLINK-29031
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kinesis
>Affects Versions: 1.16.0
>Reporter: Huang Xingbo
>Priority: Major
> Labels: test-stability
>
> {code:java}
> 2022-08-18T03:58:00.0197521Z Aug 18 03:58:00 [ERROR]
> org.apache.flink.streaming.connectors.kinesis.FlinkKinesisConsumerTest.testSourceSynchronization
> Time elapsed: 10.191 s <<< FAILURE!
> 2022-08-18T03:58:00.0198736Z Aug 18 03:58:00
> org.opentest4j.AssertionFailedError:
> 2022-08-18T03:58:00.0199434Z Aug 18 03:58:00 [first record received]
> 2022-08-18T03:58:00.0200022Z Aug 18 03:58:00 expected: 1
> 2022-08-18T03:58:00.0200577Z Aug 18 03:58:00 but was: 0
> 2022-08-18T03:58:00.0201285Z Aug 18 03:58:00 at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> 2022-08-18T03:58:00.0202337Z Aug 18 03:58:00 at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> 2022-08-18T03:58:00.0203442Z Aug 18 03:58:00 at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> 2022-08-18T03:58:00.0205001Z Aug 18 03:58:00 at
> org.apache.flink.streaming.connectors.kinesis.FlinkKinesisConsumerTest.testSourceSynchronization(FlinkKinesisConsumerTest.java:1149)
> 2022-08-18T03:58:00.0206078Z Aug 18 03:58:00 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2022-08-18T03:58:00.0206994Z Aug 18 03:58:00 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2022-08-18T03:58:00.0208019Z Aug 18 03:58:00 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2022-08-18T03:58:00.0208952Z Aug 18 03:58:00 at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2022-08-18T03:58:00.0209816Z Aug 18 03:58:00 at
> org.junit.internal.runners.TestMethod.invoke(TestMethod.java:68)
> 2022-08-18T03:58:00.0211029Z Aug 18 03:58:00 at
> org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$PowerMockJUnit44MethodRunner.runTestMethod(PowerMockJUnit44RunnerDelegateImpl.java:326)
> 2022-08-18T03:58:00.0212264Z Aug 18 03:58:00 at
> org.junit.internal.runners.MethodRoadie$2.run(MethodRoadie.java:89)
> 2022-08-18T03:58:00.0213266Z Aug 18 03:58:00 at
> org.junit.internal.runners.MethodRoadie.runBeforesThenTestThenAfters(MethodRoadie.java:97)
> 2022-08-18T03:58:00.0214530Z Aug 18 03:58:00 at
> org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$PowerMockJUnit44MethodRunner.executeTest(PowerMockJUnit44RunnerDelegateImpl.java:310)
> 2022-08-18T03:58:00.0216259Z Aug 18 03:58:00 at
> org.powermock.modules.junit4.internal.impl.PowerMockJUnit47RunnerDelegateImpl$PowerMockJUnit47MethodRunner.executeTestInSuper(PowerMockJUnit47RunnerDelegateImpl.java:131)
> 2022-08-18T03:58:00.0217769Z Aug 18 03:58:00 at
> org.powermock.modules.junit4.internal.impl.PowerMockJUnit47RunnerDelegateImpl$PowerMockJUnit47MethodRunner.access$100(PowerMockJUnit47RunnerDelegateImpl.java:59)
> 2022-08-18T03:58:00.0219348Z Aug 18 03:58:00 at
> org.powermock.modules.junit4.internal.impl.PowerMockJUnit47RunnerDelegateImpl$PowerMockJUnit47MethodRunner$TestExecutorStatement.evaluate(PowerMockJUnit47RunnerDelegateImpl.java:147)
> 2022-08-18T03:58:00.0220610Z Aug 18 03:58:00 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> 2022-08-18T03:58:00.0221543Z Aug 18 03:58:00 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> 2022-08-18T03:58:00.0222807Z Aug 18 03:58:00 at
> org.powermock.modules.junit4.internal.impl.PowerMockJUnit47RunnerDelegateImpl$PowerMockJUnit47MethodRunner.evaluateStatement(PowerMockJUnit47RunnerDelegateImpl.java:107)
> 2022-08-18T03:58:00.0224339Z Aug 18 03:58:00 at
> org.powermock.modules.junit4.internal.impl.PowerMockJUnit47RunnerDelegateImpl$PowerMockJUnit47MethodRunner.executeTest(PowerMockJUnit47RunnerDelegateImpl.java:82)
> 2022-08-18T03:58:00.0226110Z Aug 18 03:58:00 at
> org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$PowerMockJUnit44MethodRunner.runBeforesThenTestThenAfters(PowerMockJUnit44RunnerDelegateImpl.java:298)
> 2022-08-18T03:58:00.0227637Z Aug 18 03:58:00 at
> org.junit.internal.runners.MethodRoadie.runTest(MethodRoadie.java:87)
> 2022-08-18T03:58:00.0228583Z Aug 18 03:58:00 at
> org.junit.intern
[jira] [Commented] (FLINK-35877) Shade protobuf in flink
[ https://issues.apache.org/jira/browse/FLINK-35877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17868001#comment-17868001 ] Martijn Visser commented on FLINK-35877: This is tagged for Shaded, but it's not in https://github.com/apache/flink-shaded so this isn't the right component > Shade protobuf in flink > --- > > Key: FLINK-35877 > URL: https://issues.apache.org/jira/browse/FLINK-35877 > Project: Flink > Issue Type: Improvement > Components: BuildSystem / Shaded >Affects Versions: 1.19.2 >Reporter: zhuanshenbsj1 >Priority: Minor > Labels: pull-request-available > Fix For: 1.19.2 > > > Shade the classes in protobuf to avoid class conflict. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35861) The dependency of kafka sql connector seems wrong
[ https://issues.apache.org/jira/browse/FLINK-35861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17867027#comment-17867027 ] Martijn Visser commented on FLINK-35861: The docs are correct imho; the Maven dependency assumes that users will add the necessary Kafka Client themselves, while the download link for the SQL Client points to the fat JAR > The dependency of kafka sql connector seems wrong > - > > Key: FLINK-35861 > URL: https://issues.apache.org/jira/browse/FLINK-35861 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka, Documentation >Affects Versions: 1.18.1, 1.19.1 >Reporter: xuyang >Priority: Minor > > The dependency of it should be `flink-sql-connector-kafka` instead of > `flink-connector-kafka`, right? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35832) IFNULL returns incorrect result in Flink SQL
[
https://issues.apache.org/jira/browse/FLINK-35832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17866644#comment-17866644
]
Martijn Visser commented on FLINK-35832:
Thanks for helping with the fix! Will we also backport this to 1.20/1.19?
> IFNULL returns incorrect result in Flink SQL
>
>
> Key: FLINK-35832
> URL: https://issues.apache.org/jira/browse/FLINK-35832
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 2.0.0, 1.15.4, 1.16.3, 1.17.2, 1.18.1, 1.20.0, 1.19.1
>Reporter: Yu Chen
>Assignee: Dylan He
>Priority: Critical
> Labels: pull-request-available
>
> Run following SQL in sql-client:
> The correct result should be '16', but we got '1' on the master.
> {code:java}
> Flink SQL> SET 'sql-client.execution.result-mode' = 'tableau';
> [INFO] Execute statement succeeded.
> Flink SQL> select JSON_VALUE('{"a":16}','$.a'),
> IFNULL(JSON_VALUE('{"a":16}','$.a'),'0');
> ++++
> | op | EXPR$0 | EXPR$1 |
> ++++
> | +I | 16 | 1 |
> ++++
> Received a total of 1 row (0.30 seconds){code}
>
> With some quick debugging, I guess it may be caused by
> [FLINK-24413|https://issues.apache.org/jira/browse/FLINK-24413] which was
> introduced in Flink version 1.15.
>
> I think the wrong result '1' was produced because the simplifying SQL
> procedure assumed that parameter 1 and parameter 2 ('0' was char) of IFNULL
> were of the same type, and therefore implicitly cast '16' to char, resulting
> in the incorrect result.
>
> I have tested the SQL in the following version:
>
> ||Flink Version||Result||
> |1.13|16,16|
> |1.17|16,1|
> |1.19|16,1|
> |master|16,1|
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-35832) IFNULL returns incorrect result in Flink SQL
[
https://issues.apache.org/jira/browse/FLINK-35832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser updated FLINK-35832:
---
Affects Version/s: 1.19.1
1.18.1
1.17.2
1.16.3
1.15.4
1.20.0
> IFNULL returns incorrect result in Flink SQL
>
>
> Key: FLINK-35832
> URL: https://issues.apache.org/jira/browse/FLINK-35832
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 2.0.0, 1.15.4, 1.16.3, 1.17.2, 1.18.1, 1.20.0, 1.19.1
>Reporter: Yu Chen
>Assignee: Dylan He
>Priority: Critical
> Labels: pull-request-available
>
> Run following SQL in sql-client:
> The correct result should be '16', but we got '1' on the master.
> {code:java}
> Flink SQL> SET 'sql-client.execution.result-mode' = 'tableau';
> [INFO] Execute statement succeeded.
> Flink SQL> select JSON_VALUE('{"a":16}','$.a'),
> IFNULL(JSON_VALUE('{"a":16}','$.a'),'0');
> ++++
> | op | EXPR$0 | EXPR$1 |
> ++++
> | +I | 16 | 1 |
> ++++
> Received a total of 1 row (0.30 seconds){code}
>
> With some quick debugging, I guess it may be caused by
> [FLINK-24413|https://issues.apache.org/jira/browse/FLINK-24413] which was
> introduced in Flink version 1.15.
>
> I think the wrong result '1' was produced because the simplifying SQL
> procedure assumed that parameter 1 and parameter 2 ('0' was char) of IFNULL
> were of the same type, and therefore implicitly cast '16' to char, resulting
> in the incorrect result.
>
> I have tested the SQL in the following version:
>
> ||Flink Version||Result||
> |1.13|16,16|
> |1.17|16,1|
> |1.19|16,1|
> |master|16,1|
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-35829) StreamPhysicalWindowTableFunction doesn't always require watermark
[ https://issues.apache.org/jira/browse/FLINK-35829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser reassigned FLINK-35829: -- Assignee: xuyang > StreamPhysicalWindowTableFunction doesn't always require watermark > -- > > Key: FLINK-35829 > URL: https://issues.apache.org/jira/browse/FLINK-35829 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Planner >Affects Versions: 1.20.0, 1.19.1 >Reporter: xuyang >Assignee: xuyang >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35826) [SQL] Sliding window may produce unstable calculations when processing changelog data.
[ https://issues.apache.org/jira/browse/FLINK-35826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17865378#comment-17865378 ] Martijn Visser commented on FLINK-35826: [~xuyangzhong] given that you worked on FLINK-20281 I could imagine that this ticket is interesting for you? > [SQL] Sliding window may produce unstable calculations when processing > changelog data. > -- > > Key: FLINK-35826 > URL: https://issues.apache.org/jira/browse/FLINK-35826 > Project: Flink > Issue Type: Bug > Components: Table SQL / Planner >Affects Versions: 1.20.0 > Environment: flink with release-1.20 >Reporter: Yuan Kui >Priority: Major > Attachments: image-2024-07-12-14-27-58-061.png > > > Calculation results may be unstable when using a sliding window to process > changelog data. The test results are partial success and partial failure: > !image-2024-07-12-14-27-58-061.png! > See the documentation and code for more details. > [https://docs.google.com/document/d/1JmwSLs4SJvZKe7kqALqVBZ-1F1OyPmiWw8J6Ug6vqW0/edit?usp=sharing] > code： > [[BUG] Reproduce the issue of unstable sliding window calculation results · > yuchengxin/flink@c003e45 > (github.com)|https://github.com/yuchengxin/flink/commit/c003e45082e0d1464111c286ac9c7abb79527492] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35796) Ensure that MailboxExecutor.submit is used correctly
[
https://issues.apache.org/jira/browse/FLINK-35796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17864668#comment-17864668
]
Martijn Visser commented on FLINK-35796:
[~arvid] Will you backport it also to the supported 1.* branches?
> Ensure that MailboxExecutor.submit is used correctly
>
>
> Key: FLINK-35796
> URL: https://issues.apache.org/jira/browse/FLINK-35796
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 2.0.0
>Reporter: Arvid Heise
>Assignee: Arvid Heise
>Priority: Major
> Labels: pull-request-available
>
> https://issues.apache.org/jira/browse/FLINK-34470 showed that
> MailboxExecutor.submit may result in unexpected exception handling: while
> {{execute}} will bubble up the exception in the task thread and result in
> some fault, {{submit}} can hide the exception because the API assumes that
> the returned {{Future}} is checked for it explicitly or implicitly.
> We can solve the situation by improving the doc of MailboxExecutor and
> double-check the internal usages.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-35789) Allow defining watermarks & PRIMARY KEY in CREATE TABLE AS (CTAS)
[ https://issues.apache.org/jira/browse/FLINK-35789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser reassigned FLINK-35789: -- Assignee: Sergio Peña > Allow defining watermarks & PRIMARY KEY in CREATE TABLE AS (CTAS) > - > > Key: FLINK-35789 > URL: https://issues.apache.org/jira/browse/FLINK-35789 > Project: Flink > Issue Type: Sub-task >Reporter: Sergio Peña >Assignee: Sergio Peña >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-34111) Add JSON_QUOTE and JSON_UNQUOTE function
[ https://issues.apache.org/jira/browse/FLINK-34111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser reassigned FLINK-34111: -- Assignee: Anupam Aggarwal > Add JSON_QUOTE and JSON_UNQUOTE function > > > Key: FLINK-34111 > URL: https://issues.apache.org/jira/browse/FLINK-34111 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Reporter: Martijn Visser >Assignee: Anupam Aggarwal >Priority: Major > Labels: pull-request-available > > Escapes or unescapes a JSON string removing traces of offending characters > that could prevent parsing. > Proposal: > - JSON_QUOTE: Quotes a string by wrapping it with double quote characters and > escaping interior quote and other characters, then returning the result as a > utf8mb4 string. Returns NULL if the argument is NULL. > - JSON_UNQUOTE: Unquotes value and returns the result as a string. Returns > NULL if the argument is NULL. An error occurs if the value starts and ends > with double quotes but is not a valid JSON string literal. > The following characters are reserved in JSON and must be properly escaped to > be used in strings: > Backspace is replaced with \b > Form feed is replaced with \f > Newline is replaced with \n > Carriage return is replaced with \r > Tab is replaced with \t > Double quote is replaced with \" > Backslash is replaced with \\ > This function exists in MySQL: > - > https://dev.mysql.com/doc/refman/8.0/en/json-creation-functions.html#function_json-quote > - > https://dev.mysql.com/doc/refman/8.0/en/json-modification-functions.html#function_json-unquote > It's still open in Calcite CALCITE-3130 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-34111) Add JSON_QUOTE and JSON_UNQUOTE function
[ https://issues.apache.org/jira/browse/FLINK-34111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser reassigned FLINK-34111: -- Assignee: (was: Anupam) > Add JSON_QUOTE and JSON_UNQUOTE function > > > Key: FLINK-34111 > URL: https://issues.apache.org/jira/browse/FLINK-34111 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Reporter: Martijn Visser >Priority: Major > Labels: pull-request-available > > Escapes or unescapes a JSON string removing traces of offending characters > that could prevent parsing. > Proposal: > - JSON_QUOTE: Quotes a string by wrapping it with double quote characters and > escaping interior quote and other characters, then returning the result as a > utf8mb4 string. Returns NULL if the argument is NULL. > - JSON_UNQUOTE: Unquotes value and returns the result as a string. Returns > NULL if the argument is NULL. An error occurs if the value starts and ends > with double quotes but is not a valid JSON string literal. > The following characters are reserved in JSON and must be properly escaped to > be used in strings: > Backspace is replaced with \b > Form feed is replaced with \f > Newline is replaced with \n > Carriage return is replaced with \r > Tab is replaced with \t > Double quote is replaced with \" > Backslash is replaced with \\ > This function exists in MySQL: > - > https://dev.mysql.com/doc/refman/8.0/en/json-creation-functions.html#function_json-quote > - > https://dev.mysql.com/doc/refman/8.0/en/json-modification-functions.html#function_json-unquote > It's still open in Calcite CALCITE-3130 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-34111) Add JSON_QUOTE and JSON_UNQUOTE function
[ https://issues.apache.org/jira/browse/FLINK-34111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser reassigned FLINK-34111: -- Assignee: Anupam (was: Jeyhun Karimov) > Add JSON_QUOTE and JSON_UNQUOTE function > > > Key: FLINK-34111 > URL: https://issues.apache.org/jira/browse/FLINK-34111 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Reporter: Martijn Visser >Assignee: Anupam >Priority: Major > Labels: pull-request-available > > Escapes or unescapes a JSON string removing traces of offending characters > that could prevent parsing. > Proposal: > - JSON_QUOTE: Quotes a string by wrapping it with double quote characters and > escaping interior quote and other characters, then returning the result as a > utf8mb4 string. Returns NULL if the argument is NULL. > - JSON_UNQUOTE: Unquotes value and returns the result as a string. Returns > NULL if the argument is NULL. An error occurs if the value starts and ends > with double quotes but is not a valid JSON string literal. > The following characters are reserved in JSON and must be properly escaped to > be used in strings: > Backspace is replaced with \b > Form feed is replaced with \f > Newline is replaced with \n > Carriage return is replaced with \r > Tab is replaced with \t > Double quote is replaced with \" > Backslash is replaced with \\ > This function exists in MySQL: > - > https://dev.mysql.com/doc/refman/8.0/en/json-creation-functions.html#function_json-quote > - > https://dev.mysql.com/doc/refman/8.0/en/json-modification-functions.html#function_json-unquote > It's still open in Calcite CALCITE-3130 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-25217) FLIP-190: Support Version Upgrades for Table API & SQL Programs
[ https://issues.apache.org/jira/browse/FLINK-25217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17864495#comment-17864495 ] Martijn Visser commented on FLINK-25217: [~fsk119] Since Flink 1.15 actually. > FLIP-190: Support Version Upgrades for Table API & SQL Programs > --- > > Key: FLINK-25217 > URL: https://issues.apache.org/jira/browse/FLINK-25217 > Project: Flink > Issue Type: New Feature > Components: Table SQL / API, Table SQL / Planner >Reporter: Timo Walther >Priority: Minor > Labels: auto-deprioritized-major, pull-request-available > > Nowadays, the Table & SQL API is as important to Flink as the DataStream API. > It is one of the main abstractions for expressing pipelines that perform > stateful stream processing. Users expect the same backwards compatibility > guarantees when upgrading to a newer Flink version as with the DataStream API. > In particular, this means: > * once the operator topology is defined, it remains static and does not > change between Flink versions, unless resulting in better performance, > * business logic (defined using expressions and functions in queries) behaves > identical as before the version upgrade, > * the state of a Table & SQL API program can be restored from a savepoint of > a previous version, > * adding or removing stateful operators should be made possible in the > DataStream API. > The same query can remain up and running after upgrades. > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191336489 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-35794) Ability to skip kubernetes tests in e2e nightly test script
[ https://issues.apache.org/jira/browse/FLINK-35794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser closed FLINK-35794. -- Resolution: Won't Fix > Ability to skip kubernetes tests in e2e nightly test script > --- > > Key: FLINK-35794 > URL: https://issues.apache.org/jira/browse/FLINK-35794 > Project: Flink > Issue Type: Improvement > Components: Tests >Reporter: David Kornel >Priority: Major > Labels: pull-request-available > > Currently if environment for testing does not support running minikube which > is installed by script we can't run nightly test executed by script. So it > would be nice to add skip mark and then kubernetes tests are skipped. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35794) Ability to skip kubernetes tests in e2e nightly test script
[ https://issues.apache.org/jira/browse/FLINK-35794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17864159#comment-17864159 ] Martijn Visser commented on FLINK-35794: Yeah, but in that case you're running a different test then Flink CI and you could have that it works for you, but fails for Flink, and then you have to spend another cycle of testing which causes the saved resources to be gone :) Plus, you would have to modify your code already to skip the tests (since it's just adding a parameter). I'm going to close this ticket as it's not useful for the Flink project overall, but I do appreciate the discussion! > Ability to skip kubernetes tests in e2e nightly test script > --- > > Key: FLINK-35794 > URL: https://issues.apache.org/jira/browse/FLINK-35794 > Project: Flink > Issue Type: Improvement > Components: Tests >Reporter: David Kornel >Priority: Major > Labels: pull-request-available > > Currently if environment for testing does not support running minikube which > is installed by script we can't run nightly test executed by script. So it > would be nice to add skip mark and then kubernetes tests are skipped. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-35794) Ability to skip kubernetes tests in e2e nightly test script
[ https://issues.apache.org/jira/browse/FLINK-35794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17864150#comment-17864150 ] Martijn Visser edited comment on FLINK-35794 at 7/9/24 11:03 AM: - [~kornys] The CI uses run-nightly-tests.sh during the nightly runs., that's basically the entire point for that file :) It's not meant running these tests locally, that's pretty much not possible Edit: that's also what the README says: bq. NOTICE: We do not recommend executing the nightly test script on production or personal desktop systems, as tests contained there might modify the environment (leftover processes, modification of system files, request for root permissions via sudo, ...). was (Author: martijnvisser): [~kornys] The CI uses run-nightly-tests.sh during the nightly runs., that's basically the entire point for that file :) It's not meant running these tests locally, that's pretty much not possible > Ability to skip kubernetes tests in e2e nightly test script > --- > > Key: FLINK-35794 > URL: https://issues.apache.org/jira/browse/FLINK-35794 > Project: Flink > Issue Type: Improvement > Components: Tests >Reporter: David Kornel >Priority: Major > Labels: pull-request-available > > Currently if environment for testing does not support running minikube which > is installed by script we can't run nightly test executed by script. So it > would be nice to add skip mark and then kubernetes tests are skipped. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35794) Ability to skip kubernetes tests in e2e nightly test script
[ https://issues.apache.org/jira/browse/FLINK-35794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17864150#comment-17864150 ] Martijn Visser commented on FLINK-35794: [~kornys] The CI uses run-nightly-tests.sh during the nightly runs., that's basically the entire point for that file :) It's not meant running these tests locally, that's pretty much not possible > Ability to skip kubernetes tests in e2e nightly test script > --- > > Key: FLINK-35794 > URL: https://issues.apache.org/jira/browse/FLINK-35794 > Project: Flink > Issue Type: Improvement > Components: Tests >Reporter: David Kornel >Priority: Major > Labels: pull-request-available > > Currently if environment for testing does not support running minikube which > is installed by script we can't run nightly test executed by script. So it > would be nice to add skip mark and then kubernetes tests are skipped. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35794) Ability to skip kubernetes tests in e2e nightly test script
[ https://issues.apache.org/jira/browse/FLINK-35794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17864088#comment-17864088 ] Martijn Visser commented on FLINK-35794: [~kornys] This would potentially affect all CI tests, skipping those. Why would we want this as Flink? The nightly tests are specifically meant for CI testing > Ability to skip kubernetes tests in e2e nightly test script > --- > > Key: FLINK-35794 > URL: https://issues.apache.org/jira/browse/FLINK-35794 > Project: Flink > Issue Type: Improvement > Components: Tests >Reporter: David Kornel >Priority: Major > Labels: pull-request-available > > Currently if environment for testing does not support running minikube which > is installed by script we can't run nightly test executed by script. So it > would be nice to add skip mark and then kubernetes tests are skipped. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35749) Kafka sink component will lose data when kafka cluster is unavailable for a while
[ https://issues.apache.org/jira/browse/FLINK-35749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser updated FLINK-35749: --- Affects Version/s: kafka-3.2.0 > Kafka sink component will lose data when kafka cluster is unavailable for a > while > - > > Key: FLINK-35749 > URL: https://issues.apache.org/jira/browse/FLINK-35749 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka >Affects Versions: 1.16.2, 1.17.1, kafka-3.2.0 >Reporter: Jimmy Zhao >Assignee: Jimmy Zhao >Priority: Blocker > Labels: pull-request-available > > As the title described, here is the procedure to reproduce the problem: > 1. develop a simple flink stream job to consume from one kafka topic and sink > to anthoer kafka sever and topic > 2. make amount of kafka message and produce to the source kafka topic, record > the message number > 3. start the flink stream job, and config to cosume from earliest source > topic offset > 4. during the job cosuming the source topic, restart the kafka cluster(we use > aws MSK) > 5. the flink job will not throw any Exception like nothing happened, but only > print error log like : [kafka-producer-network-thread | producer-2] INFO > org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-2] Node > 2 disconnected. > 6. wait for the kafka cluster finished restarting and all the source kafka > message consumed > 7. count the target kafka topic message number, compare to the source, there > is a high probability of data loss（more than 50%） -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-35785) Executing query in SQL client results in "java.lang.ClassNotFoundException: org.apache.flink.core.execution.RestoreMode"
[
https://issues.apache.org/jira/browse/FLINK-35785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser closed FLINK-35785.
--
Resolution: Cannot Reproduce
[~Weijie Guo] [~ferenc-csaky] I'm not sure why, but after manually killing the
Flink java processes and trying it again, it does work. I guess there was some
left-over process that interfered with RC0. Apologies.
> Executing query in SQL client results in "java.lang.ClassNotFoundException:
> org.apache.flink.core.execution.RestoreMode"
>
>
> Key: FLINK-35785
> URL: https://issues.apache.org/jira/browse/FLINK-35785
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Table SQL / Client
>Reporter: Martijn Visser
>Priority: Blocker
>
> Tested with Flink 1.20 RC0
> Reproducer:
> {code:sql}
> CREATE TABLE `product` (
> id INT,
> brandId INT,
> PRIMARY KEY (id) NOT ENFORCED
> ) WITH (
> 'connector' = 'datagen',
> 'rows-per-second' = '10',
> 'fields.id.kind' = 'random',
> 'fields.brandId.min' = '1',
> 'fields.brandId.max' = '100'
> );
> {code}
> Followed by:
> {code:sql}
> SELECT * FROM product
> {code}
> Results in:
> [ERROR] Could not execute SQL statement. Reason:
> java.lang.ClassNotFoundException: org.apache.flink.core.execution.RestoreMode
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-35785) Executing query in SQL client results in "java.lang.ClassNotFoundException: org.apache.flink.core.execution.RestoreMode"
[
https://issues.apache.org/jira/browse/FLINK-35785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser updated FLINK-35785:
---
Description:
Tested with Flink 1.20 RC0
Reproducer:
{code:sql}
CREATE TABLE `product` (
id INT,
brandId INT,
PRIMARY KEY (id) NOT ENFORCED
) WITH (
'connector' = 'datagen',
'rows-per-second' = '10',
'fields.id.kind' = 'random',
'fields.brandId.min' = '1',
'fields.brandId.max' = '100'
);
{code}
Followed by:
{code:sql}
SELECT * FROM product
{code}
Results in:
[ERROR] Could not execute SQL statement. Reason:
java.lang.ClassNotFoundException: org.apache.flink.core.execution.RestoreMode
was:
Tested with Flink 1.20 RC0
Reproducer:
{code:sql}
-- Create the product table
CREATE TABLE `product` (
id INT,
brandId INT,
grade STRING,
PRIMARY KEY (id) NOT ENFORCED
) WITH (
'connector' = 'datagen',
'rows-per-second' = '10',
'fields.id.kind' = 'random',
'fields.brandId.min' = '1',
'fields.brandId.max' = '100',
'fields.grade.length' = '10'
);
{code}
Followed by:
{code:sql}
SELECT * FROM product
{code}
Results in:
[ERROR] Could not execute SQL statement. Reason:
java.lang.ClassNotFoundException: org.apache.flink.core.execution.RestoreMode
> Executing query in SQL client results in "java.lang.ClassNotFoundException:
> org.apache.flink.core.execution.RestoreMode"
>
>
> Key: FLINK-35785
> URL: https://issues.apache.org/jira/browse/FLINK-35785
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Table SQL / Client
>Reporter: Martijn Visser
>Priority: Blocker
>
> Tested with Flink 1.20 RC0
> Reproducer:
> {code:sql}
> CREATE TABLE `product` (
> id INT,
> brandId INT,
> PRIMARY KEY (id) NOT ENFORCED
> ) WITH (
> 'connector' = 'datagen',
> 'rows-per-second' = '10',
> 'fields.id.kind' = 'random',
> 'fields.brandId.min' = '1',
> 'fields.brandId.max' = '100'
> );
> {code}
> Followed by:
> {code:sql}
> SELECT * FROM product
> {code}
> Results in:
> [ERROR] Could not execute SQL statement. Reason:
> java.lang.ClassNotFoundException: org.apache.flink.core.execution.RestoreMode
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-35785) Executing query in SQL client results in "java.lang.ClassNotFoundException: org.apache.flink.core.execution.RestoreMode"
Martijn Visser created FLINK-35785:
--
Summary: Executing query in SQL client results in
"java.lang.ClassNotFoundException: org.apache.flink.core.execution.RestoreMode"
Key: FLINK-35785
URL: https://issues.apache.org/jira/browse/FLINK-35785
Project: Flink
Issue Type: Bug
Components: Runtime / Checkpointing, Table SQL / Client
Reporter: Martijn Visser
Tested with Flink 1.20 RC0
Reproducer:
{code:sql}
-- Create the product table
CREATE TABLE `product` (
id INT,
brandId INT,
grade STRING,
PRIMARY KEY (id) NOT ENFORCED
) WITH (
'connector' = 'datagen',
'rows-per-second' = '10',
'fields.id.kind' = 'random',
'fields.brandId.min' = '1',
'fields.brandId.max' = '100',
'fields.grade.length' = '10'
);
{code}
Followed by:
{code:sql}
SELECT * FROM product
{code}
Results in:
[ERROR] Could not execute SQL statement. Reason:
java.lang.ClassNotFoundException: org.apache.flink.core.execution.RestoreMode
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-35763) Bump Java version
[ https://issues.apache.org/jira/browse/FLINK-35763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser closed FLINK-35763. -- Resolution: Invalid > Bump Java version > - > > Key: FLINK-35763 > URL: https://issues.apache.org/jira/browse/FLINK-35763 > Project: Flink > Issue Type: Improvement >Affects Versions: 1.20.0 >Reporter: 911432 >Priority: Major > > Java8 is displayed as > [document|https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/java_compatibility/] > is not available in flink1.15. > However, java8 is still available on flink 1.20 at the moment. > [#1,|https://github.com/apache/flink/blob/6b08eb15d09d6c8debb609b8b3a2a241ca855155/tools/azure-pipelines/build-apache-repo.yml#L75] > [#2, > |https://github.com/apache/flink/blob/6b08eb15d09d6c8debb609b8b3a2a241ca855155/tools/azure-pipelines/build-apache-repo.yml#L108] > > [#3|https://cwiki.apache.org/confluence/display/FLINK/Creating+a+Flink+Release] > I hope you can fix this part. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35763) Bump Java version
[ https://issues.apache.org/jira/browse/FLINK-35763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17863210#comment-17863210 ] Martijn Visser commented on FLINK-35763: [~911432] The documentation doesn't state that it's not available, it states it's deprecated. That means that support for Java 8 will be removed in a future version of Flink (in this case, with Flink 2.0) > Bump Java version > - > > Key: FLINK-35763 > URL: https://issues.apache.org/jira/browse/FLINK-35763 > Project: Flink > Issue Type: Improvement >Affects Versions: 1.20.0 >Reporter: 911432 >Priority: Major > > Java8 is displayed as > [document|https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/java_compatibility/] > is not available in flink1.15. > However, java8 is still available on flink 1.20 at the moment. > [#1,|https://github.com/apache/flink/blob/6b08eb15d09d6c8debb609b8b3a2a241ca855155/tools/azure-pipelines/build-apache-repo.yml#L75] > [#2, > |https://github.com/apache/flink/blob/6b08eb15d09d6c8debb609b8b3a2a241ca855155/tools/azure-pipelines/build-apache-repo.yml#L108] > > [#3|https://cwiki.apache.org/confluence/display/FLINK/Creating+a+Flink+Release] > I hope you can fix this part. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35749) Kafka sink component will lose data when kafka cluster is unavailable for a while
[ https://issues.apache.org/jira/browse/FLINK-35749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17862991#comment-17862991 ] Martijn Visser commented on FLINK-35749: [~zjmean] It would be good to verify if this still occurs in the latest versions of Flink and the latest available version of the Flink Kafka connector Flink 1.16.2 is no longer community supported, and a lot of the internals have changed as well, especially on the API side. Can you run the same test on the latest version? > Kafka sink component will lose data when kafka cluster is unavailable for a > while > - > > Key: FLINK-35749 > URL: https://issues.apache.org/jira/browse/FLINK-35749 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka >Affects Versions: 1.16.2, 1.17.1, 1.20.0 >Reporter: Jimmy Zhao >Assignee: Jimmy Zhao >Priority: Blocker > > As the title described, here is the procedure to reproduce the problem: > 1. develop a simple flink stream job to consume from one kafka topic and sink > to anthoer kafka sever and topic > 2. make amount of kafka message and produce to the source kafka topic, record > the message number > 3. start the flink stream job, and config to cosume from earliest source > topic offset > 4. during the job cosuming the source topic, restart the kafka cluster(we use > aws MSK) > 5. the flink job will not throw any Exception like nothing happened, but only > print error log like : [kafka-producer-network-thread | producer-2] INFO > org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-2] Node > 2 disconnected. > 6. wait for the kafka cluster finished restarting and all the source kafka > message consumed > 7. count the target kafka topic message number, compare to the source, there > is a high probability of data loss（more than 50%） -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-34268) Add a test to verify if restore test exists for ExecNode
[ https://issues.apache.org/jira/browse/FLINK-34268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17862976#comment-17862976 ] Martijn Visser commented on FLINK-34268: Removed the exclusions on the tests in apache/flink:master via cc19159b489dafad71c01bf551bc224a38203eae Not backported to 1.20, but will be added into 2.0 > Add a test to verify if restore test exists for ExecNode > > > Key: FLINK-34268 > URL: https://issues.apache.org/jira/browse/FLINK-34268 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Reporter: Bonnie Varghese >Assignee: Bonnie Varghese >Priority: Major > Labels: pull-request-available > Fix For: 1.20.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35720) flink-connector-kafka fails to restore from checkpoint when using an EXACTLY_ONCE sink
[
https://issues.apache.org/jira/browse/FLINK-35720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17862473#comment-17862473
]
Martijn Visser commented on FLINK-35720:
Can you provide a reproducer in code? The thing is that I suspect a local
issue, because a) there are tests for this in the code, like
https://github.com/apache/flink-connector-kafka/blob/v3.1/flink-connector-kafka-e2e-tests/flink-end-to-end-tests-common-kafka/src/test/java/org/apache/flink/tests/util/kafka/KafkaSinkE2ECase.java
and b) if it was so broken, we would have had complaints since it was released
back in February of this year.
> flink-connector-kafka fails to restore from checkpoint when using an
> EXACTLY_ONCE sink
> --
>
> Key: FLINK-35720
> URL: https://issues.apache.org/jira/browse/FLINK-35720
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
> Environment: Flink 1.17, 1.18, 1.19
> Kubernetes
> flink-connector-kafka 3.1, 3.2
>Reporter: Nicolas Ferrario
>Priority: Critical
>
> Hi team, I noticed that the Kafka Client was changed from version 3.2.4 to to
> 3.4.0 in Flink Connector Kafka 3.1.
> This introduced a bug that prevents Flink from restoring from a checkpoint
> due to the usage of Reflection to get field {{txnPartitionMap}} when a
> Producer with Exactly Once semantics is used.
> Another side-effect is that a downgrade from 3.1 to 3.0.2 (the last working
> version) is not possible, as there was a version increase (from 1 to 2) in
> the Source state serializer, effectively forcing a full bootstrap losing all
> state.
> *All flink-connector-kafka versions starting from 3.1 are affected.*
> How to reproduce:
> # Start a pipeline with an EXACTLY ONCE Kafka Sink
> # Wait for a checkpoint
> # Stop the and restore from checkpoint
> # The sink fails to restore state, causing an exception
> The code-line causing the exception:
> [https://github.com/apache/flink-connector-kafka/blob/15d3fbd4e65dae6c334e2386dd337d2bf423c216/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/sink/FlinkKafkaInternalProducer.java#L310]
> POM 3.0: https://github.com/apache/flink-connector-kafka/blob/v3.0/pom.xml#L53
> POM 3.1: https://github.com/apache/flink-connector-kafka/blob/v3.1/pom.xml#L54
> Exception:
> {quote}2024-06-18 19:48:17
> java.lang.IllegalStateException: Failed to commit KafkaCommittable
> Unknown macro: \{producerId=67946, epoch=0,
> transactionalId=smsc-toolbox-enrichment-toolbox-sms-0-438}
> at
> org.apache.flink.streaming.runtime.operators.sink.committables.CommitRequestImpl.signalFailedWithUnknownReason(CommitRequestImpl.java:77)
> at
> org.apache.flink.connector.kafka.sink.KafkaCommitter.commit(KafkaCommitter.java:119)
> at
> org.apache.flink.streaming.runtime.operators.sink.committables.CheckpointCommittableManagerImpl.commit(CheckpointCommittableManagerImpl.java:126)
> at
> org.apache.flink.streaming.runtime.operators.sink.CommitterOperator.commitAndEmit(CommitterOperator.java:176)
> at
> org.apache.flink.streaming.runtime.operators.sink.CommitterOperator.commitAndEmitCheckpoints(CommitterOperator.java:160)
> at
> org.apache.flink.streaming.runtime.operators.sink.CommitterOperator.initializeState(CommitterOperator.java:121)
> at
> org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.initializeOperatorState(StreamOperatorStateHandler.java:122)
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:274)
> at
> org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.initializeStateAndOpenOperators(RegularOperatorChain.java:106)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:734)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:709)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:675)
> at
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:952)
> at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:921)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:745)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562)
> at java.base/java.lang.Thread.run(Unknown Source)
> Caused by: java.lang.RuntimeException: Incompatible KafkaProducer version
> at
> org.apache.flink.connector.kafka.sink.FlinkKafkaInternalProducer.getField(FlinkKafkaInternalProducer.java:285)
> at
> org.apache.flink.connector.kafka.sink.FlinkKafkaInternalProducer.getField(FlinkKafkaInternalProducer.java:272)
> at
> org.apache.flink.connector.kafka.sink.Flink
[jira] [Commented] (FLINK-35720) flink-connector-kafka fails to restore from checkpoint when using an EXACTLY_ONCE sink
[
https://issues.apache.org/jira/browse/FLINK-35720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17862455#comment-17862455
]
Martijn Visser commented on FLINK-35720:
[~nferrario] Have you also included org.apache.kafka:kafka-clients:3.4.0 ?
> flink-connector-kafka fails to restore from checkpoint when using an
> EXACTLY_ONCE sink
> --
>
> Key: FLINK-35720
> URL: https://issues.apache.org/jira/browse/FLINK-35720
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
> Environment: Flink 1.17, 1.18, 1.19
> Kubernetes
> flink-connector-kafka 3.1, 3.2
>Reporter: Nicolas Ferrario
>Priority: Critical
>
> Hi team, I noticed that the Kafka Client was changed from version 3.2.4 to to
> 3.4.0 in Flink Connector Kafka 3.1.
> This introduced a bug that prevents Flink from restoring from a checkpoint
> due to the usage of Reflection to get field {{txnPartitionMap}} when a
> Producer with Exactly Once semantics is used.
> Another side-effect is that a downgrade from 3.1 to 3.0.2 (the last working
> version) is not possible, as there was a version increase (from 1 to 2) in
> the Source state serializer, effectively forcing a full bootstrap losing all
> state.
> *All flink-connector-kafka versions starting from 3.1 are affected.*
> How to reproduce:
> # Start a pipeline with an EXACTLY ONCE Kafka Sink
> # Wait for a checkpoint
> # Stop the and restore from checkpoint
> # The sink fails to restore state, causing an exception
> The code-line causing the exception:
> [https://github.com/apache/flink-connector-kafka/blob/15d3fbd4e65dae6c334e2386dd337d2bf423c216/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/sink/FlinkKafkaInternalProducer.java#L310]
> POM 3.0: https://github.com/apache/flink-connector-kafka/blob/v3.0/pom.xml#L53
> POM 3.1: https://github.com/apache/flink-connector-kafka/blob/v3.1/pom.xml#L54
> Exception:
> {quote}2024-06-18 19:48:17
> java.lang.IllegalStateException: Failed to commit KafkaCommittable
> Unknown macro: \{producerId=67946, epoch=0,
> transactionalId=smsc-toolbox-enrichment-toolbox-sms-0-438}
> at
> org.apache.flink.streaming.runtime.operators.sink.committables.CommitRequestImpl.signalFailedWithUnknownReason(CommitRequestImpl.java:77)
> at
> org.apache.flink.connector.kafka.sink.KafkaCommitter.commit(KafkaCommitter.java:119)
> at
> org.apache.flink.streaming.runtime.operators.sink.committables.CheckpointCommittableManagerImpl.commit(CheckpointCommittableManagerImpl.java:126)
> at
> org.apache.flink.streaming.runtime.operators.sink.CommitterOperator.commitAndEmit(CommitterOperator.java:176)
> at
> org.apache.flink.streaming.runtime.operators.sink.CommitterOperator.commitAndEmitCheckpoints(CommitterOperator.java:160)
> at
> org.apache.flink.streaming.runtime.operators.sink.CommitterOperator.initializeState(CommitterOperator.java:121)
> at
> org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.initializeOperatorState(StreamOperatorStateHandler.java:122)
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:274)
> at
> org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.initializeStateAndOpenOperators(RegularOperatorChain.java:106)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:734)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:709)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:675)
> at
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:952)
> at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:921)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:745)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562)
> at java.base/java.lang.Thread.run(Unknown Source)
> Caused by: java.lang.RuntimeException: Incompatible KafkaProducer version
> at
> org.apache.flink.connector.kafka.sink.FlinkKafkaInternalProducer.getField(FlinkKafkaInternalProducer.java:285)
> at
> org.apache.flink.connector.kafka.sink.FlinkKafkaInternalProducer.getField(FlinkKafkaInternalProducer.java:272)
> at
> org.apache.flink.connector.kafka.sink.FlinkKafkaInternalProducer.resumeTransaction(FlinkKafkaInternalProducer.java:310)
> at
> org.apache.flink.connector.kafka.sink.KafkaCommitter.getRecoveryProducer(KafkaCommitter.java:143)
> at
> org.apache.flink.connector.kafka.sink.KafkaCommitter.lambda$commit$0(KafkaCommitter.java:72)
> at java.base/java.util.Optional.orElseGet(Unknown Source)
> at
> org.apache.flink.co
[jira] [Commented] (FLINK-35720) flink-connector-kafka fails to restore from checkpoint when using an EXACTLY_ONCE sink
[
https://issues.apache.org/jira/browse/FLINK-35720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17862404#comment-17862404
]
Martijn Visser commented on FLINK-35720:
[~nferrario] I don't run into any problems. I have a Flink SQL application on
Flink 1.18 that works exactly when following those steps. What API are you
using, and which JARs are you including?
> flink-connector-kafka fails to restore from checkpoint when using an
> EXACTLY_ONCE sink
> --
>
> Key: FLINK-35720
> URL: https://issues.apache.org/jira/browse/FLINK-35720
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
> Environment: Flink 1.17, 1.18, 1.19
> Kubernetes
> flink-connector-kafka 3.1, 3.2
>Reporter: Nicolas Ferrario
>Priority: Critical
>
> Hi team, I noticed that the Kafka Client was changed from version 3.2.4 to to
> 3.4.0 in Flink Connector Kafka 3.1.
> This introduced a bug that prevents Flink from restoring from a checkpoint
> due to the usage of Reflection to get field {{txnPartitionMap}} when a
> Producer with Exactly Once semantics is used.
> Another side-effect is that a downgrade from 3.1 to 3.0.2 (the last working
> version) is not possible, as there was a version increase (from 1 to 2) in
> the Source state serializer, effectively forcing a full bootstrap losing all
> state.
> *All flink-connector-kafka versions starting from 3.1 are affected.*
> How to reproduce:
> # Start a pipeline with an EXACTLY ONCE Kafka Sink
> # Wait for a checkpoint
> # Stop the and restore from checkpoint
> # The sink fails to restore state, causing an exception
> The code-line causing the exception:
> [https://github.com/apache/flink-connector-kafka/blob/15d3fbd4e65dae6c334e2386dd337d2bf423c216/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/sink/FlinkKafkaInternalProducer.java#L310]
> POM 3.0: https://github.com/apache/flink-connector-kafka/blob/v3.0/pom.xml#L53
> POM 3.1: https://github.com/apache/flink-connector-kafka/blob/v3.1/pom.xml#L54
> Exception:
> {quote}2024-06-18 19:48:17
> java.lang.IllegalStateException: Failed to commit KafkaCommittable
> Unknown macro: \{producerId=67946, epoch=0,
> transactionalId=smsc-toolbox-enrichment-toolbox-sms-0-438}
> at
> org.apache.flink.streaming.runtime.operators.sink.committables.CommitRequestImpl.signalFailedWithUnknownReason(CommitRequestImpl.java:77)
> at
> org.apache.flink.connector.kafka.sink.KafkaCommitter.commit(KafkaCommitter.java:119)
> at
> org.apache.flink.streaming.runtime.operators.sink.committables.CheckpointCommittableManagerImpl.commit(CheckpointCommittableManagerImpl.java:126)
> at
> org.apache.flink.streaming.runtime.operators.sink.CommitterOperator.commitAndEmit(CommitterOperator.java:176)
> at
> org.apache.flink.streaming.runtime.operators.sink.CommitterOperator.commitAndEmitCheckpoints(CommitterOperator.java:160)
> at
> org.apache.flink.streaming.runtime.operators.sink.CommitterOperator.initializeState(CommitterOperator.java:121)
> at
> org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.initializeOperatorState(StreamOperatorStateHandler.java:122)
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:274)
> at
> org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.initializeStateAndOpenOperators(RegularOperatorChain.java:106)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:734)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:709)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:675)
> at
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:952)
> at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:921)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:745)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562)
> at java.base/java.lang.Thread.run(Unknown Source)
> Caused by: java.lang.RuntimeException: Incompatible KafkaProducer version
> at
> org.apache.flink.connector.kafka.sink.FlinkKafkaInternalProducer.getField(FlinkKafkaInternalProducer.java:285)
> at
> org.apache.flink.connector.kafka.sink.FlinkKafkaInternalProducer.getField(FlinkKafkaInternalProducer.java:272)
> at
> org.apache.flink.connector.kafka.sink.FlinkKafkaInternalProducer.resumeTransaction(FlinkKafkaInternalProducer.java:310)
> at
> org.apache.flink.connector.kafka.sink.KafkaCommitter.getRecoveryProducer(KafkaCommitter.java:143)
> at
> org.apache.flink.connector.kafka.sink.KafkaCommitter.lambda
[jira] [Commented] (FLINK-35720) flink-connector-kafka fails to restore from checkpoint when using an EXACTLY_ONCE sink
[
https://issues.apache.org/jira/browse/FLINK-35720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17862401#comment-17862401
]
Martijn Visser commented on FLINK-35720:
[~nferrario] There is always some state involved, since offsets of Kafka are
stored in checkpoints/savepoints. In your reproduction steps, it only shows
that you've stopped and restarted from checkpoints. Can you please describe
your exact reproduction steps?
> flink-connector-kafka fails to restore from checkpoint when using an
> EXACTLY_ONCE sink
> --
>
> Key: FLINK-35720
> URL: https://issues.apache.org/jira/browse/FLINK-35720
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
> Environment: Flink 1.17, 1.18, 1.19
> Kubernetes
> flink-connector-kafka 3.1, 3.2
>Reporter: Nicolas Ferrario
>Priority: Critical
>
> Hi team, I noticed that the Kafka Client was changed from version 3.2.4 to to
> 3.4.0 in Flink Connector Kafka 3.1.
> This introduced a bug that prevents Flink from restoring from a checkpoint
> due to the usage of Reflection to get field {{txnPartitionMap}} when a
> Producer with Exactly Once semantics is used.
> Another side-effect is that a downgrade from 3.1 to 3.0.2 (the last working
> version) is not possible, as there was a version increase (from 1 to 2) in
> the Source state serializer, effectively forcing a full bootstrap losing all
> state.
> *All flink-connector-kafka versions starting from 3.1 are affected.*
> How to reproduce:
> # Start a pipeline with an EXACTLY ONCE Kafka Sink
> # Wait for a checkpoint
> # Stop the and restore from checkpoint
> # The sink fails to restore state, causing an exception
> The code-line causing the exception:
> [https://github.com/apache/flink-connector-kafka/blob/15d3fbd4e65dae6c334e2386dd337d2bf423c216/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/sink/FlinkKafkaInternalProducer.java#L310]
> POM 3.0: https://github.com/apache/flink-connector-kafka/blob/v3.0/pom.xml#L53
> POM 3.1: https://github.com/apache/flink-connector-kafka/blob/v3.1/pom.xml#L54
> Exception:
> {quote}2024-06-18 19:48:17
> java.lang.IllegalStateException: Failed to commit KafkaCommittable
> Unknown macro: \{producerId=67946, epoch=0,
> transactionalId=smsc-toolbox-enrichment-toolbox-sms-0-438}
> at
> org.apache.flink.streaming.runtime.operators.sink.committables.CommitRequestImpl.signalFailedWithUnknownReason(CommitRequestImpl.java:77)
> at
> org.apache.flink.connector.kafka.sink.KafkaCommitter.commit(KafkaCommitter.java:119)
> at
> org.apache.flink.streaming.runtime.operators.sink.committables.CheckpointCommittableManagerImpl.commit(CheckpointCommittableManagerImpl.java:126)
> at
> org.apache.flink.streaming.runtime.operators.sink.CommitterOperator.commitAndEmit(CommitterOperator.java:176)
> at
> org.apache.flink.streaming.runtime.operators.sink.CommitterOperator.commitAndEmitCheckpoints(CommitterOperator.java:160)
> at
> org.apache.flink.streaming.runtime.operators.sink.CommitterOperator.initializeState(CommitterOperator.java:121)
> at
> org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.initializeOperatorState(StreamOperatorStateHandler.java:122)
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:274)
> at
> org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.initializeStateAndOpenOperators(RegularOperatorChain.java:106)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:734)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:709)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:675)
> at
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:952)
> at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:921)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:745)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562)
> at java.base/java.lang.Thread.run(Unknown Source)
> Caused by: java.lang.RuntimeException: Incompatible KafkaProducer version
> at
> org.apache.flink.connector.kafka.sink.FlinkKafkaInternalProducer.getField(FlinkKafkaInternalProducer.java:285)
> at
> org.apache.flink.connector.kafka.sink.FlinkKafkaInternalProducer.getField(FlinkKafkaInternalProducer.java:272)
> at
> org.apache.flink.connector.kafka.sink.FlinkKafkaInternalProducer.resumeTransaction(FlinkKafkaInternalProducer.java:310)
> at
> org.apache.flink.connector.kafka.sink.KafkaCommitter.getRecoveryProducer(KafkaCommitter.java:143)
[jira] [Commented] (FLINK-35720) flink-connector-kafka fails to restore from checkpoint when using an EXACTLY_ONCE sink
[
https://issues.apache.org/jira/browse/FLINK-35720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17861371#comment-17861371
]
Martijn Visser commented on FLINK-35720:
Have you performed the additional required upgrade steps, as documented at
https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/kafka/#upgrading-to-the-latest-connector-version
?
> flink-connector-kafka fails to restore from checkpoint when using an
> EXACTLY_ONCE sink
> --
>
> Key: FLINK-35720
> URL: https://issues.apache.org/jira/browse/FLINK-35720
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
> Environment: Flink 1.17, 1.18, 1.19
> Kubernetes
> flink-connector-kafka 3.1, 3.2
>Reporter: Nicolas Ferrario
>Priority: Critical
>
> Hi team, I noticed that the Kafka Client was changed from version 3.2.4 to to
> 3.4.0 in Flink Connector Kafka 3.1.
> This introduced a bug that prevents Flink from restoring from a checkpoint
> due to the usage of Reflection to get field {{txnPartitionMap}} when a
> Producer with Exactly Once semantics is used.
> Another side-effect is that a downgrade from 3.1 to 3.0.2 (the last working
> version) is not possible, as there was a version increase (from 1 to 2) in
> the Source state serializer, effectively forcing a full bootstrap losing all
> state.
> *All flink-connector-kafka versions starting from 3.1 are affected.*
> How to reproduce:
> # Start a pipeline with an EXACTLY ONCE Kafka Sink
> # Wait for a checkpoint
> # Stop the and restore from checkpoint
> # The sink fails to restore state, causing an exception
> The code-line causing the exception:
> [https://github.com/apache/flink-connector-kafka/blob/15d3fbd4e65dae6c334e2386dd337d2bf423c216/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/sink/FlinkKafkaInternalProducer.java#L310]
> POM 3.0: https://github.com/apache/flink-connector-kafka/blob/v3.0/pom.xml#L53
> POM 3.1: https://github.com/apache/flink-connector-kafka/blob/v3.1/pom.xml#L54
> Exception:
> {quote}2024-06-18 19:48:17
> java.lang.IllegalStateException: Failed to commit KafkaCommittable
> Unknown macro: \{producerId=67946, epoch=0,
> transactionalId=smsc-toolbox-enrichment-toolbox-sms-0-438}
> at
> org.apache.flink.streaming.runtime.operators.sink.committables.CommitRequestImpl.signalFailedWithUnknownReason(CommitRequestImpl.java:77)
> at
> org.apache.flink.connector.kafka.sink.KafkaCommitter.commit(KafkaCommitter.java:119)
> at
> org.apache.flink.streaming.runtime.operators.sink.committables.CheckpointCommittableManagerImpl.commit(CheckpointCommittableManagerImpl.java:126)
> at
> org.apache.flink.streaming.runtime.operators.sink.CommitterOperator.commitAndEmit(CommitterOperator.java:176)
> at
> org.apache.flink.streaming.runtime.operators.sink.CommitterOperator.commitAndEmitCheckpoints(CommitterOperator.java:160)
> at
> org.apache.flink.streaming.runtime.operators.sink.CommitterOperator.initializeState(CommitterOperator.java:121)
> at
> org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.initializeOperatorState(StreamOperatorStateHandler.java:122)
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:274)
> at
> org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.initializeStateAndOpenOperators(RegularOperatorChain.java:106)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:734)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:709)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:675)
> at
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:952)
> at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:921)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:745)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562)
> at java.base/java.lang.Thread.run(Unknown Source)
> Caused by: java.lang.RuntimeException: Incompatible KafkaProducer version
> at
> org.apache.flink.connector.kafka.sink.FlinkKafkaInternalProducer.getField(FlinkKafkaInternalProducer.java:285)
> at
> org.apache.flink.connector.kafka.sink.FlinkKafkaInternalProducer.getField(FlinkKafkaInternalProducer.java:272)
> at
> org.apache.flink.connector.kafka.sink.FlinkKafkaInternalProducer.resumeTransaction(FlinkKafkaInternalProducer.java:310)
> at
> org.apache.flink.connector.kafka.sink.KafkaCommitter.getRecoveryProducer(KafkaCommitter.java:143)
> at
> org.apache.flink.connector.kafka.sink.KafkaComm
[jira] [Assigned] (FLINK-35726) Data Stream to Table API converts Map to RAW 'java.util.Map'
[
https://issues.apache.org/jira/browse/FLINK-35726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser reassigned FLINK-35726:
--
Assignee: Martijn Visser
> Data Stream to Table API converts Map to RAW 'java.util.Map'
>
>
> Key: FLINK-35726
> URL: https://issues.apache.org/jira/browse/FLINK-35726
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.17.2
>Reporter: David Perkins
>Assignee: Martijn Visser
>Priority: Major
>
> We have a use case where we convert from the Table API to a Data Stream with
> a class, perform some operations, and then convert back to the Table API.
> When the data contains a Map, the conversion back to the Table API converts
> the Map to {{{}RAW('java.util.Map', '...'){}}}. This causes an 'Incompatible
> types for sink column' exception.
> In this particular case, the Map contains the Kafka headers, which we need to
> preserve and write to the output topic. Both topics/table definitions use the
> same schema. We have set a {{DataTypeHint}} annotation on the Map field in
> the Java class. We are currently working around this issue by using a UDF to
> simply perform a type conversion from the RAW Java Map to the Table API Map.
> One note is that if no operations are performed on the stream, it work's
> correctly. But adding a simple identity map causes the exception.
> Here's a simple example to reproduce the problem.
> CREATE TABLE Source (
> id STRING,
> headers MAP METADATA
> ) WITH (
> 'connector' = 'kafka',
> 'topic' = 'source',
> 'properties.bootstrap.servers' = 'kafka-bootstrap-server',
> 'format' = 'json'
> );
> CREATE TABLE Target (
> id STRING,
> headers MAP METADATA
> ) WITH (
> 'connector' = 'kafka',
> 'topic' = 'target',
> 'properties.bootstrap.servers' = 'kafka-bootstrap-server',
> 'format' = 'json'
> );
> public class MyRecord {
> private String id;
> @DataTypeHint(value = "MAP")
> private Map headers;
> ...
> }
> public class MyJob {
> public static void main(String[] args) throws Exception{
> final StreamExecutionEnvironment streamEnv =
> StreamExecutionEnvironment.getExecutionEnvironment();
> final StreamTableEnvironment tableEnv =
> StreamTableEnvironment.create(streamEnv); Table sourceTable =
> tableEnv.from("Source"); var sourceStream =
> tableEnv.toDataStream(sourceTable, MyRecord.class); var mappedStream =
> sourceStream.map(row -> row); Table outputTable =
> tableEnv.fromDataStream(mappedStream);
> tableEnv.createStatementSet().add(outputTable.insertInto("Target"))
> .attachAsDataStream(); streamEnv.executeAsync("Table Datastream test");
> }
> }
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-35726) Data Stream to Table API converts Map to RAW 'java.util.Map'
[
https://issues.apache.org/jira/browse/FLINK-35726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser reassigned FLINK-35726:
--
Assignee: (was: Martijn Visser)
> Data Stream to Table API converts Map to RAW 'java.util.Map'
>
>
> Key: FLINK-35726
> URL: https://issues.apache.org/jira/browse/FLINK-35726
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.17.2
>Reporter: David Perkins
>Priority: Major
>
> We have a use case where we convert from the Table API to a Data Stream with
> a class, perform some operations, and then convert back to the Table API.
> When the data contains a Map, the conversion back to the Table API converts
> the Map to {{{}RAW('java.util.Map', '...'){}}}. This causes an 'Incompatible
> types for sink column' exception.
> In this particular case, the Map contains the Kafka headers, which we need to
> preserve and write to the output topic. Both topics/table definitions use the
> same schema. We have set a {{DataTypeHint}} annotation on the Map field in
> the Java class. We are currently working around this issue by using a UDF to
> simply perform a type conversion from the RAW Java Map to the Table API Map.
> One note is that if no operations are performed on the stream, it work's
> correctly. But adding a simple identity map causes the exception.
> Here's a simple example to reproduce the problem.
> CREATE TABLE Source (
> id STRING,
> headers MAP METADATA
> ) WITH (
> 'connector' = 'kafka',
> 'topic' = 'source',
> 'properties.bootstrap.servers' = 'kafka-bootstrap-server',
> 'format' = 'json'
> );
> CREATE TABLE Target (
> id STRING,
> headers MAP METADATA
> ) WITH (
> 'connector' = 'kafka',
> 'topic' = 'target',
> 'properties.bootstrap.servers' = 'kafka-bootstrap-server',
> 'format' = 'json'
> );
> public class MyRecord {
> private String id;
> @DataTypeHint(value = "MAP")
> private Map headers;
> ...
> }
> public class MyJob {
> public static void main(String[] args) throws Exception{
> final StreamExecutionEnvironment streamEnv =
> StreamExecutionEnvironment.getExecutionEnvironment();
> final StreamTableEnvironment tableEnv =
> StreamTableEnvironment.create(streamEnv); Table sourceTable =
> tableEnv.from("Source"); var sourceStream =
> tableEnv.toDataStream(sourceTable, MyRecord.class); var mappedStream =
> sourceStream.map(row -> row); Table outputTable =
> tableEnv.fromDataStream(mappedStream);
> tableEnv.createStatementSet().add(outputTable.insertInto("Target"))
> .attachAsDataStream(); streamEnv.executeAsync("Table Datastream test");
> }
> }
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-35611) Release Testing Instructions: Verify [FLIP-453] Promote Unified Sink API V2 to Public and Deprecate SinkFunction
[ https://issues.apache.org/jira/browse/FLINK-35611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser closed FLINK-35611. -- Resolution: Information Provided > Release Testing Instructions: Verify [FLIP-453] Promote Unified Sink API V2 > to Public and Deprecate SinkFunction > > > Key: FLINK-35611 > URL: https://issues.apache.org/jira/browse/FLINK-35611 > Project: Flink > Issue Type: Sub-task > Components: API / Core >Reporter: Rui Fan >Assignee: Martijn Visser >Priority: Blocker > Labels: release-testing > Fix For: 1.20.0 > > > Follow up the test for https://issues.apache.org/jira/browse/FLINK-35378 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35611) Release Testing Instructions: Verify [FLIP-453] Promote Unified Sink API V2 to Public and Deprecate SinkFunction
[ https://issues.apache.org/jira/browse/FLINK-35611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17861334#comment-17861334 ] Martijn Visser commented on FLINK-35611: [~fanrui] It doesn't need release testing, since it's indeed only an annotation check. Let me close this ticket given this information > Release Testing Instructions: Verify [FLIP-453] Promote Unified Sink API V2 > to Public and Deprecate SinkFunction > > > Key: FLINK-35611 > URL: https://issues.apache.org/jira/browse/FLINK-35611 > Project: Flink > Issue Type: Sub-task > Components: API / Core >Reporter: Rui Fan >Assignee: Martijn Visser >Priority: Blocker > Labels: release-testing > Fix For: 1.20.0 > > > Follow up the test for https://issues.apache.org/jira/browse/FLINK-35378 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-34554) Using EXACTLY_ONCE with KafkaSink cause broker's OOM due to newly created transactionalId per checkpoint
[
https://issues.apache.org/jira/browse/FLINK-34554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17860823#comment-17860823
]
Martijn Visser commented on FLINK-34554:
[~loserwang1024] I think that would require a discussion on the Dev mailing
list and potentially a FLIP, it would be great if you could start such a
discussion thread
> Using EXACTLY_ONCE with KafkaSink cause broker's OOM due to newly created
> transactionalId per checkpoint
>
>
> Key: FLINK-34554
> URL: https://issues.apache.org/jira/browse/FLINK-34554
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Kafka
>Affects Versions: 1.16.3, 1.17.2, 1.18.1
>Reporter: Hilmi Al Fatih
>Priority: Major
> Attachments: image (4).png, image (5).png,
> image-2024-03-16-17-17-16-152.png
>
>
> Flink version: 1.17.1
> Kafka Broker version: 2.7.1 * 4 GB heap memory for each
> Hi, We recently had an outage in our production system after we perform a
> Flink kafka-connector API upgrade. To give a brief context, our application
> is a simple kafka-to-kafka pipeline with minimal processing. We run in
> EXACTLY_ONCE mode, thus kafka transaction is involved.
> Our application runs with total around 350 sink subtask. Checkpoint period
> was set to 5 seconds to avoid blocking {{read_committed}} consumers too long.
> We recently performed an upgrade with the following details:
> Previous state:
> * Flink version: 1.14.4
> * Broker version: 2.7.1
> * kafka connector API: FlinkKafkaProducer
> Update to:
> * Flink version: 1.17.1
> * Broker version: 2.7.1
> * kafka connector API: KafkaSink
> Around 10 hours after the deployment, our kafka broker started to failing
> with OOM error. Heap dump entries are dominated by the ProducerStateEntry
> records.
> Our investigation leads to finding the total implementation change between
> FlinkKafkaProducer and KafkaSink.
> * KafkaSink generate different transactionalId for each checkpoint,
> * FlinkKafkaProducer uses constant set of transactionalId pool.
> With this behavior, KafkaSink seemed to exhaust our broker heap very fast and
> the ProducerStateEntry will only expire after
> [transactional.id.expiration.ms|http://transactional.id.expiration.ms/] ,
> which by default is set to 7 days.
> ([ref1|https://github.com/apache/kafka/blob/61dbce85d0d41457d81a4096ecaea049f3a4b3ae/core/src/main/scala/kafka/log/Log.scala#L677],
>
> [ref2|https://github.com/apache/kafka/blob/61dbce85d0d41457d81a4096ecaea049f3a4b3ae/core/src/main/scala/kafka/log/LogManager.scala#L268],
>
> [ref3|https://github.com/apache/kafka/blob/61dbce85d0d41457d81a4096ecaea049f3a4b3ae/core/src/main/scala/kafka/log/LogManager.scala#L1207])
> For our job, it means it creates roughly:
> * 10 hour running) 350 ids * 12 times/minute * 60 min/hour * 10 hour ~
> 2,520,000
> * 7 days) ~ 42mil entries.
> Attached below is the number of ProducerStateEntry entries of heap dump when
> it is OOM:
> * 505,000 (6.5%), in total it would be roughly ~ 7,000,000 entries.
> There are several things that come up in our mind to mitigate the drawbacks
> such as:
> * reduce the number of subtasks, so it reduces the number of transactionalId
> * Enlarge the checkpoint period to reduce the newly generated
> transactionalId rate.
> * Shorten
> [transactional.id.expiration.ms|http://transactional.id.expiration.ms/] to
> expire the unused transactionalId soon.
> * Increase the broker heap
> However, above mitigation might be too cumbersome and need careful tuning
> which harm our flexibility.In addition, due to the lack of maintaining
> lingering transaction state, TransactionAborter seems to abort old
> transaction naively. We might be accidentally (or purposefully) reuse the
> same transactionalIdPrefix and start the counter from 0. In that case, if the
> old transactionalId happens to have epoch >0, it will keep looping aborting
> the nonexistent transactions up to the latest checkpoint counter (which may
> be too big) and make the job stuck.
> Btw, I am aware that in Flink 2.0, you guys are putting a lot of effort on
> creating better integration with Kafka transaction
> ([FLIP-319|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=255071710]).
> In FLIP-319, it mentions something about TID pooling. However, it is seem
> that there is no relevant page yet for it, so I wonder whether there are any
> concrete plan already that I can follow, or if there is something I can
> contribute to, I will be really happy to help.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-20539) Type mismatch when using ROW in computed column
[
https://issues.apache.org/jira/browse/FLINK-20539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17860386#comment-17860386
]
Martijn Visser commented on FLINK-20539:
[~xuyangzhong] Have you been able to take a look?
> Type mismatch when using ROW in computed column
> ---
>
> Key: FLINK-20539
> URL: https://issues.apache.org/jira/browse/FLINK-20539
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: Timo Walther
>Assignee: xuyang
>Priority: Major
> Labels: auto-unassigned, pull-request-available
> Fix For: 1.19.0, 1.18.2
>
>
> The following SQL:
> {code}
> env.executeSql(
> "CREATE TABLE Orders (\n"
> + "order_number BIGINT,\n"
> + "priceINT,\n"
> + "first_name STRING,\n"
> + "last_nameSTRING,\n"
> + "buyer_name AS ROW(first_name, last_name)\n"
> + ") WITH (\n"
> + " 'connector' = 'datagen'\n"
> + ")");
> env.executeSql("SELECT * FROM Orders").print();
> {code}
> Fails with:
> {code}
> Exception in thread "main" java.lang.AssertionError: Conversion to relational
> algebra failed to preserve datatypes:
> validated type:
> RecordType(BIGINT order_number, INTEGER price, VARCHAR(2147483647) CHARACTER
> SET "UTF-16LE" first_name, VARCHAR(2147483647) CHARACTER SET "UTF-16LE"
> last_name, RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
> "UTF-16LE" EXPR$0, VARCHAR(2147483647) CHARACTER SET "UTF-16LE" EXPR$1) NOT
> NULL buyer_name) NOT NULL
> converted type:
> RecordType(BIGINT order_number, INTEGER price, VARCHAR(2147483647) CHARACTER
> SET "UTF-16LE" first_name, VARCHAR(2147483647) CHARACTER SET "UTF-16LE"
> last_name, RecordType(VARCHAR(2147483647) CHARACTER SET "UTF-16LE" EXPR$0,
> VARCHAR(2147483647) CHARACTER SET "UTF-16LE" EXPR$1) NOT NULL buyer_name) NOT
> NULL
> rel:
> LogicalProject(order_number=[$0], price=[$1], first_name=[$2],
> last_name=[$3], buyer_name=[ROW($2, $3)])
> LogicalTableScan(table=[[default_catalog, default_database, Orders]])
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.checkConvertedType(SqlToRelConverter.java:467)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:582)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-35706) FLIP-463: Schema Definition in CREATE TABLE AS Statement
[ https://issues.apache.org/jira/browse/FLINK-35706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser reassigned FLINK-35706: -- Assignee: Sergio Peña > FLIP-463: Schema Definition in CREATE TABLE AS Statement > > > Key: FLINK-35706 > URL: https://issues.apache.org/jira/browse/FLINK-35706 > Project: Flink > Issue Type: New Feature > Components: Table SQL / API >Reporter: Sergio Peña >Assignee: Sergio Peña >Priority: Major > > Extend the CTAS and RTAS statements to allow a schema definition on the > CREATE or REPLACE clauses. Such as new columns, watermarks primary and > partition keys, and table distribution. > Please refer to the FLIP for any details: > https://cwiki.apache.org/confluence/display/FLINK/FLIP-463:+Schema+Definition+in+CREATE+TABLE+AS+Statement -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-35686) Flink-connector-jdbc v3.2.0 support flink 1.17.x
[ https://issues.apache.org/jira/browse/FLINK-35686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser closed FLINK-35686. -- Resolution: Information Provided As can be seen on https://flink.apache.org/downloads/#apache-flink-jdbc-connector-320 this version has been designated as compatible with 1.18 and 1.19, not with 1.17. It might work, but isn't supported. > Flink-connector-jdbc v3.2.0 support flink 1.17.x > > > Key: FLINK-35686 > URL: https://issues.apache.org/jira/browse/FLINK-35686 > Project: Flink > Issue Type: Improvement >Reporter: ZhengJunZhou >Priority: Blocker > > Can Flink-connector-jdbc v3.2.0 support flink 1.17.x? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-20539) Type mismatch when using ROW in computed column
[
https://issues.apache.org/jira/browse/FLINK-20539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17859658#comment-17859658
]
Martijn Visser commented on FLINK-20539:
[~xuyangzhong] [~qingyue] I've re-opened the ticket because it indeed doesn't
yet work. Can you take a look?
> Type mismatch when using ROW in computed column
> ---
>
> Key: FLINK-20539
> URL: https://issues.apache.org/jira/browse/FLINK-20539
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: Timo Walther
>Assignee: xuyang
>Priority: Major
> Labels: auto-unassigned, pull-request-available
> Fix For: 1.19.0, 1.18.2
>
>
> The following SQL:
> {code}
> env.executeSql(
> "CREATE TABLE Orders (\n"
> + "order_number BIGINT,\n"
> + "priceINT,\n"
> + "first_name STRING,\n"
> + "last_nameSTRING,\n"
> + "buyer_name AS ROW(first_name, last_name)\n"
> + ") WITH (\n"
> + " 'connector' = 'datagen'\n"
> + ")");
> env.executeSql("SELECT * FROM Orders").print();
> {code}
> Fails with:
> {code}
> Exception in thread "main" java.lang.AssertionError: Conversion to relational
> algebra failed to preserve datatypes:
> validated type:
> RecordType(BIGINT order_number, INTEGER price, VARCHAR(2147483647) CHARACTER
> SET "UTF-16LE" first_name, VARCHAR(2147483647) CHARACTER SET "UTF-16LE"
> last_name, RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
> "UTF-16LE" EXPR$0, VARCHAR(2147483647) CHARACTER SET "UTF-16LE" EXPR$1) NOT
> NULL buyer_name) NOT NULL
> converted type:
> RecordType(BIGINT order_number, INTEGER price, VARCHAR(2147483647) CHARACTER
> SET "UTF-16LE" first_name, VARCHAR(2147483647) CHARACTER SET "UTF-16LE"
> last_name, RecordType(VARCHAR(2147483647) CHARACTER SET "UTF-16LE" EXPR$0,
> VARCHAR(2147483647) CHARACTER SET "UTF-16LE" EXPR$1) NOT NULL buyer_name) NOT
> NULL
> rel:
> LogicalProject(order_number=[$0], price=[$1], first_name=[$2],
> last_name=[$3], buyer_name=[ROW($2, $3)])
> LogicalTableScan(table=[[default_catalog, default_database, Orders]])
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.checkConvertedType(SqlToRelConverter.java:467)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:582)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Reopened] (FLINK-20539) Type mismatch when using ROW in computed column
[
https://issues.apache.org/jira/browse/FLINK-20539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser reopened FLINK-20539:
> Type mismatch when using ROW in computed column
> ---
>
> Key: FLINK-20539
> URL: https://issues.apache.org/jira/browse/FLINK-20539
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: Timo Walther
>Assignee: xuyang
>Priority: Major
> Labels: auto-unassigned, pull-request-available
> Fix For: 1.19.0, 1.18.2
>
>
> The following SQL:
> {code}
> env.executeSql(
> "CREATE TABLE Orders (\n"
> + "order_number BIGINT,\n"
> + "priceINT,\n"
> + "first_name STRING,\n"
> + "last_nameSTRING,\n"
> + "buyer_name AS ROW(first_name, last_name)\n"
> + ") WITH (\n"
> + " 'connector' = 'datagen'\n"
> + ")");
> env.executeSql("SELECT * FROM Orders").print();
> {code}
> Fails with:
> {code}
> Exception in thread "main" java.lang.AssertionError: Conversion to relational
> algebra failed to preserve datatypes:
> validated type:
> RecordType(BIGINT order_number, INTEGER price, VARCHAR(2147483647) CHARACTER
> SET "UTF-16LE" first_name, VARCHAR(2147483647) CHARACTER SET "UTF-16LE"
> last_name, RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
> "UTF-16LE" EXPR$0, VARCHAR(2147483647) CHARACTER SET "UTF-16LE" EXPR$1) NOT
> NULL buyer_name) NOT NULL
> converted type:
> RecordType(BIGINT order_number, INTEGER price, VARCHAR(2147483647) CHARACTER
> SET "UTF-16LE" first_name, VARCHAR(2147483647) CHARACTER SET "UTF-16LE"
> last_name, RecordType(VARCHAR(2147483647) CHARACTER SET "UTF-16LE" EXPR$0,
> VARCHAR(2147483647) CHARACTER SET "UTF-16LE" EXPR$1) NOT NULL buyer_name) NOT
> NULL
> rel:
> LogicalProject(order_number=[$0], price=[$1], first_name=[$2],
> last_name=[$3], buyer_name=[ROW($2, $3)])
> LogicalTableScan(table=[[default_catalog, default_database, Orders]])
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.checkConvertedType(SqlToRelConverter.java:467)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:582)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-35631) KafkaSource parameter partition.discovery.interval.ms with a default value of 5 minutes does not take effect
[
https://issues.apache.org/jira/browse/FLINK-35631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17856151#comment-17856151
]
Martijn Visser commented on FLINK-35631:
[~renqs] Given that you worked on the FLIP, can you give your thoughts on this
one?
> KafkaSource parameter partition.discovery.interval.ms with a default value of
> 5 minutes does not take effect
>
>
> Key: FLINK-35631
> URL: https://issues.apache.org/jira/browse/FLINK-35631
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Kafka
>Affects Versions: kafka-3.1.0
>Reporter: elon_X
>Priority: Major
>
> When I start a stream program to consume Kafka
> (flink-connector-kafka-3.1-SNAPSHOT) the Flink task does not automatically
> detect new partitions after Kafka adds partitions.
>
> *Reason*
> In the {{{}KafkaSourceBuilder{}}}, this parameter is checked to see if it has
> been overridden. Since I did not set this parameter, even though it is
> {{{}CONTINUOUS_UNBOUNDED{}}}, it still sets
> {{{}partition.discovery.interval.ms = -1{}}}.
> In the {{{}KafkaSourceEnumerator{}}}, the value of
> {{partition.discovery.interval.ms}} is then -1, instead of the default value
> of 5 minutes, so automatic partition discovery does not work, and the default
> value of 5 minutes for {{partition.discovery.interval.ms}} is meaningless.
>
> A possible solution is to set {{partition.discovery.interval.ms = -1}} only
> if {{boundedness == Boundedness.BOUNDED}} is true.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-35631) KafkaSource parameter partition.discovery.interval.ms with a default value of 5 minutes does not take effect
[
https://issues.apache.org/jira/browse/FLINK-35631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17855909#comment-17855909
]
Martijn Visser commented on FLINK-35631:
[~elon] Have you validated this with how it was implemented according to
https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source
?
> KafkaSource parameter partition.discovery.interval.ms with a default value of
> 5 minutes does not take effect
>
>
> Key: FLINK-35631
> URL: https://issues.apache.org/jira/browse/FLINK-35631
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Kafka
>Affects Versions: kafka-3.1.0
>Reporter: elon_X
>Priority: Major
>
> When I start a stream program to consume Kafka
> (flink-connector-kafka-3.1-SNAPSHOT) the Flink task does not automatically
> detect new partitions after Kafka adds partitions.
>
> *Reason*
> In the {{{}KafkaSourceBuilder{}}}, this parameter is checked to see if it has
> been overridden. Since I did not set this parameter, even though it is
> {{{}CONTINUOUS_UNBOUNDED{}}}, it still sets
> {{{}partition.discovery.interval.ms = -1{}}}.
> In the {{{}KafkaSourceEnumerator{}}}, the value of
> {{partition.discovery.interval.ms}} is then -1, instead of the default value
> of 5 minutes, so automatic partition discovery does not work, and the default
> value of 5 minutes for {{partition.discovery.interval.ms}} is meaningless.
>
> A possible solution is to set {{partition.discovery.interval.ms = -1}} only
> if {{boundedness == Boundedness.BOUNDED}} is true.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-35626) Add support for FIPS
[ https://issues.apache.org/jira/browse/FLINK-35626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser updated FLINK-35626: --- Component/s: Documentation > Add support for FIPS > > > Key: FLINK-35626 > URL: https://issues.apache.org/jira/browse/FLINK-35626 > Project: Flink > Issue Type: New Feature > Components: Documentation >Reporter: Mark >Priority: Minor > > In order to be able to use Apache Flink in certain environments with strict > compliance requirements, the application may need to be compliant with > FIPS-140-2. > The path to produce a FIPS complaint image can vary, depending on the > programming language and application itself. For Java, this may involve: > * For any bundled crypto, utilising FIPS versions, i.e the FIPS version of > BouncyCastle (bcfips). Alternatively, ensuring there is no bundled crypto, > and all operations fallback to the JRE / OpenSSL on the host > * Where keystores are required, support for creating bckfs keystore types > * Producing a base image with OpenSSL configured for FIPS, as well as a JRE > configured for FIPS (such as bcfips) > * No use of unapproved crypto algorithms, such as DES, MD5, SHA1 etc > It would be great if Flink was able to support FIPS - by that, I mean > ensuring all requirements are met, and some documentation outlining how this > is met and enforced. > From a quick scan of the codebase, I don't see any direct references to > bouncycastle (either non-fips or fips), but I do see several places where md5 > and sha1 are used for example. > Support for FIPS would increase the use-cases for deploying Flink in secure > environments. Thanks for taking this into consideration. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35626) Add support for FIPS
[ https://issues.apache.org/jira/browse/FLINK-35626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser updated FLINK-35626: --- Priority: Not a Priority (was: Minor) > Add support for FIPS > > > Key: FLINK-35626 > URL: https://issues.apache.org/jira/browse/FLINK-35626 > Project: Flink > Issue Type: New Feature > Components: Documentation >Reporter: Mark >Priority: Not a Priority > > In order to be able to use Apache Flink in certain environments with strict > compliance requirements, the application may need to be compliant with > FIPS-140-2. > The path to produce a FIPS complaint image can vary, depending on the > programming language and application itself. For Java, this may involve: > * For any bundled crypto, utilising FIPS versions, i.e the FIPS version of > BouncyCastle (bcfips). Alternatively, ensuring there is no bundled crypto, > and all operations fallback to the JRE / OpenSSL on the host > * Where keystores are required, support for creating bckfs keystore types > * Producing a base image with OpenSSL configured for FIPS, as well as a JRE > configured for FIPS (such as bcfips) > * No use of unapproved crypto algorithms, such as DES, MD5, SHA1 etc > It would be great if Flink was able to support FIPS - by that, I mean > ensuring all requirements are met, and some documentation outlining how this > is met and enforced. > From a quick scan of the codebase, I don't see any direct references to > bouncycastle (either non-fips or fips), but I do see several places where md5 > and sha1 are used for example. > Support for FIPS would increase the use-cases for deploying Flink in secure > environments. Thanks for taking this into consideration. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35626) Add support for FIPS
[ https://issues.apache.org/jira/browse/FLINK-35626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17855815#comment-17855815 ] Martijn Visser commented on FLINK-35626: Realistically I don't think that we will do this, unless there are multiple maintainers who are willing to take this one. But looking at various other security/compliance topics (not always using the latest versions of dependencies, no RBAC etc), I don't think this is something that will happen. > Add support for FIPS > > > Key: FLINK-35626 > URL: https://issues.apache.org/jira/browse/FLINK-35626 > Project: Flink > Issue Type: New Feature >Reporter: Mark >Priority: Minor > > In order to be able to use Apache Flink in certain environments with strict > compliance requirements, the application may need to be compliant with > FIPS-140-2. > The path to produce a FIPS complaint image can vary, depending on the > programming language and application itself. For Java, this may involve: > * For any bundled crypto, utilising FIPS versions, i.e the FIPS version of > BouncyCastle (bcfips). Alternatively, ensuring there is no bundled crypto, > and all operations fallback to the JRE / OpenSSL on the host > * Where keystores are required, support for creating bckfs keystore types > * Producing a base image with OpenSSL configured for FIPS, as well as a JRE > configured for FIPS (such as bcfips) > * No use of unapproved crypto algorithms, such as DES, MD5, SHA1 etc > It would be great if Flink was able to support FIPS - by that, I mean > ensuring all requirements are met, and some documentation outlining how this > is met and enforced. > From a quick scan of the codebase, I don't see any direct references to > bouncycastle (either non-fips or fips), but I do see several places where md5 > and sha1 are used for example. > Support for FIPS would increase the use-cases for deploying Flink in secure > environments. Thanks for taking this into consideration. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35630) Kafka source may reset the consume offset to earliest when the partition leader changes
[ https://issues.apache.org/jira/browse/FLINK-35630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17855813#comment-17855813 ] Martijn Visser commented on FLINK-35630: [~tanjialiang] Do you have a reproducible scenario, because this would indicate a problem for all Kafka consumers, not so much Flink. Flink is just using the Kafka consumer under the hood in this case. > Kafka source may reset the consume offset to earliest when the partition > leader changes > --- > > Key: FLINK-35630 > URL: https://issues.apache.org/jira/browse/FLINK-35630 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka >Affects Versions: kafka-3.2.0 >Reporter: tanjialiang >Priority: Major > > Kafka producer using the ack=1 option to write data to a topic. > Flink Kafka source startup with the *scan.startup.mode=earliest-offset* > option to consume (the Kafka *auto.offset.reset* option will be force > override to earliest). > If a partition leader is not available, a follower may become the new leader > and this may trigger log truncation. It may cause consumers to consume offset > out of range and use the *auto.offset.reset* strategy to reset the offset. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-35378) [FLIP-453] Promote Unified Sink API V2 to Public and Deprecate SinkFunction
[ https://issues.apache.org/jira/browse/FLINK-35378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser closed FLINK-35378. -- Fix Version/s: 1.20.0 Resolution: Fixed Fixed in apache/flink:master f0b01277dd23dd0edc2a65c2634370936f95c136 > [FLIP-453] Promote Unified Sink API V2 to Public and Deprecate SinkFunction > --- > > Key: FLINK-35378 > URL: https://issues.apache.org/jira/browse/FLINK-35378 > Project: Flink > Issue Type: Technical Debt > Components: API / Core >Reporter: Martijn Visser >Assignee: Martijn Visser >Priority: Major > Labels: pull-request-available > Fix For: 1.20.0 > > > https://cwiki.apache.org/confluence/pages/resumedraft.action?draftId=303794871&draftShareId=af4ace88-98b7-4a53-aece-cd67d2f91a15&; -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35378) [FLIP-453] Promote Unified Sink API V2 to Public and Deprecate SinkFunction
[ https://issues.apache.org/jira/browse/FLINK-35378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser updated FLINK-35378: --- Summary: [FLIP-453] Promote Unified Sink API V2 to Public and Deprecate SinkFunction (was: [FLIP-453] Promote Unified Sink API V2 to Public and Deprecate SinkFunc) > [FLIP-453] Promote Unified Sink API V2 to Public and Deprecate SinkFunction > --- > > Key: FLINK-35378 > URL: https://issues.apache.org/jira/browse/FLINK-35378 > Project: Flink > Issue Type: Technical Debt > Components: API / Core >Reporter: Martijn Visser >Assignee: Martijn Visser >Priority: Major > Labels: pull-request-available > > https://cwiki.apache.org/confluence/pages/resumedraft.action?draftId=303794871&draftShareId=af4ace88-98b7-4a53-aece-cd67d2f91a15&; -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35566) Consider promoting TypeSerializer from PublicEvolving to Public
Martijn Visser created FLINK-35566: -- Summary: Consider promoting TypeSerializer from PublicEvolving to Public Key: FLINK-35566 URL: https://issues.apache.org/jira/browse/FLINK-35566 Project: Flink Issue Type: Technical Debt Components: API / Core Reporter: Martijn Visser While working on implementing FLINK-35378, I ran into the problem that TypeSerializer is still on PublicEvolving since Flink 1.0. We should consider annotating this as Public. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-12173) Optimize "SELECT DISTINCT" into Deduplicate with keep first row
[
https://issues.apache.org/jira/browse/FLINK-12173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser updated FLINK-12173:
---
Priority: Major (was: Not a Priority)
> Optimize "SELECT DISTINCT" into Deduplicate with keep first row
> ---
>
> Key: FLINK-12173
> URL: https://issues.apache.org/jira/browse/FLINK-12173
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / Planner
>Reporter: Jark Wu
>Priority: Major
> Labels: auto-deprioritized-major, auto-deprioritized-minor
>
> The following distinct query can be optimized into deduplicate on keys "a, b,
> c, d" and keep the first row.
> {code:sql}
> SELECT DISTINCT a, b, c, d;
> {code}
> We can optimize this query into Deduplicate to get a better performance than
> GroupAggregate.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-35515) Upgrade hive version to 4.0.0
[ https://issues.apache.org/jira/browse/FLINK-35515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser updated FLINK-35515: --- Issue Type: New Feature (was: Improvement) > Upgrade hive version to 4.0.0 > - > > Key: FLINK-35515 > URL: https://issues.apache.org/jira/browse/FLINK-35515 > Project: Flink > Issue Type: New Feature > Components: Connectors / Hive >Affects Versions: 1.18.1 >Reporter: vikasap >Priority: Major > > Hive version 4.0.0 was released recently. However none of the major flink > versions will work with this. Filing this so that major flink version's > flink-sql and table api will be able to work with the new version of hive > metastore. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35515) Upgrade hive version to 4.0.0
[ https://issues.apache.org/jira/browse/FLINK-35515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser updated FLINK-35515: --- Fix Version/s: (was: 1.18.2) > Upgrade hive version to 4.0.0 > - > > Key: FLINK-35515 > URL: https://issues.apache.org/jira/browse/FLINK-35515 > Project: Flink > Issue Type: Improvement > Components: Connectors / Hive >Affects Versions: 1.18.1 >Reporter: vikasap >Priority: Major > > Hive version 4.0.0 was released recently. However none of the major flink > versions will work with this. Filing this so that major flink version's > flink-sql and table api will be able to work with the new version of hive > metastore. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-25538) [JUnit5 Migration] Module: flink-connector-kafka
[ https://issues.apache.org/jira/browse/FLINK-25538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser reassigned FLINK-25538: -- Assignee: Muhammet Orazov (was: xiang1 yu) > [JUnit5 Migration] Module: flink-connector-kafka > > > Key: FLINK-25538 > URL: https://issues.apache.org/jira/browse/FLINK-25538 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Kafka >Reporter: Qingsheng Ren >Assignee: Muhammet Orazov >Priority: Minor > Labels: pull-request-available, stale-assigned, starter > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-35470) Flink 1.18.1 and 1.19.0 having build jdk 1.8 and causing incompatibilities with Java 17
[ https://issues.apache.org/jira/browse/FLINK-35470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser closed FLINK-35470. -- Resolution: Invalid Please post user questions to the User mailing list, Stackoverflow or Slack, as is explained in our docs https://flink.apache.org/how-to-contribute/getting-help/#having-a-question > Flink 1.18.1 and 1.19.0 having build jdk 1.8 and causing incompatibilities > with Java 17 > --- > > Key: FLINK-35470 > URL: https://issues.apache.org/jira/browse/FLINK-35470 > Project: Flink > Issue Type: Bug > Components: Flink CDC >Affects Versions: 1.18.1 >Reporter: Rajat Pratap Singh >Priority: Critical > > I am writing flink jobs with latest release version for flink (1.18.1). The > jobmanager is also deployed with the same version build. But we faced issues > when we deployed the jobs. On further investigation, I noticed all libraries > from flink have build jdk 1.8. is this the correct version? How to write > flink jobs with Java 17 code and which library versions to use? Is Java 17 > even supported on the client side or is it just on the server side support > for java 17? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-34746) Switching to the Apache CDN for Dockerfile
[ https://issues.apache.org/jira/browse/FLINK-34746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser closed FLINK-34746. -- Resolution: Fixed Fixed in apache/flink-docker@master 883600747505c128d97e9d25c9326f0c6f1d31e4 > Switching to the Apache CDN for Dockerfile > -- > > Key: FLINK-34746 > URL: https://issues.apache.org/jira/browse/FLINK-34746 > Project: Flink > Issue Type: Improvement > Components: flink-docker >Reporter: lincoln lee >Assignee: Martijn Visser >Priority: Major > Labels: pull-request-available > Fix For: 1.18.2, 1.20.0, 1.19.1 > > > During publishing the official image, we received some comments > for Switching to the Apache CDN > > See > https://github.com/docker-library/official-images/pull/16114 > https://github.com/docker-library/official-images/pull/16430 > > Reason for switching: [https://apache.org/history/mirror-history.html] (also > [https://www.apache.org/dyn/closer.cgi] and [https://www.apache.org/mirrors]) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35417) JobManager and TaskManager support merging and run in a single process
[ https://issues.apache.org/jira/browse/FLINK-35417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17848485#comment-17848485 ] Martijn Visser commented on FLINK-35417: Per https://flink.apache.org/how-to-contribute/contribute-code/#consensus this should first be a discussion on the Dev mailing list before opening a Jira ticket. > JobManager and TaskManager support merging and run in a single process > -- > > Key: FLINK-35417 > URL: https://issues.apache.org/jira/browse/FLINK-35417 > Project: Flink > Issue Type: Improvement > Components: API / Core >Reporter: melin >Priority: Major > > flink is widely used in data integration scenarios, where a single > concurrency is not high, and in many cases a single concurrency can run a > task. Consider the high availability, application mode, and large number of > JobManger nodes that cost a lot of resources. If the Session mode is used, > the stability is not high. > In application mode, JobManager and TaskManager can be run together to > achieve reliability and save resources. > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-20539) Type mismatch when using ROW in computed column
[
https://issues.apache.org/jira/browse/FLINK-20539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17848160#comment-17848160
]
Martijn Visser commented on FLINK-20539:
Hope you're feeling better [~qingyue] - Happy to have helped out
> Type mismatch when using ROW in computed column
> ---
>
> Key: FLINK-20539
> URL: https://issues.apache.org/jira/browse/FLINK-20539
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: Timo Walther
>Assignee: xuyang
>Priority: Major
> Labels: auto-unassigned, pull-request-available
> Fix For: 1.19.0, 1.18.2
>
>
> The following SQL:
> {code}
> env.executeSql(
> "CREATE TABLE Orders (\n"
> + "order_number BIGINT,\n"
> + "priceINT,\n"
> + "first_name STRING,\n"
> + "last_nameSTRING,\n"
> + "buyer_name AS ROW(first_name, last_name)\n"
> + ") WITH (\n"
> + " 'connector' = 'datagen'\n"
> + ")");
> env.executeSql("SELECT * FROM Orders").print();
> {code}
> Fails with:
> {code}
> Exception in thread "main" java.lang.AssertionError: Conversion to relational
> algebra failed to preserve datatypes:
> validated type:
> RecordType(BIGINT order_number, INTEGER price, VARCHAR(2147483647) CHARACTER
> SET "UTF-16LE" first_name, VARCHAR(2147483647) CHARACTER SET "UTF-16LE"
> last_name, RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
> "UTF-16LE" EXPR$0, VARCHAR(2147483647) CHARACTER SET "UTF-16LE" EXPR$1) NOT
> NULL buyer_name) NOT NULL
> converted type:
> RecordType(BIGINT order_number, INTEGER price, VARCHAR(2147483647) CHARACTER
> SET "UTF-16LE" first_name, VARCHAR(2147483647) CHARACTER SET "UTF-16LE"
> last_name, RecordType(VARCHAR(2147483647) CHARACTER SET "UTF-16LE" EXPR$0,
> VARCHAR(2147483647) CHARACTER SET "UTF-16LE" EXPR$1) NOT NULL buyer_name) NOT
> NULL
> rel:
> LogicalProject(order_number=[$0], price=[$1], first_name=[$2],
> last_name=[$3], buyer_name=[ROW($2, $3)])
> LogicalTableScan(table=[[default_catalog, default_database, Orders]])
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.checkConvertedType(SqlToRelConverter.java:467)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:582)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-32828) Partition aware watermark not handled correctly shortly after job start up from checkpoint or savepoint
[
https://issues.apache.org/jira/browse/FLINK-32828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser updated FLINK-32828:
---
Fix Version/s: (was: 2.0.0)
> Partition aware watermark not handled correctly shortly after job start up
> from checkpoint or savepoint
> ---
>
> Key: FLINK-32828
> URL: https://issues.apache.org/jira/browse/FLINK-32828
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream, Connectors / Kafka
>Affects Versions: 1.17.1, 1.19.0, 1.18.1
> Environment: Affected environments:
> * Local MiniCluster + Confluent Kafka run in docker
> ** See attached files
> * Flink job run in Kubernetes using Flink Operator 1.15 + 3 node Kafka
> cluster run in Kubernetes cluster
>Reporter: Grzegorz Liter
>Assignee: Piotr Nowojski
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.18.2, 1.20.0, 1.19.1
>
> Attachments: docker-compose.yml, test-job.java
>
>
> When using KafkaSource with partition aware watermarks. Watermarks are being
> emitted even when only one partition has some events just after job startup
> from savepoint/checkpoint. After it has some events on other partitions the
> watermark behaviour is correct and watermark is emited as a minimum watarmark
> from all partition.
>
> Steps to reproduce:
> # Setup a Kafka cluster with a topic that has 2 or more partitions. (see
> attached docker-compose.yml)
> #
> ## {{./kafka-topics.sh --bootstrap-server localhost:9092 --create --topic
> test-2 --partitions 4}}
> # Create a job that (see attached `test-job.java`):
> ## uses a KafkaSource with
> `WatermarkStrategy.forBoundedOutOfOrderness(Duration.ofSeconds(10L)`
> ## has parallelism lower than number of partitions
> ## stores checkpoint/savepoint
> # Start job
> # Send events only on single partition
> ## {{./kafka-console-producer.sh --bootstrap-server localhost:9092 --topic
> test-2 --property "parse.key=true" --property "key.separator=:"}}
>
> {{14:51:19,883 WARN com.example.TestJob6$InputSink2 [] -
> == Received: test-2/3: 1 -> a, timestamp 2023-08-10T12:51:18.849Z, watermark
> -292275055-05-16T16:47:04.192Z}}
> {{14:51:32,484 WARN com.example.TestJob6$InputSink2 [] -
> == Received: test-2/3: 1 -> a, timestamp 2023-08-10T12:51:31.475Z, watermark
> -292275055-05-16T16:47:04.192Z}}
> {{14:51:35,914 WARN com.example.TestJob6$InputSink2 [] -
> == Received: test-2/3: 1 -> a, timestamp 2023-08-10T12:51:34.909Z, watermark
> -292275055-05-16T16:47:04.192Z}}
> Expected: Watermark does not progress. Actual: Watermark does not progress.
> 5. Stop the job
> 6. Startup job from last checkpoint/savepoint
> 7. Send events only on single partitions
> {{14:53:41,693 WARN com.example.TestJob6$InputSink2 [] -
> == Received: test-2/3: 1 -> a, timestamp 2023-08-10T12:53:40.662Z, watermark
> -292275055-05-16T16:47:04.192Z}}
> {{14:53:46,088 WARN com.example.TestJob6$InputSink2 [] -
> == Received: test-2/3: 1 -> a, timestamp 2023-08-10T12:53:45.078Z, watermark
> 2023-08-10T12:53:30.661Z}}
> {{14:53:49,520 WARN com.example.TestJob6$InputSink2 [] -
> == Received: test-2/3: 1 -> a, timestamp 2023-08-10T12:53:48.511Z, watermark
> 2023-08-10T12:53:35.077Z}}
> Expected: Watermark does not progress. {color:#ff}*Actual: Watermark has
> progress*{color}
>
> {color:#172b4d}To add bit more of context:{color}
> {color:#172b4d}8. Send events on other partitions and then send events only
> on single partitions{color}
> {{{color:#172b4d}14:54:55,112 WARN com.example.TestJob6$InputSink2
> [] - == Received: test-2/0: 2 -> a, timestamp
> 2023-08-10T12:54:54.104Z, watermark 2023-08-10T12:53:38.510Z
> 14:54:57,673 WARN com.example.TestJob6$InputSink2 [] - ==
> Received: test-2/1: 4 -> a, timestamp 2023-08-10T12:54:56.665Z, watermark
> 2023-08-10T12:53:38.510Z
> 14:54:57,673 WARN com.example.TestJob6$InputSink2 [] - ==
> Received: test-2/2: 5 -> a, timestamp 2023-08-10T12:54:57.554Z, watermark
> 2023-08-10T12:53:38.510Z
> 14:55:12,821 WARN com.example.TestJob6$InputSink2 [] - ==
> Received: test-2/3: 1 -> a, timestamp 2023-08-10T12:55:11.814Z, watermark
> 2023-08-10T12:54:44.103Z
> 14:55:16,099 WARN com.example.TestJob6$InputSink2 [] - ==
> Received: test-2/3: 1 -> a, timestamp 2023-08-10T12:55:15.091Z, watermark
> 2023-08-10T12:54:44.103Z
> 14:55:19,122 WARN com.example.TestJob6$InputSink2 [] - ==
> Received: test-2/3: 1 -> a, timestamp 2023-08-10T12:55:18.114Z, watermark
> 2023-08-10T12:54:44.103Z{color}}}
>
[jira] [Closed] (FLINK-35393) Support flink kafka catalog
[ https://issues.apache.org/jira/browse/FLINK-35393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser closed FLINK-35393. -- Resolution: Duplicate > Support flink kafka catalog > --- > > Key: FLINK-35393 > URL: https://issues.apache.org/jira/browse/FLINK-35393 > Project: Flink > Issue Type: New Feature > Components: Connectors / Kafka >Reporter: melin >Priority: Major > > Support for flink kafka catalog -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-20539) Type mismatch when using ROW in computed column
[
https://issues.apache.org/jira/browse/FLINK-20539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17846968#comment-17846968
]
Martijn Visser edited comment on FLINK-20539 at 5/16/24 2:17 PM:
-
[~qingyue] Should this ticket have been closed already? Or is still open?
Edit: Checked myself: it's in 1.19.0, but not backported to 1.18.*
was (Author: martijnvisser):
[~qingyue] Should this ticket have been closed already? Or is still open?
> Type mismatch when using ROW in computed column
> ---
>
> Key: FLINK-20539
> URL: https://issues.apache.org/jira/browse/FLINK-20539
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: Timo Walther
>Assignee: xuyang
>Priority: Major
> Labels: auto-unassigned, pull-request-available
> Fix For: 1.19.0, 1.18.2
>
>
> The following SQL:
> {code}
> env.executeSql(
> "CREATE TABLE Orders (\n"
> + "order_number BIGINT,\n"
> + "priceINT,\n"
> + "first_name STRING,\n"
> + "last_nameSTRING,\n"
> + "buyer_name AS ROW(first_name, last_name)\n"
> + ") WITH (\n"
> + " 'connector' = 'datagen'\n"
> + ")");
> env.executeSql("SELECT * FROM Orders").print();
> {code}
> Fails with:
> {code}
> Exception in thread "main" java.lang.AssertionError: Conversion to relational
> algebra failed to preserve datatypes:
> validated type:
> RecordType(BIGINT order_number, INTEGER price, VARCHAR(2147483647) CHARACTER
> SET "UTF-16LE" first_name, VARCHAR(2147483647) CHARACTER SET "UTF-16LE"
> last_name, RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
> "UTF-16LE" EXPR$0, VARCHAR(2147483647) CHARACTER SET "UTF-16LE" EXPR$1) NOT
> NULL buyer_name) NOT NULL
> converted type:
> RecordType(BIGINT order_number, INTEGER price, VARCHAR(2147483647) CHARACTER
> SET "UTF-16LE" first_name, VARCHAR(2147483647) CHARACTER SET "UTF-16LE"
> last_name, RecordType(VARCHAR(2147483647) CHARACTER SET "UTF-16LE" EXPR$0,
> VARCHAR(2147483647) CHARACTER SET "UTF-16LE" EXPR$1) NOT NULL buyer_name) NOT
> NULL
> rel:
> LogicalProject(order_number=[$0], price=[$1], first_name=[$2],
> last_name=[$3], buyer_name=[ROW($2, $3)])
> LogicalTableScan(table=[[default_catalog, default_database, Orders]])
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.checkConvertedType(SqlToRelConverter.java:467)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:582)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-28220) Create Table Like support excluding physical columns
[
https://issues.apache.org/jira/browse/FLINK-28220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser updated FLINK-28220:
---
Fix Version/s: (was: 1.19.0)
(was: 1.18.2)
> Create Table Like support excluding physical columns
>
>
> Key: FLINK-28220
> URL: https://issues.apache.org/jira/browse/FLINK-28220
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / API
>Affects Versions: 1.15.0
>Reporter: JustinLee
>Priority: Major
>
> when users want to Create Table A Like B , they can choose to include or
> exclude options, computed columns ,etc. But it's mandatory that table A
> should inherit all physical columns of table B, which may cause inconvenience
> in some scenes , such as table A has its own schema and just want to inherit
> the options of table B.
> In our production case, sometimes we need to consume kafka table as RAW
> format which has an original JSON format , so we would like to Create Table A
> (data string ) with ('format'='raw') Like B, but it encounters an error
> saying table A has more than one physical columns because of inheriting these
> columns from table B.
> so I think it would be more flexible to provide the option to include or
> exclude physical columns when Using Create Table .. Like ..
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-20539) Type mismatch when using ROW in computed column
[
https://issues.apache.org/jira/browse/FLINK-20539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser updated FLINK-20539:
---
Fix Version/s: 1.18.2
1.19.0
> Type mismatch when using ROW in computed column
> ---
>
> Key: FLINK-20539
> URL: https://issues.apache.org/jira/browse/FLINK-20539
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: Timo Walther
>Assignee: xuyang
>Priority: Major
> Labels: auto-unassigned, pull-request-available
> Fix For: 1.19.0, 1.18.2
>
>
> The following SQL:
> {code}
> env.executeSql(
> "CREATE TABLE Orders (\n"
> + "order_number BIGINT,\n"
> + "priceINT,\n"
> + "first_name STRING,\n"
> + "last_nameSTRING,\n"
> + "buyer_name AS ROW(first_name, last_name)\n"
> + ") WITH (\n"
> + " 'connector' = 'datagen'\n"
> + ")");
> env.executeSql("SELECT * FROM Orders").print();
> {code}
> Fails with:
> {code}
> Exception in thread "main" java.lang.AssertionError: Conversion to relational
> algebra failed to preserve datatypes:
> validated type:
> RecordType(BIGINT order_number, INTEGER price, VARCHAR(2147483647) CHARACTER
> SET "UTF-16LE" first_name, VARCHAR(2147483647) CHARACTER SET "UTF-16LE"
> last_name, RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
> "UTF-16LE" EXPR$0, VARCHAR(2147483647) CHARACTER SET "UTF-16LE" EXPR$1) NOT
> NULL buyer_name) NOT NULL
> converted type:
> RecordType(BIGINT order_number, INTEGER price, VARCHAR(2147483647) CHARACTER
> SET "UTF-16LE" first_name, VARCHAR(2147483647) CHARACTER SET "UTF-16LE"
> last_name, RecordType(VARCHAR(2147483647) CHARACTER SET "UTF-16LE" EXPR$0,
> VARCHAR(2147483647) CHARACTER SET "UTF-16LE" EXPR$1) NOT NULL buyer_name) NOT
> NULL
> rel:
> LogicalProject(order_number=[$0], price=[$1], first_name=[$2],
> last_name=[$3], buyer_name=[ROW($2, $3)])
> LogicalTableScan(table=[[default_catalog, default_database, Orders]])
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.checkConvertedType(SqlToRelConverter.java:467)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:582)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-28220) Create Table Like support excluding physical columns
[
https://issues.apache.org/jira/browse/FLINK-28220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser updated FLINK-28220:
---
Fix Version/s: 1.18.2
1.19.0
> Create Table Like support excluding physical columns
>
>
> Key: FLINK-28220
> URL: https://issues.apache.org/jira/browse/FLINK-28220
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / API
>Affects Versions: 1.15.0
>Reporter: JustinLee
>Priority: Major
> Fix For: 1.19.0, 1.18.2
>
>
> when users want to Create Table A Like B , they can choose to include or
> exclude options, computed columns ,etc. But it's mandatory that table A
> should inherit all physical columns of table B, which may cause inconvenience
> in some scenes , such as table A has its own schema and just want to inherit
> the options of table B.
> In our production case, sometimes we need to consume kafka table as RAW
> format which has an original JSON format , so we would like to Create Table A
> (data string ) with ('format'='raw') Like B, but it encounters an error
> saying table A has more than one physical columns because of inheriting these
> columns from table B.
> so I think it would be more flexible to provide the option to include or
> exclude physical columns when Using Create Table .. Like ..
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-20539) Type mismatch when using ROW in computed column
[
https://issues.apache.org/jira/browse/FLINK-20539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17846968#comment-17846968
]
Martijn Visser commented on FLINK-20539:
[~qingyue] Should this ticket have been closed already? Or is still open?
> Type mismatch when using ROW in computed column
> ---
>
> Key: FLINK-20539
> URL: https://issues.apache.org/jira/browse/FLINK-20539
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: Timo Walther
>Assignee: xuyang
>Priority: Major
> Labels: auto-unassigned, pull-request-available
>
> The following SQL:
> {code}
> env.executeSql(
> "CREATE TABLE Orders (\n"
> + "order_number BIGINT,\n"
> + "priceINT,\n"
> + "first_name STRING,\n"
> + "last_nameSTRING,\n"
> + "buyer_name AS ROW(first_name, last_name)\n"
> + ") WITH (\n"
> + " 'connector' = 'datagen'\n"
> + ")");
> env.executeSql("SELECT * FROM Orders").print();
> {code}
> Fails with:
> {code}
> Exception in thread "main" java.lang.AssertionError: Conversion to relational
> algebra failed to preserve datatypes:
> validated type:
> RecordType(BIGINT order_number, INTEGER price, VARCHAR(2147483647) CHARACTER
> SET "UTF-16LE" first_name, VARCHAR(2147483647) CHARACTER SET "UTF-16LE"
> last_name, RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
> "UTF-16LE" EXPR$0, VARCHAR(2147483647) CHARACTER SET "UTF-16LE" EXPR$1) NOT
> NULL buyer_name) NOT NULL
> converted type:
> RecordType(BIGINT order_number, INTEGER price, VARCHAR(2147483647) CHARACTER
> SET "UTF-16LE" first_name, VARCHAR(2147483647) CHARACTER SET "UTF-16LE"
> last_name, RecordType(VARCHAR(2147483647) CHARACTER SET "UTF-16LE" EXPR$0,
> VARCHAR(2147483647) CHARACTER SET "UTF-16LE" EXPR$1) NOT NULL buyer_name) NOT
> NULL
> rel:
> LogicalProject(order_number=[$0], price=[$1], first_name=[$2],
> last_name=[$3], buyer_name=[ROW($2, $3)])
> LogicalTableScan(table=[[default_catalog, default_database, Orders]])
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.checkConvertedType(SqlToRelConverter.java:467)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:582)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-35378) [FLIP-453] Promote Unified Sink API V2 to Public and Deprecate SinkFunc
Martijn Visser created FLINK-35378: -- Summary: [FLIP-453] Promote Unified Sink API V2 to Public and Deprecate SinkFunc Key: FLINK-35378 URL: https://issues.apache.org/jira/browse/FLINK-35378 Project: Flink Issue Type: Technical Debt Components: API / Core Reporter: Martijn Visser Assignee: Martijn Visser https://cwiki.apache.org/confluence/pages/resumedraft.action?draftId=303794871&draftShareId=af4ace88-98b7-4a53-aece-cd67d2f91a15&; -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-35374) Flink 1.14 kafka connector Demo Error
[ https://issues.apache.org/jira/browse/FLINK-35374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser closed FLINK-35374. -- Resolution: Won't Fix This has already been fixed for supported versions; Flink 1.14. isn't supported anymore > Flink 1.14 kafka connector Demo Error > - > > Key: FLINK-35374 > URL: https://issues.apache.org/jira/browse/FLINK-35374 > Project: Flink > Issue Type: Bug > Components: Documentation >Affects Versions: 1.14.4 >Reporter: hongxu han >Priority: Major > Attachments: image-2024-05-16-16-14-52-414.png, > image-2024-05-16-16-16-01-621.png > > > !image-2024-05-16-16-14-52-414.png|width=249,height=139! > It should be > !image-2024-05-16-16-16-01-621.png|width=486,height=100! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35358) Breaking change when loading artifacts
[
https://issues.apache.org/jira/browse/FLINK-35358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17846536#comment-17846536
]
Martijn Visser commented on FLINK-35358:
[~ferenc-csaky] [~mbalassi] Any thoughts on this one?
> Breaking change when loading artifacts
> --
>
> Key: FLINK-35358
> URL: https://issues.apache.org/jira/browse/FLINK-35358
> Project: Flink
> Issue Type: New Feature
> Components: Client / Job Submission, flink-docker
>Affects Versions: 1.19.0
>Reporter: Rasmus Thygesen
>Priority: Not a Priority
>
> We have been using the following code snippet in our Dockerfiles for running
> a Flink job in application mode
>
> {code:java}
> FROM flink:1.18.1-scala_2.12-java17
> COPY --from=build /app/target/my-job*.jar
> /opt/flink/usrlib/artifacts/my-job.jar
> USER flink {code}
>
> Which has been working since at least around Flink 1.14, but the 1.19 update
> has broken our Dockerfiles. The fix is to put the jar file a step further out
> so the code snippet becomes
>
> {code:java}
> FROM flink:1.18.1-scala_2.12-java17
> COPY --from=build /app/target/my-job*.jar /opt/flink/usrlib/my-job.jar
> USER flink {code}
>
> We have not spent too much time looking into what the cause is, but we get
> the stack trace
>
> {code:java}
> myjob-jobmanager-1 | org.apache.flink.util.FlinkException: Could not load
> the provided entrypoint class.
> myjob-jobmanager-1 | at
> org.apache.flink.client.program.DefaultPackagedProgramRetriever.getPackagedProgram(DefaultPackagedProgramRetriever.java:230)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.getPackagedProgram(StandaloneApplicationClusterEntryPoint.java:149)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.lambda$main$0(StandaloneApplicationClusterEntryPoint.java:90)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.main(StandaloneApplicationClusterEntryPoint.java:89)
> [flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | Caused by:
> org.apache.flink.client.program.ProgramInvocationException: The program's
> entry point class 'my.company.job.MyJob' was not found in the jar file.
> myjob-jobmanager-1 | at
> org.apache.flink.client.program.PackagedProgram.loadMainClass(PackagedProgram.java:481)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.client.program.PackagedProgram.(PackagedProgram.java:153)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.client.program.PackagedProgram.(PackagedProgram.java:65)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.client.program.PackagedProgram$Builder.build(PackagedProgram.java:691)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.client.program.DefaultPackagedProgramRetriever.getPackagedProgram(DefaultPackagedProgramRetriever.java:228)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | ... 4 more
> myjob-jobmanager-1 | Caused by: java.lang.ClassNotFoundException:
> my.company.job.MyJob
> myjob-jobmanager-1 | at java.net.URLClassLoader.findClass(Unknown
> Source) ~[?:?]
> myjob-jobmanager-1 | at java.lang.ClassLoader.loadClass(Unknown Source)
> ~[?:?]
> myjob-jobmanager-1 | at
> org.apache.flink.util.FlinkUserCodeClassLoader.loadClassWithoutExceptionHandling(FlinkUserCodeClassLoader.java:67)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:74)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:51)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at java.lang.ClassLoader.loadClass(Unknown Source)
> ~[?:?]
> myjob-jobmanager-1 | at
> org.apache.flink.util.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.loadClass(FlinkUserCodeClassLoaders.java:197)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at java.lang.Class.forName0(Native Method) ~[?:?]
> myjob-jobmanager-1 | at java.lang.Class.forName(Unknown Source) ~[?:?]
> myjob-jobmanager-1 | at
> org.apache.flink.client.program.PackagedProgram.loadMainClass(PackagedProgram.java:479)
> ~[flink-dist-1.19.0.j
[jira] [Assigned] (FLINK-35350) Add documentation for Kudu
[ https://issues.apache.org/jira/browse/FLINK-35350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser reassigned FLINK-35350: -- Assignee: Ferenc Csaky > Add documentation for Kudu > -- > > Key: FLINK-35350 > URL: https://issues.apache.org/jira/browse/FLINK-35350 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Kudu >Reporter: Martijn Visser >Assignee: Ferenc Csaky >Priority: Major > Fix For: kudu-2.0.0 > > > There's currently no documentation for Kudu; this should be added -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-35114) Remove old Table API implementations
[ https://issues.apache.org/jira/browse/FLINK-35114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser reassigned FLINK-35114: -- Assignee: Ferenc Csaky > Remove old Table API implementations > > > Key: FLINK-35114 > URL: https://issues.apache.org/jira/browse/FLINK-35114 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Kudu >Reporter: Ferenc Csaky >Assignee: Ferenc Csaky >Priority: Major > > At the moment, the connector has both the old Table sink/source/catalog > implementations and the matching Dynamic... implementations as well. > Going forward, the deprecated old implementation should be removed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-34931) Update Kudu connector DataStream Sink implementation
[ https://issues.apache.org/jira/browse/FLINK-34931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser reassigned FLINK-34931: -- Assignee: Ferenc Csaky > Update Kudu connector DataStream Sink implementation > > > Key: FLINK-34931 > URL: https://issues.apache.org/jira/browse/FLINK-34931 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Kudu >Reporter: Ferenc Csaky >Assignee: Ferenc Csaky >Priority: Major > > Update the DataSource API classes to use the current interfaces. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35350) Add documentation for Kudu
Martijn Visser created FLINK-35350: -- Summary: Add documentation for Kudu Key: FLINK-35350 URL: https://issues.apache.org/jira/browse/FLINK-35350 Project: Flink Issue Type: Sub-task Components: Connectors / Kudu Reporter: Martijn Visser Fix For: kudu-2.0.0 There's currently no documentation for Kudu; this should be added -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-34930) Move existing Kudu connector code from Bahir repo to dedicated repo
[ https://issues.apache.org/jira/browse/FLINK-34930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser closed FLINK-34930. -- Fix Version/s: kudu-2.0.0 Resolution: Fixed > Move existing Kudu connector code from Bahir repo to dedicated repo > --- > > Key: FLINK-34930 > URL: https://issues.apache.org/jira/browse/FLINK-34930 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Kudu >Reporter: Ferenc Csaky >Assignee: Ferenc Csaky >Priority: Major > Labels: pull-request-available > Fix For: kudu-2.0.0 > > > Move the existing Kudu connector code from the Bahir [1] repository to the > dedicated connector repo. > Code should be moved only with necessary changes (bump version, change > groupID, integrate to common connector CI) and we sould state explicitly that > the state was forked from the Bahir repo. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-34930) Move existing Kudu connector code from Bahir repo to dedicated repo
[ https://issues.apache.org/jira/browse/FLINK-34930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser reassigned FLINK-34930: -- Assignee: Ferenc Csaky > Move existing Kudu connector code from Bahir repo to dedicated repo > --- > > Key: FLINK-34930 > URL: https://issues.apache.org/jira/browse/FLINK-34930 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Kudu >Reporter: Ferenc Csaky >Assignee: Ferenc Csaky >Priority: Major > Labels: pull-request-available > > Move the existing Kudu connector code from the Bahir [1] repository to the > dedicated connector repo. > Code should be moved only with necessary changes (bump version, change > groupID, integrate to common connector CI) and we sould state explicitly that > the state was forked from the Bahir repo. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-34930) Move existing Kudu connector code from Bahir repo to dedicated repo
[ https://issues.apache.org/jira/browse/FLINK-34930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17846193#comment-17846193 ] Martijn Visser commented on FLINK-34930: Code has been migrated > Move existing Kudu connector code from Bahir repo to dedicated repo > --- > > Key: FLINK-34930 > URL: https://issues.apache.org/jira/browse/FLINK-34930 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Kudu >Reporter: Ferenc Csaky >Assignee: Ferenc Csaky >Priority: Major > Labels: pull-request-available > > Move the existing Kudu connector code from the Bahir [1] repository to the > dedicated connector repo. > Code should be moved only with necessary changes (bump version, change > groupID, integrate to common connector CI) and we sould state explicitly that > the state was forked from the Bahir repo. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-34961) GitHub Actions runner statistcs can be monitored per workflow name
[
https://issues.apache.org/jira/browse/FLINK-34961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17846192#comment-17846192
]
Martijn Visser commented on FLINK-34961:
Merged to flink-connector-kudu@main as 409856698df1bf630c58f6e761d45e4dc8f06ad5
> GitHub Actions runner statistcs can be monitored per workflow name
> --
>
> Key: FLINK-34961
> URL: https://issues.apache.org/jira/browse/FLINK-34961
> Project: Flink
> Issue Type: Improvement
> Components: Build System / CI
>Reporter: Matthias Pohl
>Priority: Major
> Labels: pull-request-available, starter
> Fix For: kafka-4.0.0, jdbc-3.2.0, jdbc-3.1.3, kafka-3.2.0
>
>
> Apache Infra allows the monitoring of runner usage per workflow (see [report
> for
> Flink|https://infra-reports.apache.org/#ghactions&project=flink&hours=168&limit=10];
> only accessible with Apache committer rights). They accumulate the data by
> workflow name. The Flink space has multiple repositories that use the generic
> workflow name {{CI}}). That makes the differentiation in the report harder.
> This Jira issue is about identifying all Flink-related projects with a CI
> workflow (Kubernetes operator and the JDBC connector were identified, for
> instance) and adding a more distinct name.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-35336) SQL failed to restore from savepoint after change in default-parallelism
[
https://issues.apache.org/jira/browse/FLINK-35336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser updated FLINK-35336:
---
Issue Type: New Feature (was: Bug)
> SQL failed to restore from savepoint after change in default-parallelism
>
>
> Key: FLINK-35336
> URL: https://issues.apache.org/jira/browse/FLINK-35336
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / Planner
>Affects Versions: 1.18.1
> Environment: Flink SQL Client, Flink 1.18.1 on MacOS
>Reporter: Keith Lee
>Priority: Major
>
> After bumping 'table.exec.resource.default-parallelism' from 1 to 4, I am
> observing the following exception on restoring job from savepoint with an
> unmodified statement set.
>
> {quote}[ERROR] Could not execute SQL statement. Reason:
> java.lang.IllegalStateException: Failed to rollback to checkpoint/savepoint
> [file:/tmp/flink-savepoints/savepoint-4392e6-575fa6b692ff|file:///tmp/flink-savepoints/savepoint-4392e6-575fa6b692ff].
> Cannot map checkpoint/savepoint state for operator
> 46ba9b22862c3bbe9373c6abee964b2a to the new program, because the operator is
> not available in the new program. If you want to allow to skip this, you can
> set the --allowNonRestoredState option on the CLI.
> {quote}
> When started without savepoints, the jobgraph differs for the jobs despite
> identical statements being ran.
> There are 2 operators when default parallelism is 1.
> {quote}A: Source: UserBehaviourKafkaSource[68] -> (Calc[69] ->
> StreamRecordTimestampInserter[70] -> StreamingFileWriter -> Sink: end,
> Calc[71] -> LocalWindowAggregate[72])
> B: GlobalWindowAggregate[74] -> Calc[75] -> Sink: CampaignAggregationsJDBC[76]
> {quote}
> Three operators when default parallelism is 4.
> {quote}A: Source: UserBehaviourKafkaSource[86] -> (Calc[87] ->
> StreamRecordTimestampInserter[88] -> StreamingFileWriter, Calc[89] ->
> LocalWindowAggregate[90])
> B: Sink: end
> C: GlobalWindowAggregate[92] -> Calc[93] -> Sink: CampaignAggregationsJDBC[94]
> {quote}
>
> Notice that the operator 'Sink: end' is separated out when parallelism is set
> to 4, causing the incompatibility in job graph. EXPLAIN PLAN did not show any
> difference between syntax tree, physical plan or execution plan.
> I have attempted various configurations in `table.optimizer.*`.
> Steps to reproduce
> {quote}SET 'table.exec.resource.default-parallelism' = '1';
> EXECUTE STATEMENT SET BEGIN
> INSERT INTO UserErrorExperienceS3Sink (user_id, user_session,
> interaction_type, interaction_target, interaction_tags, event_date,
> event_hour, event_time)
> SELECT
> user_id,
> user_session,
> interaction_type,
> interaction_target,
> interaction_tags,
> DATE_FORMAT(event_time , '-MM-dd'),
> DATE_FORMAT(event_time , 'HH'),
> event_time
> FROM UserBehaviourKafkaSource
> WHERE
> interaction_result Like '%ERROR%';
> INSERT INTO CampaignAggregationsJDBC
> SELECT
> CONCAT_WS('/', interaction_tags, interaction_result,
> DATE_FORMAT(window_start, '-MM-DD HH:mm:ss.SSS'), DATE_FORMAT(window_end,
> '-MM-DD HH:mm:ss.SSS')) AS id,
> interaction_tags as campaign,
> interaction_result,
> COUNT(*) AS interaction_count,
> window_start,
> window_end
> FROM
> TABLE(TUMBLE(TABLE UserBehaviourKafkaSource, DESCRIPTOR(event_time),
> INTERVAL '10' SECONDS))
> GROUP BY window_start, window_end, interaction_tags, interaction_result;
> END;
> STOP JOB '' WITH SAVEPOINT;
> SET 'execution.savepoint.path' = '//';
> SET 'table.exec.resource.default-parallelism' = '4';
>
> {quote}
> DDLs
> {quote}– S3 Sink
> CREATE TABLE UserErrorExperienceS3Sink (
> user_id BIGINT,
> user_session STRING,
> interaction_type STRING,
> interaction_target STRING,
> interaction_tags STRING,
> event_date STRING,
> event_hour STRING,
> event_time TIMESTAMP(3) WITHOUT TIME ZONE)
> PARTITIONED BY (event_date, event_hour)
> WITH (
> 'connector' = 'filesystem',
> 'path' = 's3:///userErrorExperience/',
> 'format' = 'json');
> – Kafka Source
> ADD JAR
> 'file:///Users/leekei/Downloads/flink-sql-connector-kafka-3.1.0-1.18.jar';
> CREATE TABLE UserBehaviourKafkaSource (
> user_id BIGINT,
> user_session STRING,
> interaction_type STRING,
> interaction_target STRING,
> interaction_tags STRING,
> interaction_result STRING,
> event_time TIMESTAMP(3) WITHOUT TIME ZONE METADATA FROM 'timestamp',
> WATERMARK FOR event_time AS event_time - INTERVAL '5' SECOND)
> WITH (
> 'connector' = 'kafka',
> 'topic' = 'user_behaviour',
> 'properties.bootstrap.servers' = 'localhost:9092',
> 'properties.group.id' = 'demoGroup',
> 's
[jira] [Commented] (FLINK-35336) SQL failed to restore from savepoint after change in default-parallelism
[
https://issues.apache.org/jira/browse/FLINK-35336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845901#comment-17845901
]
Martijn Visser commented on FLINK-35336:
That's not a bug: changing the parallelism is generates a new jobgraph and that
will lead to state incompatibility, as outlined in
https://nightlies.apache.org/flink/flink-docs-master/docs/ops/upgrading/#table-api--sql
See https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191336489
for more details as well
> SQL failed to restore from savepoint after change in default-parallelism
>
>
> Key: FLINK-35336
> URL: https://issues.apache.org/jira/browse/FLINK-35336
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.18.1
> Environment: Flink SQL Client, Flink 1.18.1 on MacOS
>Reporter: Keith Lee
>Priority: Major
>
> After bumping 'table.exec.resource.default-parallelism' from 1 to 4, I am
> observing the following exception on restoring job from savepoint with an
> unmodified statement set.
>
> {quote}[ERROR] Could not execute SQL statement. Reason:
> java.lang.IllegalStateException: Failed to rollback to checkpoint/savepoint
> [file:/tmp/flink-savepoints/savepoint-4392e6-575fa6b692ff|file:///tmp/flink-savepoints/savepoint-4392e6-575fa6b692ff].
> Cannot map checkpoint/savepoint state for operator
> 46ba9b22862c3bbe9373c6abee964b2a to the new program, because the operator is
> not available in the new program. If you want to allow to skip this, you can
> set the --allowNonRestoredState option on the CLI.
> {quote}
> When started without savepoints, the jobgraph differs for the jobs despite
> identical statements being ran.
> There are 2 operators when default parallelism is 1.
> {quote}A: Source: UserBehaviourKafkaSource[68] -> (Calc[69] ->
> StreamRecordTimestampInserter[70] -> StreamingFileWriter -> Sink: end,
> Calc[71] -> LocalWindowAggregate[72])
> B: GlobalWindowAggregate[74] -> Calc[75] -> Sink: CampaignAggregationsJDBC[76]
> {quote}
> Three operators when default parallelism is 4.
> {quote}A: Source: UserBehaviourKafkaSource[86] -> (Calc[87] ->
> StreamRecordTimestampInserter[88] -> StreamingFileWriter, Calc[89] ->
> LocalWindowAggregate[90])
> B: Sink: end
> C: GlobalWindowAggregate[92] -> Calc[93] -> Sink: CampaignAggregationsJDBC[94]
> {quote}
>
> Notice that the operator 'Sink: end' is separated out when parallelism is set
> to 4, causing the incompatibility in job graph. EXPLAIN PLAN did not show any
> difference between syntax tree, physical plan or execution plan.
> I have attempted various configurations in `table.optimizer.*`.
> Steps to reproduce
> {quote}SET 'table.exec.resource.default-parallelism' = '1';
> EXECUTE STATEMENT SET BEGIN
> INSERT INTO UserErrorExperienceS3Sink (user_id, user_session,
> interaction_type, interaction_target, interaction_tags, event_date,
> event_hour, event_time)
> SELECT
> user_id,
> user_session,
> interaction_type,
> interaction_target,
> interaction_tags,
> DATE_FORMAT(event_time , '-MM-dd'),
> DATE_FORMAT(event_time , 'HH'),
> event_time
> FROM UserBehaviourKafkaSource
> WHERE
> interaction_result Like '%ERROR%';
> INSERT INTO CampaignAggregationsJDBC
> SELECT
> CONCAT_WS('/', interaction_tags, interaction_result,
> DATE_FORMAT(window_start, '-MM-DD HH:mm:ss.SSS'), DATE_FORMAT(window_end,
> '-MM-DD HH:mm:ss.SSS')) AS id,
> interaction_tags as campaign,
> interaction_result,
> COUNT(*) AS interaction_count,
> window_start,
> window_end
> FROM
> TABLE(TUMBLE(TABLE UserBehaviourKafkaSource, DESCRIPTOR(event_time),
> INTERVAL '10' SECONDS))
> GROUP BY window_start, window_end, interaction_tags, interaction_result;
> END;
> STOP JOB '' WITH SAVEPOINT;
> SET 'execution.savepoint.path' = '//';
> SET 'table.exec.resource.default-parallelism' = '4';
>
> {quote}
> DDLs
> {quote}– S3 Sink
> CREATE TABLE UserErrorExperienceS3Sink (
> user_id BIGINT,
> user_session STRING,
> interaction_type STRING,
> interaction_target STRING,
> interaction_tags STRING,
> event_date STRING,
> event_hour STRING,
> event_time TIMESTAMP(3) WITHOUT TIME ZONE)
> PARTITIONED BY (event_date, event_hour)
> WITH (
> 'connector' = 'filesystem',
> 'path' = 's3:///userErrorExperience/',
> 'format' = 'json');
> – Kafka Source
> ADD JAR
> 'file:///Users/leekei/Downloads/flink-sql-connector-kafka-3.1.0-1.18.jar';
> CREATE TABLE UserBehaviourKafkaSource (
> user_id BIGINT,
> user_session STRING,
> interaction_type STRING,
> interaction_target STRING,
> interaction_tags STRING,
>
[jira] [Created] (FLINK-35333) JdbcXaSinkTestBase fails in weekly Flink JDBC Connector tests
Martijn Visser created FLINK-35333:
--
Summary: JdbcXaSinkTestBase fails in weekly Flink JDBC Connector
tests
Key: FLINK-35333
URL: https://issues.apache.org/jira/browse/FLINK-35333
Project: Flink
Issue Type: Bug
Components: Connectors / JDBC
Affects Versions: jdbc-3.2.0
Reporter: Martijn Visser
https://github.com/apache/flink-connector-jdbc/actions/runs/9047366679/job/24859224407#step:15:147
{code:java}
Error: Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.8.0:testCompile
(default-testCompile) on project flink-connector-jdbc: Compilation failure
Error:
/home/runner/work/flink-connector-jdbc/flink-connector-jdbc/flink-connector-jdbc/src/test/java/org/apache/flink/connector/jdbc/xa/JdbcXaSinkTestBase.java:[164,37]
is not
abstract and does not override abstract method getTaskInfo() in
org.apache.flink.api.common.functions.RuntimeContext
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-34108) Add URL_ENCODE and URL_DECODE function
[ https://issues.apache.org/jira/browse/FLINK-34108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser reassigned FLINK-34108: -- Assignee: chesterxu > Add URL_ENCODE and URL_DECODE function > -- > > Key: FLINK-34108 > URL: https://issues.apache.org/jira/browse/FLINK-34108 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Reporter: Martijn Visser >Assignee: chesterxu >Priority: Major > Labels: pull-request-available > > Add URL_ENCODE and URL_DECODE function > URL_ENCODE(str) - Translates a string into > 'application/x-www-form-urlencoded' format using a specific encoding scheme. > URL_DECODE(str) - Decodes a string in 'application/x-www-form-urlencoded' > format using a specific encoding scheme. > Related ticket from Calcite: CALCITE-5825 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-35280) Migrate HBase Sink connector to use the ASync Sink API
[ https://issues.apache.org/jira/browse/FLINK-35280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser reassigned FLINK-35280: -- Assignee: Ferenc Csaky > Migrate HBase Sink connector to use the ASync Sink API > -- > > Key: FLINK-35280 > URL: https://issues.apache.org/jira/browse/FLINK-35280 > Project: Flink > Issue Type: Technical Debt > Components: Connectors / HBase >Affects Versions: hbase-3.0.0, hbase-3.0.1, hbase-4.0.0 >Reporter: Martijn Visser >Assignee: Ferenc Csaky >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33183) Enable metadata columns in NduAnalyzer with retract if non-virtual
[ https://issues.apache.org/jira/browse/FLINK-33183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17843175#comment-17843175 ] Martijn Visser commented on FLINK-33183: [~lincoln.86xy] Friendly ping, what do you think of this ticket? > Enable metadata columns in NduAnalyzer with retract if non-virtual > -- > > Key: FLINK-33183 > URL: https://issues.apache.org/jira/browse/FLINK-33183 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Reporter: Timo Walther >Priority: Major > > Currently, the NduAnalyzer is very strict about metadata columns in updating > sources. Compared to append and upsert sources (see also FLINK-33182), > retract sources are tricky. And the analyzer is actually correct. > However, for retract sources we should expose more functionality to the user > and add a warning to the documentation that retract mode could potentially > cause NDU problems if not enough attention is paid. We should only throw an > error on virtual metadata columns. Persisted metadata columns can be > considered “safe“. When a metadata column is persisted, we can assume that an > upstream Flink job fills its content thus likely also fills its correct > retraction. -- This message was sent by Atlassian Jira (v8.20.10#820010)
