[jira] [Commented] (FLINK-28915) Make application mode could support remote DFS schema(e.g. S3, OSS, HDFS, etc.)
[
https://issues.apache.org/jira/browse/FLINK-28915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17855184#comment-17855184
]
Ferenc Csaky commented on FLINK-28915:
--
I played with it a bit more and {{s3://}} also works after I added the
{{flink-s3-fs-hadoop}} JAR as a plugin to the Flink img. I tested it with a
local minio deployed to k8s, created a
[gist|https://gist.github.com/ferenc-csaky/c2f2b601e6547e497778c4e336c62ff6]
from the {{Dockerfile}} and job YAML.
So the only thing missing for this to work officially is to have a K8s operator
release that is built on top of Flink 1.19, which is not true ATM.
> Make application mode could support remote DFS schema(e.g. S3, OSS, HDFS,
> etc.)
> ---
>
> Key: FLINK-28915
> URL: https://issues.apache.org/jira/browse/FLINK-28915
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / Kubernetes, flink-contrib
>Reporter: hjw
>Assignee: Ferenc Csaky
>Priority: Major
> Labels: pull-request-available, stale-assigned
> Fix For: 1.19.0
>
>
> As the Flink document show , local is the only supported scheme in Native k8s
> deployment.
> Is there have a plan to support s3 filesystem? thx.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (FLINK-34914) FLIP-436: Introduce Catalog-related Syntax
[ https://issues.apache.org/jira/browse/FLINK-34914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jane Chan resolved FLINK-34914. --- Fix Version/s: 1.20.0 Resolution: Fixed > FLIP-436: Introduce Catalog-related Syntax > -- > > Key: FLINK-34914 > URL: https://issues.apache.org/jira/browse/FLINK-34914 > Project: Flink > Issue Type: New Feature > Components: Table SQL / API >Affects Versions: 1.20.0 >Reporter: Yubin Li >Assignee: Yubin Li >Priority: Major > Fix For: 1.20.0 > > > Umbrella issue for: > https://cwiki.apache.org/confluence/display/FLINK/FLIP-436%3A+Introduce+Catalog-related+Syntax -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] [cdc][postgres]PostgresDialect removes useless code [flink-cdc]
ChengJie1053 opened a new pull request, #3416: URL: https://github.com/apache/flink-cdc/pull/3416   -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-32086][checkpointing] Cleanup useless file-merging managed directory on exit of TM [flink]
Zakelly commented on PR #24933: URL: https://github.com/apache/flink/pull/24933#issuecomment-2169093495 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-32086][checkpointing] Cleanup useless file-merging managed directory on exit of TM [flink]
Zakelly commented on PR #24933: URL: https://github.com/apache/flink/pull/24933#issuecomment-2169093315 Ah, the CI is not stable. There seems to be some network issue between agents. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-32229) Implement metrics and logging for Initial implementation
[ https://issues.apache.org/jira/browse/FLINK-32229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-32229: --- Labels: pull-request-available (was: ) > Implement metrics and logging for Initial implementation > > > Key: FLINK-32229 > URL: https://issues.apache.org/jira/browse/FLINK-32229 > Project: Flink > Issue Type: Sub-task >Reporter: Hong Liang Teoh >Assignee: Burak Ozakinci >Priority: Major > Labels: pull-request-available > > Add/Ensure Kinesis specific metrics for MillisBehindLatest/numRecordsIn are > published. > More metrics here: > [https://cwiki.apache.org/confluence/display/FLINK/FLIP-33%3A+Standardize+Connector+Metrics] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35621) Release Testing Instructions: Verify FLIP-436: Introduce Catalog-related Syntax
[ https://issues.apache.org/jira/browse/FLINK-35621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weijie Guo updated FLINK-35621: --- Component/s: Table SQL / API (was: Connectors / Common) > Release Testing Instructions: Verify FLIP-436: Introduce Catalog-related > Syntax > --- > > Key: FLINK-35621 > URL: https://issues.apache.org/jira/browse/FLINK-35621 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Reporter: Weijie Guo >Assignee: Yubin Li >Priority: Blocker > Labels: release-testing > Fix For: 1.20.0 > > > Follow up the test for https://issues.apache.org/jira/browse/FLINK-34914 -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-32229][Connector/Kinesis] Add millisBehindLatest metric to Kinesis source implementation [flink-connector-aws]
boring-cyborg[bot] commented on PR #143: URL: https://github.com/apache/flink-connector-aws/pull/143#issuecomment-2168811002 Thanks for opening this pull request! Please check out our contributing guidelines. (https://flink.apache.org/contributing/how-to-contribute.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [FLINK-32229][Connector/Kinesis] Add millisBehindLatest metric to Kinesis source implementation [flink-connector-aws]
karubian opened a new pull request, #143: URL: https://github.com/apache/flink-connector-aws/pull/143 ## Purpose of the change Add millisBehindLatest metric to the Kinesis source implementation. ## Verifying this change This change added tests and can be verified as follows: - *Added unit tests* - *Manually verified by running the Kinesis connector on a local Flink cluster.* ## Significant changes *(Please check any boxes [x] if the answer is "yes". You can first publish the PR and check them afterwards, for convenience.)* - [ ] Dependencies have been added or upgraded - [ ] Public API has been changed (Public API is any class annotated with `@Public(Evolving)`) - [ ] Serializers have been changed - [X] New feature has been introduced - If yes, how is this documented? (not applicable / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-35621) Release Testing Instructions: Verify FLIP-436: Introduce Catalog-related Syntax
[ https://issues.apache.org/jira/browse/FLINK-35621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weijie Guo updated FLINK-35621: --- Description: Follow up the test for https://issues.apache.org/jira/browse/FLINK-34914 (was: Follow up the test for [https://issues.apache.org/jira/browse/FLINK-34914|https://issues.apache.org/jira/projects/FLINK/issues/FLINK-34918]) > Release Testing Instructions: Verify FLIP-436: Introduce Catalog-related > Syntax > --- > > Key: FLINK-35621 > URL: https://issues.apache.org/jira/browse/FLINK-35621 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Common >Reporter: Weijie Guo >Assignee: Yubin Li >Priority: Blocker > Labels: release-testing > Fix For: 1.20.0 > > > Follow up the test for https://issues.apache.org/jira/browse/FLINK-34914 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35621) Release Testing Instructions: Verify FLIP-436: Introduce Catalog-related Syntax
[ https://issues.apache.org/jira/browse/FLINK-35621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weijie Guo updated FLINK-35621: --- Description: Follow up the test for [https://issues.apache.org/jira/browse/FLINK-34914|https://issues.apache.org/jira/projects/FLINK/issues/FLINK-34918] (was: Follow up the test for [https://issues.apache.org/jira/browse/FLINK-35435|https://issues.apache.org/jira/projects/FLINK/issues/FLINK-34918]) > Release Testing Instructions: Verify FLIP-436: Introduce Catalog-related > Syntax > --- > > Key: FLINK-35621 > URL: https://issues.apache.org/jira/browse/FLINK-35621 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Common >Reporter: Weijie Guo >Assignee: Yubin Li >Priority: Blocker > Labels: release-testing > Fix For: 1.20.0 > > > Follow up the test for > [https://issues.apache.org/jira/browse/FLINK-34914|https://issues.apache.org/jira/projects/FLINK/issues/FLINK-34918] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35621) Release Testing Instructions: Verify FLIP-436: Introduce Catalog-related Syntax
[ https://issues.apache.org/jira/browse/FLINK-35621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weijie Guo updated FLINK-35621: --- Description: Follow up the test for [https://issues.apache.org/jira/browse/FLINK-35435|https://issues.apache.org/jira/projects/FLINK/issues/FLINK-34918] (was: Follow up the test for https://issues.apache.org/jira/browse/FLINK-35435) > Release Testing Instructions: Verify FLIP-436: Introduce Catalog-related > Syntax > --- > > Key: FLINK-35621 > URL: https://issues.apache.org/jira/browse/FLINK-35621 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Common >Reporter: Weijie Guo >Assignee: Yubin Li >Priority: Blocker > Labels: release-testing > Fix For: 1.20.0 > > > Follow up the test for > [https://issues.apache.org/jira/browse/FLINK-35435|https://issues.apache.org/jira/projects/FLINK/issues/FLINK-34918] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-35621) Release Testing Instructions: Verify FLIP-436: Introduce Catalog-related Syntax
[ https://issues.apache.org/jira/browse/FLINK-35621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weijie Guo reassigned FLINK-35621: -- Assignee: Yubin Li (was: Ahmed Hamdy) > Release Testing Instructions: Verify FLIP-436: Introduce Catalog-related > Syntax > --- > > Key: FLINK-35621 > URL: https://issues.apache.org/jira/browse/FLINK-35621 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Common >Reporter: Weijie Guo >Assignee: Yubin Li >Priority: Blocker > Labels: release-testing > Fix For: 1.20.0 > > > Follow up the test for https://issues.apache.org/jira/browse/FLINK-35435 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35621) Release Testing Instructions: Verify FLIP-436: Introduce Catalog-related Syntax
Weijie Guo created FLINK-35621: -- Summary: Release Testing Instructions: Verify FLIP-436: Introduce Catalog-related Syntax Key: FLINK-35621 URL: https://issues.apache.org/jira/browse/FLINK-35621 Project: Flink Issue Type: Sub-task Components: Connectors / Common Reporter: Weijie Guo Assignee: Ahmed Hamdy Fix For: 1.20.0 Follow up the test for https://issues.apache.org/jira/browse/FLINK-35435 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-34918) Support `ALTER CATALOG COMMENT` syntax
[ https://issues.apache.org/jira/browse/FLINK-34918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weijie Guo closed FLINK-34918. -- Fix Version/s: 1.20.0 Resolution: Done master(1.20) via 2385cc05e311797d209e85ce9b9e668a7dd3c51c. > Support `ALTER CATALOG COMMENT` syntax > -- > > Key: FLINK-34918 > URL: https://issues.apache.org/jira/browse/FLINK-34918 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Affects Versions: 1.20.0 >Reporter: Yubin Li >Assignee: Yubin Li >Priority: Major > Labels: pull-request-available > Fix For: 1.20.0 > > Attachments: image-2024-06-13-18-01-34-910.png > > > Set comment in the specified catalog. If the comment is already set in the > catalog, override the old value with the new one. > !image-2024-06-13-18-01-34-910.png|width=715,height=523! -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-34918][table] Support `ALTER CATALOG COMMENT` syntax [flink]
reswqa merged PR #24932: URL: https://github.com/apache/flink/pull/24932 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-28915) Make application mode could support remote DFS schema(e.g. S3, OSS, HDFS, etc.)
[ https://issues.apache.org/jira/browse/FLINK-28915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17855124#comment-17855124 ] Robert Metzger commented on FLINK-28915: Thanks a lot for looking into it. Let me try to come up with a minimal reproducer using only vanilla Flink releases. > Make application mode could support remote DFS schema(e.g. S3, OSS, HDFS, > etc.) > --- > > Key: FLINK-28915 > URL: https://issues.apache.org/jira/browse/FLINK-28915 > Project: Flink > Issue Type: Improvement > Components: Deployment / Kubernetes, flink-contrib >Reporter: hjw >Assignee: Ferenc Csaky >Priority: Major > Labels: pull-request-available, stale-assigned > Fix For: 1.19.0 > > > As the Flink document show , local is the only supported scheme in Native k8s > deployment. > Is there have a plan to support s3 filesystem? thx. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-32086][checkpointing] Cleanup useless file-merging managed directory on exit of TM [flink]
Zakelly commented on PR #24933: URL: https://github.com/apache/flink/pull/24933#issuecomment-2168543823 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-32086][checkpointing] Cleanup useless file-merging managed directory on exit of TM [flink]
Zakelly commented on PR #24933: URL: https://github.com/apache/flink/pull/24933#issuecomment-2168527783 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Flink Kubernetes Operator 1.9.0 [flink-web]
mateczagany opened a new pull request, #747: URL: https://github.com/apache/flink-web/pull/747 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34918][table] Support `ALTER CATALOG COMMENT` syntax [flink]
reswqa commented on PR #24932: URL: https://github.com/apache/flink/pull/24932#issuecomment-2168431495 I didn't review every detail since @LadyForest has already carefully reviewed this, but this basically looks good to me. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-35371) Allow the keystore and truststore type to configured for SSL
[
https://issues.apache.org/jira/browse/FLINK-35371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Somogyi closed FLINK-35371.
-
> Allow the keystore and truststore type to configured for SSL
>
>
> Key: FLINK-35371
> URL: https://issues.apache.org/jira/browse/FLINK-35371
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Network
>Affects Versions: 1.19.0
>Reporter: Ammar Master
>Assignee: Ammar Master
>Priority: Minor
> Labels: SSL, pull-request-available
> Fix For: 1.20.0
>
>
> Flink always creates a keystore and trustore using the [default
> type|https://github.com/apache/flink/blob/b87ead743dca161cdae8a1fef761954d206b81fb/flink-runtime/src/main/java/org/apache/flink/runtime/net/SSLUtils.java#L236]
> defined in the JDK, which in most cases is JKS.
> {code}
> KeyStore trustStore = KeyStore.getInstance(KeyStore.getDefaultType());
> {code}
> We should add other configuration options to set the type explicitly to
> support other custom formats, and match the options provided by other
> applications by
> [Spark|https://spark.apache.org/docs/latest/security.html#:~:text=the%20key%20store.-,%24%7Bns%7D.keyStoreType,-JKS]
> and
> [Kafka|https://kafka.apache.org/documentation/#:~:text=per%2Dbroker-,ssl.keystore.type,-The%20file%20format]
> already. The default would continue to be specified by the JDK.
>
> The SSLContext for the REST API can read the configuration option directly,
> and we need to add extra logic to the
> [CustomSSLEngineProvider|https://github.com/apache/flink/blob/master/flink-rpc/flink-rpc-akka/src/main/java/org/apache/flink/runtime/rpc/pekko/CustomSSLEngineProvider.java]
> for Pekko.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (FLINK-35371) Allow the keystore and truststore type to configured for SSL
[
https://issues.apache.org/jira/browse/FLINK-35371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Somogyi resolved FLINK-35371.
---
Fix Version/s: 1.20.0
Resolution: Fixed
0919ff2 on master
> Allow the keystore and truststore type to configured for SSL
>
>
> Key: FLINK-35371
> URL: https://issues.apache.org/jira/browse/FLINK-35371
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Network
>Affects Versions: 1.19.0
>Reporter: Ammar Master
>Assignee: Ammar Master
>Priority: Minor
> Labels: SSL, pull-request-available
> Fix For: 1.20.0
>
>
> Flink always creates a keystore and trustore using the [default
> type|https://github.com/apache/flink/blob/b87ead743dca161cdae8a1fef761954d206b81fb/flink-runtime/src/main/java/org/apache/flink/runtime/net/SSLUtils.java#L236]
> defined in the JDK, which in most cases is JKS.
> {code}
> KeyStore trustStore = KeyStore.getInstance(KeyStore.getDefaultType());
> {code}
> We should add other configuration options to set the type explicitly to
> support other custom formats, and match the options provided by other
> applications by
> [Spark|https://spark.apache.org/docs/latest/security.html#:~:text=the%20key%20store.-,%24%7Bns%7D.keyStoreType,-JKS]
> and
> [Kafka|https://kafka.apache.org/documentation/#:~:text=per%2Dbroker-,ssl.keystore.type,-The%20file%20format]
> already. The default would continue to be specified by the JDK.
>
> The SSLContext for the REST API can read the configuration option directly,
> and we need to add extra logic to the
> [CustomSSLEngineProvider|https://github.com/apache/flink/blob/master/flink-rpc/flink-rpc-akka/src/main/java/org/apache/flink/runtime/rpc/pekko/CustomSSLEngineProvider.java]
> for Pekko.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] Add GPG key for 1.19.1 release [flink-docker]
hlteoh37 merged PR #194: URL: https://github.com/apache/flink-docker/pull/194 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Update Dockerfiles for 1.19.1 release [flink-docker]
hlteoh37 merged PR #195: URL: https://github.com/apache/flink-docker/pull/195 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-28915) Make application mode could support remote DFS schema(e.g. S3, OSS, HDFS, etc.)
[
https://issues.apache.org/jira/browse/FLINK-28915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17855099#comment-17855099
]
Ferenc Csaky edited comment on FLINK-28915 at 6/14/24 4:16 PM:
---
[~rmetzger] are you using the default {{flink}} image as {{spec.image}}? I did
not have enough time to verify it completely, but my hunch is that the default
img do not contain the S3 FS plugin, hence the error. I also guess you are
using a modified operator, cause the current operator is compiled against Flink
1.18 and the {{FlinkDeploymentController}} does not even deploy if the
{{jarURI}} scheme is not local.
After I compiled the operator with Flink 1.19, I managed to deploy a job
successfully via a HTTP served JAR ({{jarURI:
http://host.minikube.internal:/data/TopSpeedWindowing.jar}}), so I am
pretty sure the DFS variant should work as well.
was (Author: ferenc-csaky):
[~rmetzger] are you using the default {{flink}} image as {{spec.image}}? I did
not have enough time to verify it completely, but my hunch is that the default
img do not contain the S3 FS plugin, hence the error. I also guess you are
using a modified operator, cause the current operator is compiled against Flink
1.18 and the {{FlinkDeploymentController}} executes and does not even deploy if
the {{jarURI}} scheme is not local. After I compiled the operator with Flink
1.19, I managed to deploy a job successfully via a HTTP served JAR ({{jarURI:
http://host.minikube.internal:/data/TopSpeedWindowing.jar}}), so I am
pretty sure the DFS variant should work as well.
> Make application mode could support remote DFS schema(e.g. S3, OSS, HDFS,
> etc.)
> ---
>
> Key: FLINK-28915
> URL: https://issues.apache.org/jira/browse/FLINK-28915
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / Kubernetes, flink-contrib
>Reporter: hjw
>Assignee: Ferenc Csaky
>Priority: Major
> Labels: pull-request-available, stale-assigned
> Fix For: 1.19.0
>
>
> As the Flink document show , local is the only supported scheme in Native k8s
> deployment.
> Is there have a plan to support s3 filesystem? thx.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-28915) Make application mode could support remote DFS schema(e.g. S3, OSS, HDFS, etc.)
[
https://issues.apache.org/jira/browse/FLINK-28915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17855099#comment-17855099
]
Ferenc Csaky commented on FLINK-28915:
--
[~rmetzger] are you using the default {{flink}} image as {{spec.image}}? I did
not have enough time to verify it completely, but my hunch is that the default
img do not contain the S3 FS plugin, hence the error. I also guess you are
using a modified operator, cause the current operator is compiled against Flink
1.18 and the {{FlinkDeploymentController}} executes and does not even deploy if
the {{jarURI}} scheme is not local. After I compiled the operator with Flink
1.19, I managed to deploy a job successfully via a HTTP served JAR ({{jarURI:
http://host.minikube.internal:/data/TopSpeedWindowing.jar}}), so I am
pretty sure the DFS variant should work as well.
> Make application mode could support remote DFS schema(e.g. S3, OSS, HDFS,
> etc.)
> ---
>
> Key: FLINK-28915
> URL: https://issues.apache.org/jira/browse/FLINK-28915
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / Kubernetes, flink-contrib
>Reporter: hjw
>Assignee: Ferenc Csaky
>Priority: Major
> Labels: pull-request-available, stale-assigned
> Fix For: 1.19.0
>
>
> As the Flink document show , local is the only supported scheme in Native k8s
> deployment.
> Is there have a plan to support s3 filesystem? thx.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [hotfix] Fix exception when setting 'state.backend.forst.compression.per.level' in yaml [flink]
Zakelly closed pull request #24938: [hotfix] Fix exception when setting 'state.backend.forst.compression.per.level' in yaml URL: https://github.com/apache/flink/pull/24938 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-35620) Parquet writer creates wrong file for nested fields
[
https://issues.apache.org/jira/browse/FLINK-35620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17855098#comment-17855098
]
Vicky Papavasileiou commented on FLINK-35620:
-
[~lzljs3620320] [~jingge] FYI: The previous PR
[https://github.com/apache/flink/pull/24795] did not completely address the
issue of supporting nested arrays
> Parquet writer creates wrong file for nested fields
> ---
>
> Key: FLINK-35620
> URL: https://issues.apache.org/jira/browse/FLINK-35620
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>Affects Versions: 1.19.0
>Reporter: Vicky Papavasileiou
>Priority: Major
>
> After PR [https://github.com/apache/flink/pull/24795] got merged that added
> support for nested arrays, the parquet writer produces wrong parquet files
> that cannot be read. Note, the reader (both flink and iceberg) don't throw an
> exception but return `null` for the nested field.
> The error is in how the field `max_definition_level` is populated for nested
> fields.
> Consider Avro schema:
> {code:java}
> {
> "namespace": "com.test",
> "type": "record",
> "name": "RecordData",
> "fields": [
> {
> "name": "Field1",
> "type": {
> "type": "array",
> "items": {
> "type": "record",
> "name": "NestedField2",
> "fields": [
> { "name": "NestedField3", "type": "double" }
> ]
> }
> }
> }
> ]
> } {code}
>
> Consider the excerpt below of a parquet file produced by Flink for the above
> schema:
> {code:java}
> Column(SegmentStartTime)
> name: NestedField3
> path: Field1.list.element.NestedField3
> max_definition_level: 1
> max_repetition_level: 1
> physical_type: DOUBLE
> logical_type: None
> converted_type (legacy): NONE
> compression: SNAPPY (space_saved: 7%) {code}
>
> The max_definition_level should be 4 but is 1
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-35620) Parquet writer creates wrong file for nested fields
[
https://issues.apache.org/jira/browse/FLINK-35620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vicky Papavasileiou updated FLINK-35620:
Description:
After PR [https://github.com/apache/flink/pull/24795] got merged that added
support for nested arrays, the parquet writer produces wrong parquet files that
cannot be read. Note, the reader (both flink and iceberg) don't throw an
exception but return `null` for the nested field.
The error is in how the field `max_definition_level` is populated for nested
fields.
Consider Avro schema:
{code:java}
{
"namespace": "com.test",
"type": "record",
"name": "RecordData",
"fields": [
{
"name": "Field1",
"type": {
"type": "array",
"items": {
"type": "record",
"name": "NestedField2",
"fields": [
{ "name": "NestedField3", "type": "double" }
]
}
}
}
]
} {code}
Consider the excerpt below of a parquet file produced by Flink for the above
schema:
{code:java}
Column(NestedField3)
name: NestedField3
path: Field1.list.element.NestedField3
max_definition_level: 1
max_repetition_level: 1
physical_type: DOUBLE
logical_type: None
converted_type (legacy): NONE
compression: SNAPPY (space_saved: 7%) {code}
The max_definition_level should be 4 but is 1
was:
After PR [https://github.com/apache/flink/pull/24795] got merged that added
support for nested arrays, the parquet writer produces wrong parquet files that
cannot be read. Note, the reader (both flink and iceberg) don't throw an
exception but return `null` for the nested field.
The error is in how the field `max_definition_level` is populated for nested
fields.
Consider Avro schema:
{code:java}
{
"namespace": "com.test",
"type": "record",
"name": "RecordData",
"fields": [
{
"name": "Field1",
"type": {
"type": "array",
"items": {
"type": "record",
"name": "NestedField2",
"fields": [
{ "name": "NestedField3", "type": "double" }
]
}
}
}
]
} {code}
Consider the excerpt below of a parquet file produced by Flink for the above
schema:
{code:java}
Column(SegmentStartTime)
name: NestedField3
path: Field1.list.element.NestedField3
max_definition_level: 1
max_repetition_level: 1
physical_type: DOUBLE
logical_type: None
converted_type (legacy): NONE
compression: SNAPPY (space_saved: 7%) {code}
The max_definition_level should be 4 but is 1
> Parquet writer creates wrong file for nested fields
> ---
>
> Key: FLINK-35620
> URL: https://issues.apache.org/jira/browse/FLINK-35620
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>Affects Versions: 1.19.0
>Reporter: Vicky Papavasileiou
>Priority: Major
>
> After PR [https://github.com/apache/flink/pull/24795] got merged that added
> support for nested arrays, the parquet writer produces wrong parquet files
> that cannot be read. Note, the reader (both flink and iceberg) don't throw an
> exception but return `null` for the nested field.
> The error is in how the field `max_definition_level` is populated for nested
> fields.
> Consider Avro schema:
> {code:java}
> {
> "namespace": "com.test",
> "type": "record",
> "name": "RecordData",
> "fields": [
> {
> "name": "Field1",
> "type": {
> "type": "array",
> "items": {
> "type": "record",
> "name": "NestedField2",
> "fields": [
> { "name": "NestedField3", "type": "double" }
> ]
> }
> }
> }
> ]
> } {code}
>
> Consider the excerpt below of a parquet file produced by Flink for the above
> schema:
> {code:java}
> Column(NestedField3)
> name: NestedField3
> path: Field1.list.element.NestedField3
> max_definition_level: 1
> max_repetition_level: 1
> physical_type: DOUBLE
> logical_type: None
> converted_type (legacy): NONE
> compression: SNAPPY (space_saved: 7%) {code}
>
> The max_definition_level should be 4 but is 1
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-34918][table] Support `ALTER CATALOG COMMENT` syntax [flink]
liyubin117 commented on PR #24932: URL: https://github.com/apache/flink/pull/24932#issuecomment-2168329905 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35371][security] Add configuration for SSL keystore and truststore type [flink]
gaborgsomogyi merged PR #24919: URL: https://github.com/apache/flink/pull/24919 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-16784) Support KeyedBroadcastProcessFunction state bootstrapping.
[ https://issues.apache.org/jira/browse/FLINK-16784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17855093#comment-17855093 ] Or Keren commented on FLINK-16784: -- Can I take this one? > Support KeyedBroadcastProcessFunction state bootstrapping. > --- > > Key: FLINK-16784 > URL: https://issues.apache.org/jira/browse/FLINK-16784 > Project: Flink > Issue Type: Improvement > Components: API / State Processor >Reporter: Seth Wiesman >Priority: Not a Priority > Labels: auto-deprioritized-major, auto-deprioritized-minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-34918][table] Support `ALTER CATALOG COMMENT` syntax [flink]
liyubin117 commented on PR #24932: URL: https://github.com/apache/flink/pull/24932#issuecomment-2168280767 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-32086][checkpointing] Cleanup useless file-merging managed directory on exit of TM [flink]
zoltar9264 commented on PR #24933: URL: https://github.com/apache/flink/pull/24933#issuecomment-2168264529 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34918][table] Support `ALTER CATALOG COMMENT` syntax [flink]
liyubin117 commented on code in PR #24932:
URL: https://github.com/apache/flink/pull/24932#discussion_r1639976276
##
flink-table/flink-table-common/src/main/java/org/apache/flink/table/catalog/CatalogDescriptor.java:
##
@@ -59,6 +59,10 @@ public Optional getComment() {
return Optional.ofNullable(comment);
}
+public CatalogDescriptor setComment(String comment) {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-34918][table] Support `ALTER CATALOG COMMENT` syntax [flink]
liyubin117 commented on code in PR #24932:
URL: https://github.com/apache/flink/pull/24932#discussion_r1639976003
##
flink-table/flink-table-common/src/main/java/org/apache/flink/table/catalog/CatalogChange.java:
##
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.catalog;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.Configuration;
+
+import java.util.function.Consumer;
+
+/** {@link CatalogChange} represents the modification of the table. */
+@Internal
+public interface CatalogChange {
+
+CatalogDescriptor applyChange(CatalogDescriptor descriptor);
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-35619] Window rank query fails with 'must call validate first' [flink]
dawidwys commented on code in PR #24940:
URL: https://github.com/apache/flink/pull/24940#discussion_r1639946641
##
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/MiscTests.java:
##
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.nodes.exec.stream;
+
+import org.apache.flink.table.api.EnvironmentSettings;
+import org.apache.flink.table.api.TableEnvironment;
+import org.apache.flink.table.planner.factories.TestValuesTableFactory;
+import
org.apache.flink.table.planner.plan.nodes.exec.testutils.RestoreTestBase;
+import org.apache.flink.table.test.program.SinkTestStep;
+import org.apache.flink.table.test.program.SourceTestStep;
+import org.apache.flink.table.test.program.TableTestProgram;
+import org.apache.flink.table.test.program.TableTestProgramRunner;
+import org.apache.flink.table.test.program.TestStep;
+import org.apache.flink.test.junit5.MiniClusterExtension;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.TestInstance;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+/**
+ * Miscellaneous tests that do not fall into {@link RestoreTestBase} category,
but use the {@link
+ * TableTestProgram} infrastructure.
+ */
+@ExtendWith(MiniClusterExtension.class)
+@TestInstance(TestInstance.Lifecycle.PER_CLASS)
+public class MiscTests implements TableTestProgramRunner {
Review Comment:
My idea was to create a place already where we could put tests like this
one. Tests that just need to run the program e2e.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-35619] Window rank query fails with 'must call validate first' [flink]
dawidwys commented on code in PR #24940:
URL: https://github.com/apache/flink/pull/24940#discussion_r1639952320
##
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/MiscTests.java:
##
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.nodes.exec.stream;
+
+import org.apache.flink.table.api.EnvironmentSettings;
+import org.apache.flink.table.api.TableEnvironment;
+import org.apache.flink.table.planner.factories.TestValuesTableFactory;
+import
org.apache.flink.table.planner.plan.nodes.exec.testutils.RestoreTestBase;
+import org.apache.flink.table.test.program.SinkTestStep;
+import org.apache.flink.table.test.program.SourceTestStep;
+import org.apache.flink.table.test.program.TableTestProgram;
+import org.apache.flink.table.test.program.TableTestProgramRunner;
+import org.apache.flink.table.test.program.TestStep;
+import org.apache.flink.test.junit5.MiniClusterExtension;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.TestInstance;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+/**
+ * Miscellaneous tests that do not fall into {@link RestoreTestBase} category,
but use the {@link
+ * TableTestProgram} infrastructure.
+ */
+@ExtendWith(MiniClusterExtension.class)
+@TestInstance(TestInstance.Lifecycle.PER_CLASS)
+public class MiscTests implements TableTestProgramRunner {
Review Comment:
> Otherwise the whole range if test will match this name...
Generally speaking that was my idea. Once there are too many of those we can
split it, but I don't think we get there any time soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-35619] Window rank query fails with 'must call validate first' [flink]
snuyanzin commented on code in PR #24940:
URL: https://github.com/apache/flink/pull/24940#discussion_r1639927000
##
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/MiscTests.java:
##
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.nodes.exec.stream;
+
+import org.apache.flink.table.api.EnvironmentSettings;
+import org.apache.flink.table.api.TableEnvironment;
+import org.apache.flink.table.planner.factories.TestValuesTableFactory;
+import
org.apache.flink.table.planner.plan.nodes.exec.testutils.RestoreTestBase;
+import org.apache.flink.table.test.program.SinkTestStep;
+import org.apache.flink.table.test.program.SourceTestStep;
+import org.apache.flink.table.test.program.TableTestProgram;
+import org.apache.flink.table.test.program.TableTestProgramRunner;
+import org.apache.flink.table.test.program.TestStep;
+import org.apache.flink.test.junit5.MiniClusterExtension;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.TestInstance;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+/**
+ * Miscellaneous tests that do not fall into {@link RestoreTestBase} category,
but use the {@link
+ * TableTestProgram} infrastructure.
+ */
+@ExtendWith(MiniClusterExtension.class)
+@TestInstance(TestInstance.Lifecycle.PER_CLASS)
+public class MiscTests implements TableTestProgramRunner {
Review Comment:
```suggestion
class MiscTests implements TableTestProgramRunner {
```
nit: no need for public modifier in junit5
##
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/MiscTests.java:
##
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.nodes.exec.stream;
+
+import org.apache.flink.table.api.EnvironmentSettings;
+import org.apache.flink.table.api.TableEnvironment;
+import org.apache.flink.table.planner.factories.TestValuesTableFactory;
+import
org.apache.flink.table.planner.plan.nodes.exec.testutils.RestoreTestBase;
+import org.apache.flink.table.test.program.SinkTestStep;
+import org.apache.flink.table.test.program.SourceTestStep;
+import org.apache.flink.table.test.program.TableTestProgram;
+import org.apache.flink.table.test.program.TableTestProgramRunner;
+import org.apache.flink.table.test.program.TestStep;
+import org.apache.flink.test.junit5.MiniClusterExtension;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.TestInstance;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+/**
+ * Miscellaneous tests that do not fall into {@link RestoreTestBase} category,
but use the {@link
+ * TableTestProgram} infrastructure.
+ */
+@ExtendWith(MiniClusterExtension.class)
+@TestInst
Re: [PR] [FLINK-34918][table] Support `ALTER CATALOG COMMENT` syntax [flink]
LadyForest commented on code in PR #24932:
URL: https://github.com/apache/flink/pull/24932#discussion_r1639931090
##
flink-table/flink-table-common/src/main/java/org/apache/flink/table/catalog/CatalogChange.java:
##
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.catalog;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.Configuration;
+
+import java.util.function.Consumer;
+
+/** {@link CatalogChange} represents the modification of the table. */
+@Internal
+public interface CatalogChange {
+
+CatalogDescriptor applyChange(CatalogDescriptor descriptor);
Review Comment:
Lack Javadoc comment
##
flink-table/flink-table-common/src/main/java/org/apache/flink/table/catalog/CatalogDescriptor.java:
##
@@ -59,6 +59,10 @@ public Optional getComment() {
return Optional.ofNullable(comment);
}
+public CatalogDescriptor setComment(String comment) {
Review Comment:
Nit: better add a nonnull annotation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-35619] Window rank query fails with 'must call validate first' [flink]
snuyanzin commented on code in PR #24940:
URL: https://github.com/apache/flink/pull/24940#discussion_r1639927000
##
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/MiscTests.java:
##
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.nodes.exec.stream;
+
+import org.apache.flink.table.api.EnvironmentSettings;
+import org.apache.flink.table.api.TableEnvironment;
+import org.apache.flink.table.planner.factories.TestValuesTableFactory;
+import
org.apache.flink.table.planner.plan.nodes.exec.testutils.RestoreTestBase;
+import org.apache.flink.table.test.program.SinkTestStep;
+import org.apache.flink.table.test.program.SourceTestStep;
+import org.apache.flink.table.test.program.TableTestProgram;
+import org.apache.flink.table.test.program.TableTestProgramRunner;
+import org.apache.flink.table.test.program.TestStep;
+import org.apache.flink.test.junit5.MiniClusterExtension;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.TestInstance;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+/**
+ * Miscellaneous tests that do not fall into {@link RestoreTestBase} category,
but use the {@link
+ * TableTestProgram} infrastructure.
+ */
+@ExtendWith(MiniClusterExtension.class)
+@TestInstance(TestInstance.Lifecycle.PER_CLASS)
+public class MiscTests implements TableTestProgramRunner {
Review Comment:
```suggestion
class MiscTests implements TableTestProgramRunner {
```
no need for public modifier in junit5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-35585] Add documentation for distribution [flink]
twalthr commented on code in PR #24929: URL: https://github.com/apache/flink/pull/24929#discussion_r1639919185 ## docs/content/docs/dev/table/sql/create.md: ## @@ -406,6 +413,14 @@ Flink will assume correctness of the primary key by assuming that the columns nu Partition the created table by the specified columns. A directory is created for each partition if this table is used as a filesystem sink. +### `DISTRIBUTED BY / DISTRIBUTED INTO` Review Comment: You can simplify the text a bit. So that it better fits to a first-level API rather then a second-level API. First-level API is used by less experienced users. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35619] Window rank query fails with 'must call validate first' [flink]
snuyanzin commented on code in PR #24940:
URL: https://github.com/apache/flink/pull/24940#discussion_r1639926653
##
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/MiscTests.java:
##
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.nodes.exec.stream;
+
+import org.apache.flink.table.api.EnvironmentSettings;
+import org.apache.flink.table.api.TableEnvironment;
+import org.apache.flink.table.planner.factories.TestValuesTableFactory;
+import
org.apache.flink.table.planner.plan.nodes.exec.testutils.RestoreTestBase;
+import org.apache.flink.table.test.program.SinkTestStep;
+import org.apache.flink.table.test.program.SourceTestStep;
+import org.apache.flink.table.test.program.TableTestProgram;
+import org.apache.flink.table.test.program.TableTestProgramRunner;
+import org.apache.flink.table.test.program.TestStep;
+import org.apache.flink.test.junit5.MiniClusterExtension;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.TestInstance;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+/**
+ * Miscellaneous tests that do not fall into {@link RestoreTestBase} category,
but use the {@link
+ * TableTestProgram} infrastructure.
+ */
+@ExtendWith(MiniClusterExtension.class)
+@TestInstance(TestInstance.Lifecycle.PER_CLASS)
+public class MiscTests implements TableTestProgramRunner {
+@Override
+public List programs() {
+return Collections.singletonList(
+WindowRankTestPrograms.WINDOW_RANK_HOP_TVF_NAMED_MIN_TOP_1);
+}
+
+@Override
+public EnumSet supportedSetupSteps() {
+return EnumSet.of(
+TestStep.TestKind.CONFIG,
+TestStep.TestKind.SOURCE_WITH_DATA,
+TestStep.TestKind.SINK_WITH_DATA);
+}
+
+@Override
+public EnumSet supportedRunSteps() {
+return EnumSet.of(TestStep.TestKind.SQL);
+}
+
+@AfterEach
+public void clearData() {
+TestValuesTableFactory.clearAllData();
+}
+
+@ParameterizedTest
+@MethodSource("supportedPrograms")
+public void generateTestSetupFiles(TableTestProgram program) throws
Exception {
Review Comment:
```suggestion
void generateTestSetupFiles(TableTestProgram program) throws Exception {
```
no need for public modifier in junit5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-35619] Window rank query fails with 'must call validate first' [flink]
snuyanzin commented on code in PR #24940:
URL: https://github.com/apache/flink/pull/24940#discussion_r1639924796
##
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/MiscTests.java:
##
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.nodes.exec.stream;
+
+import org.apache.flink.table.api.EnvironmentSettings;
+import org.apache.flink.table.api.TableEnvironment;
+import org.apache.flink.table.planner.factories.TestValuesTableFactory;
+import
org.apache.flink.table.planner.plan.nodes.exec.testutils.RestoreTestBase;
+import org.apache.flink.table.test.program.SinkTestStep;
+import org.apache.flink.table.test.program.SourceTestStep;
+import org.apache.flink.table.test.program.TableTestProgram;
+import org.apache.flink.table.test.program.TableTestProgramRunner;
+import org.apache.flink.table.test.program.TestStep;
+import org.apache.flink.test.junit5.MiniClusterExtension;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.TestInstance;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+/**
+ * Miscellaneous tests that do not fall into {@link RestoreTestBase} category,
but use the {@link
+ * TableTestProgram} infrastructure.
+ */
+@ExtendWith(MiniClusterExtension.class)
+@TestInstance(TestInstance.Lifecycle.PER_CLASS)
+public class MiscTests implements TableTestProgramRunner {
Review Comment:
Should we have a less abstract name e.g. `WindowRankMiscTests` or something
like that?
Otherwise the whole range if test will match this name...
##
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/MiscTests.java:
##
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.nodes.exec.stream;
+
+import org.apache.flink.table.api.EnvironmentSettings;
+import org.apache.flink.table.api.TableEnvironment;
+import org.apache.flink.table.planner.factories.TestValuesTableFactory;
+import
org.apache.flink.table.planner.plan.nodes.exec.testutils.RestoreTestBase;
+import org.apache.flink.table.test.program.SinkTestStep;
+import org.apache.flink.table.test.program.SourceTestStep;
+import org.apache.flink.table.test.program.TableTestProgram;
+import org.apache.flink.table.test.program.TableTestProgramRunner;
+import org.apache.flink.table.test.program.TestStep;
+import org.apache.flink.test.junit5.MiniClusterExtension;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.TestInstance;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+/**
+ * Miscellaneous tests that do not fall into {@link RestoreTestBase} category,
but use the {@link
+ * TableTestProgram} infrastructure.
+ */
+@ExtendWith(MiniClusterExtens
Re: [PR] [FLINK-32086][checkpointing] Cleanup useless file-merging managed directory on exit of TM [flink]
zoltar9264 commented on PR #24933: URL: https://github.com/apache/flink/pull/24933#issuecomment-2168176482 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-25537] [JUnit5 Migration] Module: flink-core [flink]
GOODBOY008 commented on PR #24939: URL: https://github.com/apache/flink/pull/24939#issuecomment-2168138750 @1996fanrui @Jiabao-Sun The last pr of core module. PTAL -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-35620) Parquet writer creates wrong file for nested fields
[
https://issues.apache.org/jira/browse/FLINK-35620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vicky Papavasileiou updated FLINK-35620:
Description:
After PR [https://github.com/apache/flink/pull/24795] got merged that added
support for nested arrays, the parquet writer produces wrong parquet files that
cannot be read. Note, the reader (both flink and iceberg) don't throw an
exception but return `null` for the nested field.
The error is in how the field `max_definition_level` is populated for nested
fields.
Consider Avro schema:
{code:java}
{
"namespace": "com.test",
"type": "record",
"name": "RecordData",
"fields": [
{
"name": "Field1",
"type": {
"type": "array",
"items": {
"type": "record",
"name": "NestedField2",
"fields": [
{ "name": "NestedField3", "type": "double" }
]
}
}
}
]
} {code}
Consider the excerpt below of a parquet file produced by Flink for the above
schema:
{code:java}
Column(SegmentStartTime)
name: NestedField3
path: Field1.list.element.NestedField3
max_definition_level: 1
max_repetition_level: 1
physical_type: DOUBLE
logical_type: None
converted_type (legacy): NONE
compression: SNAPPY (space_saved: 7%) {code}
The max_definition_level should be 4 but is 1
was:
After PR [https://github.com/apache/flink/pull/24795] got merged that added
support for nested arrays, the parquet writer produces wrong parquet files that
cannot be read. Note, the reader (both flink and iceberg) don't throw an
exception but return `null` for the nested field.
The error is in how the field `max_definition_level` is populated for nested
fields.
Consider Avro schema:
```
{
"namespace": "com.test",
"type": "record",
"name": "RecordData",
"fields": [
{
"name": "Field1",
"type": {
"type": "array",
"items": {
"type": "record",
"name": "NestedField2",
"fields": [
{ "name": "NestedField3", "type": "double" }
]
}
}
}
]
}
```
Consider the excerpt below of a parquet file produced by Flink for the above
schema:
```
Column(SegmentStartTime)
name: NestedField3
path: Field1.list.element.NestedField3
max_definition_level: 1
max_repetition_level: 1
physical_type: DOUBLE
logical_type: None
converted_type (legacy): NONE
compression: SNAPPY (space_saved: 7%)
```
The max_definition_level should be 4 but is 1
> Parquet writer creates wrong file for nested fields
> ---
>
> Key: FLINK-35620
> URL: https://issues.apache.org/jira/browse/FLINK-35620
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>Affects Versions: 1.19.0
>Reporter: Vicky Papavasileiou
>Priority: Major
>
> After PR [https://github.com/apache/flink/pull/24795] got merged that added
> support for nested arrays, the parquet writer produces wrong parquet files
> that cannot be read. Note, the reader (both flink and iceberg) don't throw an
> exception but return `null` for the nested field.
> The error is in how the field `max_definition_level` is populated for nested
> fields.
> Consider Avro schema:
> {code:java}
> {
> "namespace": "com.test",
> "type": "record",
> "name": "RecordData",
> "fields": [
> {
> "name": "Field1",
> "type": {
> "type": "array",
> "items": {
> "type": "record",
> "name": "NestedField2",
> "fields": [
> { "name": "NestedField3", "type": "double" }
> ]
> }
> }
> }
> ]
> } {code}
>
> Consider the excerpt below of a parquet file produced by Flink for the above
> schema:
> {code:java}
> Column(SegmentStartTime)
> name: NestedField3
> path: Field1.list.element.NestedField3
> max_definition_level: 1
> max_repetition_level: 1
> physical_type: DOUBLE
> logical_type: None
> converted_type (legacy): NONE
> compression: SNAPPY (space_saved: 7%) {code}
>
> The max_definition_level should be 4 but is 1
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-35620) Parquet writer creates wrong file for nested fields
[
https://issues.apache.org/jira/browse/FLINK-35620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vicky Papavasileiou updated FLINK-35620:
Description:
After PR [https://github.com/apache/flink/pull/24795] got merged that added
support for nested arrays, the parquet writer produces wrong parquet files that
cannot be read. Note, the reader (both flink and iceberg) don't throw an
exception but return `null` for the nested field.
The error is in how the field `max_definition_level` is populated for nested
fields.
Consider Avro schema:
```
{
"namespace": "com.test",
"type": "record",
"name": "RecordData",
"fields": [
{
"name": "Field1",
"type": {
"type": "array",
"items": {
"type": "record",
"name": "NestedField2",
"fields": [
{ "name": "NestedField3", "type": "double" }
]
}
}
}
]
}
```
Consider the excerpt below of a parquet file produced by Flink for the above
schema:
```
Column(SegmentStartTime)
name: NestedField3
path: Field1.list.element.NestedField3
max_definition_level: 1
max_repetition_level: 1
physical_type: DOUBLE
logical_type: None
converted_type (legacy): NONE
compression: SNAPPY (space_saved: 7%)
```
The max_definition_level should be 4 but is 1
was:
After PR [https://github.com/apache/flink/pull/24795] got merged that added
support for nested arrays, the parquet writer produces wrong parquet files that
cannot be read. Note, the reader (both flink and iceberg) don't throw an
exception but return `null` for the nested field.
The error is in how the field `max_definition_level` is populated for nested
fields.
Consider Avro schema:
```
{
"namespace": "com.test",
"type": "record",
"name": "RecordData",
"fields": [
{
"name": "Field1",
"type": {
"type": "array",
"items": {
"type": "record",
"name": "NestedField2",
"fields": [
{ "name": "NestedField3", "type": "double" }
]
}
}
}
]
}
```
Consider the excerpt below of a parquet file produced by Flink for the above
schema:
```
Column(SegmentStartTime)
name: NestedField3
path: Field1.list.element.NestedField3
max_definition_level: 1
max_repetition_level: 1
physical_type: DOUBLE
logical_type: None
converted_type (legacy): NONE
compression: SNAPPY (space_saved: 7%)
```
The max_definition_level should be 4 but is 1
> Parquet writer creates wrong file for nested fields
> ---
>
> Key: FLINK-35620
> URL: https://issues.apache.org/jira/browse/FLINK-35620
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>Affects Versions: 1.19.0
>Reporter: Vicky Papavasileiou
>Priority: Major
>
> After PR [https://github.com/apache/flink/pull/24795] got merged that added
> support for nested arrays, the parquet writer produces wrong parquet files
> that cannot be read. Note, the reader (both flink and iceberg) don't throw an
> exception but return `null` for the nested field.
> The error is in how the field `max_definition_level` is populated for nested
> fields.
> Consider Avro schema:
> ```
> {
> "namespace": "com.test",
> "type": "record",
> "name": "RecordData",
> "fields": [
> {
> "name": "Field1",
> "type": {
> "type": "array",
> "items": {
> "type": "record",
> "name": "NestedField2",
> "fields": [
> { "name": "NestedField3", "type": "double" }
> ]
> }
> }
> }
> ]
> }
> ```
> Consider the excerpt below of a parquet file produced by Flink for the above
> schema:
> ```
> Column(SegmentStartTime)
> name: NestedField3
> path: Field1.list.element.NestedField3
> max_definition_level: 1
> max_repetition_level: 1
> physical_type: DOUBLE
> logical_type: None
> converted_type (legacy): NONE
> compression: SNAPPY (space_saved: 7%)
> ```
> The max_definition_level should be 4 but is 1
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-35620) Parquet writer creates wrong file for nested fields
Vicky Papavasileiou created FLINK-35620:
---
Summary: Parquet writer creates wrong file for nested fields
Key: FLINK-35620
URL: https://issues.apache.org/jira/browse/FLINK-35620
Project: Flink
Issue Type: Bug
Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
Affects Versions: 1.19.0
Reporter: Vicky Papavasileiou
After PR [https://github.com/apache/flink/pull/24795] got merged that added
support for nested arrays, the parquet writer produces wrong parquet files that
cannot be read. Note, the reader (both flink and iceberg) don't throw an
exception but return `null` for the nested field.
The error is in how the field `max_definition_level` is populated for nested
fields.
Consider Avro schema:
```
{
"namespace": "com.test",
"type": "record",
"name": "RecordData",
"fields": [
{
"name": "Field1",
"type": {
"type": "array",
"items": {
"type": "record",
"name": "NestedField2",
"fields": [
{ "name": "NestedField3", "type": "double" }
]
}
}
}
]
}
```
Consider the excerpt below of a parquet file produced by Flink for the above
schema:
```
Column(SegmentStartTime)
name: NestedField3
path: Field1.list.element.NestedField3
max_definition_level: 1
max_repetition_level: 1
physical_type: DOUBLE
logical_type: None
converted_type (legacy): NONE
compression: SNAPPY (space_saved: 7%)
```
The max_definition_level should be 4 but is 1
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-34252][table] Fix lastRecordTime tracking in WatermarkAssignerOperator [flink]
pnowojski closed pull request #24211: [FLINK-34252][table] Fix lastRecordTime tracking in WatermarkAssignerOperator URL: https://github.com/apache/flink/pull/24211 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-34252) WatermarkAssignerOperator should not emit WatermarkStatus.IDLE under continuous data flow
[
https://issues.apache.org/jira/browse/FLINK-34252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski reassigned FLINK-34252:
--
Assignee: Piotr Nowojski (was: David Christle)
> WatermarkAssignerOperator should not emit WatermarkStatus.IDLE under
> continuous data flow
> -
>
> Key: FLINK-34252
> URL: https://issues.apache.org/jira/browse/FLINK-34252
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.16.3, 1.17.2, 1.18.1
>Reporter: David Christle
>Assignee: Piotr Nowojski
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.18.2, 1.20.0
>
>
> The WatermarkAssignerOperator in the table runtime incorrectly transitions to
> an IDLE state even when data is continuously flowing. This behavior, observed
> under normal operating conditions where the interval between data elements is
> shorter than the configured idleTimeout, leads to regular transitions between
> ACTIVE and IDLE states, which are unnecessary.
> _Detail:_
> In the current implementation, the lastRecordTime variable, which tracks the
> time of the last received data element, is updated only when the
> WatermarkStatus transitions from IDLE to ACTIVE. However, it is not updated
> when WatermarkStatus is ACTIVE, which means even under continuous data flow,
> the condition `(currentTime - lastRecordTime > idleTimeout)` will eventually
> always become true, and the WatermarkStatus will erroneously be marked IDLE.
> It is unclear to me if this bug produces any incorrectness downstream, since
> when the WatermarkStatus is in in the IDLE state, the next processElement
> will cause a WatermarkStatus.ACTIVE to be emitted. Nevertheless, we should
> eliminate this flip-flop behavior between states.
> The test I wrote fails without the fix and illustrates the flip-flops:
> {noformat}
> [ERROR] Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.030
> s <<< FAILURE! -- in
> org.apache.flink.table.runtime.operators.wmassigners.WatermarkAssignerOperatorTest
> [ERROR]
> org.apache.flink.table.runtime.operators.wmassigners.WatermarkAssignerOperatorTest.testIdleStateAvoidanceWithConsistentDataFlow
> -- Time elapsed: 0.013 s <<< FAILURE!
> java.lang.AssertionError:
> Expecting
> [WatermarkStatus(IDLE),
> WatermarkStatus(ACTIVE),
> WatermarkStatus(IDLE),
> WatermarkStatus(ACTIVE),
> WatermarkStatus(IDLE),
> WatermarkStatus(ACTIVE),
> WatermarkStatus(IDLE),
> WatermarkStatus(ACTIVE),
> WatermarkStatus(IDLE)]
> not to contain
> [WatermarkStatus(IDLE)]
> but found
> [WatermarkStatus(IDLE)]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-34252][table] Fix lastRecordTime tracking in WatermarkAssignerOperator [flink]
pnowojski commented on PR #24211: URL: https://github.com/apache/flink/pull/24211#issuecomment-2168078012 I've prepared an improved version of this PR: https://github.com/apache/flink/pull/24941 Closing this in favour the newer one. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [optimised][FLINK-34252][table] Fix lastRecordTime tracking in WatermarkAssignerOperator [flink]
flinkbot commented on PR #24941: URL: https://github.com/apache/flink/pull/24941#issuecomment-2168025371 ## CI report: * 0d9257b66d6fb3d41f31e01a5cdb35fe92b2e55d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-32086][checkpointing] Cleanup useless file-merging managed directory on exit of TM [flink]
Zakelly commented on code in PR #24933:
URL: https://github.com/apache/flink/pull/24933#discussion_r1639554722
##
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/filemerging/FileMergingSnapshotManagerBase.java:
##
@@ -153,6 +152,22 @@ public abstract class FileMergingSnapshotManagerBase
implements FileMergingSnaps
/** The current space statistic, updated on file creation/deletion. */
protected SpaceStat spaceStat;
+/**
+ * This map records shared state dirs which need be clean up when the
FileMergingSnapshotManager
+ * close. The key is SubtaskKey the shared state dir belong to, and the
value is the count of
+ * the ongoing checkpoint which reference the dir. If a checkpoint which
reference the shared
+ * dir complete, the corresponding shared dir will be removed from this
map, because the
+ * ownership is transferred to JobManager.
+ */
+private final Map sharedDirToCleanRef = new
ConcurrentHashMap<>();
Review Comment:
I'd suggest a new class that bundle `DirectoryStreamStateHandle` and its
reference count.
##
flink-runtime/src/main/java/org/apache/flink/runtime/state/filemerging/DirectoryStreamStateHandle.java:
##
@@ -19,23 +19,31 @@
package org.apache.flink.runtime.state.filemerging;
import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.core.fs.FileSystem;
+import org.apache.flink.core.fs.Path;
import org.apache.flink.runtime.state.DirectoryStateHandle;
import org.apache.flink.runtime.state.PhysicalStateHandleID;
import org.apache.flink.runtime.state.SharedStateRegistryKey;
import org.apache.flink.runtime.state.StreamStateHandle;
import javax.annotation.Nonnull;
-import java.nio.file.Path;
import java.util.Optional;
/** Wrap {@link DirectoryStateHandle} to a {@link StreamStateHandle}. */
-public class DirectoryStreamStateHandle extends DirectoryStateHandle
implements StreamStateHandle {
+public class DirectoryStreamStateHandle implements StreamStateHandle {
private static final long serialVersionUID = 1L;
-public DirectoryStreamStateHandle(@Nonnull Path directory, long
directorySize) {
-super(directory, directorySize);
+/** The path that describes the directory, as a string, to be
serializable. */
+private final String directoryString;
+
+/** Transient path cache, to avoid re-parsing the string. */
+private transient Path directory;
Review Comment:
Seems unnecessary? I mean we could just hold the `Path` with `final`
keyword, and no need for the `String directoryString`?
##
flink-runtime/src/main/java/org/apache/flink/runtime/state/filesystem/FsMergingCheckpointStorageLocation.java:
##
@@ -80,6 +81,12 @@ public FsMergingCheckpointStorageLocation(
reference,
fileStateSizeThreshold,
writeBufferSize);
+
+// Record file-merging managed dir reference when
FsMergingCheckpointStorageLocation create.
+if (fileMergingSnapshotManager instanceof
FileMergingSnapshotManagerBase) {
+((FileMergingSnapshotManagerBase) fileMergingSnapshotManager)
+.recordManagedDirReference(subtaskKey, checkpointId);
Review Comment:
How about making this as an interface of `FileMergingSnapshotManager`? And
how about a new name `retainManagedDirectory`
##
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/filemerging/FileMergingSnapshotManagerBase.java:
##
@@ -496,14 +497,34 @@ protected void discardCheckpoint(long checkpointId)
throws IOException {
// Checkpoint Listener
//
+/**
+ * {@link FsMergingCheckpointStorageLocation} use this method let the file
merging manager know
+ * an ongoing checkpoint may reference the managed dirs.
+ */
+public void notifyCheckpointStart(SubtaskKey subtaskKey, long
checkpointId) {
+managedSharedStateDirHandles
+.getOrDefault(subtaskKey, NON_HANDLE_INSTANCE)
+.increaseRefCountWhenCheckpointStart(checkpointId);
Review Comment:
use `computeIfPresent` ?
##
flink-runtime/src/main/java/org/apache/flink/runtime/state/filemerging/DirectoryStreamStateHandle.java:
##
@@ -19,23 +19,26 @@
package org.apache.flink.runtime.state.filemerging;
import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.core.fs.FileSystem;
+import org.apache.flink.core.fs.Path;
import org.apache.flink.runtime.state.DirectoryStateHandle;
import org.apache.flink.runtime.state.PhysicalStateHandleID;
import org.apache.flink.runtime.state.SharedStateRegistryKey;
import org.apache.flink.runtime.state.StreamStateHandle;
import javax.annotation.Nonnull;
-import java.nio.file.Path;
import java.util.Optional;
/** Wrap {@link DirectoryStateHandle} to a {@
[PR] [optimised][FLINK-34252][table] Fix lastRecordTime tracking in WatermarkAssignerOperator [flink]
pnowojski opened a new pull request, #24941: URL: https://github.com/apache/flink/pull/24941 This is an optimised version of https://github.com/apache/flink/pull/24211 that supersedes it. This newer version avoids having to check current time per each record, as syscalls are quite expensive. ## What is the purpose of the change In the current implementation, the lastRecordTime variable, which tracks the time of the last received data element, is updated only when the WatermarkStatus transitions from IDLE to ACTIVE. However, it is not updated when WatermarkStatus is ACTIVE, which means even under continuous data flow, the condition `(currentTime - lastRecordTime > idleTimeout)` will eventually always become true, and the WatermarkStatus will erroneously be marked IDLE. I believe this bug technically causes incorrect outputs since downstream watermarks advance earlier than they otherwise would. The incorrect state doesn't last forever, though, since when the WatermarkStatus is in in the IDLE state, the next processElement will cause a WatermarkStatus.ACTIVE to be emitted. The new unit test illustrates the flip-flop behavior before the fix: ``` [ERROR] org.apache.flink.table.runtime.operators.wmassigners.WatermarkAssignerOperatorTest.testIdleStateAvoidanceWithConsistentDataFlow -- Time elapsed: 0.013 s <<< FAILURE! java.lang.AssertionError: Expecting [WatermarkStatus(IDLE), WatermarkStatus(ACTIVE), WatermarkStatus(IDLE), WatermarkStatus(ACTIVE), WatermarkStatus(IDLE), WatermarkStatus(ACTIVE), WatermarkStatus(IDLE), WatermarkStatus(ACTIVE), WatermarkStatus(IDLE)] not to contain [WatermarkStatus(IDLE)] but found [WatermarkStatus(IDLE)] ``` ## Brief change log - Update lastRecordTime in table WatermarkAssignerOperator on each record to prevent the stream from incorrectly being marked idle ## Verifying this change This change added tests and can be verified as follows: - Added test that validates the WatermarkStatus is not set to idle when records are sent more frequently than the idleTimeout ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (yes) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-34917) Support `CREATE CATALOG IF NOT EXISTS `with comment
[
https://issues.apache.org/jira/browse/FLINK-34917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jane Chan closed FLINK-34917.
-
> Support `CREATE CATALOG IF NOT EXISTS `with comment
>
>
> Key: FLINK-34917
> URL: https://issues.apache.org/jira/browse/FLINK-34917
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Affects Versions: 1.20.0
>Reporter: Yubin Li
>Assignee: Yubin Li
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.20.0
>
> Attachments: image-2024-03-22-18-31-59-632.png
>
>
> We propose to introduce `getComment()` method in `CatalogDescriptor`, and the
> reasons are as follows.
> 1. For the sake of design consistency, follow the design of FLIP-295 [1]
> which introduced `CatalogStore` component, `CatalogDescriptor` includes names
> and attributes, both of which are used to describe the catalog, and `comment`
> can be added smoothly.
> 2. Extending the existing class rather than add new method to the existing
> interface, Especially, the `Catalog` interface, as a core interface, is used
> by a series of important components such as `CatalogFactory`,
> `CatalogManager` and `FactoryUtil`, and is implemented by a large number of
> connectors such as JDBC, Paimon, and Hive. Adding methods to it will greatly
> increase the implementation complexity, and more importantly, increase the
> cost of iteration, maintenance, and verification.
>
> {{IF NOT EXISTS}} clause: If the catalog already exists, nothing happens.
> {{COMMENT}} clause: An optional string literal. The description for the
> catalog.
> NOTICE: we just need to introduce the '[IF NOT EXISTS]' and '[COMMENT]'
> clause to the 'create catalog' statement.
> !image-2024-03-22-18-31-59-632.png|width=795,height=87!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (FLINK-34917) Support `CREATE CATALOG IF NOT EXISTS `with comment
[
https://issues.apache.org/jira/browse/FLINK-34917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jane Chan resolved FLINK-34917.
---
Fix Version/s: 1.20.0
Resolution: Fixed
Fixed in master 4044b9ea42aa6cf4a638b4ed46219fed94ce84bf
> Support `CREATE CATALOG IF NOT EXISTS `with comment
>
>
> Key: FLINK-34917
> URL: https://issues.apache.org/jira/browse/FLINK-34917
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Affects Versions: 1.20.0
>Reporter: Yubin Li
>Assignee: Yubin Li
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.20.0
>
> Attachments: image-2024-03-22-18-31-59-632.png
>
>
> We propose to introduce `getComment()` method in `CatalogDescriptor`, and the
> reasons are as follows.
> 1. For the sake of design consistency, follow the design of FLIP-295 [1]
> which introduced `CatalogStore` component, `CatalogDescriptor` includes names
> and attributes, both of which are used to describe the catalog, and `comment`
> can be added smoothly.
> 2. Extending the existing class rather than add new method to the existing
> interface, Especially, the `Catalog` interface, as a core interface, is used
> by a series of important components such as `CatalogFactory`,
> `CatalogManager` and `FactoryUtil`, and is implemented by a large number of
> connectors such as JDBC, Paimon, and Hive. Adding methods to it will greatly
> increase the implementation complexity, and more importantly, increase the
> cost of iteration, maintenance, and verification.
>
> {{IF NOT EXISTS}} clause: If the catalog already exists, nothing happens.
> {{COMMENT}} clause: An optional string literal. The description for the
> catalog.
> NOTICE: we just need to introduce the '[IF NOT EXISTS]' and '[COMMENT]'
> clause to the 'create catalog' statement.
> !image-2024-03-22-18-31-59-632.png|width=795,height=87!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-34917][table] Introduce comment for CatalogStore & Support enhanced `CREATE CATALOG` syntax [flink]
LadyForest merged PR #24934: URL: https://github.com/apache/flink/pull/24934 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-34917) Support `CREATE CATALOG IF NOT EXISTS `with comment
[
https://issues.apache.org/jira/browse/FLINK-34917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jane Chan updated FLINK-34917:
--
Summary: Support `CREATE CATALOG IF NOT EXISTS `with comment (was:
Introduce comment for CatalogStore & Support enhanced `CREATE CATALOG` syntax)
> Support `CREATE CATALOG IF NOT EXISTS `with comment
>
>
> Key: FLINK-34917
> URL: https://issues.apache.org/jira/browse/FLINK-34917
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Affects Versions: 1.20.0
>Reporter: Yubin Li
>Assignee: Yubin Li
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2024-03-22-18-31-59-632.png
>
>
> We propose to introduce `getComment()` method in `CatalogDescriptor`, and the
> reasons are as follows.
> 1. For the sake of design consistency, follow the design of FLIP-295 [1]
> which introduced `CatalogStore` component, `CatalogDescriptor` includes names
> and attributes, both of which are used to describe the catalog, and `comment`
> can be added smoothly.
> 2. Extending the existing class rather than add new method to the existing
> interface, Especially, the `Catalog` interface, as a core interface, is used
> by a series of important components such as `CatalogFactory`,
> `CatalogManager` and `FactoryUtil`, and is implemented by a large number of
> connectors such as JDBC, Paimon, and Hive. Adding methods to it will greatly
> increase the implementation complexity, and more importantly, increase the
> cost of iteration, maintenance, and verification.
>
> {{IF NOT EXISTS}} clause: If the catalog already exists, nothing happens.
> {{COMMENT}} clause: An optional string literal. The description for the
> catalog.
> NOTICE: we just need to introduce the '[IF NOT EXISTS]' and '[COMMENT]'
> clause to the 'create catalog' statement.
> !image-2024-03-22-18-31-59-632.png|width=795,height=87!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-35598][sql-parser] Fix error comparison type ExtendedSqlRowTypeNameSpec#equalsDeep [flink]
wzx140 commented on PR #24937: URL: https://github.com/apache/flink/pull/24937#issuecomment-2167901878 @dawidwys Could you please review it? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35619] Window rank query fails with 'must call validate first' [flink]
flinkbot commented on PR #24940: URL: https://github.com/apache/flink/pull/24940#issuecomment-2167896279 ## CI report: * 6bf34deca7f4a8bd7c8ed9a2b988a47f263a83d3 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35598][sql-parser] Fix error comparison type ExtendedSqlRowTypeNameSpec#equalsDeep [flink]
wzx140 commented on PR #24937: URL: https://github.com/apache/flink/pull/24937#issuecomment-216707 > > Could you please review it? Really thanks! > > Sorry, I'm not familiar with sql or table module, I'm unable to review this PR. Thank you for your response! I understand. I'll find someone else who is more familiar with SQL module to review this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [FLINK-35619] Window rank query fails with 'must call validate first' [flink]
dawidwys opened a new pull request, #24940: URL: https://github.com/apache/flink/pull/24940 ## What is the purpose of the change Fix the bug that it is not possible to name arguments in a windowed rank query if the results are inserted into explicit columns of the sink. ## Verifying this change Added a test in `MiscTests` ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-35619) Window rank query fails with "must call validate first"
[
https://issues.apache.org/jira/browse/FLINK-35619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-35619:
---
Labels: pull-request-available (was: )
> Window rank query fails with "must call validate first"
> ---
>
> Key: FLINK-35619
> URL: https://issues.apache.org/jira/browse/FLINK-35619
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Reporter: Dawid Wysakowicz
>Assignee: Dawid Wysakowicz
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.20.0
>
>
> A program:
> {code}
> static final TableTestProgram WINDOW_RANK_HOP_TVF_NAMED_MIN_TOP_1 =
> TableTestProgram.of(
> "window-rank-hop-tvf-named-min-top-n",
> "validates window min top-n follows after hop
> window")
> .setupTableSource(SourceTestStep.newBuilder("bid_t")
> .addSchema(
> "ts STRING",
> "price DECIMAL(10,2)",
> "supplier_id STRING",
> "`bid_time` AS TO_TIMESTAMP(`ts`)",
> "WATERMARK for `bid_time` AS `bid_time` -
> INTERVAL '1' SECOND")
> .producedValues(
> Row.of(
> "2020-04-15 08:00:05",
> new BigDecimal(4.00),
> "supplier1"))
> .build())
> .setupTableSink(
> SinkTestStep.newBuilder("sink_t")
> .addSchema("bid_time TIMESTAMP(3)",
> "supplier_id STRING")
> .consumedValues(
> "+I[2020-04-15T08:00:05,
> supplier1]",
> "+I[2020-04-15T08:00:05,
> supplier1]")
> .build())
> .runSql("INSERT INTO sink_t(bid_time, supplier_id) "
> + "SELECT bid_time, supplier_id\n"
> + " FROM (\n"
> + "SELECT\n"
> + " bid_time,\n"
> + " supplier_id,\n"
> + " ROW_NUMBER() OVER (PARTITION BY
> window_start, window_end ORDER BY price ASC) AS row_num\n"
> + "FROM TABLE(HOP(\n"
> + " DATA => TABLE bid_t,\n"
> + " TIMECOL => DESCRIPTOR(`bid_time`),\n"
> + " SLIDE => INTERVAL '5' SECOND,\n"
> + " SIZE => INTERVAL '10' SECOND))\n"
> + " ) WHERE row_num <= 3")
> .build();
> {code}
> fails with:
> {code}
> java.lang.AssertionError: must call validate first
> at
> org.apache.calcite.sql.validate.IdentifierNamespace.resolve(IdentifierNamespace.java:256)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertIdentifier(SqlToRelConverter.java:2871)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2464)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2378)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2323)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl(SqlToRelConverter.java:730)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertSelect(SqlToRelConverter.java:716)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryRecursive(SqlToRelConverter.java:3880)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryOrInList(SqlToRelConverter.java:1912)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertExists(SqlToRelConverter.java:1895)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.substituteSubQuery(SqlToRelConverter.java:1421)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.replaceSubQueries(SqlToRelConverter.java:1161)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertCollectionTable(SqlToRelConverter.java:2928)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2511)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2378)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2323)
> at
> o
Re: [PR] [FLINK-34545][cdc-pipeline-connector]Add OceanBase pipeline connector to Flink CDC [flink-cdc]
GOODBOY008 commented on code in PR #3360:
URL: https://github.com/apache/flink-cdc/pull/3360#discussion_r1639708043
##
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-oceanbase/src/main/java/org/apache/flink/cdc/connectors/oceanbase/factory/OceanBaseDataSinkFactory.java:
##
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.cdc.connectors.oceanbase.factory;
+
+import org.apache.flink.cdc.common.annotation.Internal;
+import org.apache.flink.cdc.common.configuration.ConfigOption;
+import org.apache.flink.cdc.common.configuration.Configuration;
+import org.apache.flink.cdc.common.factories.DataSinkFactory;
+import org.apache.flink.cdc.common.sink.DataSink;
+import org.apache.flink.cdc.connectors.oceanbase.sink.OceanBaseDataSink;
+import org.apache.flink.cdc.connectors.oceanbase.sink.OceanBaseDataSinkOptions;
+
+import com.oceanbase.connector.flink.OceanBaseConnectorOptions;
+
+import java.time.ZoneId;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+import static
org.apache.flink.cdc.common.pipeline.PipelineOptions.PIPELINE_LOCAL_TIME_ZONE;
+
+/** A {@link DataSinkFactory} to create {@link OceanBaseDataSink}. */
+@Internal
+public class OceanBaseDataSinkFactory implements DataSinkFactory {
+
+@Override
+public DataSink createDataSink(Context context) {
+Configuration config = context.getFactoryConfiguration();
+OceanBaseConnectorOptions connectorOptions =
+new OceanBaseConnectorOptions(buildOceanBaseOptions(config));
+String zoneStr =
context.getFactoryConfiguration().get(PIPELINE_LOCAL_TIME_ZONE);
+ZoneId zoneId =
+PIPELINE_LOCAL_TIME_ZONE.defaultValue().equals(zoneStr)
+? ZoneId.systemDefault()
Review Comment:
add `OptionUtils.printOptions(IDENTIFIER, config.toMap());` for usr to know
the config info.
##
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-oceanbase/src/main/java/org/apache/flink/cdc/connectors/oceanbase/sink/OceanBaseEventSerializationSchema.java:
##
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.cdc.connectors.oceanbase.sink;
+
+import org.apache.flink.cdc.common.data.RecordData;
+import org.apache.flink.cdc.common.event.CreateTableEvent;
+import org.apache.flink.cdc.common.event.DataChangeEvent;
+import org.apache.flink.cdc.common.event.Event;
+import org.apache.flink.cdc.common.event.OperationType;
+import org.apache.flink.cdc.common.event.SchemaChangeEvent;
+import org.apache.flink.cdc.common.event.TableId;
+import org.apache.flink.cdc.common.schema.Column;
+import org.apache.flink.cdc.common.schema.Schema;
+import org.apache.flink.cdc.common.utils.Preconditions;
+import org.apache.flink.cdc.common.utils.SchemaUtils;
+
+import org.apache.flink.shaded.guava31.com.google.common.collect.Lists;
+
+import com.oceanbase.connector.flink.table.DataChangeRecord;
+import com.oceanbase.connector.flink.table.Record;
+import com.oceanbase.connector.flink.table.RecordSerializationSchema;
+import com.oceanbase.connector.flink.table.TableInfo;
+
+import java.time.ZoneId;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+/** A serializer for Event to Record. */
+public class OceanBaseEventSerializationSc
[jira] [Created] (FLINK-35619) Window rank query fails with "must call validate first"
Dawid Wysakowicz created FLINK-35619:
Summary: Window rank query fails with "must call validate first"
Key: FLINK-35619
URL: https://issues.apache.org/jira/browse/FLINK-35619
Project: Flink
Issue Type: Bug
Components: Table SQL / Planner
Reporter: Dawid Wysakowicz
Assignee: Dawid Wysakowicz
Fix For: 1.20.0
A program:
{code}
static final TableTestProgram WINDOW_RANK_HOP_TVF_NAMED_MIN_TOP_1 =
TableTestProgram.of(
"window-rank-hop-tvf-named-min-top-n",
"validates window min top-n follows after hop
window")
.setupTableSource(SourceTestStep.newBuilder("bid_t")
.addSchema(
"ts STRING",
"price DECIMAL(10,2)",
"supplier_id STRING",
"`bid_time` AS TO_TIMESTAMP(`ts`)",
"WATERMARK for `bid_time` AS `bid_time` -
INTERVAL '1' SECOND")
.producedValues(
Row.of(
"2020-04-15 08:00:05",
new BigDecimal(4.00),
"supplier1"))
.build())
.setupTableSink(
SinkTestStep.newBuilder("sink_t")
.addSchema("bid_time TIMESTAMP(3)",
"supplier_id STRING")
.consumedValues(
"+I[2020-04-15T08:00:05,
supplier1]",
"+I[2020-04-15T08:00:05,
supplier1]")
.build())
.runSql("INSERT INTO sink_t(bid_time, supplier_id) "
+ "SELECT bid_time, supplier_id\n"
+ " FROM (\n"
+ "SELECT\n"
+ " bid_time,\n"
+ " supplier_id,\n"
+ " ROW_NUMBER() OVER (PARTITION BY
window_start, window_end ORDER BY price ASC) AS row_num\n"
+ "FROM TABLE(HOP(\n"
+ " DATA => TABLE bid_t,\n"
+ " TIMECOL => DESCRIPTOR(`bid_time`),\n"
+ " SLIDE => INTERVAL '5' SECOND,\n"
+ " SIZE => INTERVAL '10' SECOND))\n"
+ " ) WHERE row_num <= 3")
.build();
{code}
fails with:
{code}
java.lang.AssertionError: must call validate first
at
org.apache.calcite.sql.validate.IdentifierNamespace.resolve(IdentifierNamespace.java:256)
at
org.apache.calcite.sql2rel.SqlToRelConverter.convertIdentifier(SqlToRelConverter.java:2871)
at
org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2464)
at
org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2378)
at
org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2323)
at
org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl(SqlToRelConverter.java:730)
at
org.apache.calcite.sql2rel.SqlToRelConverter.convertSelect(SqlToRelConverter.java:716)
at
org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryRecursive(SqlToRelConverter.java:3880)
at
org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryOrInList(SqlToRelConverter.java:1912)
at
org.apache.calcite.sql2rel.SqlToRelConverter.convertExists(SqlToRelConverter.java:1895)
at
org.apache.calcite.sql2rel.SqlToRelConverter.substituteSubQuery(SqlToRelConverter.java:1421)
at
org.apache.calcite.sql2rel.SqlToRelConverter.replaceSubQueries(SqlToRelConverter.java:1161)
at
org.apache.calcite.sql2rel.SqlToRelConverter.convertCollectionTable(SqlToRelConverter.java:2928)
at
org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2511)
at
org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2378)
at
org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2323)
at
org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl(SqlToRelConverter.java:730)
at
org.apache.calcite.sql2rel.SqlToRelConverter.convertSelect(SqlToRelConverter.java:716)
at
org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryRecursive(SqlToRelConverter.java:3880)
at
org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2490)
at
org.apache.calci
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
morazow commented on PR #24426: URL: https://github.com/apache/flink/pull/24426#issuecomment-2167770793 Thanks @HuangXingBo for the feedback! Should we still support `manylinux1`? It seems already end of life for some years -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
morazow commented on code in PR #24426:
URL: https://github.com/apache/flink/pull/24426#discussion_r1639647084
##
.github/workflows/nightly.yml:
##
@@ -28,69 +28,131 @@ jobs:
name: "Pre-compile Checks"
uses: ./.github/workflows/template.pre-compile-checks.yml
- java8:
-name: "Java 8"
-uses: ./.github/workflows/template.flink-ci.yml
-with:
- workflow-caller-id: java8
- environment: 'PROFILE="-Dinclude_hadoop_aws"'
- jdk_version: 8
-secrets:
- s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
- s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
- s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
- java11:
-name: "Java 11"
-uses: ./.github/workflows/template.flink-ci.yml
-with:
- workflow-caller-id: java11
- environment: 'PROFILE="-Dinclude_hadoop_aws -Djdk11 -Pjava11-target"'
- jdk_version: 11
-secrets:
- s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
- s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
- s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
- java17:
-name: "Java 17"
-uses: ./.github/workflows/template.flink-ci.yml
-with:
- workflow-caller-id: java17
- environment: 'PROFILE="-Dinclude_hadoop_aws -Djdk11 -Djdk17
-Pjava17-target"'
- jdk_version: 17
-secrets:
- s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
- s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
- s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
- java21:
-name: "Java 21"
-uses: ./.github/workflows/template.flink-ci.yml
-with:
- workflow-caller-id: java21
- environment: 'PROFILE="-Dinclude_hadoop_aws -Djdk11 -Djdk17 -Djdk21
-Pjava21-target"'
- jdk_version: 21
-secrets:
- s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
- s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
- s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
- hadoop313:
-name: "Hadoop 3.1.3"
-uses: ./.github/workflows/template.flink-ci.yml
-with:
- workflow-caller-id: hadoop313
- environment: 'PROFILE="-Dflink.hadoop.version=3.2.3
-Phadoop3-tests,hive3"'
- jdk_version: 8
-secrets:
- s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
- s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
- s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
- adaptive-scheduler:
-name: "AdaptiveScheduler"
-uses: ./.github/workflows/template.flink-ci.yml
-with:
- workflow-caller-id: adaptive-scheduler
- environment: 'PROFILE="-Penable-adaptive-scheduler"'
- jdk_version: 8
-secrets:
- s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
- s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
- s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+# java8:
Review Comment:
Yes, I will put it back once we finalize the PR, commented just to run to
create the wheels.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Closed] (FLINK-32084) Migrate current file merging of channel state into the file merging framework
[ https://issues.apache.org/jira/browse/FLINK-32084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanfei Lei closed FLINK-32084. -- > Migrate current file merging of channel state into the file merging framework > - > > Key: FLINK-32084 > URL: https://issues.apache.org/jira/browse/FLINK-32084 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing, Runtime / State Backends >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Yanfei Lei >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-32087) Space amplification statistics of file merging
[ https://issues.apache.org/jira/browse/FLINK-32087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanfei Lei closed FLINK-32087. -- > Space amplification statistics of file merging > -- > > Key: FLINK-32087 > URL: https://issues.apache.org/jira/browse/FLINK-32087 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing, Runtime / State Backends >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Yanfei Lei >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-32082) Documentation of checkpoint file-merging
[ https://issues.apache.org/jira/browse/FLINK-32082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanfei Lei closed FLINK-32082. -- > Documentation of checkpoint file-merging > > > Key: FLINK-32082 > URL: https://issues.apache.org/jira/browse/FLINK-32082 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing, Runtime / State Backends >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Yanfei Lei >Priority: Major > Labels: pull-request-available > Fix For: 1.20.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-32440) Introduce file merging configuration
[ https://issues.apache.org/jira/browse/FLINK-32440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanfei Lei closed FLINK-32440. -- > Introduce file merging configuration > > > Key: FLINK-32440 > URL: https://issues.apache.org/jira/browse/FLINK-32440 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing, Runtime / State Backends >Affects Versions: 1.18.0 >Reporter: Yanfei Lei >Assignee: Yanfei Lei >Priority: Major > Labels: pull-request-available, stale-assigned > > Introduce file merging configuration and config FileMergingSnapshotManager. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-32091) Add necessary metrics for file-merging
[ https://issues.apache.org/jira/browse/FLINK-32091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanfei Lei closed FLINK-32091. -- > Add necessary metrics for file-merging > -- > > Key: FLINK-32091 > URL: https://issues.apache.org/jira/browse/FLINK-32091 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing, Runtime / State Backends >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Yanfei Lei >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-32092) Integrate snapshot file-merging with existing IT cases
[ https://issues.apache.org/jira/browse/FLINK-32092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanfei Lei closed FLINK-32092. -- > Integrate snapshot file-merging with existing IT cases > -- > > Key: FLINK-32092 > URL: https://issues.apache.org/jira/browse/FLINK-32092 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing, Runtime / State Backends >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Yanfei Lei >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-32091) Add necessary metrics for file-merging
[ https://issues.apache.org/jira/browse/FLINK-32091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanfei Lei resolved FLINK-32091. Resolution: Resolved > Add necessary metrics for file-merging > -- > > Key: FLINK-32091 > URL: https://issues.apache.org/jira/browse/FLINK-32091 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing, Runtime / State Backends >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Yanfei Lei >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-32091) Add necessary metrics for file-merging
[ https://issues.apache.org/jira/browse/FLINK-32091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17855029#comment-17855029 ] Yanfei Lei commented on FLINK-32091: Merged into master via 5497868. > Add necessary metrics for file-merging > -- > > Key: FLINK-32091 > URL: https://issues.apache.org/jira/browse/FLINK-32091 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing, Runtime / State Backends >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Yanfei Lei >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-32091][checkpoint] Add file size metrics for file-merging [flink]
fredia merged PR #24922: URL: https://github.com/apache/flink/pull/24922 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-25537] [JUnit5 Migration] Module: flink-core [flink]
flinkbot commented on PR #24939: URL: https://github.com/apache/flink/pull/24939#issuecomment-2167744579 ## CI report: * 299695761f9ed43f73dc5bbca025e2bd29a8477c UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-35618) Flink CDC add MongoDB pipeline data sink connector
Jiabao Sun created FLINK-35618: -- Summary: Flink CDC add MongoDB pipeline data sink connector Key: FLINK-35618 URL: https://issues.apache.org/jira/browse/FLINK-35618 Project: Flink Issue Type: New Feature Components: Flink CDC Affects Versions: cdc-3.2.0 Reporter: Jiabao Sun Assignee: Jiabao Sun Flink CDC add MongoDB pipeline data sink connector -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] [FLINK-25537] [JUnit5 Migration] Module: flink-core [flink]
GOODBOY008 opened a new pull request, #24939: URL: https://github.com/apache/flink/pull/24939 The left in core module. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-35593) Apache Kubernetes Operator Docker image does not contain Apache LICENSE
[
https://issues.apache.org/jira/browse/FLINK-35593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17855022#comment-17855022
]
Robert Metzger commented on FLINK-35593:
Merge for main:
https://github.com/apache/flink-kubernetes-operator/commit/5039e0c4785c4dece473904bf65f18525ca9dd39
merge for 1.8:
https://github.com/apache/flink-kubernetes-operator/commit/569ed381d590d8c2533b39b685d6f90f8a568fcb
> Apache Kubernetes Operator Docker image does not contain Apache LICENSE
> ---
>
> Key: FLINK-35593
> URL: https://issues.apache.org/jira/browse/FLINK-35593
> Project: Flink
> Issue Type: Improvement
> Components: Kubernetes Operator
>Affects Versions: 1.8.0
>Reporter: Anupam Aggarwal
>Assignee: Anupam Aggarwal
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.9.0
>
>
> The Apache
> [LICENSE|https://github.com/apache/flink-kubernetes-operator/blob/main/LICENSE]
> is not bundled along with the Apache Flink Kubernetes Operator docker image.
> {code:java}
> ❯ docker run -it apache/flink-kubernetes-operator:1.8.0 bash
> flink@cc372b31d067:/flink-kubernetes-operator$ ls -latr
> total 104732
> -rw-r--r-- 1 flink flink 40962 Mar 14 15:19
> flink-kubernetes-standalone-1.8.0.jar
> -rw-r--r-- 1 flink flink 107055161 Mar 14 15:21
> flink-kubernetes-operator-1.8.0-shaded.jar
> -rw-r--r-- 1 flink flink 62402 Mar 14 15:21
> flink-kubernetes-webhook-1.8.0-shaded.jar
> -rw-r--r-- 1 flink flink 63740 Mar 14 15:21 NOTICE
> drwxr-xr-x 2 flink flink 4096 Mar 14 15:21 licenses
> drwxr-xr-x 1 root root 4096 Mar 14 15:21 .
> drwxr-xr-x 1 root root 4096 Jun 13 12:49 .. {code}
> The Apache Flink docker image by contrast bundles the license (LICENSE)
> {code:java}
> ❯ docker run -it apache/flink:latest bash
> sed: can't read /config.yaml: No such file or directory
> lflink@24c2dff32a45:~$ ls -latr
> total 224
> -rw-r--r-- 1 flink flink 1309 Mar 4 15:34 README.txt
> drwxrwxr-x 2 flink flink 4096 Mar 4 15:34 log
> -rw-r--r-- 1 flink flink 11357 Mar 4 15:34 LICENSE
> drwxrwxr-x 2 flink flink 4096 Mar 7 05:49 lib
> drwxrwxr-x 6 flink flink 4096 Mar 7 05:49 examples
> drwxrwxr-x 1 flink flink 4096 Mar 7 05:49 conf
> drwxrwxr-x 2 flink flink 4096 Mar 7 05:49 bin
> drwxrwxr-x 10 flink flink 4096 Mar 7 05:49 plugins
> drwxrwxr-x 3 flink flink 4096 Mar 7 05:49 opt
> -rw-rw-r-- 1 flink flink 156327 Mar 7 05:49 NOTICE
> drwxrwxr-x 2 flink flink 4096 Mar 7 05:49 licenses
> drwxr-xr-x 1 root root 4096 Mar 19 05:01 ..
> drwxr-xr-x 1 flink flink 4096 Mar 19 05:02 .
> flink@24c2dff32a45:~$ {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-35617) Support object reuse in async state execution
Zakelly Lan created FLINK-35617:
---
Summary: Support object reuse in async state execution
Key: FLINK-35617
URL: https://issues.apache.org/jira/browse/FLINK-35617
Project: Flink
Issue Type: Sub-task
Components: Runtime / Task
Reporter: Zakelly Lan
Assignee: Zakelly Lan
The record processor of {{AEC}} in async state execution model should consider
object reuse and copy if needed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [hotfix][ForSt] Fix exception when setting 'state.backend.forst.compression.per.level' in yaml [flink]
flinkbot commented on PR #24938: URL: https://github.com/apache/flink/pull/24938#issuecomment-2167713775 ## CI report: * 762958f1dab95a0e3c7dc8433cb04880c4b17e9a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-35616) Support upsert into sharded collections
Jiabao Sun created FLINK-35616:
--
Summary: Support upsert into sharded collections
Key: FLINK-35616
URL: https://issues.apache.org/jira/browse/FLINK-35616
Project: Flink
Issue Type: Improvement
Components: Connectors / MongoDB
Affects Versions: mongodb-1.2.0
Reporter: Jiabao Sun
Assignee: Jiabao Sun
{panel:}
For a db.collection.update() operation that includes upsert: true and is on a
sharded collection, the full sharded key must be included in the filter:
* For an update operation.
* For a replace document operation (starting in MongoDB 4.2).
{panel}
https://www.mongodb.com/docs/manual/reference/method/db.collection.update/#upsert-on-a-sharded-collection
We need to allow users to configure the full sharded key field names to upsert
into the sharded collection.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[PR] [hotfix][ForSt] Fix exception when setting 'state.backend.forst.compression.per.level' in yaml [flink]
Zakelly opened a new pull request, #24938: URL: https://github.com/apache/flink/pull/24938 ## What is the purpose of the change Just like #24902, the bug also exist in forst side, which should also be fixed. ## Brief change log - Fix `ForStStateBackend#mergeConfigurableOptions` - Added test `testConfigureForStCompressionPerLevel` ## Verifying this change Newly added test `testConfigureForStCompressionPerLevel`. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-35593) Apache Kubernetes Operator Docker image does not contain Apache LICENSE
[
https://issues.apache.org/jira/browse/FLINK-35593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger resolved FLINK-35593.
Fix Version/s: 1.9.0
Resolution: Fixed
> Apache Kubernetes Operator Docker image does not contain Apache LICENSE
> ---
>
> Key: FLINK-35593
> URL: https://issues.apache.org/jira/browse/FLINK-35593
> Project: Flink
> Issue Type: Improvement
> Components: Kubernetes Operator
>Affects Versions: 1.8.0
>Reporter: Anupam Aggarwal
>Assignee: Anupam Aggarwal
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.9.0
>
>
> The Apache
> [LICENSE|https://github.com/apache/flink-kubernetes-operator/blob/main/LICENSE]
> is not bundled along with the Apache Flink Kubernetes Operator docker image.
> {code:java}
> ❯ docker run -it apache/flink-kubernetes-operator:1.8.0 bash
> flink@cc372b31d067:/flink-kubernetes-operator$ ls -latr
> total 104732
> -rw-r--r-- 1 flink flink 40962 Mar 14 15:19
> flink-kubernetes-standalone-1.8.0.jar
> -rw-r--r-- 1 flink flink 107055161 Mar 14 15:21
> flink-kubernetes-operator-1.8.0-shaded.jar
> -rw-r--r-- 1 flink flink 62402 Mar 14 15:21
> flink-kubernetes-webhook-1.8.0-shaded.jar
> -rw-r--r-- 1 flink flink 63740 Mar 14 15:21 NOTICE
> drwxr-xr-x 2 flink flink 4096 Mar 14 15:21 licenses
> drwxr-xr-x 1 root root 4096 Mar 14 15:21 .
> drwxr-xr-x 1 root root 4096 Jun 13 12:49 .. {code}
> The Apache Flink docker image by contrast bundles the license (LICENSE)
> {code:java}
> ❯ docker run -it apache/flink:latest bash
> sed: can't read /config.yaml: No such file or directory
> lflink@24c2dff32a45:~$ ls -latr
> total 224
> -rw-r--r-- 1 flink flink 1309 Mar 4 15:34 README.txt
> drwxrwxr-x 2 flink flink 4096 Mar 4 15:34 log
> -rw-r--r-- 1 flink flink 11357 Mar 4 15:34 LICENSE
> drwxrwxr-x 2 flink flink 4096 Mar 7 05:49 lib
> drwxrwxr-x 6 flink flink 4096 Mar 7 05:49 examples
> drwxrwxr-x 1 flink flink 4096 Mar 7 05:49 conf
> drwxrwxr-x 2 flink flink 4096 Mar 7 05:49 bin
> drwxrwxr-x 10 flink flink 4096 Mar 7 05:49 plugins
> drwxrwxr-x 3 flink flink 4096 Mar 7 05:49 opt
> -rw-rw-r-- 1 flink flink 156327 Mar 7 05:49 NOTICE
> drwxrwxr-x 2 flink flink 4096 Mar 7 05:49 licenses
> drwxr-xr-x 1 root root 4096 Mar 19 05:01 ..
> drwxr-xr-x 1 flink flink 4096 Mar 19 05:02 .
> flink@24c2dff32a45:~$ {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-35593][Kubernetes Operator] Add Apache 2 License to docker image [flink-kubernetes-operator]
rmetzger merged PR #839: URL: https://github.com/apache/flink-kubernetes-operator/pull/839 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-35597) Fix unstable LocatableSplitAssignerTest#testConcurrentSplitAssignmentForMultipleHosts
[
https://issues.apache.org/jira/browse/FLINK-35597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17855018#comment-17855018
]
Rui Fan commented on FLINK-35597:
-
Merged to master(1.20.0) via: bc5c4b867f65f87ad8661ac8821c6dc87af41dd9
> Fix unstable
> LocatableSplitAssignerTest#testConcurrentSplitAssignmentForMultipleHosts
> -
>
> Key: FLINK-35597
> URL: https://issues.apache.org/jira/browse/FLINK-35597
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 1.20.0
>Reporter: Yubin Li
>Assignee: Yubin Li
>Priority: Major
> Labels: pull-request-available
>
> To ensure at least one fraction of hosts to be local, it should be
> {{ia.getNumberOfLocalAssignments()}} as before.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (FLINK-35597) Fix unstable LocatableSplitAssignerTest#testConcurrentSplitAssignmentForMultipleHosts
[
https://issues.apache.org/jira/browse/FLINK-35597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Fan resolved FLINK-35597.
-
Fix Version/s: 1.20.0
Resolution: Fixed
> Fix unstable
> LocatableSplitAssignerTest#testConcurrentSplitAssignmentForMultipleHosts
> -
>
> Key: FLINK-35597
> URL: https://issues.apache.org/jira/browse/FLINK-35597
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 1.20.0
>Reporter: Yubin Li
>Assignee: Yubin Li
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.20.0
>
>
> To ensure at least one fraction of hosts to be local, it should be
> {{ia.getNumberOfLocalAssignments()}} as before.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-35597][test] Fix unstable LocatableSplitAssignerTest#testConcurrentSplitAssignmentForMultipleHosts [flink]
1996fanrui merged PR #24936: URL: https://github.com/apache/flink/pull/24936 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-35613) Release Testing Instructions: Verify [FLIP-451] Introduce timeout configuration to AsyncSink
[ https://issues.apache.org/jira/browse/FLINK-35613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Fan updated FLINK-35613: Labels: release-testing (was: ) > Release Testing Instructions: Verify [FLIP-451] Introduce timeout > configuration to AsyncSink > > > Key: FLINK-35613 > URL: https://issues.apache.org/jira/browse/FLINK-35613 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Common >Reporter: Rui Fan >Assignee: Ahmed Hamdy >Priority: Blocker > Labels: release-testing > Fix For: 1.20.0 > > > Follow up the test for https://issues.apache.org/jira/browse/FLINK-35435 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35614) Release Testing Instructions: Verify FLIP-443: Interruptible timers firing
[ https://issues.apache.org/jira/browse/FLINK-35614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Fan updated FLINK-35614: Labels: release-testing (was: ) > Release Testing Instructions: Verify FLIP-443: Interruptible timers firing > > > Key: FLINK-35614 > URL: https://issues.apache.org/jira/browse/FLINK-35614 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Task >Reporter: Rui Fan >Assignee: Piotr Nowojski >Priority: Blocker > Labels: release-testing > Fix For: 1.20.0 > > > Follow up the test for https://issues.apache.org/jira/browse/FLINK-20217 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35612) Release Testing Instructions: Verify FLIP-445: Support dynamic parallelism inference for HiveSource
[ https://issues.apache.org/jira/browse/FLINK-35612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Fan updated FLINK-35612: Labels: release-testing (was: ) > Release Testing Instructions: Verify FLIP-445: Support dynamic parallelism > inference for HiveSource > --- > > Key: FLINK-35612 > URL: https://issues.apache.org/jira/browse/FLINK-35612 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Hive >Reporter: Rui Fan >Assignee: xingbe >Priority: Blocker > Labels: release-testing > Fix For: 1.20.0 > > > Follow up the test for https://issues.apache.org/jira/browse/FLINK-35293 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35609) Release Testing Instructions: Verify FLIP-435: Introduce a New Materialized Table for Simplifying Data Pipelines
[ https://issues.apache.org/jira/browse/FLINK-35609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Fan updated FLINK-35609: Labels: release-testing (was: ) > Release Testing Instructions: Verify FLIP-435: Introduce a New Materialized > Table for Simplifying Data Pipelines > > > Key: FLINK-35609 > URL: https://issues.apache.org/jira/browse/FLINK-35609 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Reporter: Rui Fan >Assignee: dalongliu >Priority: Blocker > Labels: release-testing > Fix For: 1.20.0 > > > Follow up the test for https://issues.apache.org/jira/browse/FLINK-35187 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35608) Release Testing Instructions: Verify FLIP-376: Add DISTRIBUTED BY clause
[ https://issues.apache.org/jira/browse/FLINK-35608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Fan updated FLINK-35608: Labels: release-testing (was: ) > Release Testing Instructions: Verify FLIP-376: Add DISTRIBUTED BY clause > > > Key: FLINK-35608 > URL: https://issues.apache.org/jira/browse/FLINK-35608 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Reporter: Rui Fan >Assignee: Timo Walther >Priority: Blocker > Labels: release-testing > Fix For: 1.20.0 > > > Follow up the test for https://issues.apache.org/jira/browse/FLINK-33494 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35607) Release Testing Instructions: Verify FLIP-441: Show the JobType and remove Execution Mode on Flink WebUI
[ https://issues.apache.org/jira/browse/FLINK-35607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Fan updated FLINK-35607: Labels: release-testing (was: ) > Release Testing Instructions: Verify FLIP-441: Show the JobType and remove > Execution Mode on Flink WebUI > -- > > Key: FLINK-35607 > URL: https://issues.apache.org/jira/browse/FLINK-35607 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Web Frontend >Reporter: Rui Fan >Assignee: Rui Fan >Priority: Blocker > Labels: release-testing > Fix For: 1.20.0 > > > Follow up the test for https://issues.apache.org/jira/browse/FLINK-29481 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35611) Release Testing Instructions: Verify [FLIP-453] Promote Unified Sink API V2 to Public and Deprecate SinkFunction
[ https://issues.apache.org/jira/browse/FLINK-35611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Fan updated FLINK-35611: Labels: release-testing (was: ) > Release Testing Instructions: Verify [FLIP-453] Promote Unified Sink API V2 > to Public and Deprecate SinkFunction > > > Key: FLINK-35611 > URL: https://issues.apache.org/jira/browse/FLINK-35611 > Project: Flink > Issue Type: Sub-task > Components: API / Core >Reporter: Rui Fan >Assignee: Martijn Visser >Priority: Blocker > Labels: release-testing > Fix For: 1.20.0 > > > Follow up the test for https://issues.apache.org/jira/browse/FLINK-35378 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35610) Release Testing Instructions: Verify FLIP-448: Introduce Pluggable Workflow Scheduler Interface for Materialized Table
[ https://issues.apache.org/jira/browse/FLINK-35610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Fan updated FLINK-35610: Labels: release-testing (was: ) > Release Testing Instructions: Verify FLIP-448: Introduce Pluggable Workflow > Scheduler Interface for Materialized Table > -- > > Key: FLINK-35610 > URL: https://issues.apache.org/jira/browse/FLINK-35610 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Reporter: Rui Fan >Assignee: dalongliu >Priority: Blocker > Labels: release-testing > Fix For: 1.20.0 > > > Follow up the test for https://issues.apache.org/jira/browse/FLINK-35345 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35605) Release Testing Instructions: Verify FLIP-306 Unified File Merging Mechanism for Checkpoints
[ https://issues.apache.org/jira/browse/FLINK-35605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Fan updated FLINK-35605: Labels: release-testing (was: ) > Release Testing Instructions: Verify FLIP-306 Unified File Merging Mechanism > for Checkpoints > > > Key: FLINK-35605 > URL: https://issues.apache.org/jira/browse/FLINK-35605 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing >Reporter: Rui Fan >Assignee: Zakelly Lan >Priority: Blocker > Labels: release-testing > Fix For: 1.20.0 > > > Follow up the test for https://issues.apache.org/jira/browse/FLINK-32070 -- This message was sent by Atlassian Jira (v8.20.10#820010)
