[GitHub] [flink] flinkbot edited a comment on pull request #17503: [FLINK-24334][Deployment/Kubernetes] Configuration 'kubernetes.flink.log.dir' compatible
flinkbot edited a comment on pull request #17503: URL: https://github.com/apache/flink/pull/17503#issuecomment-945138587 ## CI report: * 491311a65ed7959663e9cf0db6890e8026e10bcc Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29608) * 6729dba080503a3f34f6441355d5c25091f8b375 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29618) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-24668) Pulsar consumption data repeated output
[ https://issues.apache.org/jira/browse/FLINK-24668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser closed FLINK-24668. -- Resolution: Cannot Reproduce [~余嘉悦] Since we don't have the necessary information to reproduce, I've closed this ticket. Please re-open if you still are experiencing the issues with information to reproduce. > Pulsar consumption data repeated output > --- > > Key: FLINK-24668 > URL: https://issues.apache.org/jira/browse/FLINK-24668 > Project: Flink > Issue Type: Bug > Components: Connectors / Pulsar >Affects Versions: 1.14.0 >Reporter: jiayue.yu >Assignee: Yufan Sheng >Priority: Major > > Flink-connector-pulsar uses the RecordsBySplits of Flink-connector-base to > process data, But recordsInCurrentSplit and splitsIterator are not empty when > nextSplit splitSiterator.hasNext () = false. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-20733) Introduce the upsert-pulsar Connector
[ https://issues.apache.org/jira/browse/FLINK-20733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser updated FLINK-20733: --- Component/s: Connectors / Pulsar (was: Connectors / Common) > Introduce the upsert-pulsar Connector > - > > Key: FLINK-20733 > URL: https://issues.apache.org/jira/browse/FLINK-20733 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Pulsar >Affects Versions: 1.13.0 >Reporter: Jianyun Zhao >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-24246) Bump Pulsar to 2.9.0
[ https://issues.apache.org/jira/browse/FLINK-24246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17477634#comment-17477634 ] Martijn Visser commented on FLINK-24246: [~affe] [~syhily] Will you pick up this ticket before the Flink 1.15 release branch is cut (on February 6th)? > Bump Pulsar to 2.9.0 > > > Key: FLINK-24246 > URL: https://issues.apache.org/jira/browse/FLINK-24246 > Project: Flink > Issue Type: Improvement > Components: Connectors / Pulsar >Affects Versions: 1.14.0, 1.15.0 >Reporter: Yufan Sheng >Assignee: Yufan Sheng >Priority: Major > Labels: pull-request-available, stale-assigned > Fix For: 1.15.0 > > > Pulsar 2.8.1 has been released, the hack for getting TxnId from Pulsar > Transaction could be removed after bump flink-connector-pulsar's > pulsar-client-all to 2.8.1. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-25686) Support Pulsar Schema evolution in Pulsar Connector
[ https://issues.apache.org/jira/browse/FLINK-25686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser updated FLINK-25686: --- Issue Type: Technical Debt (was: New Feature) > Support Pulsar Schema evolution in Pulsar Connector > --- > > Key: FLINK-25686 > URL: https://issues.apache.org/jira/browse/FLINK-25686 > Project: Flink > Issue Type: Technical Debt > Components: Connectors / Pulsar >Reporter: Yufei Zhang >Assignee: Yufei Zhang >Priority: Minor > Labels: Pulsar > > Currently the source connector reads all messages as Message, and > implements a deserialization abstraction to deserialize the byte array > message. This deserialization abstraction supports both flink schema and > pulsar schema. However because of this extra layer of abstraction, even when > using pulsar schema, the schema info is not provided to the consumer > (client), so it is not using any pulsar’s schema validation mechanism. (If > pulsar client has a schema, it will send the schema to broker on connect and > broker will validate if it is a valid schema according to a preset > compatibility rule :[ > https://pulsar.apache.org/docs/en/schema-evolution-compatibility/|https://pulsar.apache.org/docs/en/schema-evolution-compatibility/]). > we need to support this schema evolution -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] (FLINK-25686) Support Pulsar Schema evolution in Pulsar Connector
[ https://issues.apache.org/jira/browse/FLINK-25686 ] Martijn Visser deleted comment on FLINK-25686: was (Author: martijnvisser): [~affe] [~syhily] Will you pick up this ticket before the Flink 1.15 release branch is cut (on February 6th)? > Support Pulsar Schema evolution in Pulsar Connector > --- > > Key: FLINK-25686 > URL: https://issues.apache.org/jira/browse/FLINK-25686 > Project: Flink > Issue Type: Technical Debt > Components: Connectors / Pulsar >Reporter: Yufei Zhang >Assignee: Yufei Zhang >Priority: Minor > Labels: Pulsar > > Currently the source connector reads all messages as Message, and > implements a deserialization abstraction to deserialize the byte array > message. This deserialization abstraction supports both flink schema and > pulsar schema. However because of this extra layer of abstraction, even when > using pulsar schema, the schema info is not provided to the consumer > (client), so it is not using any pulsar’s schema validation mechanism. (If > pulsar client has a schema, it will send the schema to broker on connect and > broker will validate if it is a valid schema according to a preset > compatibility rule :[ > https://pulsar.apache.org/docs/en/schema-evolution-compatibility/|https://pulsar.apache.org/docs/en/schema-evolution-compatibility/]). > we need to support this schema evolution -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-25686) Support Pulsar Schema evolution in Pulsar Connector
[ https://issues.apache.org/jira/browse/FLINK-25686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17477633#comment-17477633 ] Martijn Visser commented on FLINK-25686: [~affe] [~syhily] Will you pick up this ticket before the Flink 1.15 release branch is cut (on February 6th)? > Support Pulsar Schema evolution in Pulsar Connector > --- > > Key: FLINK-25686 > URL: https://issues.apache.org/jira/browse/FLINK-25686 > Project: Flink > Issue Type: New Feature > Components: Connectors / Pulsar >Reporter: Yufei Zhang >Assignee: Yufei Zhang >Priority: Minor > Labels: Pulsar > > Currently the source connector reads all messages as Message, and > implements a deserialization abstraction to deserialize the byte array > message. This deserialization abstraction supports both flink schema and > pulsar schema. However because of this extra layer of abstraction, even when > using pulsar schema, the schema info is not provided to the consumer > (client), so it is not using any pulsar’s schema validation mechanism. (If > pulsar client has a schema, it will send the schema to broker on connect and > broker will validate if it is a valid schema according to a preset > compatibility rule :[ > https://pulsar.apache.org/docs/en/schema-evolution-compatibility/|https://pulsar.apache.org/docs/en/schema-evolution-compatibility/]). > we need to support this schema evolution -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18349: [FLINK-25609][table] Anonymous/inline tables don't require ObjectIdentifier anymore
flinkbot edited a comment on pull request #18349: URL: https://github.com/apache/flink/pull/18349#issuecomment-1011991081 ## CI report: * 0aa7220300adc4f5f9ef0aa59608f299b3b9ac29 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29584) * 8dd59b65de6e71d57da22d35686b0758ff3aed24 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29620) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-25686) Support Pulsar Schema evolution in Pulsar Connector
[ https://issues.apache.org/jira/browse/FLINK-25686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabian Paul reassigned FLINK-25686: --- Assignee: Yufei Zhang > Support Pulsar Schema evolution in Pulsar Connector > --- > > Key: FLINK-25686 > URL: https://issues.apache.org/jira/browse/FLINK-25686 > Project: Flink > Issue Type: New Feature > Components: Connectors / Pulsar >Reporter: Yufei Zhang >Assignee: Yufei Zhang >Priority: Minor > Labels: Pulsar > > Currently the source connector reads all messages as Message, and > implements a deserialization abstraction to deserialize the byte array > message. This deserialization abstraction supports both flink schema and > pulsar schema. However because of this extra layer of abstraction, even when > using pulsar schema, the schema info is not provided to the consumer > (client), so it is not using any pulsar’s schema validation mechanism. (If > pulsar client has a schema, it will send the schema to broker on connect and > broker will validate if it is a valid schema according to a preset > compatibility rule :[ > https://pulsar.apache.org/docs/en/schema-evolution-compatibility/|https://pulsar.apache.org/docs/en/schema-evolution-compatibility/]). > we need to support this schema evolution -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18350: [FLINK-25636][network] Change some default config values of blocking shuffle for better usability
flinkbot edited a comment on pull request #18350: URL: https://github.com/apache/flink/pull/18350#issuecomment-1012086932 ## CI report: * 9012a5147a0a2414ff9d6d9b8f447601e3dd98c3 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29523) * 6c413ec28bb033a2dee3c06a60279971da1f324a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29619) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18349: [FLINK-25609][table] Anonymous/inline tables don't require ObjectIdentifier anymore
flinkbot edited a comment on pull request #18349: URL: https://github.com/apache/flink/pull/18349#issuecomment-1011991081 ## CI report: * 0aa7220300adc4f5f9ef0aa59608f299b3b9ac29 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29584) * 8dd59b65de6e71d57da22d35686b0758ff3aed24 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-table-store] tsreaper commented on a change in pull request #11: [FLINK-25687] DropDelete is incorrect in CompactManager when outputLevel is zero
tsreaper commented on a change in pull request #11:
URL: https://github.com/apache/flink-table-store/pull/11#discussion_r786480063
##
File path:
flink-table-store-core/src/main/java/org/apache/flink/table/store/file/mergetree/compact/CompactManager.java
##
@@ -75,7 +75,8 @@ public void submitCompaction(Levels levels) {
return;
}
boolean dropDelete =
-unit.outputLevel() >=
levels.nonEmptyHighestLevel();
+unit.outputLevel() != 0
+&& unit.outputLevel() >=
levels.nonEmptyHighestLevel();
Review comment:
add comments to explain the logic (for example level 0 is special
because key ranges from different files may overlap, and that compaction are
based on runs instead of files)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18350: [FLINK-25636][network] Change some default config values of blocking shuffle for better usability
flinkbot edited a comment on pull request #18350: URL: https://github.com/apache/flink/pull/18350#issuecomment-1012086932 ## CI report: * 9012a5147a0a2414ff9d6d9b8f447601e3dd98c3 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29523) * 6c413ec28bb033a2dee3c06a60279971da1f324a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17503: [FLINK-24334][Deployment/Kubernetes] Configuration 'kubernetes.flink.log.dir' compatible
flinkbot edited a comment on pull request #17503: URL: https://github.com/apache/flink/pull/17503#issuecomment-945138587 ## CI report: * 2bad195fedf4b3acfa4067a029c63a7e60c3d65a Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29604) * 491311a65ed7959663e9cf0db6890e8026e10bcc Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29608) * 6729dba080503a3f34f6441355d5c25091f8b375 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29618) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-25687) DropDelete is incorrect in CompactManager when outputLevel is zero
[ https://issues.apache.org/jira/browse/FLINK-25687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-25687: --- Labels: pull-request-available (was: ) > DropDelete is incorrect in CompactManager when outputLevel is zero > -- > > Key: FLINK-25687 > URL: https://issues.apache.org/jira/browse/FLINK-25687 > Project: Flink > Issue Type: Bug > Components: Table Store >Reporter: Jingsong Lee >Assignee: Jingsong Lee >Priority: Major > Labels: pull-request-available > Fix For: table-store-0.1.0 > > > When output level is zero, there may be have other files in level 0, we can > not drop delete. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink-table-store] JingsongLi opened a new pull request #11: [FLINK-25687] DropDelete is incorrect in CompactManager when outputLevel is zero
JingsongLi opened a new pull request #11: URL: https://github.com/apache/flink-table-store/pull/11 When output level is zero, there may be have other files in level 0, we can not drop delete. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17503: [FLINK-24334][Deployment/Kubernetes] Configuration 'kubernetes.flink.log.dir' compatible
flinkbot edited a comment on pull request #17503: URL: https://github.com/apache/flink/pull/17503#issuecomment-945138587 ## CI report: * 2bad195fedf4b3acfa4067a029c63a7e60c3d65a Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29604) * 491311a65ed7959663e9cf0db6890e8026e10bcc Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29608) * 6729dba080503a3f34f6441355d5c25091f8b375 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17503: [FLINK-24334][Deployment/Kubernetes] Configuration 'kubernetes.flink.log.dir' compatible
flinkbot edited a comment on pull request #17503: URL: https://github.com/apache/flink/pull/17503#issuecomment-945138587 ## CI report: * 2bad195fedf4b3acfa4067a029c63a7e60c3d65a Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29604) * 491311a65ed7959663e9cf0db6890e8026e10bcc Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29608) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-25687) DropDelete is incorrect in CompactManager when outputLevel is zero
Jingsong Lee created FLINK-25687: Summary: DropDelete is incorrect in CompactManager when outputLevel is zero Key: FLINK-25687 URL: https://issues.apache.org/jira/browse/FLINK-25687 Project: Flink Issue Type: Bug Components: Table Store Reporter: Jingsong Lee Assignee: Jingsong Lee Fix For: table-store-0.1.0 When output level is zero, there may be have other files in level 0, we can not drop delete. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18386: [FLINK-25684][table] Support enhanced `show databases` syntax
flinkbot edited a comment on pull request #18386: URL: https://github.com/apache/flink/pull/18386#issuecomment-1015100174 ## CI report: * 5655ebda80a059eae6b0bf62200b444057c026bf Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29611) * e0a1238340c360eaf676afa6e8a7fd0f9a9ede84 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29617) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18386: [FLINK-25684][table] Support enhanced `show databases` syntax
flinkbot edited a comment on pull request #18386: URL: https://github.com/apache/flink/pull/18386#issuecomment-1015100174 ## CI report: * 5655ebda80a059eae6b0bf62200b444057c026bf Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29611) * e0a1238340c360eaf676afa6e8a7fd0f9a9ede84 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18386: [FLINK-25684][table] Support enhanced `show databases` syntax
flinkbot edited a comment on pull request #18386: URL: https://github.com/apache/flink/pull/18386#issuecomment-1015100174 ## CI report: * 5655ebda80a059eae6b0bf62200b444057c026bf Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29611) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17098: [FLINK-15648][flink-kubernetes] Support to configure limit for CPU & memory
flinkbot edited a comment on pull request #17098: URL: https://github.com/apache/flink/pull/17098#issuecomment-910306059 ## CI report: * d3683df7f1e89088d8d6c255c92f6be38ff65fb3 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23618) * 70fca6164f1fa6d430fe74168f70dcc754ea70c9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29615) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-24899) Enable data compression for blocking shuffle by default
[ https://issues.apache.org/jira/browse/FLINK-24899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingjie Cao closed FLINK-24899. --- Resolution: Fixed > Enable data compression for blocking shuffle by default > --- > > Key: FLINK-24899 > URL: https://issues.apache.org/jira/browse/FLINK-24899 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Network >Reporter: Yingjie Cao >Assignee: Nicholas Jiang >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > > Currently, shuffle data compression is not enabled by default. Shuffle data > compression is important for blocking data shuffle and enabling shuffle data > compression by default can improve the usability. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-24899) Enable data compression for blocking shuffle by default
[ https://issues.apache.org/jira/browse/FLINK-24899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17477616#comment-17477616 ] Yingjie Cao commented on FLINK-24899: - Merged into master via a14c55482f60b73d3894a86a81c2ac6725acbb00. > Enable data compression for blocking shuffle by default > --- > > Key: FLINK-24899 > URL: https://issues.apache.org/jira/browse/FLINK-24899 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Network >Reporter: Yingjie Cao >Assignee: Nicholas Jiang >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > > Currently, shuffle data compression is not enabled by default. Shuffle data > compression is important for blocking data shuffle and enabling shuffle data > compression by default can improve the usability. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] wsry closed pull request #17814: [FLINK-24899][runtime] Enable data compression for blocking shuffle by default
wsry closed pull request #17814: URL: https://github.com/apache/flink/pull/17814 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18386: [FLINK-25684][table] Support enhanced `show databases` syntax
flinkbot edited a comment on pull request #18386: URL: https://github.com/apache/flink/pull/18386#issuecomment-1015100174 ## CI report: * 5655ebda80a059eae6b0bf62200b444057c026bf Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29611) * e0a1238340c360eaf676afa6e8a7fd0f9a9ede84 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17814: [FLINK-24899][runtime] Enable data compression for blocking shuffle by default
flinkbot edited a comment on pull request #17814: URL: https://github.com/apache/flink/pull/17814#issuecomment-971129699 ## CI report: * 96cd76d4486a8096148b9c87d7f4cee71f54c17b Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29602) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17098: [FLINK-15648][flink-kubernetes] Support to configure limit for CPU & memory
flinkbot edited a comment on pull request #17098: URL: https://github.com/apache/flink/pull/17098#issuecomment-910306059 ## CI report: * d3683df7f1e89088d8d6c255c92f6be38ff65fb3 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23618) * 70fca6164f1fa6d430fe74168f70dcc754ea70c9 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18386: [FLINK-25684][table] Support enhanced `show databases` syntax
flinkbot edited a comment on pull request #18386: URL: https://github.com/apache/flink/pull/18386#issuecomment-1015100174 ## CI report: * 5655ebda80a059eae6b0bf62200b444057c026bf Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29611) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18303: [FLINK-25085][runtime] Add a scheduled thread pool for periodic tasks in RpcEndpoint
flinkbot edited a comment on pull request #18303: URL: https://github.com/apache/flink/pull/18303#issuecomment-1008005239 ## CI report: * 0e27f71b22e902f655820375325456a28ce378ec Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29601) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #18102: [FLINK-25033][runtime] Let some scheduler components updatable
zhuzhurk commented on a change in pull request #18102:
URL: https://github.com/apache/flink/pull/18102#discussion_r786442228
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/adapter/DefaultExecutionTopology.java
##
@@ -159,50 +162,79 @@ public static DefaultExecutionTopology fromExecutionGraph(
topologicallySortedJobVertices)
.getAllPipelinedRegions();
-ExecutionGraphIndex executionGraphIndex =
-computeExecutionGraphIndex(
-executionGraph.getAllExecutionVertices(),
-logicalPipelinedRegions,
-edgeManager);
-
-IndexedPipelinedRegions indexedPipelinedRegions =
-computePipelinedRegions(
-
executionGraphIndex.sortedExecutionVerticesInPipelinedRegion.keySet(),
-
executionGraphIndex.sortedExecutionVerticesInPipelinedRegion::get,
-executionGraphIndex.executionVerticesById::get,
-executionGraphIndex.resultPartitionsById::get);
-
-ensureCoLocatedVerticesInSameRegion(
-indexedPipelinedRegions.pipelinedRegions, executionGraph);
-
-return new DefaultExecutionTopology(
-executionGraphIndex.executionVerticesById,
-executionGraphIndex.executionVerticesList,
-executionGraphIndex.resultPartitionsById,
-indexedPipelinedRegions.pipelinedRegionsByVertex,
-indexedPipelinedRegions.pipelinedRegions,
-edgeManager);
-}
-
-private static ExecutionGraphIndex computeExecutionGraphIndex(
-Iterable executionVertices,
-Iterable logicalPipelinedRegions,
-EdgeManager edgeManager) {
-Map executionVerticesById =
new HashMap<>();
-List executionVerticesList = new ArrayList<>();
-Map
resultPartitionsById =
-new HashMap<>();
-Map>
-sortedExecutionVerticesInPipelinedRegion = new
IdentityHashMap<>();
-
-Map
logicalPipelinedRegionByJobVertexId =
+Map
logicalPipelinedRegionsByJobVertexId =
new HashMap<>();
for (DefaultLogicalPipelinedRegion logicalPipelinedRegion :
logicalPipelinedRegions) {
for (LogicalVertex vertex : logicalPipelinedRegion.getVertices()) {
-logicalPipelinedRegionByJobVertexId.put(vertex.getId(),
logicalPipelinedRegion);
+logicalPipelinedRegionsByJobVertexId.put(vertex.getId(),
logicalPipelinedRegion);
}
}
+return logicalPipelinedRegionsByJobVertexId;
+}
+
+public void notifyExecutionGraphUpdated(
+final DefaultExecutionGraph executionGraph,
+final List newJobVertices) {
+
+checkNotNull(executionGraph, "execution graph can not be null");
+
+final Set newVertexSet =
Review comment:
NIT: newVertexSet -> newJobVertexIds
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/adapter/DefaultExecutionTopology.java
##
@@ -159,50 +162,79 @@ public static DefaultExecutionTopology fromExecutionGraph(
topologicallySortedJobVertices)
.getAllPipelinedRegions();
-ExecutionGraphIndex executionGraphIndex =
-computeExecutionGraphIndex(
-executionGraph.getAllExecutionVertices(),
-logicalPipelinedRegions,
-edgeManager);
-
-IndexedPipelinedRegions indexedPipelinedRegions =
-computePipelinedRegions(
-
executionGraphIndex.sortedExecutionVerticesInPipelinedRegion.keySet(),
-
executionGraphIndex.sortedExecutionVerticesInPipelinedRegion::get,
-executionGraphIndex.executionVerticesById::get,
-executionGraphIndex.resultPartitionsById::get);
-
-ensureCoLocatedVerticesInSameRegion(
-indexedPipelinedRegions.pipelinedRegions, executionGraph);
-
-return new DefaultExecutionTopology(
-executionGraphIndex.executionVerticesById,
-executionGraphIndex.executionVerticesList,
-executionGraphIndex.resultPartitionsById,
-indexedPipelinedRegions.pipelinedRegionsByVertex,
-indexedPipelinedRegions.pipelinedRegions,
-edgeManager);
-}
-
-private static ExecutionGraphIndex computeExecutionGraphIndex(
-Iterable executionVertices,
-Iterable logicalPipelinedRegions,
-EdgeManager edgeManager) {
-Map executionVerticesById =
new HashMap<>();
-List executionVerticesList = new ArrayList<>();
-Map
resultPartitionsById =
-new
[GitHub] [flink] wangyang0918 commented on a change in pull request #17098: [FLINK-15648][flink-kubernetes] Support to configure limit for CPU & memory
wangyang0918 commented on a change in pull request #17098:
URL: https://github.com/apache/flink/pull/17098#discussion_r786443356
##
File path:
flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/parameters/KubernetesJobManagerParameters.java
##
@@ -122,6 +122,26 @@ public double getJobManagerCPU() {
return flinkConfig.getDouble(KubernetesConfigOptions.JOB_MANAGER_CPU);
}
+public double getJobManagerCPULimitFactor() {
+double limitFactor =
Review comment:
nit: `limitFactor` could be final.
##
File path:
flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/parameters/KubernetesJobManagerParameters.java
##
@@ -122,6 +122,26 @@ public double getJobManagerCPU() {
return flinkConfig.getDouble(KubernetesConfigOptions.JOB_MANAGER_CPU);
}
+public double getJobManagerCPULimitFactor() {
+double limitFactor =
+
flinkConfig.getDouble(KubernetesConfigOptions.JOB_MANAGER_CPU_LIMIT_FACTOR);
+checkArgument(
+limitFactor >= 1,
+"%s should be greater or equal to 1.",
+KubernetesConfigOptions.JOB_MANAGER_CPU_LIMIT_FACTOR.key());
+return limitFactor;
+}
+
+public double getJobManagerMemoryLimitFactor() {
+double limitFactor =
Review comment:
nit: limitFactor could be final.
##
File path:
flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/parameters/KubernetesTaskManagerParameters.java
##
@@ -122,6 +122,26 @@ public double getTaskManagerCPU() {
.doubleValue();
}
+public double getTaskManagerCPULimitFactor() {
+double limitFactor =
Review comment:
nit: limitFactor could be final.
##
File path:
flink-kubernetes/src/main/java/org/apache/flink/kubernetes/utils/KubernetesUtils.java
##
@@ -310,26 +310,33 @@ public static CompletedCheckpointStore
createCompletedCheckpointStore(
*
* @param resourceRequirements resource requirements in pod template
* @param mem Memory in mb.
+ * @param memoryLimitFactor limit factor for the memory, used to set the
limit resources.
* @param cpu cpu.
+ * @param cpuLimitFactor limit factor for the cpu, used to set the limit
resources.
* @param externalResources external resources
* @param externalResourceConfigKeys config keys of external resources
* @return KubernetesResource requirements.
*/
public static ResourceRequirements getResourceRequirements(
ResourceRequirements resourceRequirements,
int mem,
+double memoryLimitFactor,
double cpu,
+double cpuLimitFactor,
Map externalResources,
Map externalResourceConfigKeys) {
final Quantity cpuQuantity = new Quantity(String.valueOf(cpu));
+final Quantity cpuLimitQuantity = new Quantity(String.valueOf(cpu *
cpuLimitFactor));
final Quantity memQuantity = new Quantity(mem +
Constants.RESOURCE_UNIT_MB);
+final Quantity memQuantityLimit =
+new Quantity(((int) (mem * memoryLimitFactor)) +
Constants.RESOURCE_UNIT_MB);
Review comment:
Maybe we do not need to use the `int` here since K8s could also support
decimal amount for memory resource. e.g. `1024.5m` `1.5g`. But the current
implementation also make sense to me.
##
File path:
flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/parameters/KubernetesTaskManagerParameters.java
##
@@ -122,6 +122,26 @@ public double getTaskManagerCPU() {
.doubleValue();
}
+public double getTaskManagerCPULimitFactor() {
+double limitFactor =
+
flinkConfig.getDouble(KubernetesConfigOptions.TASK_MANAGER_CPU_LIMIT_FACTOR);
+checkArgument(
+limitFactor >= 1,
+"%s should be greater or equal to 1.",
+KubernetesConfigOptions.TASK_MANAGER_CPU_LIMIT_FACTOR.key());
+return limitFactor;
+}
+
+public double getTaskManagerMemoryLimitFactor() {
+double limitFactor =
Review comment:
nit: limitFactor could be final.
##
File path:
flink-kubernetes/src/test/java/org/apache/flink/kubernetes/kubeclient/Fabric8FlinkKubeClientTest.java
##
@@ -116,6 +117,8 @@ protected void setupFlinkConfig() {
flinkConfig.set(JobManagerOptions.PORT, RPC_PORT);
flinkConfig.set(BlobServerOptions.PORT,
Integer.toString(BLOB_SERVER_PORT));
flinkConfig.set(KubernetesConfigOptions.JOB_MANAGER_CPU,
JOB_MANAGER_CPU);
+flinkConfig.set(
Review comment:
I think we could leave the `KubernetesConfigOptions.JOB_MANAGER_CPU` as
default since we do not verify the resource limit in the tests of
`Fabric8FlinkKubeClientTest`.
--
This is an automated message from the Apache Git Service.
To respond to the
[GitHub] [flink] flinkbot edited a comment on pull request #18386: [FLINK-25684][table] Support enhanced `show databases` syntax
flinkbot edited a comment on pull request #18386: URL: https://github.com/apache/flink/pull/18386#issuecomment-1015100174 ## CI report: * 5655ebda80a059eae6b0bf62200b444057c026bf Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29611) * e0a1238340c360eaf676afa6e8a7fd0f9a9ede84 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18380: [FLINK-25676][python] Support set_description in DataStream API
flinkbot edited a comment on pull request #18380: URL: https://github.com/apache/flink/pull/18380#issuecomment-1014348700 ## CI report: * 5e49df6104317779a59d57e2e4ecf08d7e36eff4 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29600) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18363: [Flink-25600][table-planner] Support new statement set syntax in sql client and update docs
flinkbot edited a comment on pull request #18363: URL: https://github.com/apache/flink/pull/18363#issuecomment-1012985353 ## CI report: * 023810c97360cbec41e825735054561aa9ed2dcd Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29599) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18386: [FLINK-25684][table] Support enhanced `show databases` syntax
flinkbot edited a comment on pull request #18386: URL: https://github.com/apache/flink/pull/18386#issuecomment-1015100174 ## CI report: * 5655ebda80a059eae6b0bf62200b444057c026bf Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29611) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18157: [FLINK-17808] Rename checkpoint meta file to "_metadata" until it has…
flinkbot edited a comment on pull request #18157: URL: https://github.com/apache/flink/pull/18157#issuecomment-998436589 ## CI report: * a9175f7c992d1cb30579dc88d4406c228e06a67a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29560) * dcebc46b47336a4f7e41003f091f4c4fb6a28bdc Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29605) * 8515ccb41dc778e38a97aa5cd7e80440a9ef389e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29609) * 7a1ba6d584788f1b07bc5bd82d879327fbdc48e6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29614) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-25073) Introduce Tree Mode description for job vertex
[ https://issues.apache.org/jira/browse/FLINK-25073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao closed FLINK-25073. --- Resolution: Fixed Merged on master via f2d03b0baea08ed59eabb72afad6ebcc4c525036 > Introduce Tree Mode description for job vertex > -- > > Key: FLINK-25073 > URL: https://issues.apache.org/jira/browse/FLINK-25073 > Project: Flink > Issue Type: Sub-task > Components: API / DataStream >Reporter: Wenlong Lyu >Assignee: Wenlong Lyu >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18386: [FLINK-25684][table] Support enhanced `show databases` syntax
flinkbot edited a comment on pull request #18386: URL: https://github.com/apache/flink/pull/18386#issuecomment-1015100174 ## CI report: * 5655ebda80a059eae6b0bf62200b444057c026bf Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29611) * e0a1238340c360eaf676afa6e8a7fd0f9a9ede84 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18157: [FLINK-17808] Rename checkpoint meta file to "_metadata" until it has…
flinkbot edited a comment on pull request #18157: URL: https://github.com/apache/flink/pull/18157#issuecomment-998436589 ## CI report: * a9175f7c992d1cb30579dc88d4406c228e06a67a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29560) * dcebc46b47336a4f7e41003f091f4c4fb6a28bdc Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29605) * 8515ccb41dc778e38a97aa5cd7e80440a9ef389e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29609) * 7a1ba6d584788f1b07bc5bd82d879327fbdc48e6 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18386: [FLINK-25684][table] Support enhanced `show databases` syntax
flinkbot edited a comment on pull request #18386: URL: https://github.com/apache/flink/pull/18386#issuecomment-1015100174 ## CI report: * 5655ebda80a059eae6b0bf62200b444057c026bf Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29611) * 5575d8841ea45e8af178b2bb756c7d410ea82ae4 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-25686) Support Pulsar Schema evolution in Pulsar Connector
[ https://issues.apache.org/jira/browse/FLINK-25686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17477605#comment-17477605 ] Yufei Zhang commented on FLINK-25686: - Please assign this ticket to me [~fpaul] ~ Thank you ! > Support Pulsar Schema evolution in Pulsar Connector > --- > > Key: FLINK-25686 > URL: https://issues.apache.org/jira/browse/FLINK-25686 > Project: Flink > Issue Type: New Feature > Components: Connectors / Pulsar >Reporter: Yufei Zhang >Priority: Minor > Labels: Pulsar > > Currently the source connector reads all messages as Message, and > implements a deserialization abstraction to deserialize the byte array > message. This deserialization abstraction supports both flink schema and > pulsar schema. However because of this extra layer of abstraction, even when > using pulsar schema, the schema info is not provided to the consumer > (client), so it is not using any pulsar’s schema validation mechanism. (If > pulsar client has a schema, it will send the schema to broker on connect and > broker will validate if it is a valid schema according to a preset > compatibility rule :[ > https://pulsar.apache.org/docs/en/schema-evolution-compatibility/|https://pulsar.apache.org/docs/en/schema-evolution-compatibility/]). > we need to support this schema evolution -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-25686) Support Pulsar Schema evolution in Pulsar Connector
Yufei Zhang created FLINK-25686: --- Summary: Support Pulsar Schema evolution in Pulsar Connector Key: FLINK-25686 URL: https://issues.apache.org/jira/browse/FLINK-25686 Project: Flink Issue Type: New Feature Components: Connectors / Pulsar Reporter: Yufei Zhang Currently the source connector reads all messages as Message, and implements a deserialization abstraction to deserialize the byte array message. This deserialization abstraction supports both flink schema and pulsar schema. However because of this extra layer of abstraction, even when using pulsar schema, the schema info is not provided to the consumer (client), so it is not using any pulsar’s schema validation mechanism. (If pulsar client has a schema, it will send the schema to broker on connect and broker will validate if it is a valid schema according to a preset compatibility rule :[ https://pulsar.apache.org/docs/en/schema-evolution-compatibility/|https://pulsar.apache.org/docs/en/schema-evolution-compatibility/]). we need to support this schema evolution -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18386: [FLINK-25684][table] Support enhanced `show databases` syntax
flinkbot edited a comment on pull request #18386: URL: https://github.com/apache/flink/pull/18386#issuecomment-1015100174 ## CI report: * 5655ebda80a059eae6b0bf62200b444057c026bf Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29611) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18386: [FLINK-25684][table] Support enhanced `show databases` syntax
flinkbot edited a comment on pull request #18386: URL: https://github.com/apache/flink/pull/18386#issuecomment-1015100174 ## CI report: * 5655ebda80a059eae6b0bf62200b444057c026bf Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29611) * 5575d8841ea45e8af178b2bb756c7d410ea82ae4 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17601: [FLINK-24697][flink-connectors-kafka] add auto.offset.reset configuration for group-offsets startup mode
flinkbot edited a comment on pull request #17601: URL: https://github.com/apache/flink/pull/17601#issuecomment-954546978 ## CI report: * 4ea3c9f8fd437a0e6d183bfb62101f54f89de34e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29219) * a65b12ce87fe15a5474b0a740e9a054e7685941d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29612) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] gaoyunhaii closed pull request #17938: [FLINK-25073][streaming] Introduce TreeMode description for vertices
gaoyunhaii closed pull request #17938: URL: https://github.com/apache/flink/pull/17938 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18386: [FLINK-25684][table] Support enhanced `show databases` syntax
flinkbot edited a comment on pull request #18386: URL: https://github.com/apache/flink/pull/18386#issuecomment-1015100174 ## CI report: * 5655ebda80a059eae6b0bf62200b444057c026bf Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29611) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] ruanhang1993 commented on a change in pull request #17601: [FLINK-24697][flink-connectors-kafka] add auto.offset.reset configuration for group-offsets startup mode
ruanhang1993 commented on a change in pull request #17601:
URL: https://github.com/apache/flink/pull/17601#discussion_r786445498
##
File path:
flink-connectors/flink-connector-kafka/src/test/java/org/apache/flink/streaming/connectors/kafka/table/KafkaTableITCase.java
##
@@ -821,6 +826,175 @@ public void testPerPartitionWatermarkWithIdleSource()
throws Exception {
deleteTestTopic(topic);
}
+@Test
+public void testStartFromGroupOffsetsLatest() throws Exception {
+testStartFromGroupOffsets("latest", Collections.emptyList());
+}
+
+@Test
+public void testStartFromGroupOffsetsEarliest() throws Exception {
+final List expected =
+Arrays.asList(
+"+I[0, 0]", "+I[0, 1]", "+I[0, 2]", "+I[1, 3]", "+I[1,
4]", "+I[1, 5]");
+testStartFromGroupOffsets("earliest", expected);
+}
+
+@Test
+public void testStartFromGroupOffsetsNone() {
+try {
+testStartFromGroupOffsetsWithException("none",
Collections.emptyList());
+fail("None offset reset error.");
+} catch (Exception e) {
+Throwable rootCause = ExceptionUtils.getRootCause(e);
+assertTrue(rootCause instanceof NoOffsetForPartitionException);
+}
+}
+
+private void testStartFromGroupOffsets(String reset, List expected)
Review comment:
I have fixed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17601: [FLINK-24697][flink-connectors-kafka] add auto.offset.reset configuration for group-offsets startup mode
flinkbot edited a comment on pull request #17601: URL: https://github.com/apache/flink/pull/17601#issuecomment-954546978 ## CI report: * 4ea3c9f8fd437a0e6d183bfb62101f54f89de34e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29219) * a65b12ce87fe15a5474b0a740e9a054e7685941d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] ruanhang1993 commented on a change in pull request #17601: [FLINK-24697][flink-connectors-kafka] add auto.offset.reset configuration for group-offsets startup mode
ruanhang1993 commented on a change in pull request #17601:
URL: https://github.com/apache/flink/pull/17601#discussion_r786445231
##
File path:
flink-connectors/flink-connector-kafka/src/test/java/org/apache/flink/streaming/connectors/kafka/table/KafkaTableITCase.java
##
@@ -821,6 +826,175 @@ public void testPerPartitionWatermarkWithIdleSource()
throws Exception {
deleteTestTopic(topic);
}
+@Test
+public void testStartFromGroupOffsetsLatest() throws Exception {
+testStartFromGroupOffsets("latest", Collections.emptyList());
+}
+
+@Test
+public void testStartFromGroupOffsetsEarliest() throws Exception {
+final List expected =
+Arrays.asList(
+"+I[0, 0]", "+I[0, 1]", "+I[0, 2]", "+I[1, 3]", "+I[1,
4]", "+I[1, 5]");
+testStartFromGroupOffsets("earliest", expected);
+}
+
+@Test
+public void testStartFromGroupOffsetsNone() {
+try {
+testStartFromGroupOffsetsWithException("none",
Collections.emptyList());
+fail("None offset reset error.");
+} catch (Exception e) {
+Throwable rootCause = ExceptionUtils.getRootCause(e);
+assertTrue(rootCause instanceof NoOffsetForPartitionException);
+}
+}
+
+private void testStartFromGroupOffsets(String reset, List expected)
+throws ExecutionException, InterruptedException {
+// we always use a different topic name for each parameterized topic,
+// in order to make sure the topic can be created.
+final String tableName = "Table" + format + reset;
+final String topic = "groupOffset_" + format + reset;
+createTestTopic(topic, 4, 1);
+
+// -- Produce an event time stream into Kafka
---
+String groupId = format + reset;
+String bootstraps = getBootstrapServers();
+tEnv.getConfig()
+.getConfiguration()
+.set(TABLE_EXEC_SOURCE_IDLE_TIMEOUT, Duration.ofMillis(100));
+
+final String createTableSql =
+"CREATE TABLE %s (\n"
++ " `partition_id` INT,\n"
++ " `value` INT\n"
++ ") WITH (\n"
++ " 'connector' = 'kafka',\n"
++ " 'topic' = '%s',\n"
++ " 'properties.bootstrap.servers' = '%s',\n"
++ " 'properties.group.id' = '%s',\n"
++ " 'scan.startup.mode' = 'group-offsets',\n"
++ " 'properties.auto.offset.reset' = '%s',\n"
++ " 'format' = '%s'\n"
++ ")";
+tEnv.executeSql(
+String.format(
+createTableSql, tableName, topic, bootstraps, groupId,
reset, format));
+
+String initialValues =
+"INSERT INTO "
++ tableName
++ "\n"
++ "VALUES\n"
++ " (0, 0),\n"
++ " (0, 1),\n"
++ " (0, 2),\n"
++ " (1, 3),\n"
++ " (1, 4),\n"
++ " (1, 5)\n";
+tEnv.executeSql(initialValues).await();
+
+// -- Consume stream from Kafka ---
+
+env.setParallelism(1);
+String sinkName = "mySink" + format + reset;
+String createSink =
+"CREATE TABLE "
++ sinkName
++ "(\n"
++ " `partition_id` INT,\n"
++ " `value` INT\n"
++ ") WITH (\n"
++ " 'connector' = 'values'\n"
++ ")";
+tEnv.executeSql(createSink);
+
+TableResult tableResult =
+tEnv.executeSql("INSERT INTO " + sinkName + " SELECT * FROM "
+ tableName);
+
+if (expected.size() == 0) {
+KafkaTableTestUtils.waitingEmptyResults(sinkName,
Duration.ofSeconds(5));
Review comment:
fixed. I have refactored them with the `CommonTestUtils#waitUntil`.
This code aims to make sure that we will not get any result from the test. I
have reimplemented it with `CommonTestUtils#waitUntil` and give it a more
meaningful name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18386: [FLINK-25684][table] Support enhanced `show databases` syntax
flinkbot edited a comment on pull request #18386: URL: https://github.com/apache/flink/pull/18386#issuecomment-1015100174 ## CI report: * 5655ebda80a059eae6b0bf62200b444057c026bf Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29611) * 58aa1f21a711c788290c6971028118a97bf7df45 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] ruanhang1993 commented on a change in pull request #17601: [FLINK-24697][flink-connectors-kafka] add auto.offset.reset configuration for group-offsets startup mode
ruanhang1993 commented on a change in pull request #17601:
URL: https://github.com/apache/flink/pull/17601#discussion_r786444064
##
File path:
flink-connectors/flink-connector-kafka/src/test/java/org/apache/flink/streaming/connectors/kafka/table/KafkaDynamicTableFactoryTest.java
##
@@ -380,6 +389,53 @@ public void testTableSourceCommitOnCheckpointDisabled() {
.noDefaultValue()));
}
+@ParameterizedTest
Review comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] ruanhang1993 commented on a change in pull request #17601: [FLINK-24697][flink-connectors-kafka] add auto.offset.reset configuration for group-offsets startup mode
ruanhang1993 commented on a change in pull request #17601:
URL: https://github.com/apache/flink/pull/17601#discussion_r786443847
##
File path:
flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/table/KafkaDynamicSource.java
##
@@ -393,7 +396,27 @@ public int hashCode() {
kafkaSourceBuilder.setStartingOffsets(OffsetsInitializer.latest());
break;
case GROUP_OFFSETS:
-
kafkaSourceBuilder.setStartingOffsets(OffsetsInitializer.committedOffsets());
+String offsetResetConfig =
+properties.getProperty(
+ConsumerConfig.AUTO_OFFSET_RESET_CONFIG,
+OffsetResetStrategy.NONE.name());
+OffsetResetStrategy offsetResetStrategy =
+Arrays.stream(OffsetResetStrategy.values())
+.filter(
+ors ->
+ors.name()
+.equals(
+
offsetResetConfig.toUpperCase(
+
Locale.ROOT)))
+.findAny()
+.orElse(null);
+Preconditions.checkArgument(
Review comment:
fixed
##
File path:
flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/table/KafkaDynamicSource.java
##
@@ -393,7 +396,27 @@ public int hashCode() {
kafkaSourceBuilder.setStartingOffsets(OffsetsInitializer.latest());
break;
case GROUP_OFFSETS:
-
kafkaSourceBuilder.setStartingOffsets(OffsetsInitializer.committedOffsets());
+String offsetResetConfig =
+properties.getProperty(
+ConsumerConfig.AUTO_OFFSET_RESET_CONFIG,
+OffsetResetStrategy.NONE.name());
+OffsetResetStrategy offsetResetStrategy =
+Arrays.stream(OffsetResetStrategy.values())
+.filter(
+ors ->
+ors.name()
+.equals(
+
offsetResetConfig.toUpperCase(
+
Locale.ROOT)))
Review comment:
fixed
##
File path:
flink-connectors/flink-connector-kafka/src/test/java/org/apache/flink/streaming/connectors/kafka/table/KafkaDynamicTableFactoryTest.java
##
@@ -380,6 +389,53 @@ public void testTableSourceCommitOnCheckpointDisabled() {
.noDefaultValue()));
}
+@ParameterizedTest
+@MethodSource("offsetResetStrategy")
Review comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18386: [FLINK-25684][table] Support enhanced `show databases` syntax
flinkbot edited a comment on pull request #18386: URL: https://github.com/apache/flink/pull/18386#issuecomment-1015100174 ## CI report: * 5655ebda80a059eae6b0bf62200b444057c026bf Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29611) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #18386: [FLINK-25684][table] Support enhanced `show databases` syntax
flinkbot commented on pull request #18386: URL: https://github.com/apache/flink/pull/18386#issuecomment-1015100250 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 5655ebda80a059eae6b0bf62200b444057c026bf (Tue Jan 18 06:11:12 UTC 2022) **Warnings:** * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-25684).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #18386: [FLINK-25684][table] Support enhanced `show databases` syntax
flinkbot commented on pull request #18386: URL: https://github.com/apache/flink/pull/18386#issuecomment-1015100174 ## CI report: * 5655ebda80a059eae6b0bf62200b444057c026bf UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-15648) Support to configure limit for CPU and memory requirement
[
https://issues.apache.org/jira/browse/FLINK-15648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Wang reassigned FLINK-15648:
-
Assignee: Denis Nutiu
> Support to configure limit for CPU and memory requirement
> -
>
> Key: FLINK-15648
> URL: https://issues.apache.org/jira/browse/FLINK-15648
> Project: Flink
> Issue Type: New Feature
> Components: Deployment / Kubernetes
>Reporter: Canbin Zheng
>Assignee: Denis Nutiu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently, the native Kubernetes mode could support same request and limit
> for the cpu and memory resources. It will be a good improvement to allow
> different resource requests and limits. And it is very useful especially for
> the CPU resource since it heavily depends on the upstream workloads.
>
> For the implementation, we could have the four config options of
> limit-factor, which could make the JM/TM benefit from burst resources.
> {code:java}
> kubernetes.jobmanager.cpu.limit-factor
> kubernetes.jobmanager.memory.limit-factor
> kubernetes.taskmanager.cpu.limit-factor
> kubernetes.taskmanager.memory.limit-factor{code}
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-25684) Support enhanced show databases syntax
[ https://issues.apache.org/jira/browse/FLINK-25684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-25684: --- Labels: pull-request-available (was: ) > Support enhanced show databases syntax > -- > > Key: FLINK-25684 > URL: https://issues.apache.org/jira/browse/FLINK-25684 > Project: Flink > Issue Type: Improvement > Components: Table SQL / API >Reporter: Moses >Priority: Major > Labels: pull-request-available > > Enhanced `show databases` statement like ` show databasesfrom like 'db%' ` > has been supported broadly in many popular SQL engine like Spark SQL/MySQL. > We could use such statement to easily show the databases that we wannted. > h3. SHOW DATABSES [ LIKE regex_pattern ] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] ZhangChaoming opened a new pull request #18386: [FLINK-25684][table] Support enhanced show databases syntax
ZhangChaoming opened a new pull request #18386: URL: https://github.com/apache/flink/pull/18386 ## What is the purpose of the change Enhanced `show databases` statement like ` show databasesfrom like 'db%' ` has been supported broadly in many popular SQL engine like Spark SQL/MySQL. We could use such statement to easily show the databases that we wannted. ## Brief change log Support enhanced show databases syntax: ```sql SHOW DATABSES [ LIKE regex_pattern ] ``` ## Verifying this change This change added tests and can be verified as follows: flink-table/flink-sql-client/src/test/resources/sql/catalog_database.q ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (**yes** / no) - If yes, how is the feature documented? (not applicable / **docs** / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18365: [FLINK-22643][network] Reuse tpc connections between taskmanagers
flinkbot edited a comment on pull request #18365: URL: https://github.com/apache/flink/pull/18365#issuecomment-1013081147 ## CI report: * Unknown: [CANCELED](TBD) * 51ec010668d8237bb8e02de3848126a9a7f19620 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29610) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18365: [FLINK-22643][network] Reuse tpc connections between taskmanagers
flinkbot edited a comment on pull request #18365: URL: https://github.com/apache/flink/pull/18365#issuecomment-1013081147 ## CI report: * Unknown: [CANCELED](TBD) * 51ec010668d8237bb8e02de3848126a9a7f19620 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] 1996fanrui removed a comment on pull request #18365: [FLINK-22643][network] Reuse tpc connections between taskmanagers
1996fanrui removed a comment on pull request #18365: URL: https://github.com/apache/flink/pull/18365#issuecomment-1015007385 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] 1996fanrui commented on pull request #18365: [FLINK-22643][network] Reuse tpc connections between taskmanagers
1996fanrui commented on pull request #18365: URL: https://github.com/apache/flink/pull/18365#issuecomment-1015094837 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-25685) RestClusterClient gets stuck on submitting job with local user artifact
Xuannan Su created FLINK-25685:
--
Summary: RestClusterClient gets stuck on submitting job with local
user artifact
Key: FLINK-25685
URL: https://issues.apache.org/jira/browse/FLINK-25685
Project: Flink
Issue Type: Bug
Components: Runtime / REST
Affects Versions: 1.14.3
Reporter: Xuannan Su
I found that a job submission gets stuck if
StreamExecutionEnvironment#registerCachedFile is called with a local file.
After some digging, I found that it gets stuck when the RestClusterClient sends
the job-submission request to the JobManager.
Below is the unit test added to the `RestClusterClientTest` to reproduce the
problem on my local machine.
{code:java}
@Test
public void testJobSubmissionWithUserArtifact() throws Exception {
try (final TestRestServerEndpoint restServerEndpoint =
createRestServerEndpoint(new TestJobSubmitHandler())) {
try (RestClusterClient restClusterClient =
createRestClusterClient(restServerEndpoint.getServerAddress().getPort())) {
TemporaryFolder temporaryFolder = new TemporaryFolder();
temporaryFolder.create();
File file = temporaryFolder.newFile();
Files.write(file.toPath(), "hello
world".getBytes(ConfigConstants.DEFAULT_CHARSET));
jobGraph.addUserArtifact("file",
new
DistributedCache.DistributedCacheEntry(file.toURI().toString(),
false));
restClusterClient
.submitJob(jobGraph)
.get();
}
}
}
{code}
The test can pass if the `jobGraph.addUserArtifact` is not called.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-25683) wrong result if table transfrom to DataStream then window process in batch mode
[ https://issues.apache.org/jira/browse/FLINK-25683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangzh updated FLINK-25683: Summary: wrong result if table transfrom to DataStream then window process in batch mode (was: wrong result if table transfrom to DataStream then window process grouped to mini batch in batch mode) > wrong result if table transfrom to DataStream then window process in batch > mode > --- > > Key: FLINK-25683 > URL: https://issues.apache.org/jira/browse/FLINK-25683 > Project: Flink > Issue Type: Bug > Components: Table SQL / API, Table SQL / Runtime >Affects Versions: 1.14.2 > Environment: mac book pro m1 > jdk 8 > scala 2.11 > flink 1.14.2 > idea 2020 >Reporter: zhangzh >Priority: Major > Attachments: TableToDataStreamBatchWindowTest.scala, pom.xml > > > I have 5 line datas, > i first need to transform current data with SQL > then mix current data and historical data which is batch get from hbase > for some special reason the program must run in batch mode > i think the correct result should be like this： > (BOB,1) > (EMA,1) > (DOUG,1) > (ALICE,1) > (CENDI,1) > but the result is : > (EMA,1) > > if i set different parallelism ,the result is different. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-25683) wrong result if table transfrom to DataStream then window process in batch mode
[ https://issues.apache.org/jira/browse/FLINK-25683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17477584#comment-17477584 ] zhangzh commented on FLINK-25683: - hi @[~paul8263] ： I found a new problem similar to FLINK-25471。 can you help me？ > wrong result if table transfrom to DataStream then window process in batch > mode > --- > > Key: FLINK-25683 > URL: https://issues.apache.org/jira/browse/FLINK-25683 > Project: Flink > Issue Type: Bug > Components: Table SQL / API, Table SQL / Runtime >Affects Versions: 1.14.2 > Environment: mac book pro m1 > jdk 8 > scala 2.11 > flink 1.14.2 > idea 2020 >Reporter: zhangzh >Priority: Major > Attachments: TableToDataStreamBatchWindowTest.scala, pom.xml > > > I have 5 line datas, > i first need to transform current data with SQL > then mix current data and historical data which is batch get from hbase > for some special reason the program must run in batch mode > i think the correct result should be like this： > (BOB,1) > (EMA,1) > (DOUG,1) > (ALICE,1) > (CENDI,1) > but the result is : > (EMA,1) > > if i set different parallelism ,the result is different. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-25509) FLIP-208: Add RecordEvaluator to dynamically stop source based on de-serialized records
[ https://issues.apache.org/jira/browse/FLINK-25509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin updated FLINK-25509: - Description: This feature is needed to migrate applications which uses KafkaDeserializationSchema::isEndOfStream() from using FlinkKafkaConsumer to using KafkaSource. Please checkout https://cwiki.apache.org/confluence/display/FLINK/FLIP-208%3A+Add+RecordEvaluator+to+dynamically+stop+source+based+on+de-serialized+records for the motivation and the proposed changes. was:Please checkout https://cwiki.apache.org/confluence/display/FLINK/FLIP-208%3A+Update+KafkaSource+to+detect+EOF+based+on+de-serialized+records for the motivation and the proposed changes. > FLIP-208: Add RecordEvaluator to dynamically stop source based on > de-serialized records > --- > > Key: FLINK-25509 > URL: https://issues.apache.org/jira/browse/FLINK-25509 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common, Connectors / Kafka >Reporter: Dong Lin >Priority: Major > > This feature is needed to migrate applications which uses > KafkaDeserializationSchema::isEndOfStream() from using FlinkKafkaConsumer to > using KafkaSource. > Please checkout > https://cwiki.apache.org/confluence/display/FLINK/FLIP-208%3A+Add+RecordEvaluator+to+dynamically+stop+source+based+on+de-serialized+records > for the motivation and the proposed changes. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-25683) wrong result if table transfrom to DataStream then window process grouped to mini batch in batch mode
[ https://issues.apache.org/jira/browse/FLINK-25683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangzh updated FLINK-25683: Attachment: TableToDataStreamBatchWindowTest.scala > wrong result if table transfrom to DataStream then window process grouped to > mini batch in batch mode > - > > Key: FLINK-25683 > URL: https://issues.apache.org/jira/browse/FLINK-25683 > Project: Flink > Issue Type: Bug > Components: Table SQL / API, Table SQL / Runtime >Affects Versions: 1.14.2 > Environment: mac book pro m1 > jdk 8 > scala 2.11 > flink 1.14.2 > idea 2020 >Reporter: zhangzh >Priority: Major > Attachments: TableToDataStreamBatchWindowTest.scala, pom.xml > > > I have 5 line datas, > i first need to transform current data with SQL > then mix current data and historical data which is batch get from hbase > for some special reason the program must run in batch mode > i think the correct result should be like this： > (BOB,1) > (EMA,1) > (DOUG,1) > (ALICE,1) > (CENDI,1) > but the result is : > (EMA,1) > > if i set different parallelism ,the result is different. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-25684) Support enhanced show databases syntax
[ https://issues.apache.org/jira/browse/FLINK-25684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17477583#comment-17477583 ] Moses commented on FLINK-25684: --- cc [~jark] > Support enhanced show databases syntax > -- > > Key: FLINK-25684 > URL: https://issues.apache.org/jira/browse/FLINK-25684 > Project: Flink > Issue Type: Improvement > Components: Table SQL / API >Reporter: Moses >Priority: Major > > Enhanced `show databases` statement like ` show databasesfrom like 'db%' ` > has been supported broadly in many popular SQL engine like Spark SQL/MySQL. > We could use such statement to easily show the databases that we wannted. > h3. SHOW DATABSES [ LIKE regex_pattern ] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-25684) Support enhanced show databases syntax
Moses created FLINK-25684: - Summary: Support enhanced show databases syntax Key: FLINK-25684 URL: https://issues.apache.org/jira/browse/FLINK-25684 Project: Flink Issue Type: Improvement Components: Table SQL / API Reporter: Moses Enhanced `show databases` statement like ` show databasesfrom like 'db%' ` has been supported broadly in many popular SQL engine like Spark SQL/MySQL. We could use such statement to easily show the databases that we wannted. h3. SHOW DATABSES [ LIKE regex_pattern ] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-25509) FLIP-208: Add RecordEvaluator to dynamically stop source based on de-serialized records
[ https://issues.apache.org/jira/browse/FLINK-25509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin updated FLINK-25509: - Summary: FLIP-208: Add RecordEvaluator to dynamically stop source based on de-serialized records (was: FLIP-208: Update KafkaSource to detect EOF based on de-serialized records) > FLIP-208: Add RecordEvaluator to dynamically stop source based on > de-serialized records > --- > > Key: FLINK-25509 > URL: https://issues.apache.org/jira/browse/FLINK-25509 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common, Connectors / Kafka >Reporter: Dong Lin >Priority: Major > > Please checkout > https://cwiki.apache.org/confluence/display/FLINK/FLIP-208%3A+Update+KafkaSource+to+detect+EOF+based+on+de-serialized+records > for the motivation and the proposed changes. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-25683) wrong result if table transfrom to DataStream then window process grouped to mini batch in batch mode
[ https://issues.apache.org/jira/browse/FLINK-25683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangzh updated FLINK-25683: Attachment: (was: TableToDataStreamBatchWindowTest.scala) > wrong result if table transfrom to DataStream then window process grouped to > mini batch in batch mode > - > > Key: FLINK-25683 > URL: https://issues.apache.org/jira/browse/FLINK-25683 > Project: Flink > Issue Type: Bug > Components: Table SQL / API, Table SQL / Runtime >Affects Versions: 1.14.2 > Environment: mac book pro m1 > jdk 8 > scala 2.11 > flink 1.14.2 > idea 2020 >Reporter: zhangzh >Priority: Major > Attachments: pom.xml > > > I have 5 line datas, > i first need to transform current data with SQL > then mix current data and historical data which is batch get from hbase > for some special reason the program must run in batch mode > i think the correct result should be like this： > (BOB,1) > (EMA,1) > (DOUG,1) > (ALICE,1) > (CENDI,1) > but the result is : > (EMA,1) > > if i set different parallelism ,the result is different. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-25683) wrong result if table transfrom to DataStream then window process grouped to mini batch in batch mode
[ https://issues.apache.org/jira/browse/FLINK-25683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangzh updated FLINK-25683: Description: I have 5 line datas, i first need to transform current data with SQL then mix current data and historical data which is batch get from hbase for some special reason the program must run in batch mode i think the correct result should be like this： (BOB,1) (EMA,1) (DOUG,1) (ALICE,1) (CENDI,1) but the result is : (EMA,1) if i set different parallelism ,the result is different. was: I have 5 line datas, i first need to transform current data with SQL then mix current data and historical data which is batch get from hbase for some special reason the program must run in batch mode i think the correct result should be like this： (BOB,1) (EMA,1) (DOUG,1) (ALICE,1) (CENDI,1) but the result is : (EMA,1) > wrong result if table transfrom to DataStream then window process grouped to > mini batch in batch mode > - > > Key: FLINK-25683 > URL: https://issues.apache.org/jira/browse/FLINK-25683 > Project: Flink > Issue Type: Bug > Components: Table SQL / API, Table SQL / Runtime >Affects Versions: 1.14.2 > Environment: mac book pro m1 > jdk 8 > scala 2.11 > flink 1.14.2 > idea 2020 >Reporter: zhangzh >Priority: Major > Attachments: pom.xml > > > I have 5 line datas, > i first need to transform current data with SQL > then mix current data and historical data which is batch get from hbase > for some special reason the program must run in batch mode > i think the correct result should be like this： > (BOB,1) > (EMA,1) > (DOUG,1) > (ALICE,1) > (CENDI,1) > but the result is : > (EMA,1) > > if i set different parallelism ,the result is different. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] wsry commented on pull request #18365: [FLINK-22643][network] Reuse tpc connections between taskmanagers
wsry commented on pull request #18365: URL: https://github.com/apache/flink/pull/18365#issuecomment-1015082277 @1996fanrui Maybe rebasing the latest master can make the tests pass. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-25683) wrong result if table transfrom to DataStream then window process grouped to mini batch in batch mode
[ https://issues.apache.org/jira/browse/FLINK-25683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangzh updated FLINK-25683: Description: I have 5 line datas, i first need to transform current data with SQL then mix current data and historical data which is batch get from hbase for some special reason the program must run in batch mode i think the correct result should be like this： (BOB,1) (EMA,1) (DOUG,1) (ALICE,1) (CENDI,1) but the result is : (EMA,1) > wrong result if table transfrom to DataStream then window process grouped to > mini batch in batch mode > - > > Key: FLINK-25683 > URL: https://issues.apache.org/jira/browse/FLINK-25683 > Project: Flink > Issue Type: Bug > Components: Table SQL / API, Table SQL / Runtime >Affects Versions: 1.14.2 > Environment: mac book pro m1 > jdk 8 > scala 2.11 > flink 1.14.2 > idea 2020 >Reporter: zhangzh >Priority: Major > Attachments: TableToDataStreamBatchWindowTest.scala, pom.xml > > > I have 5 line datas, > i first need to transform current data with SQL > then mix current data and historical data which is batch get from hbase > for some special reason the program must run in batch mode > i think the correct result should be like this： > (BOB,1) > (EMA,1) > (DOUG,1) > (ALICE,1) > (CENDI,1) > but the result is : > (EMA,1) -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] MartijnVisser commented on pull request #18366: [FLINK-25656][Kinesis] Upgrade software.amazon.glue:schema-registry-common and software.amazon.glue:schema-registry-serde dependency fr
MartijnVisser commented on pull request #18366: URL: https://github.com/apache/flink/pull/18366#issuecomment-1015080853 @dannycranmer Will you merge this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] MartijnVisser commented on pull request #18367: [FLINK-25657][Kinesis] Upgrade com.amazonaws:amazon-kinesis-client dependency from 1.14.1 to 1.14.7
MartijnVisser commented on pull request #18367: URL: https://github.com/apache/flink/pull/18367#issuecomment-1015080680 @dannycranmer Will you merge this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-25562) hive sql connector 依赖冲突
[ https://issues.apache.org/jira/browse/FLINK-25562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17477571#comment-17477571 ] Martijn Visser commented on FLINK-25562: [~luoyuxia] The documentation mentions that Hive 1.1.0 is supported, see [https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/table/hive/overview/] This looks like a misconfiguration which results in the error message > hive sql connector 依赖冲突 > --- > > Key: FLINK-25562 > URL: https://issues.apache.org/jira/browse/FLINK-25562 > Project: Flink > Issue Type: Improvement > Components: Connectors / Hive >Affects Versions: 1.13.3 > Environment: CDH：5.15.0 > hadoop：2.6.0 > hive：1.1.0 > flink：1.3.3 >Reporter: ziqiang.wang >Priority: Minor > Labels: easyfix > Fix For: 1.13.3 > > Attachments: image-2022-01-07-14-54-37-874.png, > image-2022-01-13-16-08-20-176.png, image-2022-01-13-16-18-14-830.png, > image-2022-01-13-16-21-09-495.png > > Original Estimate: 12h > Remaining Estimate: 12h > > [https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-connector-hive-1.2.2_2.11/1.13.5/flink-sql-connector-hive-1.2.2_2.11-1.13.5.jar] > Flink jar flink-sql-connector-hive-1.2.2_2.11.jar are not compatible with > Hive 1.1.0. > I cloned the source code of Flink, and then modified the version of Hive to > 1.1.0 in the POM file of Hive-sql-connector. Then I repackaged > Flink-SQL-connector-Hive, and put it online, so there was no version conflict > problem. > Version conflicts mainly occur in the Hive-exe.jar package. > For conflicts details, see the following figure. > !image-2022-01-07-14-54-37-874.png! > I hope the official website can provided other packaged jar for hive 1.1.x。 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-25560) Add "sink.delete.mode" in HBase sql connector for retracting the latest version or all versions in changelog mode
[
https://issues.apache.org/jira/browse/FLINK-25560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17477570#comment-17477570
]
Martijn Visser commented on FLINK-25560:
[~Bruce Wong] If I look at your PR, there have already been two reviewers. You
should reach consensus with them on the requested changes and not ask for more
reviewers.
> Add "sink.delete.mode" in HBase sql connector for retracting the latest
> version or all versions in changelog mode
> -
>
> Key: FLINK-25560
> URL: https://issues.apache.org/jira/browse/FLINK-25560
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / HBase
>Reporter: Bruce Wong
>Assignee: Jing Ge
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2022-01-11-20-02-17-780.png,
> image-2022-01-11-20-04-48-299.png, image-2022-01-11-20-05-53-217.png,
> image-2022-01-11-20-07-43-900.png, image-2022-01-11-20-09-29-074.png
>
>
> h1. Motivation
> When we synchronize data from mysql to HBase, we find that when deleting data
> from mysql, HBase cannot delete all versions, which leads to incorrect
> semantics. So we want to add a parameter to control deleting the latest
> version or deleting all versions.
> h1. Usage
> The test code is as follows.
> {code:java}
> package com.bruce;
> import org.apache.flink.configuration.Configuration;
> import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
> import org.apache.flink.table.api.EnvironmentSettings;
> import org.apache.flink.table.api.TableConfig;
> import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
> import static
> org.apache.flink.configuration.ConfigConstants.LOCAL_START_WEBSERVER;
> public class KafkaToHBase {
> public static void main(String[] args) {
> Configuration cfg = new Configuration();
> cfg.setBoolean(LOCAL_START_WEBSERVER, true);
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.createLocalEnvironmentWithWebUI(cfg);
> env.setParallelism(1);
> EnvironmentSettings envSettings = EnvironmentSettings.newInstance()
> .inStreamingMode()
> .build();
> StreamTableEnvironment tEnv = StreamTableEnvironment.create(env,
> envSettings);
> //TableConfig config = tEnv.getConfig();
> //config.setIdleStateRetention(Duration.ofHours(2));
> String source = "CREATE TEMPORARY TABLE IF NOT EXISTS
> kafka_llspay_bundles(\n" +
> " id STRING,\n" +
> " category_id STRING,\n" +
> " upc STRING,\n" +
> " `name`STRING,\n" +
> " price_cents STRING,\n" +
> " original_price_cents STRING,\n" +
> " short_descSTRING,\n" +
> " desc STRING,\n" +
> " cover_url STRING,\n" +
> " created_atSTRING,\n" +
> " updated_atSTRING,\n" +
> " deleted_atSTRING,\n" +
> " extra STRING,\n" +
> " statusSTRING,\n" +
> " scholarship_cents STRING,\n" +
> " is_paybackSTRING,\n" +
> " is_support_iapSTRING,\n" +
> " iap_product_idSTRING,\n" +
> " neo_product_code STRING,\n" +
> " paid_redirect_url STRING,\n" +
> " subscription_type STRING\n" +
> ") WITH (\n" +
> " 'connector' = 'kafka',\n" +
> " 'topic' = 'dim-bundles',\n" +
> " 'properties.bootstrap.servers' = 'localhost:9092',\n" +
> " 'properties.group.id' = 'vvp_dev',\n" +
> " 'scan.startup.mode' = 'latest-offset',\n" +
> " 'value.debezium-json.schema-include' = 'true',\n" +
> " 'value.format' = 'debezium-json',\n" +
> " 'value.debezium-json.ignore-parse-errors' = 'true'\n" +
> ")";
> String sink = "CREATE TEMPORARY TABLE IF NOT EXISTS dim_hbase (\n" +
> "rowkey STRING,\n" +
> "cf ROW STRING, price_cents STRING, original_price_cents STRING, short_desc STRING,
> `desc` STRING, cover_url STRING, created_at STRING, updated_at STRING,
> deleted_at STRING, extra STRING, status STRING, scholarship_cents STRING,
> is_payback STRING, is_support_iap STRING, iap_product_id STRING,
> neo_product_code STRING, paid_redirect_url STRING,
[jira] [Commented] (FLINK-25070) FLIP-195: Improve the name and structure of vertex and operator name for job
[ https://issues.apache.org/jira/browse/FLINK-25070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17477569#comment-17477569 ] Martijn Visser commented on FLINK-25070: Thanks for clarifying [~wenlong.lwl] > FLIP-195: Improve the name and structure of vertex and operator name for job > > > Key: FLINK-25070 > URL: https://issues.apache.org/jira/browse/FLINK-25070 > Project: Flink > Issue Type: Improvement > Components: API / DataStream, Runtime / Web Frontend, Table SQL / > Runtime >Reporter: Wenlong Lyu >Assignee: Wenlong Lyu >Priority: Major > Fix For: 1.15.0 > > > this is an umbrella issue tracking the improvement of operator/vertex names > in flink: > https://cwiki.apache.org/confluence/display/FLINK/FLIP-195%3A+Improve+the+name+and+structure+of+vertex+and+operator+name+for+job -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18157: [FLINK-17808] Rename checkpoint meta file to "_metadata" until it has…
flinkbot edited a comment on pull request #18157: URL: https://github.com/apache/flink/pull/18157#issuecomment-998436589 ## CI report: * a9175f7c992d1cb30579dc88d4406c228e06a67a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29560) * dcebc46b47336a4f7e41003f091f4c4fb6a28bdc Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29605) * 8515ccb41dc778e38a97aa5cd7e80440a9ef389e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29609) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18157: [FLINK-17808] Rename checkpoint meta file to "_metadata" until it has…
flinkbot edited a comment on pull request #18157: URL: https://github.com/apache/flink/pull/18157#issuecomment-998436589 ## CI report: * a9175f7c992d1cb30579dc88d4406c228e06a67a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29560) * dcebc46b47336a4f7e41003f091f4c4fb6a28bdc Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29605) * 8515ccb41dc778e38a97aa5cd7e80440a9ef389e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17503: [FLINK-24334][Deployment/Kubernetes] Configuration 'kubernetes.flink.log.dir' compatible
flinkbot edited a comment on pull request #17503: URL: https://github.com/apache/flink/pull/17503#issuecomment-945138587 ## CI report: * 4d30cc1db8de1a547c0908f6a634c647a6f4f69a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29586) * 2bad195fedf4b3acfa4067a029c63a7e60c3d65a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29604) * 491311a65ed7959663e9cf0db6890e8026e10bcc Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29608) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17503: [FLINK-24334][Deployment/Kubernetes] Configuration 'kubernetes.flink.log.dir' compatible
flinkbot edited a comment on pull request #17503: URL: https://github.com/apache/flink/pull/17503#issuecomment-945138587 ## CI report: * 4d30cc1db8de1a547c0908f6a634c647a6f4f69a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29586) * 2bad195fedf4b3acfa4067a029c63a7e60c3d65a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29604) * 491311a65ed7959663e9cf0db6890e8026e10bcc UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17503: [FLINK-24334][Deployment/Kubernetes] Configuration 'kubernetes.flink.log.dir' compatible
flinkbot edited a comment on pull request #17503: URL: https://github.com/apache/flink/pull/17503#issuecomment-945138587 ## CI report: * 4d30cc1db8de1a547c0908f6a634c647a6f4f69a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29586) * 2bad195fedf4b3acfa4067a029c63a7e60c3d65a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29604) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-25683) wrong result if table transfrom to DataStream then window process grouped to mini batch in batch mode
zhangzh created FLINK-25683: --- Summary: wrong result if table transfrom to DataStream then window process grouped to mini batch in batch mode Key: FLINK-25683 URL: https://issues.apache.org/jira/browse/FLINK-25683 Project: Flink Issue Type: Bug Components: Table SQL / API, Table SQL / Runtime Affects Versions: 1.14.2 Environment: mac book pro m1 jdk 8 scala 2.11 flink 1.14.2 idea 2020 Reporter: zhangzh Attachments: TableToDataStreamBatchWindowTest.scala, pom.xml -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #17503: [FLINK-24334][Deployment/Kubernetes] Configuration 'kubernetes.flink.log.dir' compatible
flinkbot edited a comment on pull request #17503: URL: https://github.com/apache/flink/pull/17503#issuecomment-945138587 ## CI report: * 4d30cc1db8de1a547c0908f6a634c647a6f4f69a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29586) * 2bad195fedf4b3acfa4067a029c63a7e60c3d65a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29604) * 491311a65ed7959663e9cf0db6890e8026e10bcc UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17503: [FLINK-24334][Deployment/Kubernetes] Configuration 'kubernetes.flink.log.dir' compatible
flinkbot edited a comment on pull request #17503: URL: https://github.com/apache/flink/pull/17503#issuecomment-945138587 ## CI report: * 4d30cc1db8de1a547c0908f6a634c647a6f4f69a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29586) * 2bad195fedf4b3acfa4067a029c63a7e60c3d65a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29604) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17503: [FLINK-24334][Deployment/Kubernetes] Configuration 'kubernetes.flink.log.dir' compatible
flinkbot edited a comment on pull request #17503: URL: https://github.com/apache/flink/pull/17503#issuecomment-945138587 ## CI report: * 4d30cc1db8de1a547c0908f6a634c647a6f4f69a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29586) * 2bad195fedf4b3acfa4067a029c63a7e60c3d65a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29604) * 491311a65ed7959663e9cf0db6890e8026e10bcc UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17503: [FLINK-24334][Deployment/Kubernetes] Configuration 'kubernetes.flink.log.dir' compatible
flinkbot edited a comment on pull request #17503: URL: https://github.com/apache/flink/pull/17503#issuecomment-945138587 ## CI report: * 4d30cc1db8de1a547c0908f6a634c647a6f4f69a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29586) * 2bad195fedf4b3acfa4067a029c63a7e60c3d65a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29604) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18385: [FLINK-22643][network] Reuse tpc connections between taskmanagers
flinkbot edited a comment on pull request #18385: URL: https://github.com/apache/flink/pull/18385#issuecomment-1015053663 ## CI report: * 1ee2f3b49616b0c1aaaf8d8f78d93c93ae6fc2db Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29607) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17503: [FLINK-24334][Deployment/Kubernetes] Configuration 'kubernetes.flink.log.dir' compatible
flinkbot edited a comment on pull request #17503: URL: https://github.com/apache/flink/pull/17503#issuecomment-945138587 ## CI report: * 4d30cc1db8de1a547c0908f6a634c647a6f4f69a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29586) * 2bad195fedf4b3acfa4067a029c63a7e60c3d65a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29604) * 491311a65ed7959663e9cf0db6890e8026e10bcc UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #18385: [FLINK-22643][network] Reuse tpc connections between taskmanagers
flinkbot commented on pull request #18385: URL: https://github.com/apache/flink/pull/18385#issuecomment-1015053663 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17503: [FLINK-24334][Deployment/Kubernetes] Configuration 'kubernetes.flink.log.dir' compatible
flinkbot edited a comment on pull request #17503: URL: https://github.com/apache/flink/pull/17503#issuecomment-945138587 ## CI report: * 4d30cc1db8de1a547c0908f6a634c647a6f4f69a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29586) * 2bad195fedf4b3acfa4067a029c63a7e60c3d65a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29604) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18384: [FLINK-22643][network] Reuse tpc connections between taskmanagers
flinkbot edited a comment on pull request #18384: URL: https://github.com/apache/flink/pull/18384#issuecomment-1015050232 ## CI report: * 3cbd0a1b5e27a8699edb9f7bb33e44f4aa1ab72d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29606) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] 1996fanrui opened a new pull request #18385: [FLINK-22643][network] Reuse tpc connections between taskmanagers
1996fanrui opened a new pull request #18385: URL: https://github.com/apache/flink/pull/18385 ## What is the purpose of the change Reuse tpc connections between taskmanagers ## Brief change log Reuse tpc connections between taskmanagers. ## Verifying this change Please make sure both new and modified tests in this PR follows the conventions defined in our code quality guide: https://flink.apache.org/contributing/code-style-and-quality-common.html#testing *(Please pick either of the following options)* This change added tests and can be verified as follows: Added test that PartitionRequestClientFactoryTest#testReuseNettyPartitionRequestClient. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? yes - If yes, how is the feature documented? JavaDocs -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17503: [FLINK-24334][Deployment/Kubernetes] Configuration 'kubernetes.flink.log.dir' compatible
flinkbot edited a comment on pull request #17503: URL: https://github.com/apache/flink/pull/17503#issuecomment-945138587 ## CI report: * 4d30cc1db8de1a547c0908f6a634c647a6f4f69a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29586) * 2bad195fedf4b3acfa4067a029c63a7e60c3d65a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29604) * 491311a65ed7959663e9cf0db6890e8026e10bcc UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #18384: [FLINK-22643][network] Reuse tpc connections between taskmanagers
flinkbot commented on pull request #18384: URL: https://github.com/apache/flink/pull/18384#issuecomment-1015051197 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 3cbd0a1b5e27a8699edb9f7bb33e44f4aa1ab72d (Tue Jan 18 04:02:03 UTC 2022) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
