[GitHub] [flink] flinkbot commented on pull request #18002: [FLINK-25150][API] Fix the violation of api annotation
flinkbot commented on pull request #18002: URL: https://github.com/apache/flink/pull/18002#issuecomment-985293724 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit ccaf8bc1a612ade65b53ab38bc71359f4b063c55 (Fri Dec 03 07:59:00 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-25150).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] matriv commented on a change in pull request #17962: [FLINK-22113][table-planner] Implement column uniqueness checking for TableSourceTable
matriv commented on a change in pull request #17962:
URL: https://github.com/apache/flink/pull/17962#discussion_r761716124
##
File path:
flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/plan/metadata/MetadataTestUtil.scala
##
@@ -298,6 +301,124 @@ object MetadataTestUtil {
)
}
+ private def createTableSourceTable1(): Table = {
+val catalogTable = CatalogTable.fromProperties(
+ Map(
Review comment:
Done, thx, I missed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Resolved] (FLINK-24859) Document new File formats
[ https://issues.apache.org/jira/browse/FLINK-24859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser resolved FLINK-24859. Fix Version/s: 1.15.0 Resolution: Fixed Resolved in master: [FLINK-24859][doc][formats] Make new formats name coherent 15ba358fd1281514cf9225e2966899f373c79080 [FLINK-24859][doc][formats] document text file reading 3999e56f72ca6f5ff44a83cdfeda5abd9b3dee23 [FLINK-24859][doc][formats] document parquet file reading f2ecabb994bc2a6f68deb97c507a5dd87f13b5d6 Address review comments 84a894248f2d88c9e0938bdaabd73a533bfd78c2 > Document new File formats > - > > Key: FLINK-24859 > URL: https://issues.apache.org/jira/browse/FLINK-24859 > Project: Flink > Issue Type: Technical Debt > Components: Documentation >Reporter: Etienne Chauchot >Assignee: Etienne Chauchot >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > > The project recently introduced new formats: _BulkFormat_ and _StreamFormat_ > interfaces. > There are already implementations of these formats: hive, parquet, orc and > textLine formats that need to be documented. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot commented on pull request #18002: [FLINK-25150][API] Fix the violation of api annotation
flinkbot commented on pull request #18002: URL: https://github.com/apache/flink/pull/18002#issuecomment-985293370 ## CI report: * ccaf8bc1a612ade65b53ab38bc71359f4b063c55 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-25150) org.apache.flink.connector.file.src.reader.TextLineInputFormat.createReader failed due to violate ApiAnnotationRules
[
https://issues.apache.org/jira/browse/FLINK-25150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-25150:
---
Labels: pull-request-available test-stability (was: test-stability)
> org.apache.flink.connector.file.src.reader.TextLineInputFormat.createReader
> failed due to violate ApiAnnotationRules
>
>
> Key: FLINK-25150
> URL: https://issues.apache.org/jira/browse/FLINK-25150
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Reporter: Yun Gao
>Priority: Critical
> Labels: pull-request-available, test-stability
>
> {code:java}
> 021-12-02T21:44:11.4722598Z Dec 02 21:44:11 [ERROR] Tests run: 4, Failures:
> 1, Errors: 0, Skipped: 0, Time elapsed: 38.799 s <<< FAILURE! - in
> org.apache.flink.architecture.rules.ApiAnnotationRules
> 2021-12-02T21:44:11.4724167Z Dec 02 21:44:11 [ERROR]
> ApiAnnotationRules.PUBLIC_EVOLVING_API_METHODS_USE_ONLY_PUBLIC_EVOLVING_API_TYPES
> Time elapsed: 0.231 s <<< FAILURE!
> 2021-12-02T21:44:11.4725222Z Dec 02 21:44:11 java.lang.AssertionError:
> 2021-12-02T21:44:11.4726849Z Dec 02 21:44:11 Architecture Violation
> [Priority: MEDIUM] - Rule 'Return and argument types of methods annotated
> with @PublicEvolving must be annotated with @Public(Evolving).' was violated
> (2 times):
> 2021-12-02T21:44:11.4730100Z Dec 02 21:44:11
> org.apache.flink.connector.file.src.reader.TextLineInputFormat.createReader(org.apache.flink.configuration.Configuration,
> org.apache.flink.core.fs.FSDataInputStream): Returned leaf type
> org.apache.flink.connector.file.src.reader.StreamFormat$Reader does not
> satisfy: reside outside of package 'org.apache.flink..' or annotated with
> @Public or annotated with @PublicEvolving or annotated with @Deprecated
> 2021-12-02T21:44:11.4734006Z Dec 02 21:44:11
> org.apache.flink.connector.file.src.reader.TextLineInputFormat.createReader(org.apache.flink.configuration.Configuration,
> org.apache.flink.core.fs.FSDataInputStream): Returned leaf type
> org.apache.flink.connector.file.src.reader.TextLineInputFormat$Reader does
> not satisfy: reside outside of package 'org.apache.flink..' or annotated with
> @Public or annotated with @PublicEvolving or annotated with @Deprecated
> 2021-12-02T21:44:11.4736377Z Dec 02 21:44:11 at
> com.tngtech.archunit.lang.ArchRule$Assertions.assertNoViolation(ArchRule.java:94)
> 2021-12-02T21:44:11.4737400Z Dec 02 21:44:11 at
> com.tngtech.archunit.lang.ArchRule$Assertions.check(ArchRule.java:82)
> 2021-12-02T21:44:11.4738529Z Dec 02 21:44:11 at
> com.tngtech.archunit.library.freeze.FreezingArchRule.check(FreezingArchRule.java:96)
> 2021-12-02T21:44:11.4739712Z Dec 02 21:44:11 at
> com.tngtech.archunit.junit.ArchUnitTestDescriptor$ArchUnitRuleDescriptor.execute(ArchUnitTestDescriptor.java:159)
> 2021-12-02T21:44:11.4740983Z Dec 02 21:44:11 at
> com.tngtech.archunit.junit.ArchUnitTestDescriptor$ArchUnitRuleDescriptor.execute(ArchUnitTestDescriptor.java:142)
> 2021-12-02T21:44:11.4742309Z Dec 02 21:44:11 at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
> 2021-12-02T21:44:11.4743532Z Dec 02 21:44:11 at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> 2021-12-02T21:44:11.4744736Z Dec 02 21:44:11 at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
> 2021-12-02T21:44:11.4745846Z Dec 02 21:44:11 at
> org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
> 2021-12-02T21:44:11.4747137Z Dec 02 21:44:11 at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
> 2021-12-02T21:44:11.4748480Z Dec 02 21:44:11 at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> 2021-12-02T21:44:11.4749659Z Dec 02 21:44:11 at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
> 2021-12-02T21:44:11.4750777Z Dec 02 21:44:11 at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
> 2021-12-02T21:44:11.4751822Z Dec 02 21:44:11 at
> java.util.ArrayList.forEach(ArrayList.java:1259)
> 2021-12-02T21:44:11.4752983Z Dec 02 21:44:11 at
> org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
> 2021-12-02T21:44:11.4754345Z Dec 02 21:44:11 at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
> 2021-12-02T21:44:11.470Z Dec 02 21:44:11 at
[GitHub] [flink] gaoyunhaii opened a new pull request #18002: [FLINK-25150][API] Fix the violation of api annotation
gaoyunhaii opened a new pull request #18002: URL: https://github.com/apache/flink/pull/18002 ## What is the purpose of the change This PR tries to fix the api annotation violation. ## Brief change log - ccaf8bc1a612ade65b53ab38bc71359f4b063c55 adds the @PublicEvolving to the `org.apache.flink.connector.file.src.reader.StreamFormat$Reader`. ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): **no** - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: **no** - The serializers: **no** - The runtime per-record code paths (performance sensitive): **no** - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: **no** - The S3 file system connector: **no** ## Documentation - Does this pull request introduce a new feature? **no** - If yes, how is the feature documented? **not applicable** -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-25150) org.apache.flink.connector.file.src.reader.TextLineInputFormat.createReader failed due to violate ApiAnnotationRules
[
https://issues.apache.org/jira/browse/FLINK-25150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17452802#comment-17452802
]
Yun Gao commented on FLINK-25150:
-
I'll first fix the violation.
> org.apache.flink.connector.file.src.reader.TextLineInputFormat.createReader
> failed due to violate ApiAnnotationRules
>
>
> Key: FLINK-25150
> URL: https://issues.apache.org/jira/browse/FLINK-25150
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Reporter: Yun Gao
>Priority: Critical
> Labels: test-stability
>
> {code:java}
> 021-12-02T21:44:11.4722598Z Dec 02 21:44:11 [ERROR] Tests run: 4, Failures:
> 1, Errors: 0, Skipped: 0, Time elapsed: 38.799 s <<< FAILURE! - in
> org.apache.flink.architecture.rules.ApiAnnotationRules
> 2021-12-02T21:44:11.4724167Z Dec 02 21:44:11 [ERROR]
> ApiAnnotationRules.PUBLIC_EVOLVING_API_METHODS_USE_ONLY_PUBLIC_EVOLVING_API_TYPES
> Time elapsed: 0.231 s <<< FAILURE!
> 2021-12-02T21:44:11.4725222Z Dec 02 21:44:11 java.lang.AssertionError:
> 2021-12-02T21:44:11.4726849Z Dec 02 21:44:11 Architecture Violation
> [Priority: MEDIUM] - Rule 'Return and argument types of methods annotated
> with @PublicEvolving must be annotated with @Public(Evolving).' was violated
> (2 times):
> 2021-12-02T21:44:11.4730100Z Dec 02 21:44:11
> org.apache.flink.connector.file.src.reader.TextLineInputFormat.createReader(org.apache.flink.configuration.Configuration,
> org.apache.flink.core.fs.FSDataInputStream): Returned leaf type
> org.apache.flink.connector.file.src.reader.StreamFormat$Reader does not
> satisfy: reside outside of package 'org.apache.flink..' or annotated with
> @Public or annotated with @PublicEvolving or annotated with @Deprecated
> 2021-12-02T21:44:11.4734006Z Dec 02 21:44:11
> org.apache.flink.connector.file.src.reader.TextLineInputFormat.createReader(org.apache.flink.configuration.Configuration,
> org.apache.flink.core.fs.FSDataInputStream): Returned leaf type
> org.apache.flink.connector.file.src.reader.TextLineInputFormat$Reader does
> not satisfy: reside outside of package 'org.apache.flink..' or annotated with
> @Public or annotated with @PublicEvolving or annotated with @Deprecated
> 2021-12-02T21:44:11.4736377Z Dec 02 21:44:11 at
> com.tngtech.archunit.lang.ArchRule$Assertions.assertNoViolation(ArchRule.java:94)
> 2021-12-02T21:44:11.4737400Z Dec 02 21:44:11 at
> com.tngtech.archunit.lang.ArchRule$Assertions.check(ArchRule.java:82)
> 2021-12-02T21:44:11.4738529Z Dec 02 21:44:11 at
> com.tngtech.archunit.library.freeze.FreezingArchRule.check(FreezingArchRule.java:96)
> 2021-12-02T21:44:11.4739712Z Dec 02 21:44:11 at
> com.tngtech.archunit.junit.ArchUnitTestDescriptor$ArchUnitRuleDescriptor.execute(ArchUnitTestDescriptor.java:159)
> 2021-12-02T21:44:11.4740983Z Dec 02 21:44:11 at
> com.tngtech.archunit.junit.ArchUnitTestDescriptor$ArchUnitRuleDescriptor.execute(ArchUnitTestDescriptor.java:142)
> 2021-12-02T21:44:11.4742309Z Dec 02 21:44:11 at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
> 2021-12-02T21:44:11.4743532Z Dec 02 21:44:11 at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> 2021-12-02T21:44:11.4744736Z Dec 02 21:44:11 at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
> 2021-12-02T21:44:11.4745846Z Dec 02 21:44:11 at
> org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
> 2021-12-02T21:44:11.4747137Z Dec 02 21:44:11 at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
> 2021-12-02T21:44:11.4748480Z Dec 02 21:44:11 at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> 2021-12-02T21:44:11.4749659Z Dec 02 21:44:11 at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
> 2021-12-02T21:44:11.4750777Z Dec 02 21:44:11 at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
> 2021-12-02T21:44:11.4751822Z Dec 02 21:44:11 at
> java.util.ArrayList.forEach(ArrayList.java:1259)
> 2021-12-02T21:44:11.4752983Z Dec 02 21:44:11 at
> org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
> 2021-12-02T21:44:11.4754345Z Dec 02 21:44:11 at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
> 2021-12-02T21:44:11.470Z Dec 02 21:44:11 at
>
[jira] [Closed] (FLINK-25151) FLIP-188: Introduce Built-in Dynamic Table Storage
[ https://issues.apache.org/jira/browse/FLINK-25151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-25151. Resolution: Duplicate Network problem, duplicated, see FLINK-25152 > FLIP-188: Introduce Built-in Dynamic Table Storage > -- > > Key: FLINK-25151 > URL: https://issues.apache.org/jira/browse/FLINK-25151 > Project: Flink > Issue Type: New Feature > Components: Table SQL / API, Table SQL / Ecosystem, Table SQL / > Planner >Reporter: Jingsong Lee >Priority: Major > Fix For: 1.15.0 > > > Introduce built-in storage support for dynamic table, a truly unified > changelog & table representation, from Flink SQL’s perspective. The storage > will improve the usability a lot. > More detail see: > [https://cwiki.apache.org/confluence/display/FLINK/FLIP-188%3A+Introduce+Built-in+Dynamic+Table+Storage] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-25152) [DISCUSS] FLIP-188: Introduce Built-in Dynamic Table Storage
Jingsong Lee created FLINK-25152: Summary: [DISCUSS] FLIP-188: Introduce Built-in Dynamic Table Storage Key: FLINK-25152 URL: https://issues.apache.org/jira/browse/FLINK-25152 Project: Flink Issue Type: New Feature Components: Table SQL / API, Table SQL / Ecosystem Reporter: Jingsong Lee Fix For: 1.15.0 introduce built-in storage support for dynamic table, a truly unified changelog & table representation, from Flink SQL’s perspective. tHE storage will improve the usability a lot. More detail see: https://cwiki.apache.org/confluence/display/FLINK/FLIP-188%3A+Introduce+Built-in+Dynamic+Table+Storage -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-25152) FLIP-188: Introduce Built-in Dynamic Table Storage
[ https://issues.apache.org/jira/browse/FLINK-25152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee updated FLINK-25152: - Summary: FLIP-188: Introduce Built-in Dynamic Table Storage (was: [DISCUSS] FLIP-188: Introduce Built-in Dynamic Table Storage) > FLIP-188: Introduce Built-in Dynamic Table Storage > -- > > Key: FLINK-25152 > URL: https://issues.apache.org/jira/browse/FLINK-25152 > Project: Flink > Issue Type: New Feature > Components: Table SQL / API, Table SQL / Ecosystem >Reporter: Jingsong Lee >Priority: Major > Fix For: 1.15.0 > > > introduce built-in storage support for dynamic table, a truly unified > changelog & table representation, from Flink SQL’s perspective. tHE storage > will improve the usability a lot. > More detail see: > https://cwiki.apache.org/confluence/display/FLINK/FLIP-188%3A+Introduce+Built-in+Dynamic+Table+Storage -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-25151) FLIP-188: Introduce Built-in Dynamic Table Storage
Jingsong Lee created FLINK-25151: Summary: FLIP-188: Introduce Built-in Dynamic Table Storage Key: FLINK-25151 URL: https://issues.apache.org/jira/browse/FLINK-25151 Project: Flink Issue Type: New Feature Components: Table SQL / API, Table SQL / Ecosystem, Table SQL / Planner Reporter: Jingsong Lee Fix For: 1.15.0 Introduce built-in storage support for dynamic table, a truly unified changelog & table representation, from Flink SQL’s perspective. The storage will improve the usability a lot. More detail see: [https://cwiki.apache.org/confluence/display/FLINK/FLIP-188%3A+Introduce+Built-in+Dynamic+Table+Storage] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #17958: FLINK-25029: Hadoop Caller Context Setting in Flink
flinkbot edited a comment on pull request #17958: URL: https://github.com/apache/flink/pull/17958#issuecomment-982461620 ## CI report: * 8377539867f18f1bda4fba411f961558f926ee7e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27267) * 7f59432fc9130ed007411f2be331cb58f12d14f8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27495) * 8235aed4e30f0bc4388cd3b4d92b5c868dc154e0 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27497) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-25150) org.apache.flink.connector.file.src.reader.TextLineInputFormat.createReader failed due to violate ApiAnnotationRules
[
https://issues.apache.org/jira/browse/FLINK-25150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17452793#comment-17452793
]
Yun Gao commented on FLINK-25150:
-
Hi [~airblader] , it seems the method does violate the rules, but may I double
confirm that it seems the tests has enabled for some days and the related code
seems also have no change, so why the issue starts to pop up from today?
> org.apache.flink.connector.file.src.reader.TextLineInputFormat.createReader
> failed due to violate ApiAnnotationRules
>
>
> Key: FLINK-25150
> URL: https://issues.apache.org/jira/browse/FLINK-25150
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Reporter: Yun Gao
>Priority: Critical
> Labels: test-stability
>
> {code:java}
> 021-12-02T21:44:11.4722598Z Dec 02 21:44:11 [ERROR] Tests run: 4, Failures:
> 1, Errors: 0, Skipped: 0, Time elapsed: 38.799 s <<< FAILURE! - in
> org.apache.flink.architecture.rules.ApiAnnotationRules
> 2021-12-02T21:44:11.4724167Z Dec 02 21:44:11 [ERROR]
> ApiAnnotationRules.PUBLIC_EVOLVING_API_METHODS_USE_ONLY_PUBLIC_EVOLVING_API_TYPES
> Time elapsed: 0.231 s <<< FAILURE!
> 2021-12-02T21:44:11.4725222Z Dec 02 21:44:11 java.lang.AssertionError:
> 2021-12-02T21:44:11.4726849Z Dec 02 21:44:11 Architecture Violation
> [Priority: MEDIUM] - Rule 'Return and argument types of methods annotated
> with @PublicEvolving must be annotated with @Public(Evolving).' was violated
> (2 times):
> 2021-12-02T21:44:11.4730100Z Dec 02 21:44:11
> org.apache.flink.connector.file.src.reader.TextLineInputFormat.createReader(org.apache.flink.configuration.Configuration,
> org.apache.flink.core.fs.FSDataInputStream): Returned leaf type
> org.apache.flink.connector.file.src.reader.StreamFormat$Reader does not
> satisfy: reside outside of package 'org.apache.flink..' or annotated with
> @Public or annotated with @PublicEvolving or annotated with @Deprecated
> 2021-12-02T21:44:11.4734006Z Dec 02 21:44:11
> org.apache.flink.connector.file.src.reader.TextLineInputFormat.createReader(org.apache.flink.configuration.Configuration,
> org.apache.flink.core.fs.FSDataInputStream): Returned leaf type
> org.apache.flink.connector.file.src.reader.TextLineInputFormat$Reader does
> not satisfy: reside outside of package 'org.apache.flink..' or annotated with
> @Public or annotated with @PublicEvolving or annotated with @Deprecated
> 2021-12-02T21:44:11.4736377Z Dec 02 21:44:11 at
> com.tngtech.archunit.lang.ArchRule$Assertions.assertNoViolation(ArchRule.java:94)
> 2021-12-02T21:44:11.4737400Z Dec 02 21:44:11 at
> com.tngtech.archunit.lang.ArchRule$Assertions.check(ArchRule.java:82)
> 2021-12-02T21:44:11.4738529Z Dec 02 21:44:11 at
> com.tngtech.archunit.library.freeze.FreezingArchRule.check(FreezingArchRule.java:96)
> 2021-12-02T21:44:11.4739712Z Dec 02 21:44:11 at
> com.tngtech.archunit.junit.ArchUnitTestDescriptor$ArchUnitRuleDescriptor.execute(ArchUnitTestDescriptor.java:159)
> 2021-12-02T21:44:11.4740983Z Dec 02 21:44:11 at
> com.tngtech.archunit.junit.ArchUnitTestDescriptor$ArchUnitRuleDescriptor.execute(ArchUnitTestDescriptor.java:142)
> 2021-12-02T21:44:11.4742309Z Dec 02 21:44:11 at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
> 2021-12-02T21:44:11.4743532Z Dec 02 21:44:11 at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> 2021-12-02T21:44:11.4744736Z Dec 02 21:44:11 at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
> 2021-12-02T21:44:11.4745846Z Dec 02 21:44:11 at
> org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
> 2021-12-02T21:44:11.4747137Z Dec 02 21:44:11 at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
> 2021-12-02T21:44:11.4748480Z Dec 02 21:44:11 at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> 2021-12-02T21:44:11.4749659Z Dec 02 21:44:11 at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
> 2021-12-02T21:44:11.4750777Z Dec 02 21:44:11 at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
> 2021-12-02T21:44:11.4751822Z Dec 02 21:44:11 at
> java.util.ArrayList.forEach(ArrayList.java:1259)
> 2021-12-02T21:44:11.4752983Z Dec 02 21:44:11 at
> org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
> 2021-12-02T21:44:11.4754345Z Dec 02 21:44:11 at
>
[GitHub] [flink] flinkbot edited a comment on pull request #17958: FLINK-25029: Hadoop Caller Context Setting in Flink
flinkbot edited a comment on pull request #17958: URL: https://github.com/apache/flink/pull/17958#issuecomment-982461620 ## CI report: * 8377539867f18f1bda4fba411f961558f926ee7e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27267) * 7f59432fc9130ed007411f2be331cb58f12d14f8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27495) * 8235aed4e30f0bc4388cd3b4d92b5c868dc154e0 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] JingsongLi commented on a change in pull request #17542: [FLINK-17782] Add array,map,row types support for parquet row writer
JingsongLi commented on a change in pull request #17542:
URL: https://github.com/apache/flink/pull/17542#discussion_r761700468
##
File path:
flink-formats/flink-parquet/src/test/java/org/apache/flink/formats/parquet/row/ParquetRowDataWriterTest.java
##
@@ -158,6 +190,65 @@ private void innerTest(Configuration conf, boolean
utcTimestamp) throws IOExcept
Assert.assertEquals(number, cnt);
}
+public void complexTypeTest(Configuration conf, boolean utcTimestamp)
throws Exception {
+Path path = new Path(TEMPORARY_FOLDER.newFolder().getPath(),
UUID.randomUUID().toString());
+int number = 1000;
+List rows = new ArrayList<>(number);
+Map mapData = new HashMap<>();
+mapData.put("k1", "v1");
+mapData.put(null, "v2");
+mapData.put("k2", null);
+
+for (int i = 0; i < number; i++) {
+Integer v = i;
+rows.add(Row.of(new Integer[] {v}, mapData,
Row.of(String.valueOf(v), v)));
+}
+
+ParquetWriterFactory factory =
+ParquetRowDataBuilder.createWriterFactory(ROW_TYPE_COMPLEX,
conf, utcTimestamp);
+BulkWriter writer =
+factory.create(path.getFileSystem().create(path,
FileSystem.WriteMode.OVERWRITE));
+for (int i = 0; i < number; i++) {
+writer.addElement(CONVERTER_COMPLEX.toInternal(rows.get(i)));
+}
+writer.flush();
+writer.finish();
+
+File file = new File(path.getPath());
+final List fileContent = readParquetFile(file);
+assertEquals(rows, fileContent);
+}
+
+private static List readParquetFile(File file) throws IOException {
+InputFile inFile =
+HadoopInputFile.fromPath(
+new org.apache.hadoop.fs.Path(file.toURI()), new
Configuration());
+
+ArrayList results = new ArrayList<>();
+try (ParquetReader reader =
+AvroParquetReader.builder(inFile).build()) {
+GenericRecord next;
+while ((next = reader.read()) != null) {
+Integer c0 = (Integer) ((ArrayList)
next.get(0)).get(0).get(0);
+HashMap map = ((HashMap) next.get(1));
+String c21 = ((GenericData.Record)
next.get(2)).get(0).toString();
+Integer c22 = (Integer) ((GenericData.Record)
next.get(2)).get(1);
+
+Map c1 = new HashMap<>();
+for (Utf8 key : map.keySet()) {
+String k = Strings.isEmpty(key) ? null : key.toString();
Review comment:
> avro GenericRecord will auto convert null to empty string.
Do you have a reference stating this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] JingsongLi commented on a change in pull request #17542: [FLINK-17782] Add array,map,row types support for parquet row writer
JingsongLi commented on a change in pull request #17542:
URL: https://github.com/apache/flink/pull/17542#discussion_r761699867
##
File path:
flink-formats/flink-parquet/src/test/java/org/apache/flink/formats/parquet/row/ParquetRowDataWriterTest.java
##
@@ -158,6 +190,65 @@ private void innerTest(Configuration conf, boolean
utcTimestamp) throws IOExcept
Assert.assertEquals(number, cnt);
}
+public void complexTypeTest(Configuration conf, boolean utcTimestamp)
throws Exception {
+Path path = new Path(TEMPORARY_FOLDER.newFolder().getPath(),
UUID.randomUUID().toString());
+int number = 1000;
+List rows = new ArrayList<>(number);
+Map mapData = new HashMap<>();
+mapData.put("k1", "v1");
+mapData.put(null, "v2");
+mapData.put("k2", null);
+
+for (int i = 0; i < number; i++) {
+Integer v = i;
+rows.add(Row.of(new Integer[] {v}, mapData,
Row.of(String.valueOf(v), v)));
+}
+
+ParquetWriterFactory factory =
+ParquetRowDataBuilder.createWriterFactory(ROW_TYPE_COMPLEX,
conf, utcTimestamp);
+BulkWriter writer =
+factory.create(path.getFileSystem().create(path,
FileSystem.WriteMode.OVERWRITE));
+for (int i = 0; i < number; i++) {
+writer.addElement(CONVERTER_COMPLEX.toInternal(rows.get(i)));
+}
+writer.flush();
+writer.finish();
+
+File file = new File(path.getPath());
+final List fileContent = readParquetFile(file);
+assertEquals(rows, fileContent);
+}
+
+private static List readParquetFile(File file) throws IOException {
+InputFile inFile =
+HadoopInputFile.fromPath(
+new org.apache.hadoop.fs.Path(file.toURI()), new
Configuration());
+
+ArrayList results = new ArrayList<>();
+try (ParquetReader reader =
+AvroParquetReader.builder(inFile).build()) {
+GenericRecord next;
+while ((next = reader.read()) != null) {
+Integer c0 = (Integer) ((ArrayList)
next.get(0)).get(0).get(0);
+HashMap map = ((HashMap) next.get(1));
+String c21 = ((GenericData.Record)
next.get(2)).get(0).toString();
+Integer c22 = (Integer) ((GenericData.Record)
next.get(2)).get(1);
+
+Map c1 = new HashMap<>();
+for (Utf8 key : map.keySet()) {
+String k = Strings.isEmpty(key) ? null : key.toString();
Review comment:
I change `Strings.isEmpty(key)` to `key == null`, the tests passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-25150) org.apache.flink.connector.file.src.reader.TextLineInputFormat.createReader failed due to violate ApiAnnotationRules
[
https://issues.apache.org/jira/browse/FLINK-25150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-25150:
Priority: Critical (was: Major)
> org.apache.flink.connector.file.src.reader.TextLineInputFormat.createReader
> failed due to violate ApiAnnotationRules
>
>
> Key: FLINK-25150
> URL: https://issues.apache.org/jira/browse/FLINK-25150
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Reporter: Yun Gao
>Priority: Critical
> Labels: test-stability
>
> {code:java}
> 021-12-02T21:44:11.4722598Z Dec 02 21:44:11 [ERROR] Tests run: 4, Failures:
> 1, Errors: 0, Skipped: 0, Time elapsed: 38.799 s <<< FAILURE! - in
> org.apache.flink.architecture.rules.ApiAnnotationRules
> 2021-12-02T21:44:11.4724167Z Dec 02 21:44:11 [ERROR]
> ApiAnnotationRules.PUBLIC_EVOLVING_API_METHODS_USE_ONLY_PUBLIC_EVOLVING_API_TYPES
> Time elapsed: 0.231 s <<< FAILURE!
> 2021-12-02T21:44:11.4725222Z Dec 02 21:44:11 java.lang.AssertionError:
> 2021-12-02T21:44:11.4726849Z Dec 02 21:44:11 Architecture Violation
> [Priority: MEDIUM] - Rule 'Return and argument types of methods annotated
> with @PublicEvolving must be annotated with @Public(Evolving).' was violated
> (2 times):
> 2021-12-02T21:44:11.4730100Z Dec 02 21:44:11
> org.apache.flink.connector.file.src.reader.TextLineInputFormat.createReader(org.apache.flink.configuration.Configuration,
> org.apache.flink.core.fs.FSDataInputStream): Returned leaf type
> org.apache.flink.connector.file.src.reader.StreamFormat$Reader does not
> satisfy: reside outside of package 'org.apache.flink..' or annotated with
> @Public or annotated with @PublicEvolving or annotated with @Deprecated
> 2021-12-02T21:44:11.4734006Z Dec 02 21:44:11
> org.apache.flink.connector.file.src.reader.TextLineInputFormat.createReader(org.apache.flink.configuration.Configuration,
> org.apache.flink.core.fs.FSDataInputStream): Returned leaf type
> org.apache.flink.connector.file.src.reader.TextLineInputFormat$Reader does
> not satisfy: reside outside of package 'org.apache.flink..' or annotated with
> @Public or annotated with @PublicEvolving or annotated with @Deprecated
> 2021-12-02T21:44:11.4736377Z Dec 02 21:44:11 at
> com.tngtech.archunit.lang.ArchRule$Assertions.assertNoViolation(ArchRule.java:94)
> 2021-12-02T21:44:11.4737400Z Dec 02 21:44:11 at
> com.tngtech.archunit.lang.ArchRule$Assertions.check(ArchRule.java:82)
> 2021-12-02T21:44:11.4738529Z Dec 02 21:44:11 at
> com.tngtech.archunit.library.freeze.FreezingArchRule.check(FreezingArchRule.java:96)
> 2021-12-02T21:44:11.4739712Z Dec 02 21:44:11 at
> com.tngtech.archunit.junit.ArchUnitTestDescriptor$ArchUnitRuleDescriptor.execute(ArchUnitTestDescriptor.java:159)
> 2021-12-02T21:44:11.4740983Z Dec 02 21:44:11 at
> com.tngtech.archunit.junit.ArchUnitTestDescriptor$ArchUnitRuleDescriptor.execute(ArchUnitTestDescriptor.java:142)
> 2021-12-02T21:44:11.4742309Z Dec 02 21:44:11 at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
> 2021-12-02T21:44:11.4743532Z Dec 02 21:44:11 at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> 2021-12-02T21:44:11.4744736Z Dec 02 21:44:11 at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
> 2021-12-02T21:44:11.4745846Z Dec 02 21:44:11 at
> org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
> 2021-12-02T21:44:11.4747137Z Dec 02 21:44:11 at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
> 2021-12-02T21:44:11.4748480Z Dec 02 21:44:11 at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> 2021-12-02T21:44:11.4749659Z Dec 02 21:44:11 at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
> 2021-12-02T21:44:11.4750777Z Dec 02 21:44:11 at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
> 2021-12-02T21:44:11.4751822Z Dec 02 21:44:11 at
> java.util.ArrayList.forEach(ArrayList.java:1259)
> 2021-12-02T21:44:11.4752983Z Dec 02 21:44:11 at
> org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
> 2021-12-02T21:44:11.4754345Z Dec 02 21:44:11 at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
> 2021-12-02T21:44:11.470Z Dec 02 21:44:11 at
>
[GitHub] [flink] flinkbot edited a comment on pull request #17958: FLINK-25029: Hadoop Caller Context Setting in Flink
flinkbot edited a comment on pull request #17958: URL: https://github.com/apache/flink/pull/17958#issuecomment-982461620 ## CI report: * 8377539867f18f1bda4fba411f961558f926ee7e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27267) * 7f59432fc9130ed007411f2be331cb58f12d14f8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27495) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-25150) org.apache.flink.connector.file.src.reader.TextLineInputFormat.createReader failed due to violate ApiAnnotationRules
Yun Gao created FLINK-25150:
---
Summary:
org.apache.flink.connector.file.src.reader.TextLineInputFormat.createReader
failed due to violate ApiAnnotationRules
Key: FLINK-25150
URL: https://issues.apache.org/jira/browse/FLINK-25150
Project: Flink
Issue Type: Bug
Components: API / DataStream
Reporter: Yun Gao
{code:java}
021-12-02T21:44:11.4722598Z Dec 02 21:44:11 [ERROR] Tests run: 4, Failures: 1,
Errors: 0, Skipped: 0, Time elapsed: 38.799 s <<< FAILURE! - in
org.apache.flink.architecture.rules.ApiAnnotationRules
2021-12-02T21:44:11.4724167Z Dec 02 21:44:11 [ERROR]
ApiAnnotationRules.PUBLIC_EVOLVING_API_METHODS_USE_ONLY_PUBLIC_EVOLVING_API_TYPES
Time elapsed: 0.231 s <<< FAILURE!
2021-12-02T21:44:11.4725222Z Dec 02 21:44:11 java.lang.AssertionError:
2021-12-02T21:44:11.4726849Z Dec 02 21:44:11 Architecture Violation [Priority:
MEDIUM] - Rule 'Return and argument types of methods annotated with
@PublicEvolving must be annotated with @Public(Evolving).' was violated (2
times):
2021-12-02T21:44:11.4730100Z Dec 02 21:44:11
org.apache.flink.connector.file.src.reader.TextLineInputFormat.createReader(org.apache.flink.configuration.Configuration,
org.apache.flink.core.fs.FSDataInputStream): Returned leaf type
org.apache.flink.connector.file.src.reader.StreamFormat$Reader does not
satisfy: reside outside of package 'org.apache.flink..' or annotated with
@Public or annotated with @PublicEvolving or annotated with @Deprecated
2021-12-02T21:44:11.4734006Z Dec 02 21:44:11
org.apache.flink.connector.file.src.reader.TextLineInputFormat.createReader(org.apache.flink.configuration.Configuration,
org.apache.flink.core.fs.FSDataInputStream): Returned leaf type
org.apache.flink.connector.file.src.reader.TextLineInputFormat$Reader does not
satisfy: reside outside of package 'org.apache.flink..' or annotated with
@Public or annotated with @PublicEvolving or annotated with @Deprecated
2021-12-02T21:44:11.4736377Z Dec 02 21:44:11at
com.tngtech.archunit.lang.ArchRule$Assertions.assertNoViolation(ArchRule.java:94)
2021-12-02T21:44:11.4737400Z Dec 02 21:44:11at
com.tngtech.archunit.lang.ArchRule$Assertions.check(ArchRule.java:82)
2021-12-02T21:44:11.4738529Z Dec 02 21:44:11at
com.tngtech.archunit.library.freeze.FreezingArchRule.check(FreezingArchRule.java:96)
2021-12-02T21:44:11.4739712Z Dec 02 21:44:11at
com.tngtech.archunit.junit.ArchUnitTestDescriptor$ArchUnitRuleDescriptor.execute(ArchUnitTestDescriptor.java:159)
2021-12-02T21:44:11.4740983Z Dec 02 21:44:11at
com.tngtech.archunit.junit.ArchUnitTestDescriptor$ArchUnitRuleDescriptor.execute(ArchUnitTestDescriptor.java:142)
2021-12-02T21:44:11.4742309Z Dec 02 21:44:11at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
2021-12-02T21:44:11.4743532Z Dec 02 21:44:11at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
2021-12-02T21:44:11.4744736Z Dec 02 21:44:11at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
2021-12-02T21:44:11.4745846Z Dec 02 21:44:11at
org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
2021-12-02T21:44:11.4747137Z Dec 02 21:44:11at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
2021-12-02T21:44:11.4748480Z Dec 02 21:44:11at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
2021-12-02T21:44:11.4749659Z Dec 02 21:44:11at
org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
2021-12-02T21:44:11.4750777Z Dec 02 21:44:11at
org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
2021-12-02T21:44:11.4751822Z Dec 02 21:44:11at
java.util.ArrayList.forEach(ArrayList.java:1259)
2021-12-02T21:44:11.4752983Z Dec 02 21:44:11at
org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
2021-12-02T21:44:11.4754345Z Dec 02 21:44:11at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
2021-12-02T21:44:11.470Z Dec 02 21:44:11at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
2021-12-02T21:44:11.4756753Z Dec 02 21:44:11at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
2021-12-02T21:44:11.4757869Z Dec 02 21:44:11at
org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
2021-12-02T21:44:11.4759122Z Dec 02 21:44:11at
[GitHub] [flink] flinkbot edited a comment on pull request #17958: FLINK-25029: Hadoop Caller Context Setting in Flink
flinkbot edited a comment on pull request #17958: URL: https://github.com/apache/flink/pull/17958#issuecomment-982461620 ## CI report: * 8377539867f18f1bda4fba411f961558f926ee7e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27267) * 7f59432fc9130ed007411f2be331cb58f12d14f8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27495) * 8235aed4e30f0bc4388cd3b4d92b5c868dc154e0 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17919: [FLINK-24419][table-planner] Trim to precision when casting to BINARY/VARBINARY
flinkbot edited a comment on pull request #17919: URL: https://github.com/apache/flink/pull/17919#issuecomment-979659835 ## CI report: * bd5b6272b266dfe8dc494ee6decac510d1c73f2b Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27487) * 56515bd322459d965610005e3f3da140aa9ce1a7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27494) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17958: FLINK-25029: Hadoop Caller Context Setting in Flink
flinkbot edited a comment on pull request #17958: URL: https://github.com/apache/flink/pull/17958#issuecomment-982461620 ## CI report: * 8377539867f18f1bda4fba411f961558f926ee7e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27267) * 7f59432fc9130ed007411f2be331cb58f12d14f8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27495) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17958: FLINK-25029: Hadoop Caller Context Setting in Flink
flinkbot edited a comment on pull request #17958: URL: https://github.com/apache/flink/pull/17958#issuecomment-982461620 ## CI report: * 8377539867f18f1bda4fba411f961558f926ee7e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27267) * 7f59432fc9130ed007411f2be331cb58f12d14f8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27495) * 8235aed4e30f0bc4388cd3b4d92b5c868dc154e0 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-25149) getScanRuntimeProvider method called before the pushDown method,
[
https://issues.apache.org/jira/browse/FLINK-25149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17452788#comment-17452788
]
Wenlong Lyu commented on FLINK-25149:
-
[~waittting] currently the table planner will validate the DynamicTableSource
before doing the optimization, you should make sure that the original table
source created by factory can work well without push down.
> getScanRuntimeProvider method called before the pushDown method,
>
>
> Key: FLINK-25149
> URL: https://issues.apache.org/jira/browse/FLINK-25149
> Project: Flink
> Issue Type: Bug
>Reporter: waittting
>Priority: Major
>
> ```
> public class CCDynamicTableSource implements ScanTableSource,
> SupportsLimitPushDown, SupportsFilterPushDown, SupportsProjectionPushDown {}
> ```
> Why is the getScanRuntimeProvider method called before the pushDown method,
> so that the parameters I get in the pushDown are not available in the
> getScanRuntimeProvider?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #17958: FLINK-25029: Hadoop Caller Context Setting in Flink
flinkbot edited a comment on pull request #17958: URL: https://github.com/apache/flink/pull/17958#issuecomment-982461620 ## CI report: * 8377539867f18f1bda4fba411f961558f926ee7e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27267) * 7f59432fc9130ed007411f2be331cb58f12d14f8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27495) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17892: [FLINK-25038][testutils] Refactor FlinkContainer to split JM and TMs to individual containers and support HA
flinkbot edited a comment on pull request #17892: URL: https://github.com/apache/flink/pull/17892#issuecomment-977710431 ## CI report: * 0c477cc849835f21dace860974ff6ebe61740893 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27492) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17958: FLINK-25029: Hadoop Caller Context Setting in Flink
flinkbot edited a comment on pull request #17958: URL: https://github.com/apache/flink/pull/17958#issuecomment-982461620 ## CI report: * 8377539867f18f1bda4fba411f961558f926ee7e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27267) * 7f59432fc9130ed007411f2be331cb58f12d14f8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27495) * 8235aed4e30f0bc4388cd3b4d92b5c868dc154e0 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-20928) KafkaSourceReaderTest.testOffsetCommitOnCheckpointComplete:189->pollUntil:270 » Timeout
[
https://issues.apache.org/jira/browse/FLINK-20928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17452785#comment-17452785
]
Qingsheng Ren commented on FLINK-20928:
---
Sorry for my late response! I took a look in the log of the latest instance. I
think it is not quite related to KafkaSource but Kafka in-memory brokers. The
log indicates that consumer cannot connect to coordinator on broker side
because of timeout, and the test case has kept trying for 1 minute, then
failed. I'll keep watching this issue and see if there's any other instance in
the future.
Also it'll be great to have some input from [~lindong] ~
> KafkaSourceReaderTest.testOffsetCommitOnCheckpointComplete:189->pollUntil:270
> » Timeout
> ---
>
> Key: FLINK-20928
> URL: https://issues.apache.org/jira/browse/FLINK-20928
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.13.0, 1.14.0, 1.15.0
>Reporter: Robert Metzger
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.15.0, 1.14.1, 1.13.4
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=11861=logs=c5f0071e-1851-543e-9a45-9ac140befc32=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5
> {code}
> [ERROR] Tests run: 8, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 93.992 s <<< FAILURE! - in
> org.apache.flink.connector.kafka.source.reader.KafkaSourceReaderTest
> [ERROR]
> testOffsetCommitOnCheckpointComplete(org.apache.flink.connector.kafka.source.reader.KafkaSourceReaderTest)
> Time elapsed: 60.086 s <<< ERROR!
> java.util.concurrent.TimeoutException: The offset commit did not finish
> before timeout.
> at
> org.apache.flink.core.testutils.CommonTestUtils.waitUtil(CommonTestUtils.java:210)
> at
> org.apache.flink.connector.kafka.source.reader.KafkaSourceReaderTest.pollUntil(KafkaSourceReaderTest.java:270)
> at
> org.apache.flink.connector.kafka.source.reader.KafkaSourceReaderTest.testOffsetCommitOnCheckpointComplete(KafkaSourceReaderTest.java:189)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] godfreyhe commented on a change in pull request #17962: [FLINK-22113][table-planner] Implement column uniqueness checking for TableSourceTable
godfreyhe commented on a change in pull request #17962:
URL: https://github.com/apache/flink/pull/17962#discussion_r761692575
##
File path:
flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/plan/metadata/MetadataTestUtil.scala
##
@@ -298,6 +301,124 @@ object MetadataTestUtil {
)
}
+ private def createTableSourceTable1(): Table = {
+val catalogTable = CatalogTable.fromProperties(
+ Map(
Review comment:
Could you also update the
`MetadataTestUtil#createProjectedTableSourceTable`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17958: FLINK-25029: Hadoop Caller Context Setting in Flink
flinkbot edited a comment on pull request #17958: URL: https://github.com/apache/flink/pull/17958#issuecomment-982461620 ## CI report: * 8377539867f18f1bda4fba411f961558f926ee7e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27267) * 7f59432fc9130ed007411f2be331cb58f12d14f8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27495) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17958: FLINK-25029: Hadoop Caller Context Setting in Flink
flinkbot edited a comment on pull request #17958: URL: https://github.com/apache/flink/pull/17958#issuecomment-982461620 ## CI report: * 8377539867f18f1bda4fba411f961558f926ee7e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27267) * 7f59432fc9130ed007411f2be331cb58f12d14f8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27495) * 8235aed4e30f0bc4388cd3b4d92b5c868dc154e0 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
flinkbot edited a comment on pull request #17462: URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789 ## CI report: * 26811181b8b06860cf50b28a8193114fc455bf37 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26510) * e5dc41ce526335e5a1ef9adf8a434f89dd3e4679 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27496) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] Myasuka commented on a change in pull request #17945: [FLINK-21027][state] Introduce KeyedStateBackend#isSafeToReuseState for opmitization hint
Myasuka commented on a change in pull request #17945:
URL: https://github.com/apache/flink/pull/17945#discussion_r761683066
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/state/KeyedStateBackend.java
##
@@ -137,10 +137,25 @@
*/
boolean deregisterKeySelectionListener(KeySelectionListener listener);
+@Deprecated
default boolean isStateImmutableInStateBackend(CheckpointType
checkpointOptions) {
return false;
}
+/**
+ * Whether it's safe to reuse key-value from the state-backend, e.g for
the purpose of
+ * optimization.
+ *
+ * NOTE: this method should not be used to check for {@link
InternalPriorityQueue}, as the
+ * priority queue could be stored on different locations, e.g RocksDB
state-backend could store
+ * that on JVM heap if configuring HEAP as the time-service factory.
+ *
+ * @return returns ture if safe to reuse the key-value from the
state-backend.
+ */
+default boolean isSafeToReuseState() {
Review comment:
I think these two methods serveing for different purposes. The method of
`isSafeToReuseState` is used to hint for SQL like operators to decide whether
they need to copy the state data from the state-backend. This actually focus on
the normal state access and has no relationship with checkpoint or timers.
On the other hand, `requiresLegacySynchronousTimerSnapshots` is only used to
represent that whether we should use raw checkpoint stream to backup the
timers. As you can see, this method serves for checkpoint and timers. If we
could drop the support HEAP timers in RocksDB, we could remove
`requiresLegacySynchronousTimerSnapshots` but not the `isSafeToReuseState`
method.
Maybe we could rename the method `KeyedStateBackend.isSafeToReuseState` to
`KeyedStateBackend.isSafeToReuseKVState`, however, I don't think
`KeyedStateBackend.isSafeToReusePQState` could meet the purpose of current
`AbstractKeyedStateBackend.requiresLegacySynchronousTimerSnapshots
`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17462: [FLINK-23170] Write metadata after materialization
flinkbot edited a comment on pull request #17462: URL: https://github.com/apache/flink/pull/17462#issuecomment-942024789 ## CI report: * 26811181b8b06860cf50b28a8193114fc455bf37 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26510) * e5dc41ce526335e5a1ef9adf8a434f89dd3e4679 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17919: [FLINK-24419][table-planner] Trim to precision when casting to BINARY/VARBINARY
flinkbot edited a comment on pull request #17919: URL: https://github.com/apache/flink/pull/17919#issuecomment-979659835 ## CI report: * 734f7484f6d2a72901fce53e14a49adc845816b7 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27486) * bd5b6272b266dfe8dc494ee6decac510d1c73f2b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27487) * 56515bd322459d965610005e3f3da140aa9ce1a7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27494) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-25029) Hadoop Caller Context Setting In Flink
[ https://issues.apache.org/jira/browse/FLINK-25029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17452777#comment-17452777 ] 刘方奇 commented on FLINK-25029: - [~arvid], Hi, I get your advice in the PR, that 's valuable for me. I try to reply all your comment and switch the code to the better, could you help to review again when you are free? BTW, I still has some questions in the PR. ( e.g : what file should I put the option into, that may help me slove the CI problem.) > Hadoop Caller Context Setting In Flink > -- > > Key: FLINK-25029 > URL: https://issues.apache.org/jira/browse/FLINK-25029 > Project: Flink > Issue Type: Improvement > Components: FileSystems >Reporter: 刘方奇 >Assignee: 刘方奇 >Priority: Major > Labels: pull-request-available > > For a given HDFS operation (e.g. delete file), it's very helpful to track > which upper level job issues it. The upper level callers may be specific > Oozie tasks, MR jobs, and hive queries. One scenario is that the namenode > (NN) is abused/spammed, the operator may want to know immediately which MR > job should be blamed so that she can kill it. To this end, the caller context > contains at least the application-dependent "tracking id". > The above is the main effect of the Caller Context. HDFS Client set Caller > Context, then name node get it in audit log to do some work. > Now the Spark and hive have the Caller Context to meet the HDFS Job Audit > requirement. > In my company, flink jobs often cause some problems for HDFS, so we did it > for preventing some cases. > If the feature is general enough. Should we support it, then I can submit a > PR for this. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink-ml] zhipeng93 commented on a change in pull request #28: [Flink-24556] Add Estimator and Transformer for logistic regression
zhipeng93 commented on a change in pull request #28:
URL: https://github.com/apache/flink-ml/pull/28#discussion_r761677853
##
File path:
flink-ml-lib/src/main/java/org/apache/flink/ml/classification/linear/LogisticRegression.java
##
@@ -0,0 +1,488 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.linear;
+
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.BasicTypeInfo;
+import org.apache.flink.api.common.typeinfo.PrimitiveArrayTypeInfo;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.iteration.DataStreamList;
+import org.apache.flink.iteration.IterationBody;
+import org.apache.flink.iteration.IterationBodyResult;
+import org.apache.flink.iteration.IterationConfig;
+import org.apache.flink.iteration.IterationConfig.OperatorLifeCycle;
+import org.apache.flink.iteration.IterationListener;
+import org.apache.flink.iteration.Iterations;
+import org.apache.flink.iteration.ReplayableDataStreamList;
+import org.apache.flink.iteration.operator.OperatorStateUtils;
+import org.apache.flink.ml.api.Estimator;
+import org.apache.flink.ml.common.datastream.DataStreamUtils;
+import org.apache.flink.ml.common.feature.LabeledPointWithWeight;
+import org.apache.flink.ml.common.iteration.TerminateOnMaxIterOrTol;
+import org.apache.flink.ml.linalg.BLAS;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.BoundedOneInput;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.OutputTag;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.commons.collections.IteratorUtils;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+
+/**
+ * This class implements methods to train a logistic regression model. For
details, see
+ * https://en.wikipedia.org/wiki/Logistic_regression.
+ */
+public class LogisticRegression
+implements Estimator,
+LogisticRegressionParams {
+
+private Map, Object> paramMap = new HashMap<>();
+
+private static final OutputTag MODEL_OUTPUT =
Review comment:
Hmm, thanks for the comment here!! I misunderstood the cost of creating
output tag. I have made it a local variable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17958: FLINK-25029: Hadoop Caller Context Setting in Flink
flinkbot edited a comment on pull request #17958: URL: https://github.com/apache/flink/pull/17958#issuecomment-982461620 ## CI report: * 8377539867f18f1bda4fba411f961558f926ee7e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27267) * 7f59432fc9130ed007411f2be331cb58f12d14f8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27495) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17958: FLINK-25029: Hadoop Caller Context Setting in Flink
flinkbot edited a comment on pull request #17958: URL: https://github.com/apache/flink/pull/17958#issuecomment-982461620 ## CI report: * 8377539867f18f1bda4fba411f961558f926ee7e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27267) * 7f59432fc9130ed007411f2be331cb58f12d14f8 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-13395) Add source and sink connector for Alibaba Log Service
[ https://issues.apache.org/jira/browse/FLINK-13395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17452771#comment-17452771 ] ZhuoYu Chen commented on FLINK-13395: - Hi [~liketic] I am very interested in this，and I want do some job for flink，can I help to do that？ Thank you > Add source and sink connector for Alibaba Log Service > - > > Key: FLINK-13395 > URL: https://issues.apache.org/jira/browse/FLINK-13395 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Reporter: Ke Li >Priority: Not a Priority > Labels: auto-deprioritized-major, auto-deprioritized-minor > > Alibaba Log Service is a big data service which has been widely used in > Alibaba Group and thousands of customers of Alibaba Cloud. The core storage > engine of Log Service is named Loghub which is a large scale distributed > storage system which provides producer and consumer to push and pull data > like Kafka, AWS Kinesis and Azure Eventhub does. > Log Service provides a complete solution to help user collect data from both > on premise and cloud data sources. More than 10 PB data is sent to and > consumed from Loghub every day. And hundreds of thousands of users > implemented their DevOPS and big data system based on Log Service. > Log Service and Flink/Blink has became the de facto standard of big data > architecture for unified data processing in Alibaba Group and more users of > Alibaba Cloud. > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] liufangqi commented on a change in pull request #17958: FLINK-25029: Hadoop Caller Context Setting in Flink
liufangqi commented on a change in pull request #17958:
URL: https://github.com/apache/flink/pull/17958#discussion_r761672254
##
File path:
flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/util/HadoopUtils.java
##
@@ -247,4 +248,43 @@ private static boolean addHadoopConfIfFound(
}
return foundHadoopConfiguration;
}
+
+/**
+ * Set up the caller context [[callerContext]] by invoking Hadoop
CallerContext API of
+ * [[org.apache.hadoop.ipc.CallerContext]], which was added in hadoop 2.8.
+ */
+public static void setCallerContext(String callerContext,
+
org.apache.flink.configuration.Configuration flinkConfig) {
+if (isMinHadoopVersion(2,8)) {
+try {
+callerContext = truncateCallerContext(callerContext,
flinkConfig);
+Class callerContextClass =
Class.forName("org.apache.hadoop.ipc.CallerContext");
+Class builder =
Class.forName("org.apache.hadoop.ipc.CallerContext$Builder");
+Constructor builderInst =
builder.getConstructor(callerContext.getClass());
+callerContextClass.getMethod("setCurrent", callerContextClass)
+.invoke(null, builder.getMethod("build")
+
.invoke(builderInst.newInstance(callerContext)));
Review comment:
I know it is a graceful way to do this feature. But as you see, that's
probably a big effort for this feature. Above all, I think reflection is a
transitional way to do this work. And I can do the Hadoop version bumping after
this feature, it may take some time, but i think i can take this ticket.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] liufangqi commented on a change in pull request #17958: FLINK-25029: Hadoop Caller Context Setting in Flink
liufangqi commented on a change in pull request #17958:
URL: https://github.com/apache/flink/pull/17958#discussion_r761670505
##
File path:
flink-core/src/main/java/org/apache/flink/configuration/ExecutionOptions.java
##
@@ -105,6 +105,14 @@
+ "throughput"))
.build());
+public static final ConfigOption CALLER_CONTEXT_APP_ID =
+ConfigOptions.key("execution.caller-context-app-id")
+.stringType()
+.defaultValue("")
Review comment:
Get it, THX for your advice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] liufangqi commented on a change in pull request #17958: FLINK-25029: Hadoop Caller Context Setting in Flink
liufangqi commented on a change in pull request #17958: URL: https://github.com/apache/flink/pull/17958#discussion_r761670210 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/taskmanager/Task.java ## @@ -431,6 +433,14 @@ public Task( // finally, create the executing thread, but do not start it executingThread = new Thread(TASK_THREADS_GROUP, this, taskNameWithSubtask); + +//add CallerContext +//JID => jobID, JName => jobName, TName => TaskName, and with attempt number end. Review comment: I will change this soon. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] liufangqi commented on a change in pull request #17958: FLINK-25029: Hadoop Caller Context Setting in Flink
liufangqi commented on a change in pull request #17958: URL: https://github.com/apache/flink/pull/17958#discussion_r761670108 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/taskmanager/Task.java ## @@ -431,6 +433,14 @@ public Task( // finally, create the executing thread, but do not start it executingThread = new Thread(TASK_THREADS_GROUP, this, taskNameWithSubtask); + +//add CallerContext +//JID => jobID, JName => jobName, TName => TaskName, and with attempt number end. +String callerContext = "Flink_Task_"+ "JID_" + jobId + +"_TName_" + taskInfo.getTaskName() + +"_" + taskInfo.getAttemptNumber(); + HadoopUtils.setCallerContext(tmConfig.getOptional(ExecutionOptions.CALLER_CONTEXT_APP_ID) Review comment: @AHeise I think I get your meaning. But maybe a new module/ new File/ new interface can cause more influence for the project. I see many same cases in .I think i can do it in a simple way that I can check the hadoop classpath before. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-24152) Exactly-once semantics should be configurable through Flink configuration only
[ https://issues.apache.org/jira/browse/FLINK-24152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17452768#comment-17452768 ] ZhuoYu Chen commented on FLINK-24152: - Hi [~mapohl] I am very interested in this，and I want do some job for flink，can I help to do that？ Thank you > Exactly-once semantics should be configurable through Flink configuration only > -- > > Key: FLINK-24152 > URL: https://issues.apache.org/jira/browse/FLINK-24152 > Project: Flink > Issue Type: Improvement > Components: API / Core, Connectors / Common >Reporter: Matthias >Priority: Major > > Exactly-once constraint needs to be configured in multiple locations: > 1. Flink configuration > 2. Kafka connector configuration > As a user, I would expect to set this configuration parameter in Flink. The > Kafka connector should be able to derive this configuration parameter from > the Flink configuration by default. Setting it on the connector level could > be considered as an advanced configuration parameter which would overwrite > the default configuration. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] meetjunsu commented on a change in pull request #17542: [FLINK-17782] Add array,map,row types support for parquet row writer
meetjunsu commented on a change in pull request #17542:
URL: https://github.com/apache/flink/pull/17542#discussion_r761667459
##
File path:
flink-formats/flink-parquet/src/main/java/org/apache/flink/formats/parquet/row/ParquetRowDataWriter.java
##
@@ -224,6 +277,153 @@ private TimestampWriter(int precision) {
public void write(RowData row, int ordinal) {
recordConsumer.addBinary(timestampToInt96(row.getTimestamp(ordinal,
precision)));
}
+
+@Override
+public void write(ArrayData arrayData, int ordinal) {
+
recordConsumer.addBinary(timestampToInt96(arrayData.getTimestamp(ordinal,
precision)));
+}
+}
+
+/** It writes a map field to parquet, both key and value are nullable. */
+private class MapWriter implements FieldWriter {
+
+private String repeatedGroupName;
+private String keyName, valueName;
+private FieldWriter keyWriter, valueWriter;
+
+private MapWriter(LogicalType keyType, LogicalType valueType,
GroupType groupType) {
+// Get the internal map structure (MAP_KEY_VALUE)
+GroupType repeatedType = groupType.getType(0).asGroupType();
+this.repeatedGroupName = repeatedType.getName();
+
+// Get key element information
+Type type = repeatedType.getType(0);
+this.keyName = type.getName();
+this.keyWriter = createWriter(keyType, type);
+
+// Get value element information
+Type valuetype = repeatedType.getType(1);
+this.valueName = valuetype.getName();
+this.valueWriter = createWriter(valueType, valuetype);
+}
+
+@Override
+public void write(RowData row, int ordinal) {
+recordConsumer.startGroup();
+
+MapData mapData = row.getMap(ordinal);
+
+if (mapData != null && mapData.size() > 0) {
+recordConsumer.startField(repeatedGroupName, 0);
+
+ArrayData keyArray = mapData.keyArray();
+ArrayData valueArray = mapData.valueArray();
+for (int i = 0; i < keyArray.size(); i++) {
+recordConsumer.startGroup();
+// write key element
+recordConsumer.startField(keyName, 0);
+keyWriter.write(keyArray, i);
+recordConsumer.endField(keyName, 0);
+// write value element
+recordConsumer.startField(valueName, 1);
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] meetjunsu commented on a change in pull request #17542: [FLINK-17782] Add array,map,row types support for parquet row writer
meetjunsu commented on a change in pull request #17542:
URL: https://github.com/apache/flink/pull/17542#discussion_r761667191
##
File path:
flink-formats/flink-parquet/src/main/java/org/apache/flink/formats/parquet/row/ParquetRowDataWriter.java
##
@@ -165,6 +187,11 @@ public void write(RowData row, int ordinal) {
public void write(RowData row, int ordinal) {
recordConsumer.addLong(row.getLong(ordinal));
}
+
+@Override
+public void write(ArrayData arrayData, int ordinal) {
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Comment Edited] (FLINK-24794) AZP kills docker containers with exit code 137 (maybe OOM)
[
https://issues.apache.org/jira/browse/FLINK-24794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17452766#comment-17452766
]
Yun Gao edited comment on FLINK-24794 at 12/3/21, 5:58 AM:
---
It seems the problem has been reproduced yesterday:
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=27415=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=9753]
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=27418=logs=0da23115-68bb-5dcd-192c-bd4c8adebde1=24c3384f-1bcb-57b3-224f-51bf973bbee8=7046]
[~chesnay] It seems we have already fixed it, right?
was (Author: gaoyunhaii):
It seems the problem has been reproduced:
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=27415=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=9753]
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=27418=logs=0da23115-68bb-5dcd-192c-bd4c8adebde1=24c3384f-1bcb-57b3-224f-51bf973bbee8=7046
> AZP kills docker containers with exit code 137 (maybe OOM)
> --
>
> Key: FLINK-24794
> URL: https://issues.apache.org/jira/browse/FLINK-24794
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines
>Affects Versions: 1.14.0, 1.15.0
>Reporter: Till Rohrmann
>Assignee: Chesnay Schepler
>Priority: Critical
> Labels: test-stability
> Fix For: 1.15.0, 1.14.1
>
>
> The test {{pyflink/datastream/tests/test_data_stream.py}} fails on AZP with
> {code}
> ##[error]Exit code 137 returned from process: file name '/bin/docker',
> arguments 'exec -i -u 1004
> c51e2746884724c100e2ea1ff39c2552ed90ad0edaf87b3c460f70c7780efd0c
> /__a/externals/node/bin/node /__w/_temp/containerHandlerInvoker.js'.
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=25995=logs=9cada3cb-c1d3-5621-16da-0f718fb86602=c67e71ed-6451-5d26-8920-5a8cf9651901=21843
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-24794) AZP kills docker containers with exit code 137 (maybe OOM)
[
https://issues.apache.org/jira/browse/FLINK-24794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17452766#comment-17452766
]
Yun Gao commented on FLINK-24794:
-
It seems the problem has been reproduced:
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=27415=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=9753]
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=27418=logs=0da23115-68bb-5dcd-192c-bd4c8adebde1=24c3384f-1bcb-57b3-224f-51bf973bbee8=7046
> AZP kills docker containers with exit code 137 (maybe OOM)
> --
>
> Key: FLINK-24794
> URL: https://issues.apache.org/jira/browse/FLINK-24794
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines
>Affects Versions: 1.14.0, 1.15.0
>Reporter: Till Rohrmann
>Assignee: Chesnay Schepler
>Priority: Critical
> Labels: test-stability
> Fix For: 1.15.0, 1.14.1
>
>
> The test {{pyflink/datastream/tests/test_data_stream.py}} fails on AZP with
> {code}
> ##[error]Exit code 137 returned from process: file name '/bin/docker',
> arguments 'exec -i -u 1004
> c51e2746884724c100e2ea1ff39c2552ed90ad0edaf87b3c460f70c7780efd0c
> /__a/externals/node/bin/node /__w/_temp/containerHandlerInvoker.js'.
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=25995=logs=9cada3cb-c1d3-5621-16da-0f718fb86602=c67e71ed-6451-5d26-8920-5a8cf9651901=21843
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Closed] (FLINK-23525) Docker command fails on Azure: Exit code 137 returned from process: file name '/usr/bin/docker'
[
https://issues.apache.org/jira/browse/FLINK-23525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao closed FLINK-23525.
---
Resolution: Duplicate
> Docker command fails on Azure: Exit code 137 returned from process: file name
> '/usr/bin/docker'
> ---
>
> Key: FLINK-23525
> URL: https://issues.apache.org/jira/browse/FLINK-23525
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines
>Affects Versions: 1.14.0, 1.13.1
>Reporter: Dawid Wysakowicz
>Priority: Major
> Labels: auto-deprioritized-blocker, auto-deprioritized-critical,
> test-stability
> Attachments: screenshot-1.png
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=21053=logs=4d4a0d10-fca2-5507-8eed-c07f0bdf4887=7b25afdf-cc6c-566f-5459-359dc2585798=10034
> {code}
> ##[error]Exit code 137 returned from process: file name '/usr/bin/docker',
> arguments 'exec -i -u 1001 -w /home/vsts_azpcontainer
> 9dca235e075b70486fac576ee17cee722940edf575e5478e0a52def5b46c28b5
> /__a/externals/node/bin/node /__w/_temp/containerHandlerInvoker.js'.
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-24396) Add @Public annotations to Table & SQL API classes
[
https://issues.apache.org/jira/browse/FLINK-24396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17452765#comment-17452765
]
ZhuoYu Chen commented on FLINK-24396:
-
[~twalthr] One module, one subtask?
> Add @Public annotations to Table & SQL API classes
> --
>
> Key: FLINK-24396
> URL: https://issues.apache.org/jira/browse/FLINK-24396
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / API
>Reporter: Timo Walther
>Priority: Major
>
> Many parts of the Table & SQL API have stabilized and we can mark them as
> {{@Public}} which gives both users and downstream projects more confidence
> when using Flink.
> A concrete list of classes and methods needs to be compiled. Some parts of
> the API might stay {{@PublicEvolving}} for now.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-23525) Docker command fails on Azure: Exit code 137 returned from process: file name '/usr/bin/docker'
[
https://issues.apache.org/jira/browse/FLINK-23525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17452764#comment-17452764
]
Yun Gao commented on FLINK-23525:
-
I'll close this issue since it should be duplicated with
https://issues.apache.org/jira/browse/FLINK-24794. Let's continue the
discussion there.
> Docker command fails on Azure: Exit code 137 returned from process: file name
> '/usr/bin/docker'
> ---
>
> Key: FLINK-23525
> URL: https://issues.apache.org/jira/browse/FLINK-23525
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines
>Affects Versions: 1.14.0, 1.13.1
>Reporter: Dawid Wysakowicz
>Priority: Major
> Labels: auto-deprioritized-blocker, auto-deprioritized-critical,
> test-stability
> Attachments: screenshot-1.png
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=21053=logs=4d4a0d10-fca2-5507-8eed-c07f0bdf4887=7b25afdf-cc6c-566f-5459-359dc2585798=10034
> {code}
> ##[error]Exit code 137 returned from process: file name '/usr/bin/docker',
> arguments 'exec -i -u 1001 -w /home/vsts_azpcontainer
> 9dca235e075b70486fac576ee17cee722940edf575e5478e0a52def5b46c28b5
> /__a/externals/node/bin/node /__w/_temp/containerHandlerInvoker.js'.
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Reopened] (FLINK-24794) AZP kills docker containers with exit code 137 (maybe OOM)
[
https://issues.apache.org/jira/browse/FLINK-24794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao reopened FLINK-24794:
-
> AZP kills docker containers with exit code 137 (maybe OOM)
> --
>
> Key: FLINK-24794
> URL: https://issues.apache.org/jira/browse/FLINK-24794
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines
>Affects Versions: 1.14.0, 1.15.0
>Reporter: Till Rohrmann
>Assignee: Chesnay Schepler
>Priority: Critical
> Labels: test-stability
> Fix For: 1.15.0, 1.14.1
>
>
> The test {{pyflink/datastream/tests/test_data_stream.py}} fails on AZP with
> {code}
> ##[error]Exit code 137 returned from process: file name '/bin/docker',
> arguments 'exec -i -u 1004
> c51e2746884724c100e2ea1ff39c2552ed90ad0edaf87b3c460f70c7780efd0c
> /__a/externals/node/bin/node /__w/_temp/containerHandlerInvoker.js'.
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=25995=logs=9cada3cb-c1d3-5621-16da-0f718fb86602=c67e71ed-6451-5d26-8920-5a8cf9651901=21843
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] liufangqi commented on pull request #17958: FLINK-25029: Hadoop Caller Context Setting in Flink
liufangqi commented on pull request #17958: URL: https://github.com/apache/flink/pull/17958#issuecomment-985233869 @AHeise THX for your reply, these were worthy advice for me. I will handle it case by case. Give me some time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] liufangqi commented on pull request #17958: FLINK-25029: Hadoop Caller Context Setting in Flink
liufangqi commented on pull request #17958: URL: https://github.com/apache/flink/pull/17958#issuecomment-985232979 > @AHeise , Sorry to bother, it seems that i put this option into a wrong file, can I ask for advice what file should I put into? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17919: [FLINK-24419][table-planner] Trim to precision when casting to BINARY/VARBINARY
flinkbot edited a comment on pull request #17919: URL: https://github.com/apache/flink/pull/17919#issuecomment-979659835 ## CI report: * e7d585c7aab70989fe340f5f14b359952433ddf2 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27327) * 734f7484f6d2a72901fce53e14a49adc845816b7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27486) * bd5b6272b266dfe8dc494ee6decac510d1c73f2b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27487) * 56515bd322459d965610005e3f3da140aa9ce1a7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27494) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17919: [FLINK-24419][table-planner] Trim to precision when casting to BINARY/VARBINARY
flinkbot edited a comment on pull request #17919: URL: https://github.com/apache/flink/pull/17919#issuecomment-979659835 ## CI report: * e7d585c7aab70989fe340f5f14b359952433ddf2 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27327) * 734f7484f6d2a72901fce53e14a49adc845816b7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27486) * bd5b6272b266dfe8dc494ee6decac510d1c73f2b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27487) * 56515bd322459d965610005e3f3da140aa9ce1a7 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-24959) Add a BitMap function to FlinkSQL
[
https://issues.apache.org/jira/browse/FLINK-24959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ZhuoYu Chen updated FLINK-24959:
Description:
BITMAP_AND :{color:#33}Computes the intersection of two input bitmaps and
returns the new bitmap{color}
{code:java}
select bitmap_count(bitmap_and(to_bitmap(1), to_bitmap(2))) cnt; {code}
{color:#33} {color}
{color:#33}{{TO_BITMAP:input for the value of 0 ~ 18446744073709551615
unsigned bigint in the range, the output for the bitmap containing the element.
This function is mainly used for stream load task to import integer fields into
the bitmap field of the table}}{color}
{code:java}
select bitmap_count(to_bitmap(10)); {code}
{color:#30323e}BITMAP_ANDNOT:Computes the set (difference set) that is in A but
not in B.{color}
{code:java}
select bitmap_to_string(bitmap_andnot(bitmap_from_string('1, 3'),
bitmap_from_string('2'))) cnt; {code}
Bitmap functions related to join operations, etc
was:
BITMAP_AND :{color:#33}Computes the intersection of two input bitmaps and
returns the new bitmap
{color}
select bitmap_count(bitmap_and(to_bitmap(1), to_bitmap(2))) cnt;
{color:#33} {color}
{color:#33}{{TO_BITMAP:input for the value of 0 ~ 18446744073709551615
unsigned bigint in the range, the output for the bitmap containing the element.
This function is mainly used for stream load task to import integer fields into
the bitmap field of the table}}
{color}
{code:java}
select bitmap_count(to_bitmap(10)); {code}
{color:#30323e}BITMAP_ANDNOT:Computes the set (difference set) that is in A but
not in B.{color}
{code:java}
select bitmap_to_string(bitmap_andnot(bitmap_from_string('1, 3'),
bitmap_from_string('2'))) cnt; {code}
Bitmap functions related to join operations, etc
> Add a BitMap function to FlinkSQL
> -
>
> Key: FLINK-24959
> URL: https://issues.apache.org/jira/browse/FLINK-24959
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / API
>Affects Versions: 1.15.0
>Reporter: ZhuoYu Chen
>Priority: Minor
>
> BITMAP_AND :{color:#33}Computes the intersection of two input bitmaps and
> returns the new bitmap{color}
> {code:java}
> select bitmap_count(bitmap_and(to_bitmap(1), to_bitmap(2))) cnt; {code}
> {color:#33} {color}
> {color:#33}{{TO_BITMAP:input for the value of 0 ~ 18446744073709551615
> unsigned bigint in the range, the output for the bitmap containing the
> element. This function is mainly used for stream load task to import integer
> fields into the bitmap field of the table}}{color}
> {code:java}
> select bitmap_count(to_bitmap(10)); {code}
>
> {color:#30323e}BITMAP_ANDNOT:Computes the set (difference set) that is in A
> but not in B.{color}
> {code:java}
> select bitmap_to_string(bitmap_andnot(bitmap_from_string('1, 3'),
> bitmap_from_string('2'))) cnt; {code}
>
> Bitmap functions related to join operations, etc
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-24959) Add a BitMap function to FlinkSQL
[
https://issues.apache.org/jira/browse/FLINK-24959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ZhuoYu Chen updated FLINK-24959:
Description:
BITMAP_AND :{color:#33}Computes the intersection of two input bitmaps and
returns the new bitmap
{color}
select bitmap_count(bitmap_and(to_bitmap(1), to_bitmap(2))) cnt;
{color:#33} {color}
{color:#33}{{TO_BITMAP:input for the value of 0 ~ 18446744073709551615
unsigned bigint in the range, the output for the bitmap containing the element.
This function is mainly used for stream load task to import integer fields into
the bitmap field of the table}}
{color}
{code:java}
select bitmap_count(to_bitmap(10)); {code}
{color:#30323e}BITMAP_ANDNOT:Computes the set (difference set) that is in A but
not in B.{color}
{code:java}
select bitmap_to_string(bitmap_andnot(bitmap_from_string('1, 3'),
bitmap_from_string('2'))) cnt; {code}
Bitmap functions related to join operations, etc
was:
bitmap_and :{color:#33}Computes the intersection of two input bitmaps and
returns the new bitmap{color}
{color:#30323e}bitmap_andnot:{color:#33}Computes the set (difference set)
that is in A but not in B.{color}{color}
{color:#30323e}{color:#33}Bitmap functions related to join operations,
etc{color}{color}
> Add a BitMap function to FlinkSQL
> -
>
> Key: FLINK-24959
> URL: https://issues.apache.org/jira/browse/FLINK-24959
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / API
>Affects Versions: 1.15.0
>Reporter: ZhuoYu Chen
>Priority: Minor
>
> BITMAP_AND :{color:#33}Computes the intersection of two input bitmaps and
> returns the new bitmap
> {color}
> select bitmap_count(bitmap_and(to_bitmap(1), to_bitmap(2))) cnt;
> {color:#33} {color}
> {color:#33}{{TO_BITMAP:input for the value of 0 ~ 18446744073709551615
> unsigned bigint in the range, the output for the bitmap containing the
> element. This function is mainly used for stream load task to import integer
> fields into the bitmap field of the table}}
> {color}
> {code:java}
> select bitmap_count(to_bitmap(10)); {code}
> {color:#30323e}BITMAP_ANDNOT:Computes the set (difference set) that is in A
> but not in B.{color}
> {code:java}
> select bitmap_to_string(bitmap_andnot(bitmap_from_string('1, 3'),
> bitmap_from_string('2'))) cnt; {code}
> Bitmap functions related to join operations, etc
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] JingsongLi commented on a change in pull request #17542: [FLINK-17782] Add array,map,row types support for parquet row writer
JingsongLi commented on a change in pull request #17542:
URL: https://github.com/apache/flink/pull/17542#discussion_r761651214
##
File path:
flink-formats/flink-parquet/src/main/java/org/apache/flink/formats/parquet/row/ParquetRowDataWriter.java
##
@@ -224,6 +277,153 @@ private TimestampWriter(int precision) {
public void write(RowData row, int ordinal) {
recordConsumer.addBinary(timestampToInt96(row.getTimestamp(ordinal,
precision)));
}
+
+@Override
+public void write(ArrayData arrayData, int ordinal) {
+
recordConsumer.addBinary(timestampToInt96(arrayData.getTimestamp(ordinal,
precision)));
+}
+}
+
+/** It writes a map field to parquet, both key and value are nullable. */
+private class MapWriter implements FieldWriter {
+
+private String repeatedGroupName;
+private String keyName, valueName;
+private FieldWriter keyWriter, valueWriter;
+
+private MapWriter(LogicalType keyType, LogicalType valueType,
GroupType groupType) {
+// Get the internal map structure (MAP_KEY_VALUE)
+GroupType repeatedType = groupType.getType(0).asGroupType();
+this.repeatedGroupName = repeatedType.getName();
+
+// Get key element information
+Type type = repeatedType.getType(0);
+this.keyName = type.getName();
+this.keyWriter = createWriter(keyType, type);
+
+// Get value element information
+Type valuetype = repeatedType.getType(1);
+this.valueName = valuetype.getName();
+this.valueWriter = createWriter(valueType, valuetype);
+}
+
+@Override
+public void write(RowData row, int ordinal) {
+recordConsumer.startGroup();
+
+MapData mapData = row.getMap(ordinal);
+
+if (mapData != null && mapData.size() > 0) {
+recordConsumer.startField(repeatedGroupName, 0);
+
+ArrayData keyArray = mapData.keyArray();
+ArrayData valueArray = mapData.valueArray();
+for (int i = 0; i < keyArray.size(); i++) {
+recordConsumer.startGroup();
+// write key element
+recordConsumer.startField(keyName, 0);
Review comment:
It is dangerous to `getXX` without `isNullAt` returned false. Maybe it
will produce NPE, maybe it will have an unexpected effect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Comment Edited] (FLINK-24959) Add a BitMap function to FlinkSQL
[
https://issues.apache.org/jira/browse/FLINK-24959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17452755#comment-17452755
]
ZhuoYu Chen edited comment on FLINK-24959 at 12/3/21, 5:12 AM:
---
hi [~jark] clickhouse，starRocks，doris all support bitmap
Traditional Count distinct calculation, due to the need for multiple data
shuffle (transferring data between different nodes and calculating
de-duplication) during query execution, can lead to a linear decrease in
performance as the amount of data increases.
Advantages of bitmap de-duplication
Space advantage: The use of one bit of a bitmap to indicate the existence
of the corresponding subscript has a great space advantage; for example, for
int32 de-duplication, the storage space required by a normal bitmap is only
1/32 of the traditional de-duplication. With the optimized implementation of
Roaring Bitmap, the storage space is further reduced significantly for sparse
bitmaps.
Time advantage: The computation involved in bitmap de-duplication includes
bit placement for a given subscript and counting the number of bits in a
bitmap, which is an O(1) operation and an O(n) operation, respectively, and the
latter can be efficiently computed using clz, ctz, and other instructions.
was (Author: monster#12):
hi [~jark] clickhouse，starRocks，doris all support bitmap
Traditional Count distinct calculation, due to the need for multiple data
shuffle (transferring data between different nodes and calculating
de-duplication) during query execution, can lead to a linear decrease in
performance as the amount of data increases.
Advantages of bitmap de-duplication
Space advantage: The use of one bit of a bitmap to indicate the existence
of the corresponding subscript has a great space advantage; for example, for
int32 de-duplication, the storage space required by a normal bitmap is only
1/32 of the traditional de-duplication. With the optimized implementation of
Roaring Bitmap, the storage space is further reduced significantly for sparse
bitmaps.
Time advantage: The computation involved in bitmap de-duplication includes
bit placement for a given subscript and counting the number of bits in a
bitmap, which is an O(1) operation and an O(n) operation, respectively, and the
latter can be efficiently computed using clz, ctz, and other instructions. In
addition, bitmap de-duplication can be accelerated in parallel in the MPP
execution engine, where each computation node computes a local sub-bitmap and
uses the bitor operation to merge these sub-bitmaps into the final bitmap. The
bitor operation is more efficient than sort-based and hash-based
de-duplication, has no conditional dependencies and data dependencies, and can
be executed quantitatively.
> Add a BitMap function to FlinkSQL
> -
>
> Key: FLINK-24959
> URL: https://issues.apache.org/jira/browse/FLINK-24959
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / API
>Affects Versions: 1.15.0
>Reporter: ZhuoYu Chen
>Priority: Minor
>
> bitmap_and :{color:#33}Computes the intersection of two input bitmaps and
> returns the new bitmap{color}
> {color:#30323e}bitmap_andnot:{color:#33}Computes the set (difference set)
> that is in A but not in B.{color}{color}
> {color:#30323e}{color:#33}Bitmap functions related to join operations,
> etc{color}{color}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-24959) Add a BitMap function to FlinkSQL
[
https://issues.apache.org/jira/browse/FLINK-24959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17452755#comment-17452755
]
ZhuoYu Chen commented on FLINK-24959:
-
hi [~jark] clickhouse，starRocks，doris all support bitmap
Traditional Count distinct calculation, due to the need for multiple data
shuffle (transferring data between different nodes and calculating
de-duplication) during query execution, can lead to a linear decrease in
performance as the amount of data increases.
Advantages of bitmap de-duplication
Space advantage: The use of one bit of a bitmap to indicate the existence
of the corresponding subscript has a great space advantage; for example, for
int32 de-duplication, the storage space required by a normal bitmap is only
1/32 of the traditional de-duplication. With the optimized implementation of
Roaring Bitmap, the storage space is further reduced significantly for sparse
bitmaps.
Time advantage: The computation involved in bitmap de-duplication includes
bit placement for a given subscript and counting the number of bits in a
bitmap, which is an O(1) operation and an O(n) operation, respectively, and the
latter can be efficiently computed using clz, ctz, and other instructions. In
addition, bitmap de-duplication can be accelerated in parallel in the MPP
execution engine, where each computation node computes a local sub-bitmap and
uses the bitor operation to merge these sub-bitmaps into the final bitmap. The
bitor operation is more efficient than sort-based and hash-based
de-duplication, has no conditional dependencies and data dependencies, and can
be executed quantitatively.
> Add a BitMap function to FlinkSQL
> -
>
> Key: FLINK-24959
> URL: https://issues.apache.org/jira/browse/FLINK-24959
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / API
>Affects Versions: 1.15.0
>Reporter: ZhuoYu Chen
>Priority: Minor
>
> bitmap_and :{color:#33}Computes the intersection of two input bitmaps and
> returns the new bitmap{color}
> {color:#30323e}bitmap_andnot:{color:#33}Computes the set (difference set)
> that is in A but not in B.{color}{color}
> {color:#30323e}{color:#33}Bitmap functions related to join operations,
> etc{color}{color}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] gaoyunhaii commented on pull request #17924: [FLINK-25072][streaming] Introduce description on Transformation and …
gaoyunhaii commented on pull request #17924: URL: https://github.com/apache/flink/pull/17924#issuecomment-985221811 LGTM from my side. And @AHeise do you want to also have a look~? Very thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-20873) Upgrade Calcite version to 1.27
[ https://issues.apache.org/jira/browse/FLINK-20873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17452753#comment-17452753 ] ZhuoYu Chen commented on FLINK-20873: - [~jark] Are there still plans for this to go ahead? If there is a plan, I think I can be assigned this task. > Upgrade Calcite version to 1.27 > --- > > Key: FLINK-20873 > URL: https://issues.apache.org/jira/browse/FLINK-20873 > Project: Flink > Issue Type: Improvement > Components: Table SQL / API, Table SQL / Planner >Reporter: Jark Wu >Priority: Major > > The following files should be removed from the Flink code base during an > upgrade: > - org.apache.calcite.rex.RexSimplify > - org.apache.calcite.sql.SqlMatchRecognize > - org.apache.calcite.sql.SqlTableRef > - org.apache.calcite.sql2rel.RelDecorrelator > - org.apache.flink.table.planner.functions.sql.SqlJsonObjectFunction (added > in FLINK-16203) > - Adopt calcite's behaviour and add SQL tests once > [https://github.com/apache/calcite/pull/2555] is merged, (check FLINK-24576 ) -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-12429) Translate the "Generating Timestamps / Watermarks" page into Chinese
[ https://issues.apache.org/jira/browse/FLINK-12429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17452752#comment-17452752 ] ZhuoYu Chen commented on FLINK-12429: - Hi [~jark] I am very interested in this，and I want do some job for flink，can I help to do that？ Thank you > Translate the "Generating Timestamps / Watermarks" page into Chinese > > > Key: FLINK-12429 > URL: https://issues.apache.org/jira/browse/FLINK-12429 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Documentation >Affects Versions: 1.9.0 >Reporter: YangFei >Priority: Major > Labels: auto-unassigned > > file locate flink/docs/dev/event_timestamps_watermarks.zh.md > [https://ci.apache.org/projects/flink/flink-docs-master/dev/event_timestamps_watermarks.html] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-17203) Add metrics for ClickHouse sink
[ https://issues.apache.org/jira/browse/FLINK-17203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17452747#comment-17452747 ] ZhuoYu Chen commented on FLINK-17203: - Hi [~csbliss] I am very interested in this，and I want do some job for flink，can I help to do that？ Thank you > Add metrics for ClickHouse sink > --- > > Key: FLINK-17203 > URL: https://issues.apache.org/jira/browse/FLINK-17203 > Project: Flink > Issue Type: Sub-task >Reporter: jinhai >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-17202) Add SQL for ClickHouse connector
[ https://issues.apache.org/jira/browse/FLINK-17202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17452745#comment-17452745 ] ZhuoYu Chen commented on FLINK-17202: - Hi [~csbliss] I am very interested in this，and I want do some job for flink，can I help to do that？ Thank you > Add SQL for ClickHouse connector > > > Key: FLINK-17202 > URL: https://issues.apache.org/jira/browse/FLINK-17202 > Project: Flink > Issue Type: Sub-task >Reporter: jinhai >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-17201) Implement Streaming ClickHouseSink
[ https://issues.apache.org/jira/browse/FLINK-17201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17452746#comment-17452746 ] ZhuoYu Chen commented on FLINK-17201: - Hi [~csbliss] I am very interested in this，and I want do some job for flink，can I help to do that？ Thank you > Implement Streaming ClickHouseSink > -- > > Key: FLINK-17201 > URL: https://issues.apache.org/jira/browse/FLINK-17201 > Project: Flink > Issue Type: Sub-task >Reporter: jinhai >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-25141) Elasticsearch connector customize sink parallelism
[ https://issues.apache.org/jira/browse/FLINK-25141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ada Wong updated FLINK-25141: - Component/s: Table SQL / Ecosystem > Elasticsearch connector customize sink parallelism > -- > > Key: FLINK-25141 > URL: https://issues.apache.org/jira/browse/FLINK-25141 > Project: Flink > Issue Type: Improvement > Components: Connectors / ElasticSearch, Table SQL / Ecosystem >Affects Versions: 1.14.0 >Reporter: Ada Wong >Priority: Major > > Inspired by JDBC and Kafka connector, add a 'sink.parallelism' option, and > using SinkProvider#of(sink, sinkParallelism). > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #17979: [FLINK-25122] [flink-dist] Add variable expansion for FLINK_ENV_JAVA_OPTS
flinkbot edited a comment on pull request #17979: URL: https://github.com/apache/flink/pull/17979#issuecomment-983597309 ## CI report: * 98a1592d95ba0ef58c0320f8e62c926950359058 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27358) * e236a3a2508d33020a5e69512048fe4effaba936 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27490) * 7f6a57dd89e3b8e9f750115cbadb21d21641e728 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27493) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17979: [FLINK-25122] [flink-dist] Add variable expansion for FLINK_ENV_JAVA_OPTS
flinkbot edited a comment on pull request #17979: URL: https://github.com/apache/flink/pull/17979#issuecomment-983597309 ## CI report: * 98a1592d95ba0ef58c0320f8e62c926950359058 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27358) * e236a3a2508d33020a5e69512048fe4effaba936 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27490) * 7f6a57dd89e3b8e9f750115cbadb21d21641e728 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17979: [FLINK-25122] [flink-dist] Add variable expansion for FLINK_ENV_JAVA_OPTS
flinkbot edited a comment on pull request #17979: URL: https://github.com/apache/flink/pull/17979#issuecomment-983597309 ## CI report: * 98a1592d95ba0ef58c0320f8e62c926950359058 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27358) * e236a3a2508d33020a5e69512048fe4effaba936 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27490) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17979: [FLINK-25122] [flink-dist] Add variable expansion for FLINK_ENV_JAVA_OPTS
flinkbot edited a comment on pull request #17979: URL: https://github.com/apache/flink/pull/17979#issuecomment-983597309 ## CI report: * 98a1592d95ba0ef58c0320f8e62c926950359058 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27358) * e236a3a2508d33020a5e69512048fe4effaba936 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27490) * 7f6a57dd89e3b8e9f750115cbadb21d21641e728 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17979: [FLINK-25122] [flink-dist] Add variable expansion for FLINK_ENV_JAVA_OPTS
flinkbot edited a comment on pull request #17979: URL: https://github.com/apache/flink/pull/17979#issuecomment-983597309 ## CI report: * 98a1592d95ba0ef58c0320f8e62c926950359058 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27358) * e236a3a2508d33020a5e69512048fe4effaba936 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27490) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17979: [FLINK-25122] [flink-dist] Add variable expansion for FLINK_ENV_JAVA_OPTS
flinkbot edited a comment on pull request #17979: URL: https://github.com/apache/flink/pull/17979#issuecomment-983597309 ## CI report: * 98a1592d95ba0ef58c0320f8e62c926950359058 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27358) * e236a3a2508d33020a5e69512048fe4effaba936 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27490) * 7f6a57dd89e3b8e9f750115cbadb21d21641e728 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17979: [FLINK-25122] [flink-dist] Add variable expansion for FLINK_ENV_JAVA_OPTS
flinkbot edited a comment on pull request #17979: URL: https://github.com/apache/flink/pull/17979#issuecomment-983597309 ## CI report: * 98a1592d95ba0ef58c0320f8e62c926950359058 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27358) * e236a3a2508d33020a5e69512048fe4effaba936 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27490) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17979: [FLINK-25122] [flink-dist] Add variable expansion for FLINK_ENV_JAVA_OPTS
flinkbot edited a comment on pull request #17979: URL: https://github.com/apache/flink/pull/17979#issuecomment-983597309 ## CI report: * 98a1592d95ba0ef58c0320f8e62c926950359058 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27358) * e236a3a2508d33020a5e69512048fe4effaba936 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27490) * 7f6a57dd89e3b8e9f750115cbadb21d21641e728 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17892: [FLINK-25038][testutils] Refactor FlinkContainer to split JM and TMs to individual containers and support HA
flinkbot edited a comment on pull request #17892: URL: https://github.com/apache/flink/pull/17892#issuecomment-977710431 ## CI report: * ef9bf98289409eafd1c3e49e9ae96317a4397a34 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26995) * 0c477cc849835f21dace860974ff6ebe61740893 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27492) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17542: [FLINK-17782] Add array,map,row types support for parquet row writer
flinkbot edited a comment on pull request #17542: URL: https://github.com/apache/flink/pull/17542#issuecomment-949170131 ## CI report: * 20b5850a68b1463d36796ccbfb8ea3eae5bf1221 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27426) * d0d2a946486ca5c1c021b3fb81b27870ca04ec29 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27491) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-24103) Create time-based LAST_VALUE / FIRST_VALUE
[
https://issues.apache.org/jira/browse/FLINK-24103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17452738#comment-17452738
]
ZhuoYu Chen edited comment on FLINK-24103 at 12/3/21, 4:21 AM:
---
Hi [~twalthr] I am very interested in this，and I want do some job for flink，can
I help to do that？
Thank you
was (Author: monster#12):
[~twalthr]Walther I am very interested in this，and I want do some job for
flink，can I help to do that？
Thank you
> Create time-based LAST_VALUE / FIRST_VALUE
> --
>
> Key: FLINK-24103
> URL: https://issues.apache.org/jira/browse/FLINK-24103
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Runtime
>Reporter: Timo Walther
>Priority: Major
>
> LAST_VALUE and FIRST_VALUE don't support merging. As far I can see it,
> FLINK-20110 tries to solve this by using nano second timestamps internally.
> However, an easier and consistent approach could be to allow a time parameter
> in the signature:
> {code}
> LAST_VALUE(timestamp, value)
> FIRST_VALUE(timestamp, value)
> {code}
> This allows merging based on a timestamp in HOP or SESSION windows.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #17979: [FLINK-25122] [flink-dist] Add variable expansion for FLINK_ENV_JAVA_OPTS
flinkbot edited a comment on pull request #17979: URL: https://github.com/apache/flink/pull/17979#issuecomment-983597309 ## CI report: * 98a1592d95ba0ef58c0320f8e62c926950359058 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27358) * e236a3a2508d33020a5e69512048fe4effaba936 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27490) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-24103) Create time-based LAST_VALUE / FIRST_VALUE
[
https://issues.apache.org/jira/browse/FLINK-24103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17452738#comment-17452738
]
ZhuoYu Chen commented on FLINK-24103:
-
[~twalthr] 我对这个非常感兴趣，我想为flink做一些工作，我可以帮助做吗？
谢谢你
> Create time-based LAST_VALUE / FIRST_VALUE
> --
>
> Key: FLINK-24103
> URL: https://issues.apache.org/jira/browse/FLINK-24103
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Runtime
>Reporter: Timo Walther
>Priority: Major
>
> LAST_VALUE and FIRST_VALUE don't support merging. As far I can see it,
> FLINK-20110 tries to solve this by using nano second timestamps internally.
> However, an easier and consistent approach could be to allow a time parameter
> in the signature:
> {code}
> LAST_VALUE(timestamp, value)
> FIRST_VALUE(timestamp, value)
> {code}
> This allows merging based on a timestamp in HOP or SESSION windows.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-24900) Support to run multiple shuffle plugins in one session cluster
[ https://issues.apache.org/jira/browse/FLINK-24900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17452737#comment-17452737 ] ZhuoYu Chen commented on FLINK-24900: - Hi [~kevin.cyj] I am very interested in this，and I want do some job for flink，can I help to do that？ Thank you > Support to run multiple shuffle plugins in one session cluster > -- > > Key: FLINK-24900 > URL: https://issues.apache.org/jira/browse/FLINK-24900 > Project: Flink > Issue Type: Sub-task >Reporter: Yingjie Cao >Priority: Major > > Currently, one Flink cluster can only use one shuffle plugin. However, there > are cases where different jobs may need different shuffle implementations. By > loading shuffle plugin with the plugin manager and letting jobs select their > shuffle service freely, Flink can support to run multiple shuffle plugins in > one session cluster. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #17892: [FLINK-25038][testutils] Refactor FlinkContainer to split JM and TMs to individual containers and support HA
flinkbot edited a comment on pull request #17892: URL: https://github.com/apache/flink/pull/17892#issuecomment-977710431 ## CI report: * ef9bf98289409eafd1c3e49e9ae96317a4397a34 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26995) * 0c477cc849835f21dace860974ff6ebe61740893 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-24103) Create time-based LAST_VALUE / FIRST_VALUE
[
https://issues.apache.org/jira/browse/FLINK-24103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17452738#comment-17452738
]
ZhuoYu Chen edited comment on FLINK-24103 at 12/3/21, 4:20 AM:
---
[~twalthr]Walther I am very interested in this，and I want do some job for
flink，can I help to do that？
Thank you
was (Author: monster#12):
[~twalthr] 我对这个非常感兴趣，我想为flink做一些工作，我可以帮助做吗？
谢谢你
> Create time-based LAST_VALUE / FIRST_VALUE
> --
>
> Key: FLINK-24103
> URL: https://issues.apache.org/jira/browse/FLINK-24103
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Runtime
>Reporter: Timo Walther
>Priority: Major
>
> LAST_VALUE and FIRST_VALUE don't support merging. As far I can see it,
> FLINK-20110 tries to solve this by using nano second timestamps internally.
> However, an easier and consistent approach could be to allow a time parameter
> in the signature:
> {code}
> LAST_VALUE(timestamp, value)
> FIRST_VALUE(timestamp, value)
> {code}
> This allows merging based on a timestamp in HOP or SESSION windows.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #17542: [FLINK-17782] Add array,map,row types support for parquet row writer
flinkbot edited a comment on pull request #17542: URL: https://github.com/apache/flink/pull/17542#issuecomment-949170131 ## CI report: * 20b5850a68b1463d36796ccbfb8ea3eae5bf1221 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27426) * d0d2a946486ca5c1c021b3fb81b27870ca04ec29 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-24103) Create time-based LAST_VALUE / FIRST_VALUE
[
https://issues.apache.org/jira/browse/FLINK-24103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17452736#comment-17452736
]
ZhuoYu Chen commented on FLINK-24103:
-
[~twalthr] Is there a result of the discussion now?
> Create time-based LAST_VALUE / FIRST_VALUE
> --
>
> Key: FLINK-24103
> URL: https://issues.apache.org/jira/browse/FLINK-24103
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Runtime
>Reporter: Timo Walther
>Priority: Major
>
> LAST_VALUE and FIRST_VALUE don't support merging. As far I can see it,
> FLINK-20110 tries to solve this by using nano second timestamps internally.
> However, an easier and consistent approach could be to allow a time parameter
> in the signature:
> {code}
> LAST_VALUE(timestamp, value)
> FIRST_VALUE(timestamp, value)
> {code}
> This allows merging based on a timestamp in HOP or SESSION windows.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink-ml] zhipeng93 commented on a change in pull request #28: [Flink-24556] Add Estimator and Transformer for logistic regression
zhipeng93 commented on a change in pull request #28:
URL: https://github.com/apache/flink-ml/pull/28#discussion_r761636447
##
File path:
flink-ml-core/src/main/java/org/apache/flink/ml/common/datastream/DataStreamUtils.java
##
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.common.datastream;
+
+import org.apache.flink.api.common.functions.MapPartitionFunction;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.TypeComparator;
+import org.apache.flink.api.java.typeutils.TypeExtractor;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.BoundedOneInput;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+
+import org.apache.commons.collections.IteratorUtils;
+
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+/** Provides utility functions for {@link DataStream}. */
+public class DataStreamUtils {
+/**
+ * Applies allReduceSum on the input data stream. The input data stream is
supposed to contain
+ * one double array in each partition. The result data stream has the same
parallelism as the
+ * input, where each partition contains one double array that sums all of
the double arrays in
+ * the input data stream.
+ *
+ * Note that we throw exception when one of the following two cases
happen:
+ * There exists one partition that contains more than one double array.
+ * The length of the double array is not consistent among all
partitions.
+ *
+ * @param input The input data stream.
+ * @return The result data stream.
+ */
+public static DataStream allReduceSum(DataStream
input) {
+return AllReduceImpl.allReduceSum(input);
+}
+
+/**
+ * Collects distinct values in a bounded data stream. The parallelism of
the output stream is 1.
+ *
+ * @param The class type of the input data stream.
+ * @param input The bounded input data stream.
+ * @return The result data stream that contains all the distinct values.
+ */
+public static DataStream distinct(DataStream input) {
+return input.transform(
+"distinctInEachPartition",
+input.getType(),
+new DistinctPartitionOperator<>())
+.setParallelism(input.getParallelism())
+.transform(
+"distinctInFinalPartition",
+input.getType(),
+new DistinctPartitionOperator<>())
+.setParallelism(1);
+}
+
+/**
+ * Applies a {@link MapPartitionFunction} on a bounded data stream.
+ *
+ * @param input The input data stream.
+ * @param func The user defined mapPartition function.
+ * @param The class type of the input element.
+ * @param The class type of output element.
+ * @return The result data stream.
+ */
+public static DataStream mapPartition(
+DataStream input, MapPartitionFunction func) {
+TypeInformation resultType =
+TypeExtractor.getMapPartitionReturnTypes(func,
input.getType(), null, true);
+return input.transform("mapPartition", resultType, new
MapPartitionOperator<>(func))
+.setParallelism(input.getParallelism());
+}
+
+/**
+ * Sorts the elements in each partition of the input bounded data stream.
+ *
+ * @param input The input data stream.
+ * @param comparator The comparator used to sort the elements.
+ * @param The class type of input element.
+ * @return The sorted data stream.
+ */
+
[GitHub] [flink-ml] zhipeng93 commented on a change in pull request #28: [Flink-24556] Add Estimator and Transformer for logistic regression
zhipeng93 commented on a change in pull request #28:
URL: https://github.com/apache/flink-ml/pull/28#discussion_r761636447
##
File path:
flink-ml-core/src/main/java/org/apache/flink/ml/common/datastream/DataStreamUtils.java
##
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.common.datastream;
+
+import org.apache.flink.api.common.functions.MapPartitionFunction;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.TypeComparator;
+import org.apache.flink.api.java.typeutils.TypeExtractor;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.BoundedOneInput;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+
+import org.apache.commons.collections.IteratorUtils;
+
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+/** Provides utility functions for {@link DataStream}. */
+public class DataStreamUtils {
+/**
+ * Applies allReduceSum on the input data stream. The input data stream is
supposed to contain
+ * one double array in each partition. The result data stream has the same
parallelism as the
+ * input, where each partition contains one double array that sums all of
the double arrays in
+ * the input data stream.
+ *
+ * Note that we throw exception when one of the following two cases
happen:
+ * There exists one partition that contains more than one double array.
+ * The length of the double array is not consistent among all
partitions.
+ *
+ * @param input The input data stream.
+ * @return The result data stream.
+ */
+public static DataStream allReduceSum(DataStream
input) {
+return AllReduceImpl.allReduceSum(input);
+}
+
+/**
+ * Collects distinct values in a bounded data stream. The parallelism of
the output stream is 1.
+ *
+ * @param The class type of the input data stream.
+ * @param input The bounded input data stream.
+ * @return The result data stream that contains all the distinct values.
+ */
+public static DataStream distinct(DataStream input) {
+return input.transform(
+"distinctInEachPartition",
+input.getType(),
+new DistinctPartitionOperator<>())
+.setParallelism(input.getParallelism())
+.transform(
+"distinctInFinalPartition",
+input.getType(),
+new DistinctPartitionOperator<>())
+.setParallelism(1);
+}
+
+/**
+ * Applies a {@link MapPartitionFunction} on a bounded data stream.
+ *
+ * @param input The input data stream.
+ * @param func The user defined mapPartition function.
+ * @param The class type of the input element.
+ * @param The class type of output element.
+ * @return The result data stream.
+ */
+public static DataStream mapPartition(
+DataStream input, MapPartitionFunction func) {
+TypeInformation resultType =
+TypeExtractor.getMapPartitionReturnTypes(func,
input.getType(), null, true);
+return input.transform("mapPartition", resultType, new
MapPartitionOperator<>(func))
+.setParallelism(input.getParallelism());
+}
+
+/**
+ * Sorts the elements in each partition of the input bounded data stream.
+ *
+ * @param input The input data stream.
+ * @param comparator The comparator used to sort the elements.
+ * @param The class type of input element.
+ * @return The sorted data stream.
+ */
+
[GitHub] [flink-ml] zhipeng93 commented on a change in pull request #28: [Flink-24556] Add Estimator and Transformer for logistic regression
zhipeng93 commented on a change in pull request #28:
URL: https://github.com/apache/flink-ml/pull/28#discussion_r761636447
##
File path:
flink-ml-core/src/main/java/org/apache/flink/ml/common/datastream/DataStreamUtils.java
##
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.common.datastream;
+
+import org.apache.flink.api.common.functions.MapPartitionFunction;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.TypeComparator;
+import org.apache.flink.api.java.typeutils.TypeExtractor;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.BoundedOneInput;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+
+import org.apache.commons.collections.IteratorUtils;
+
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+/** Provides utility functions for {@link DataStream}. */
+public class DataStreamUtils {
+/**
+ * Applies allReduceSum on the input data stream. The input data stream is
supposed to contain
+ * one double array in each partition. The result data stream has the same
parallelism as the
+ * input, where each partition contains one double array that sums all of
the double arrays in
+ * the input data stream.
+ *
+ * Note that we throw exception when one of the following two cases
happen:
+ * There exists one partition that contains more than one double array.
+ * The length of the double array is not consistent among all
partitions.
+ *
+ * @param input The input data stream.
+ * @return The result data stream.
+ */
+public static DataStream allReduceSum(DataStream
input) {
+return AllReduceImpl.allReduceSum(input);
+}
+
+/**
+ * Collects distinct values in a bounded data stream. The parallelism of
the output stream is 1.
+ *
+ * @param The class type of the input data stream.
+ * @param input The bounded input data stream.
+ * @return The result data stream that contains all the distinct values.
+ */
+public static DataStream distinct(DataStream input) {
+return input.transform(
+"distinctInEachPartition",
+input.getType(),
+new DistinctPartitionOperator<>())
+.setParallelism(input.getParallelism())
+.transform(
+"distinctInFinalPartition",
+input.getType(),
+new DistinctPartitionOperator<>())
+.setParallelism(1);
+}
+
+/**
+ * Applies a {@link MapPartitionFunction} on a bounded data stream.
+ *
+ * @param input The input data stream.
+ * @param func The user defined mapPartition function.
+ * @param The class type of the input element.
+ * @param The class type of output element.
+ * @return The result data stream.
+ */
+public static DataStream mapPartition(
+DataStream input, MapPartitionFunction func) {
+TypeInformation resultType =
+TypeExtractor.getMapPartitionReturnTypes(func,
input.getType(), null, true);
+return input.transform("mapPartition", resultType, new
MapPartitionOperator<>(func))
+.setParallelism(input.getParallelism());
+}
+
+/**
+ * Sorts the elements in each partition of the input bounded data stream.
+ *
+ * @param input The input data stream.
+ * @param comparator The comparator used to sort the elements.
+ * @param The class type of input element.
+ * @return The sorted data stream.
+ */
+
[GitHub] [flink] flinkbot edited a comment on pull request #17979: [FLINK-25122] [flink-dist] Add variable expansion for FLINK_ENV_JAVA_OPTS
flinkbot edited a comment on pull request #17979: URL: https://github.com/apache/flink/pull/17979#issuecomment-983597309 ## CI report: * 98a1592d95ba0ef58c0320f8e62c926950359058 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27358) * e236a3a2508d33020a5e69512048fe4effaba936 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27490) * 7f6a57dd89e3b8e9f750115cbadb21d21641e728 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17979: [FLINK-25122] [flink-dist] Add variable expansion for FLINK_ENV_JAVA_OPTS
flinkbot edited a comment on pull request #17979: URL: https://github.com/apache/flink/pull/17979#issuecomment-983597309 ## CI report: * 98a1592d95ba0ef58c0320f8e62c926950359058 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27358) * e236a3a2508d33020a5e69512048fe4effaba936 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27490) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-24910) Propagate the Calcite parser config to SQL Client
[ https://issues.apache.org/jira/browse/FLINK-24910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17452734#comment-17452734 ] ZhuoYu Chen commented on FLINK-24910: - Hi [~Sergey Nuyanzin] I am very interested in this，and I want do some job for flink，can I help to do that？ Thank you > Propagate the Calcite parser config to SQL Client > - > > Key: FLINK-24910 > URL: https://issues.apache.org/jira/browse/FLINK-24910 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Client >Reporter: Sergey Nuyanzin >Priority: Major > > It's required to get Dialect specific info like keywords, sql quote identifier -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #17979: [FLINK-25122] [flink-dist] Add variable expansion for FLINK_ENV_JAVA_OPTS
flinkbot edited a comment on pull request #17979: URL: https://github.com/apache/flink/pull/17979#issuecomment-983597309 ## CI report: * 98a1592d95ba0ef58c0320f8e62c926950359058 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27358) * e236a3a2508d33020a5e69512048fe4effaba936 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27490) * 7f6a57dd89e3b8e9f750115cbadb21d21641e728 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] PatrickRen commented on a change in pull request #17892: [FLINK-25038][testutils] Refactor FlinkContainer to split JM and TMs to individual containers and support HA
PatrickRen commented on a change in pull request #17892:
URL: https://github.com/apache/flink/pull/17892#discussion_r761634524
##
File path:
flink-end-to-end-tests/flink-end-to-end-tests-common/src/main/java/org/apache/flink/tests/util/flink/container/FlinkContainer.java
##
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.tests.util.flink.container;
+
+import org.apache.flink.client.deployment.StandaloneClusterId;
+import org.apache.flink.client.program.rest.RestClusterClient;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.RestOptions;
+import org.apache.flink.core.testutils.CommonTestUtils;
+import
org.apache.flink.runtime.rest.handler.legacy.messages.ClusterOverviewWithVersion;
+import org.apache.flink.runtime.rest.messages.ClusterOverviewHeaders;
+import org.apache.flink.runtime.rest.messages.EmptyMessageParameters;
+import org.apache.flink.runtime.rest.messages.EmptyRequestBody;
+import org.apache.flink.tests.util.flink.SQLJobSubmission;
+import org.apache.flink.util.function.RunnableWithException;
+
+import org.junit.jupiter.api.extension.AfterAllCallback;
+import org.junit.jupiter.api.extension.BeforeAllCallback;
+import org.junit.jupiter.api.extension.ExtensionContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.testcontainers.containers.Container;
+import org.testcontainers.containers.GenericContainer;
+import org.testcontainers.utility.MountableFile;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkState;
+
+/** A Flink cluster running JM and TM on different containers. */
+public class FlinkContainer implements BeforeAllCallback, AfterAllCallback {
+private static final Logger LOG =
LoggerFactory.getLogger(FlinkContainer.class);
+
+// Hostname of components within container network
+public static final String JOB_MANAGER_HOSTNAME = "jobmanager";
+public static final String TASK_MANAGER_HOSTNAME_PREFIX = "taskmanager-";
+public static final String ZOOKEEPER_HOSTNAME = "zookeeper";
+
+// Directories for storing states
+public static final Path CHECKPOINT_PATH = Paths.get("/flink/checkpoint");
+public static final Path HA_STORAGE_PATH = Paths.get("/flink/recovery");
+
+private final GenericContainer jobManager;
+private final List> taskManagers;
+private final GenericContainer haService;
+private final Configuration conf;
+
+private RestClusterClient restClusterClient;
+private boolean isRunning;
+
+/** Creates a builder for {@link FlinkContainer}. */
+public static FlinkContainerBuilder builder() {
+return new FlinkContainerBuilder();
+}
+
+FlinkContainer(
+GenericContainer jobManager,
+List> taskManagers,
+@Nullable GenericContainer haService,
+Configuration conf) {
+this.jobManager = jobManager;
+this.taskManagers = taskManagers;
+this.haService = haService;
+this.conf = conf;
+}
+
+/** Starts all containers. */
+public void start() throws Exception {
+if (haService != null) {
+LOG.debug("Starting HA service container");
+this.haService.start();
+}
+LOG.debug("Starting JobManager container");
+this.jobManager.start();
+LOG.debug("Starting TaskManager containers");
+this.taskManagers.parallelStream().forEach(GenericContainer::start);
+LOG.debug("Creating REST cluster client");
+this.restClusterClient = createClusterClient();
+waitUntilAllTaskManagerConnected();
+isRunning = true;
+}
+
+/** Stops all containers. */
+public void stop() {
+isRunning = false;
+if (restClusterClient != null) {
+restClusterClient.close();
+}
+this.taskManagers.forEach(GenericContainer::stop);
+
[GitHub] [flink] flinkbot edited a comment on pull request #17979: [FLINK-25122] [flink-dist] Add variable expansion for FLINK_ENV_JAVA_OPTS
flinkbot edited a comment on pull request #17979: URL: https://github.com/apache/flink/pull/17979#issuecomment-983597309 ## CI report: * 98a1592d95ba0ef58c0320f8e62c926950359058 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27358) * e236a3a2508d33020a5e69512048fe4effaba936 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27490) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] shenzhu commented on pull request #17793: [FLINK-21565][Table SQL/API] Support more integer types in TIMESTAMPADD
shenzhu commented on pull request #17793: URL: https://github.com/apache/flink/pull/17793#issuecomment-985199167 Hey @twalthr , sorry to bother you, would you mind taking a look at this PR when you have a moment? Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17979: [FLINK-25122] [flink-dist] Add variable expansion for FLINK_ENV_JAVA_OPTS
flinkbot edited a comment on pull request #17979: URL: https://github.com/apache/flink/pull/17979#issuecomment-983597309 ## CI report: * 98a1592d95ba0ef58c0320f8e62c926950359058 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27358) * e236a3a2508d33020a5e69512048fe4effaba936 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27490) * 7f6a57dd89e3b8e9f750115cbadb21d21641e728 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a change in pull request #28: [Flink-24556] Add Estimator and Transformer for logistic regression
zhipeng93 commented on a change in pull request #28:
URL: https://github.com/apache/flink-ml/pull/28#discussion_r761633170
##
File path:
flink-ml-core/src/main/java/org/apache/flink/ml/common/datastream/DataStreamUtils.java
##
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.common.datastream;
+
+import org.apache.flink.api.common.functions.MapPartitionFunction;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.TypeComparator;
+import org.apache.flink.api.java.typeutils.TypeExtractor;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.BoundedOneInput;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+
+import org.apache.commons.collections.IteratorUtils;
+
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+/** Provides utility functions for {@link DataStream}. */
+public class DataStreamUtils {
+/**
+ * Applies allReduceSum on the input data stream. The input data stream is
supposed to contain
+ * one double array in each partition. The result data stream has the same
parallelism as the
+ * input, where each partition contains one double array that sums all of
the double arrays in
+ * the input data stream.
+ *
+ * Note that we throw exception when one of the following two cases
happen:
+ * There exists one partition that contains more than one double array.
+ * The length of the double array is not consistent among all
partitions.
+ *
+ * @param input The input data stream.
+ * @return The result data stream.
+ */
+public static DataStream allReduceSum(DataStream
input) {
+return AllReduceImpl.allReduceSum(input);
+}
+
+/**
+ * Collects distinct values in a bounded data stream. The parallelism of
the output stream is 1.
+ *
+ * @param The class type of the input data stream.
+ * @param input The bounded input data stream.
+ * @return The result data stream that contains all the distinct values.
+ */
+public static DataStream distinct(DataStream input) {
+return input.transform(
+"distinctInEachPartition",
+input.getType(),
+new DistinctPartitionOperator<>())
+.setParallelism(input.getParallelism())
+.transform(
+"distinctInFinalPartition",
+input.getType(),
+new DistinctPartitionOperator<>())
+.setParallelism(1);
+}
+
+/**
+ * Applies a {@link MapPartitionFunction} on a bounded data stream.
+ *
+ * @param input The input data stream.
+ * @param func The user defined mapPartition function.
+ * @param The class type of the input element.
+ * @param The class type of output element.
+ * @return The result data stream.
+ */
+public static DataStream mapPartition(
+DataStream input, MapPartitionFunction func) {
+TypeInformation resultType =
+TypeExtractor.getMapPartitionReturnTypes(func,
input.getType(), null, true);
+return input.transform("mapPartition", resultType, new
MapPartitionOperator<>(func))
+.setParallelism(input.getParallelism());
+}
+
+/**
+ * Sorts the elements in each partition of the input bounded data stream.
+ *
+ * @param input The input data stream.
+ * @param comparator The comparator used to sort the elements.
+ * @param The class type of input element.
+ * @return The sorted data stream.
+ */
+
[GitHub] [flink] flinkbot edited a comment on pull request #17979: [FLINK-25122] [flink-dist] Add variable expansion for FLINK_ENV_JAVA_OPTS
flinkbot edited a comment on pull request #17979: URL: https://github.com/apache/flink/pull/17979#issuecomment-983597309 ## CI report: * 98a1592d95ba0ef58c0320f8e62c926950359058 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27358) * e236a3a2508d33020a5e69512048fe4effaba936 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27490) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
