[jira] [Commented] (FLINK-20747) ClassCastException when using MAX aggregate function
[ https://issues.apache.org/jira/browse/FLINK-20747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17275506#comment-17275506 ] charlieme commented on FLINK-20747: --- What's the problem with Flink after it's taken over by Alibaba. Bugs ,disordered type systems and plans. Shouldn't it be tested? > ClassCastException when using MAX aggregate function > > > Key: FLINK-20747 > URL: https://issues.apache.org/jira/browse/FLINK-20747 > Project: Flink > Issue Type: Bug > Components: Table SQL / Planner >Affects Versions: 1.12.0 >Reporter: zengjinbo >Priority: Critical > Fix For: 1.12.2, 1.13.0 > > Attachments: image-2020-12-23-18-04-21-079.png > > > During the process of upgrading 1.12.0, I found that Flink SQL 1.11.1 is not > compatible > java.lang.ClassCastException: java.lang.Integer cannot be cast to > org.apache.flink.table.data.StringDatajava.lang.ClassCastException: > java.lang.Integer cannot be cast to org.apache.flink.table.data.StringData at > org$apache$flink$table$planner$functions$aggfunctions$MaxWithRetractAggFunction$MaxWithRetractAccumulator$Converter.toInternal(Unknown > Source) at > org.apache.flink.table.data.conversion.StructuredObjectConverter.toInternal(StructuredObjectConverter.java:92) > at > org.apache.flink.table.data.conversion.StructuredObjectConverter.toInternal(StructuredObjectConverter.java:47) > at > org.apache.flink.table.data.conversion.DataStructureConverter.toInternalOrNull(DataStructureConverter.java:59) > at GroupAggsHandler$875.getAccumulators(Unknown Source) at > org.apache.flink.table.runtime.operators.aggregate.GroupAggFunction.processElement(GroupAggFunction.java:175) > at > org.apache.flink.table.runtime.operators.aggregate.GroupAggFunction.processElement(GroupAggFunction.java:45) > at > org.apache.flink.streaming.api.operators.KeyedProcessOperator.processElement(KeyedProcessOperator.java:85) > at > org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:193) > at > org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.processElement(StreamTaskNetworkInput.java:179) > at > org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.emitNext(StreamTaskNetworkInput.java:152) > at > org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:67) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:372) > at > org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:186) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:575) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:539) > at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:722) at > org.apache.flink.runtime.taskmanager.Task.run(Task.java:547) > > > !image-2020-12-23-18-04-21-079.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14816: [FLINK-21202][python] Introduce TimeWindow and CountWindow in PyFlink
flinkbot edited a comment on pull request #14816: URL: https://github.com/apache/flink/pull/14816#issuecomment-770164177 ## CI report: * daece76bc00e4dd9aab27b59c16fd67d57cc24d6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12690) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14816: [FLINK-21202][python] Introduce TimeWindow and CountWindow in PyFlink
flinkbot commented on pull request #14816: URL: https://github.com/apache/flink/pull/14816#issuecomment-770164177 ## CI report: * daece76bc00e4dd9aab27b59c16fd67d57cc24d6 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-21202) Introduce TimeWindow and CountWindow in PyFlink
[ https://issues.apache.org/jira/browse/FLINK-21202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17275488#comment-17275488 ] Huang Xingbo commented on FLINK-21202: -- Hi [~wangpeibin] Why do you change the description? I don't think the description you added has much to do with this JIRA. > Introduce TimeWindow and CountWindow in PyFlink > --- > > Key: FLINK-21202 > URL: https://issues.apache.org/jira/browse/FLINK-21202 > Project: Flink > Issue Type: Sub-task > Components: API / Python >Affects Versions: 1.13.0 >Reporter: Huang Xingbo >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > > Now, when the lastrow function is used for de duplication, if two successive > rows are equal, a fallback row and an update row will still be generated, > which will trigger downstream repeated calculation. > Therefore, by optimizing the logic of the deduplicatefunctionhelper, when the > two rows before and after the fallback are equal, it can automatically > offset, thus reducing the amount of calculation > In order to reduce the impact on the existing logic, this change can be > opened manually by adding a configuration item as a switch -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #14816: [FLINK-21202][python] Introduce TimeWindow and CountWindow in PyFlink
flinkbot commented on pull request #14816: URL: https://github.com/apache/flink/pull/14816#issuecomment-770162262 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit daece76bc00e4dd9aab27b59c16fd67d57cc24d6 (Sat Jan 30 05:43:59 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-21202).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-21202) Introduce TimeWindow and CountWindow in PyFlink
[ https://issues.apache.org/jira/browse/FLINK-21202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-21202: --- Labels: pull-request-available (was: ) > Introduce TimeWindow and CountWindow in PyFlink > --- > > Key: FLINK-21202 > URL: https://issues.apache.org/jira/browse/FLINK-21202 > Project: Flink > Issue Type: Sub-task > Components: API / Python >Affects Versions: 1.13.0 >Reporter: Huang Xingbo >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > > Now, when the lastrow function is used for de duplication, if two successive > rows are equal, a fallback row and an update row will still be generated, > which will trigger downstream repeated calculation. > Therefore, by optimizing the logic of the deduplicatefunctionhelper, when the > two rows before and after the fallback are equal, it can automatically > offset, thus reducing the amount of calculation > In order to reduce the impact on the existing logic, this change can be > opened manually by adding a configuration item as a switch -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] HuangXingBo opened a new pull request #14816: [FLINK-21202][python] Introduce TimeWindow and CountWindow in PyFlink
HuangXingBo opened a new pull request #14816: URL: https://github.com/apache/flink/pull/14816 ## What is the purpose of the change *This pull request will Introduce TimeWindow, CountWindow and corresponding coders.* ## Brief change log - *Add `TimeWindow` and `CountWindow`* - *Add coders and implementation of `TimeWindowCoder` and `CountWindowCoder`* ## Verifying this change This change added tests and can be verified as follows: - *Add python coder test in `test_coders.py`* - *Add cython coder test in `test_fast_coders.py`* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14813: [FLINK-21207] Fix 'csv.disable-quote-character' can not take effect during deserial…
flinkbot edited a comment on pull request #14813: URL: https://github.com/apache/flink/pull/14813#issuecomment-769867480 ## CI report: * ab61d899b8b08c6c4707ba65f922c4a838bdbb6e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12667) * e5f633baea6ed961c21635feedab97683c429def Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12689) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14813: [FLINK-21207] Fix 'csv.disable-quote-character' can not take effect during deserial…
flinkbot edited a comment on pull request #14813: URL: https://github.com/apache/flink/pull/14813#issuecomment-769867480 ## CI report: * ab61d899b8b08c6c4707ba65f922c4a838bdbb6e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12667) * e5f633baea6ed961c21635feedab97683c429def UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14813: [FLINK-21207] Fix 'csv.disable-quote-character' can not take effect during deserial…
flinkbot edited a comment on pull request #14813: URL: https://github.com/apache/flink/pull/14813#issuecomment-769867480 ## CI report: * ab61d899b8b08c6c4707ba65f922c4a838bdbb6e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12667) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] sharkdtu commented on pull request #14813: [FLINK-21207] Fix 'csv.disable-quote-character' can not take effect during deserial…
sharkdtu commented on pull request #14813: URL: https://github.com/apache/flink/pull/14813#issuecomment-770156048 @flinkbot run azure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-21212) Can no longer cast INT to DATE
Rex Remind created FLINK-21212:
--
Summary: Can no longer cast INT to DATE
Key: FLINK-21212
URL: https://issues.apache.org/jira/browse/FLINK-21212
Project: Flink
Issue Type: Improvement
Components: API / Type Serialization System
Affects Versions: 1.12.1
Environment: EMR 6.1
Flink 1.12.1
Reporter: Rex Remind
I upgraded from 1.11.3 to 1.12.1 and can no longer cast int to date. Data is
arriving from Debezium. We also went from json to avro but I'd think that would
be unrelated.
Example:
{code:java}
.addOrReplaceColumns(
$"date".cast(Types.SQL_DATE()) as "date"
) {code}
Result:
{code:java}
org.apache.flink.client.program.ProgramInvocationException: The main method
caused an error: Invalid function call:
cast(INT, DATE)
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:360)
at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:213)
at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:816)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:248)
at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1058)
at
org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1136)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at
org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1136)
Caused by: org.apache.flink.table.api.ValidationException: Invalid function
call:
cast(INT, DATE)
at
org.apache.flink.table.types.inference.TypeInferenceUtil.createInvalidCallException(TypeInferenceUtil.java:190)
at
org.apache.flink.table.types.inference.TypeInferenceUtil.runTypeInference(TypeInferenceUtil.java:87)
at
org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule$ResolvingCallVisitor.runTypeInference(ResolveCallByArgumentsRule.java:222)
at
org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule$ResolvingCallVisitor.lambda$visit$1(ResolveCallByArgumentsRule.java:149)
at java.base/java.util.Optional.map(Optional.java:265)
at
org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule$ResolvingCallVisitor.visit(ResolveCallByArgumentsRule.java:147)
at
org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule$ResolvingCallVisitor.visit(ResolveCallByArgumentsRule.java:91)
at
org.apache.flink.table.expressions.ApiExpressionVisitor.visit(ApiExpressionVisitor.java:37)
at
org.apache.flink.table.expressions.UnresolvedCallExpression.accept(UnresolvedCallExpression.java:128)
at
org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule$ResolvingCallVisitor.visit(ResolveCallByArgumentsRule.java:138)
at
org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule$ResolvingCallVisitor.visit(ResolveCallByArgumentsRule.java:91)
at
org.apache.flink.table.expressions.ApiExpressionVisitor.visit(ApiExpressionVisitor.java:37)
at
org.apache.flink.table.expressions.UnresolvedCallExpression.accept(UnresolvedCallExpression.java:128)
at
org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule.lambda$apply$0(ResolveCallByArgumentsRule.java:85)
at
java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271)
at
java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
at
java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at
java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at
java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
at
java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at
java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
at
org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule.apply(ResolveCallByArgumentsRule.java:86)
at
org.apache.flink.table.expressions.resolver.ExpressionResolver.lambda$null$1(ExpressionResolver.java:212)
at java.base/java.util.function.Function.lambda$andThen$1(Function.java:88)
at
org.apache.flink.table.expressions.resolver.ExpressionResolver.resolve(ExpressionResolver.java:175)
at
org.apache.flink.table.operations.utils.OperationTreeBuilder.projectInternal(OperationTreeBuilder.java:191)
at
org.apache.flink.table.operations.utils.OperationTreeBuilder.project(OperationTreeBuilder.java:163)
at
org.apache.flink.table.operations.utils.O
[GitHub] [flink] flinkbot edited a comment on pull request #14724: [FLINK-19360] Fink startup fails when $JAVA_HOME have space
flinkbot edited a comment on pull request #14724: URL: https://github.com/apache/flink/pull/14724#issuecomment-765052737 ## CI report: * 6004f698719a1c91fc64a8a97663a361fb7d0dd8 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12536) * 79d610f87c58b7c392d4a6cd46fb24eb9f7850c8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12688) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14724: [FLINK-19360] Fink startup fails when $JAVA_HOME have space
flinkbot edited a comment on pull request #14724: URL: https://github.com/apache/flink/pull/14724#issuecomment-765052737 ## CI report: * 6004f698719a1c91fc64a8a97663a361fb7d0dd8 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12536) * 79d610f87c58b7c392d4a6cd46fb24eb9f7850c8 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-21207) 'csv.disable-quote-character' can not take effect during deserialization
[ https://issues.apache.org/jira/browse/FLINK-21207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sharkd tu updated FLINK-21207: -- Fix Version/s: 1.13.0 1.12.2 > 'csv.disable-quote-character' can not take effect during deserialization > - > > Key: FLINK-21207 > URL: https://issues.apache.org/jira/browse/FLINK-21207 > Project: Flink > Issue Type: Bug >Reporter: sharkd tu >Priority: Major > Labels: pull-request-available > Fix For: 1.12.2, 1.13.0 > > > source table with 'csv.disable-quote-character' = 'true' can not take effect. > > Caused by: > org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonParseException: > Missing closing quote for valueCaused by: > org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonParseException: > Missing closing quote for value at [Source: UNKNOWN; line: 1, column: 29] at > org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1840) > at > org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:712) > at > org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvParser._reportParsingError(CsvParser.java:1250) > at > org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.impl.CsvDecoder._nextQuotedString(CsvDecoder.java:785) > at > org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.impl.CsvDecoder.nextString(CsvDecoder.java:630) > at > org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvParser._handleNextEntry(CsvParser.java:846) > at > org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvParser.nextFieldName(CsvParser.java:665) > at > org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:249) > at > org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:68) > at > org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:15) > at > org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectReader._bindAndClose(ObjectReader.java:1719) > at > org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:1282) > at > org.apache.flink.formats.csv.CsvRowDataDeserializationSchema.deserialize(CsvRowDataDeserializationSchema.java:155) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-21207) 'csv.disable-quote-character' can not take effect during deserialization
[ https://issues.apache.org/jira/browse/FLINK-21207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sharkd tu updated FLINK-21207: -- Description: source table with 'csv.disable-quote-character' = 'true' can not take effect. Caused by: org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonParseException: Missing closing quote for valueCaused by: org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonParseException: Missing closing quote for value at [Source: UNKNOWN; line: 1, column: 29] at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1840) at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:712) at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvParser._reportParsingError(CsvParser.java:1250) at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.impl.CsvDecoder._nextQuotedString(CsvDecoder.java:785) at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.impl.CsvDecoder.nextString(CsvDecoder.java:630) at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvParser._handleNextEntry(CsvParser.java:846) at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvParser.nextFieldName(CsvParser.java:665) at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:249) at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:68) at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:15) at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectReader._bindAndClose(ObjectReader.java:1719) at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:1282) at org.apache.flink.formats.csv.CsvRowDataDeserializationSchema.deserialize(CsvRowDataDeserializationSchema.java:155) was: source table with 'csv.disable-quote-character' = 'true' can not take effect. ``` Caused by: org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonParseException: Missing closing quote for valueCaused by: org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonParseException: Missing closing quote for value at [Source: UNKNOWN; line: 1, column: 29] at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1840) at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:712) at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvParser._reportParsingError(CsvParser.java:1250) at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.impl.CsvDecoder._nextQuotedString(CsvDecoder.java:785) at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.impl.CsvDecoder.nextString(CsvDecoder.java:630) at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvParser._handleNextEntry(CsvParser.java:846) at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvParser.nextFieldName(CsvParser.java:665) at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:249) at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:68) at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:15) at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectReader._bindAndClose(ObjectReader.java:1719) at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:1282) at org.apache.flink.formats.csv.CsvRowDataDeserializationSchema.deserialize(CsvRowDataDeserializationSchema.java:155) ``` > 'csv.disable-quote-character' can not take effect during deserialization > - > > Key: FLINK-21207 > URL: https://issues.apache.org/jira/browse/FLINK-21207 > Project: Flink > Issue Type: Bug >Reporter: sharkd tu >Priority: Major > Labels: pull-request-available > > source table with 'csv.disable-quote-character' = 'true' can not take effect. > > Caused by: > org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonParseException: > Missing closing quote for valueCaused by: > org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonP
[GitHub] [flink] flinkbot edited a comment on pull request #13912: [FLINK-19466][FLINK-19467][runtime / state backends] Add Flip-142 public interfaces and methods
flinkbot edited a comment on pull request #13912: URL: https://github.com/apache/flink/pull/13912#issuecomment-721398037 ## CI report: * 5bd74e2865ec431f3bf2528f5d85969ca51714fb Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12684) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14797: [FLINK-20654][FLINK-21104][network] Fix couple bugs in the handling of unaligned checkpoints and cancellations.
flinkbot edited a comment on pull request #14797: URL: https://github.com/apache/flink/pull/14797#issuecomment-769223743 ## CI report: * f7f5fb6ed31c5ed60bdf4407e6e6b779fe77b77a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12683) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-20991) Create 1.12.1 official image
[ https://issues.apache.org/jira/browse/FLINK-20991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xintong Song closed FLINK-20991. Resolution: Done Fixed in https://github.com/docker-library/official-images/pull/9526 > Create 1.12.1 official image > > > Key: FLINK-20991 > URL: https://issues.apache.org/jira/browse/FLINK-20991 > Project: Flink > Issue Type: Task > Components: Release System >Affects Versions: 1.12.1 >Reporter: Xintong Song >Assignee: Xintong Song >Priority: Major > > This ticket is a reminder for creating 1.12.1 official image once the 1.12.0 > official image issue is resolved. > Track the progress of 1.12.0 official image at: > https://github.com/docker-library/official-images/pull/9249 > https://github.com/docker-library/official-images/pull/9345 > Notice: > Before the DockerHub official images are available, one can still access the > images via the the repository maintained by the Apache Flink project. > https://hub.docker.com/repository/docker/apache/flink -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-17061) Unset process/flink memory size from configuration once dynamic worker resource is activated.
[
https://issues.apache.org/jira/browse/FLINK-17061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xintong Song updated FLINK-17061:
-
Parent: FLINK-14187
Issue Type: Sub-task (was: Task)
> Unset process/flink memory size from configuration once dynamic worker
> resource is activated.
> -
>
> Key: FLINK-17061
> URL: https://issues.apache.org/jira/browse/FLINK-17061
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Configuration, Runtime / Coordination
>Affects Versions: 1.11.0
>Reporter: Xintong Song
>Priority: Major
>
> With FLINK-14106, memory of a TaskExecutor is decided in two steps on active
> resource managers.
> - {{SlotManager}} decides {{WorkerResourceSpec}}, including memory used by
> Flink tasks: task heap, task off-heap, network and managed memory.
> - {{ResourceManager}} derives {{TaskExecutorProcessSpec}} from
> {{WorkerResourceSpec}} and the configuration, deciding sizes of memory used
> by Flink framework and JVM: framework heap, framework off-heap, jvm metaspace
> and jvm overhead.
> This works fine for now, because both {{WorkerResourceSpec}} and
> {{TaskExecutorProcessSpec}} are derived from the same configurations.
> However, it might cause problem if later we have new {{SlotManager}}
> implementations that decides {{WorkerResourceSpec}} dynamically. In such
> cases, the process/flink sizes in configuration should be ignored, or it may
> easily lead to configuration conflicts.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-17061) Unset process/flink memory size from configuration once dynamic worker resource is activated.
[
https://issues.apache.org/jira/browse/FLINK-17061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17275439#comment-17275439
]
Xintong Song commented on FLINK-17061:
--
Thanks for the notice. I think this should be a follow-up issue for FLIP-56.
Converting it to subtask of the FLIP-56 umbrella (FLINK-14187).
> Unset process/flink memory size from configuration once dynamic worker
> resource is activated.
> -
>
> Key: FLINK-17061
> URL: https://issues.apache.org/jira/browse/FLINK-17061
> Project: Flink
> Issue Type: Task
> Components: Runtime / Configuration, Runtime / Coordination
>Affects Versions: 1.11.0
>Reporter: Xintong Song
>Priority: Major
>
> With FLINK-14106, memory of a TaskExecutor is decided in two steps on active
> resource managers.
> - {{SlotManager}} decides {{WorkerResourceSpec}}, including memory used by
> Flink tasks: task heap, task off-heap, network and managed memory.
> - {{ResourceManager}} derives {{TaskExecutorProcessSpec}} from
> {{WorkerResourceSpec}} and the configuration, deciding sizes of memory used
> by Flink framework and JVM: framework heap, framework off-heap, jvm metaspace
> and jvm overhead.
> This works fine for now, because both {{WorkerResourceSpec}} and
> {{TaskExecutorProcessSpec}} are derived from the same configurations.
> However, it might cause problem if later we have new {{SlotManager}}
> implementations that decides {{WorkerResourceSpec}} dynamically. In such
> cases, the process/flink sizes in configuration should be ignored, or it may
> easily lead to configuration conflicts.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14814: [WIP] (1.11) [FLINK-21132][runtime] Don't end input on stop with savepoint
flinkbot edited a comment on pull request #14814: URL: https://github.com/apache/flink/pull/14814#issuecomment-769949284 ## CI report: * 53237d304da31ce8b2bdba5103192007122f23c7 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12682) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14810: [FLINK-9844][client] PackagedProgram#close() closes ClassLoader
flinkbot edited a comment on pull request #14810: URL: https://github.com/apache/flink/pull/14810#issuecomment-769830895 ## CI report: * 0dee1f6c07a80da9abc5690b2a87442dc1725fd9 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12679) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-16520) Move Kafka client properties resolution in KafkaSinkProvider to KafkaEgressBuilder
[
https://issues.apache.org/jira/browse/FLINK-16520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17275407#comment-17275407
]
Rui Wang commented on FLINK-16520:
--
[~tzulitai]
I am looking for some ramp-up tasks for this project. This JIRA seems a good.
Can I work on this?
> Move Kafka client properties resolution in KafkaSinkProvider to
> KafkaEgressBuilder
> --
>
> Key: FLINK-16520
> URL: https://issues.apache.org/jira/browse/FLINK-16520
> Project: Flink
> Issue Type: Improvement
> Components: Stateful Functions
>Affects Versions: statefun-2.0.0
>Reporter: Tzu-Li (Gordon) Tai
>Priority: Minor
>
> We've moved Kafka client properties resolution from the
> {{KafkaSourceProvider}} to be consolidated in the {{KafkaIngressBuilder}} in
> FLINK-15769.
> The same should also be done for the producer side, i.e. for
> {{KafkaSinkProvider}}, by moving the properties resolution logic to
> {{KafkaEgressBuilder}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-21211) Looking for reviews on a framework based on flink-statefun
[
https://issues.apache.org/jira/browse/FLINK-21211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zixuan Rao updated FLINK-21211:
---
Description:
Hi, I am currently developing a framework targeting back end state management.
To ensure exactly-once processing of events in back end, I intend to use Flink
Stateful Functions runtime in combination with Python's asyncio. I hope to
receive some feedbacks.
The following code shows an example (draft) of writing a back end micro service
using my framework. It is intended to be equivalent (exchangeable) with
Flink-stateful examples/ridesharing. The idea is that "Event" is reducible to
an async function call, and external egress can be emitted by saving an object.
This preserves the exactly-once features of Flink-statefun while adding a great
deal of readability to the code.
Reviews are appreciated. Thank you!
{code}
"""
Equivalent implementation for flink stateful functions example - ridesharing
ref:
https://github.com/apache/flink-statefun/blob/master/statefun-examples/statefun-ridesharing-example/statefun-ridesharing-example-functions/src/main/java/org/apache/flink/statefun/examples/ridesharing/FnDriver.java
"""
from onto.models.base import Serializable
"""
Rewrite callback-style code to async-await:
ref:
https://www.coreycleary.me/how-to-rewrite-a-callback-function-in-promise-form-and-async-await-form-in-javascript
"""
from onto.domain_model import DomainModel
from onto.attrs import attrs
class RideshareBase(DomainModel):
pass
class Passenger(RideshareBase):
async def request_ride(self, start_geo_cell, end_geo_cell):
r = Ride.create() # TODO: implement create
await r.passenger_joins(
passenger=self,
start_geo_cell=start_geo_cell,
end_geo_cell=end_geo_cell
)
class PassengerMessage(DomainModel):
passenger = attrs.relation('Passenger')
class RideFailedMessage(Serializable):
ride = attrs.relation('Ride')
ride_failed = attrs.embed(RideFailedMessage).optional
class DriverHasBeenFoundMessage(Serializable):
driver = attrs.relation('Driver')
driver_geo_cell = attrs.relation('GeoCell')
driver_found = attrs.embed(RideFailedMessage).optional
class RideHasStarted(Serializable):
driver = attrs.relation('Driver')
ride_started = attrs.embed(RideHasStarted).optional
class RideHasEnded(Serializable):
pass # TODO: make sure that empty class works
ride_ended = attrs.embed(RideHasEnded).optional
async def ride_failed(self, ride: 'Ride'):
message = self.PassengerMessage.new(
passenger=self,
ride_failed=self.PassengerMessage.RideFailedMessage.new(
ride=ride
)
)
message.save()
async def driver_joins_ride(self, driver: 'Driver', driver_geo_cell:
'GeoCell'):
message = self.PassengerMessage.new(
passenger=self,
driver_found=self.PassengerMessage.DriverHasBeenFoundMessage.new(
driver=driver,
driver_geo_cell=driver_geo_cell
)
)
message.save()
async def ride_started(self, driver: 'Driver'):
message = self.PassengerMessage.new(
passenger=self,
ride_started=self.PassengerMessage.RideHasStarted.new(
driver=driver
)
)
message.save()
async def ride_ended(self):
message = self.PassengerMessage.new(
passenger=self,
ride_started=self.PassengerMessage.RideHasEnded.new()
)
message.save()
class DriverRejectsPickupError(RideshareBase, Exception):
driver = attrs.relation(dm_cls='Driver')
ride = attrs.relation(dm_cls='Ride')
class Driver(RideshareBase):
is_taken: bool = attrs.required
current_ride = attrs.relation(dm_cls='Ride').optional
current_location: 'GeoCell' = attrs.relation(dm_cls='GeoCell')
@is_taken.getter
def is_taken(self):
# TODO: make better
return self.current_ride is not None
async def pickup_passenger(self, ride: 'Ride', passenger: Passenger,
passenger_start_cell: 'GeoCell',
passenger_end_cell: 'GeoCell'):
if self.is_taken:
raise DriverRejectsPickupError(driver=self, ride=ride)
self.current_ride = ride
# "// We also need to unregister ourselves from the current geo
cell we belong to."
if geo_cell := self.current_location:
await geo_cell.leave_cell(driver=self)
await ride.driver_joins(driver=self,
driver_location=self.current_location)
message = self.DriverMessage.new(
driver=self,
pickup_passenger=self.DriverMessage.PickupPassengerMessage.new(
passenger=passenger,
[jira] [Created] (FLINK-21211) Looking for reviews on a framework based on flink-statefun
Zixuan Rao created FLINK-21211:
--
Summary: Looking for reviews on a framework based on
flink-statefun
Key: FLINK-21211
URL: https://issues.apache.org/jira/browse/FLINK-21211
Project: Flink
Issue Type: New Feature
Reporter: Zixuan Rao
Hi, I am currently developing a framework targeting back end state management.
To ensure exactly-once processing of events in back end, I intend to use Flink
Stateful Functions runtime in combination with Python's asyncio. I hope to
receive some feedbacks.
The following code shows an example (draft) of writing a back end micro service
using my framework. It is intended to be equivalent (exchangeable) with
Flink-stateful examples/ridesharing. The idea is that "Event" is reducible to
an async function call, and external egress can be emitted by saving an object.
This preserves the exactly-once features of Flink-statefun while adding a great
deal of readability to the code.
Reviews are appreciated. Thank you!
```python3
"""
Equivalent implementation for flink stateful functions example - ridesharing
ref:
https://github.com/apache/flink-statefun/blob/master/statefun-examples/statefun-ridesharing-example/statefun-ridesharing-example-functions/src/main/java/org/apache/flink/statefun/examples/ridesharing/FnDriver.java
"""
from onto.models.base import Serializable
"""
Rewrite callback-style code to async-await:
ref:
https://www.coreycleary.me/how-to-rewrite-a-callback-function-in-promise-form-and-async-await-form-in-javascript
"""
from onto.domain_model import DomainModel
from onto.attrs import attrs
class RideshareBase(DomainModel):
pass
class Passenger(RideshareBase):
async def request_ride(self, start_geo_cell, end_geo_cell):
r = Ride.create() # TODO: implement create
await r.passenger_joins(
passenger=self,
start_geo_cell=start_geo_cell,
end_geo_cell=end_geo_cell
)
class PassengerMessage(DomainModel):
passenger = attrs.relation('Passenger')
class RideFailedMessage(Serializable):
ride = attrs.relation('Ride')
ride_failed = attrs.embed(RideFailedMessage).optional
class DriverHasBeenFoundMessage(Serializable):
driver = attrs.relation('Driver')
driver_geo_cell = attrs.relation('GeoCell')
driver_found = attrs.embed(RideFailedMessage).optional
class RideHasStarted(Serializable):
driver = attrs.relation('Driver')
ride_started = attrs.embed(RideHasStarted).optional
class RideHasEnded(Serializable):
pass # TODO: make sure that empty class works
ride_ended = attrs.embed(RideHasEnded).optional
async def ride_failed(self, ride: 'Ride'):
message = self.PassengerMessage.new(
passenger=self,
ride_failed=self.PassengerMessage.RideFailedMessage.new(
ride=ride
)
)
message.save()
async def driver_joins_ride(self, driver: 'Driver', driver_geo_cell:
'GeoCell'):
message = self.PassengerMessage.new(
passenger=self,
driver_found=self.PassengerMessage.DriverHasBeenFoundMessage.new(
driver=driver,
driver_geo_cell=driver_geo_cell
)
)
message.save()
async def ride_started(self, driver: 'Driver'):
message = self.PassengerMessage.new(
passenger=self,
ride_started=self.PassengerMessage.RideHasStarted.new(
driver=driver
)
)
message.save()
async def ride_ended(self):
message = self.PassengerMessage.new(
passenger=self,
ride_started=self.PassengerMessage.RideHasEnded.new()
)
message.save()
class DriverRejectsPickupError(RideshareBase, Exception):
driver = attrs.relation(dm_cls='Driver')
ride = attrs.relation(dm_cls='Ride')
class Driver(RideshareBase):
is_taken: bool = attrs.required
current_ride = attrs.relation(dm_cls='Ride').optional
current_location: 'GeoCell' = attrs.relation(dm_cls='GeoCell')
@is_taken.getter
def is_taken(self):
# TODO: make better
return self.current_ride is not None
async def pickup_passenger(self, ride: 'Ride', passenger: Passenger,
passenger_start_cell: 'GeoCell',
passenger_end_cell: 'GeoCell'):
if self.is_taken:
raise DriverRejectsPickupError(driver=self, ride=ride)
self.current_ride = ride
# "// We also need to unregister ourselves from the current geo
cell we belong to."
if geo_cell := self.current_location:
await geo_cell.leave_cell(driver=self)
await ride.driver_joins(driver=self,
driver_location=self.current_location)
message = self.DriverMessage.new(
[GitHub] [flink] flinkbot edited a comment on pull request #13912: [FLINK-19466][FLINK-19467][runtime / state backends] Add Flip-142 public interfaces and methods
flinkbot edited a comment on pull request #13912: URL: https://github.com/apache/flink/pull/13912#issuecomment-721398037 ## CI report: * f258e7f9bd871f5d3dfdf4e9b95f76179cfdceda Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12681) * 5bd74e2865ec431f3bf2528f5d85969ca51714fb Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12684) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14797: [FLINK-20654][FLINK-21104][network] Fix couple bugs in the handling of unaligned checkpoints and cancellations.
flinkbot edited a comment on pull request #14797: URL: https://github.com/apache/flink/pull/14797#issuecomment-769223743 ## CI report: * 181d2e3246c40cb0344726d4e4eb3d4f1dd80353 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12680) * f7f5fb6ed31c5ed60bdf4407e6e6b779fe77b77a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12683) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14797: [FLINK-20654][FLINK-21104][network] Fix couple bugs in the handling of unaligned checkpoints and cancellations.
flinkbot edited a comment on pull request #14797: URL: https://github.com/apache/flink/pull/14797#issuecomment-769223743 ## CI report: * 181d2e3246c40cb0344726d4e4eb3d4f1dd80353 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12680) * f7f5fb6ed31c5ed60bdf4407e6e6b779fe77b77a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13912: [FLINK-19466][FLINK-19467][runtime / state backends] Add Flip-142 public interfaces and methods
flinkbot edited a comment on pull request #13912: URL: https://github.com/apache/flink/pull/13912#issuecomment-721398037 ## CI report: * f258e7f9bd871f5d3dfdf4e9b95f76179cfdceda Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12681) * 5bd74e2865ec431f3bf2528f5d85969ca51714fb UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14814: [WIP] (1.11) [FLINK-21132][runtime] Don't end input on stop with savepoint
flinkbot edited a comment on pull request #14814: URL: https://github.com/apache/flink/pull/14814#issuecomment-769949284 ## CI report: * cad0310547eafdfd2eee8dddb14813ce940d3abd Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12673) * 53237d304da31ce8b2bdba5103192007122f23c7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12682) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14797: [FLINK-20654][FLINK-21104][network] Fix couple bugs in the handling of unaligned checkpoints and cancellations.
flinkbot edited a comment on pull request #14797: URL: https://github.com/apache/flink/pull/14797#issuecomment-769223743 ## CI report: * 6410a58539ad960900653721fff977c2566f4d8f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12671) * 181d2e3246c40cb0344726d4e4eb3d4f1dd80353 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12680) * f7f5fb6ed31c5ed60bdf4407e6e6b779fe77b77a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13912: [FLINK-19466][FLINK-19467][runtime / state backends] Add Flip-142 public interfaces and methods
flinkbot edited a comment on pull request #13912: URL: https://github.com/apache/flink/pull/13912#issuecomment-721398037 ## CI report: * f258e7f9bd871f5d3dfdf4e9b95f76179cfdceda Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12681) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] AHeise closed pull request #14807: [FLINK-20654][FLINK-21104][network] Fix two bugs in the handling of UnalignedCheckpoints
AHeise closed pull request #14807: URL: https://github.com/apache/flink/pull/14807 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] AHeise commented on pull request #14807: [FLINK-20654][FLINK-21104][network] Fix two bugs in the handling of UnalignedCheckpoints
AHeise commented on pull request #14807: URL: https://github.com/apache/flink/pull/14807#issuecomment-770020842 Incorporated into #14797 . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14814: [WIP] (1.11) [FLINK-21132][runtime] Don't end input on stop with savepoint
flinkbot edited a comment on pull request #14814: URL: https://github.com/apache/flink/pull/14814#issuecomment-769949284 ## CI report: * cad0310547eafdfd2eee8dddb14813ce940d3abd Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12673) * 53237d304da31ce8b2bdba5103192007122f23c7 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14797: [FLINK-21104][network] Various fixes for UC
flinkbot edited a comment on pull request #14797: URL: https://github.com/apache/flink/pull/14797#issuecomment-769223743 ## CI report: * 6410a58539ad960900653721fff977c2566f4d8f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12671) * 181d2e3246c40cb0344726d4e4eb3d4f1dd80353 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12680) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13912: [FLINK-19466][FLINK-19467][runtime / state backends] Add Flip-142 public interfaces and methods
flinkbot edited a comment on pull request #13912: URL: https://github.com/apache/flink/pull/13912#issuecomment-721398037 ## CI report: * fd87002c34c28b30a5cd7297c585592fe8333d80 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12675) * f258e7f9bd871f5d3dfdf4e9b95f76179cfdceda UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #14797: [FLINK-21104][network] Various fixes for UC
AHeise commented on a change in pull request #14797:
URL: https://github.com/apache/flink/pull/14797#discussion_r567060705
##
File path:
flink-tests/src/test/java/org/apache/flink/test/checkpointing/UnalignedCheckpointTestBase.java
##
@@ -282,20 +281,39 @@ public void addSplits(List splits) {
"Tried to add " + splits + " but already got " +
split);
}
split = Iterables.getOnlyElement(splits);
+checkShouldFinish();
LOG.info(
-"Added split {} @ {} subtask ({} attempt)",
+"Added split {}, shouldFinish={} @ {} subtask ({}
attempt)",
split,
+shouldFinish,
split.nextNumber % split.increment,
numRestarts);
}
+/** Should only be called if the split has been successfully
checkpointed. */
+private void checkShouldFinish() {
+shouldFinish =
+split != null
+&& split.numCompletedCheckpoints >=
minCheckpoints
+&& numRestarts >= expectedRestarts;
+}
+
@Override
-public void notifyNoMoreSplits() {}
+public void notifyNoMoreSplits() {
+LOG.info("notifyNoMoreSplits ({} attempt)", numRestarts);
+shouldFinish = true;
+}
Review comment:
No, removing.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Comment Edited] (FLINK-21045) Support 'load module' and 'unload module' SQL syntax
[
https://issues.apache.org/jira/browse/FLINK-21045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17271508#comment-17271508
]
Jane Chan edited comment on FLINK-21045 at 1/29/21, 7:44 PM:
-
Hi [~jark], [~nicholasjiang], I apologized for the late reply.
Before creating subtasks, I'd like to clarify this improvement's goal and raise
some questions on non-trivial implementation details to ensure we're on the
same page. Please correct me if I'm wrong.
h4. The goal of this improvement
Support the SQL syntax on module operation. To be specific, support `{{LOAD
MODULE name [WITH (‘type’=’xxx’, 'prop'='myProp', ...)]`}} which corresponds to
{{tableEnv.loadModule(name, module)}} and `{{UNLOAD MODULE name`}} which
corresponds to {{tableEnv.unloadModule(name)}}.
According to FLIP-68
{quote}Objects in modules are loaded on demand instead of eagerly
{quote}
it requires all dependency jars to appear under classpath, and this improvement
will not involve dynamic module jar loading.
h4. The ground truth
# FLIP-68 proposed the SQL syntax`{{LOAD MODULE name [WITH (‘type’=’xxx’,
'prop'='myProp', ...)] }}`
# Currently, load an already-loaded module will throw an exception. (See
[ModuleManager.java#L74|https://github.com/apache/flink/blob/master/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/module/ModuleManager.java#L74])
h4. My question
Based on this implementation, if module {{X}} has been loaded before(say define
in the YAML file), should `{{LOAD MODULE X}}` throw an exception, too? If so,
it makes `{{LOAD MODULE name}}` less meaningful because <1> if the module is
loaded before, an exception will be thrown. <2> if it's a new module, then
without specifying properties the TableFactoryService cannot find the suitable
factory. Thus the optional `{{WITH}}` may be necessary instead? On the other
hand, if users want to shift the resolution order on loaded modules, it could
be concise to only provide the module names rather than a long property list,
how about providing `{{RELOAD MODULE X}}` instead of `{{LOAD MODULE X}}`, which
describes changing the resolution order of a loaded module {{X}}. (It may
beyond the scope of this improvement, I'm just wondering).
Another question is currently {{SHOW MODULES}} is not supported by
{{FlinkSqlParserImpl}} (FLINK-17396 tracks this), do we have plans to support
it?
Best, Jane
was (Author: qingyue):
Hi [~jark], [~nicholasjiang], I apologized for the late reply.
Before creating subtasks, I'd like to clarify this improvement's goal and raise
some questions on non-trivial implementation details to ensure we're on the
same page. Please correct me if I'm wrong.
h4. The goal of this improvement
Support the SQL syntax on module operation. To be specific, support `{{LOAD
MODULE name [WITH (‘type’=’xxx’, 'prop'='myProp', ...)]`}} which corresponds to
{{tableEnv.loadModule(name, module)}} and `{{UNLOAD MODULE name`}} which
corresponds to {{tableEnv.unloadModule(name)}}.
According to FLIP-68
{quote}Objects in modules are loaded on demand instead of eagerly
{quote}
it requires all dependency jars to appear under classpath, and this improvement
will not involve dynamic module jar loading.
h4. The ground truth
# FLIP-68 proposed the SQL syntax`{{LOAD MODULE name [WITH (‘type’=’xxx’,
'prop'='myProp', ...)] }}`
# Currently, load an already-loaded module will throw an exception. (See
[ModuleManager.java#L74|https://github.com/apache/flink/blob/master/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/module/ModuleManager.java#L74])
h4. My question
Based on this implementation, if module {{X}} has been loaded before(say define
in the YAML file), should `{{LOAD MODULE X}}` throw an exception, too? If so,
it makes `{{LOAD MODULE name}}` less meaningful because <1> if the module is
loaded before, an exception will be thrown. <2> if it's a new module, then
without specifying properties the TableFactoryService cannot find the suitable
factory. Thus the optional `{{WITH}}` may be necessary instead? (Or throw an
exception is ok?) On the other hand, if we want to describe an operation
performed on module name only, is it possible to`{{RELOAD MODULE X}}` instead
of `{{LOAD MODULE X}}`, which describes changing the resolution order of a
loaded module {{X}}. (It may beyond the scope of this improvement, I'm just
wondering).
Another question is currently {{SHOW MODULES}} is not supported by
{{FlinkSqlParserImpl}} (FLINK-17396 tracks this), do we have plans to support
it?
Best, Jane
> Support 'load module' and 'unload module' SQL syntax
>
>
> Key: FLINK-21045
> URL: https://issues.apache.org/jira/browse/FLINK-21045
> Project: Flink
> Issue Type: Improvement
> Components: Table S
[jira] [Comment Edited] (FLINK-21045) Support 'load module' and 'unload module' SQL syntax
[
https://issues.apache.org/jira/browse/FLINK-21045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17271508#comment-17271508
]
Jane Chan edited comment on FLINK-21045 at 1/29/21, 7:41 PM:
-
Hi [~jark], [~nicholasjiang], I apologized for the late reply.
Before creating subtasks, I'd like to clarify this improvement's goal and raise
some questions on non-trivial implementation details to ensure we're on the
same page. Please correct me if I'm wrong.
h4. The goal of this improvement
Support the SQL syntax on module operation. To be specific, support `{{LOAD
MODULE name [WITH (‘type’=’xxx’, 'prop'='myProp', ...)]`}} which corresponds to
{{tableEnv.loadModule(name, module)}} and `{{UNLOAD MODULE name`}} which
corresponds to {{tableEnv.unloadModule(name)}}.
According to FLIP-68
{quote}Objects in modules are loaded on demand instead of eagerly
{quote}
it requires all dependency jars to appear under classpath, and this improvement
will not involve dynamic module jar loading.
h4. The ground truth
# FLIP-68 proposed the SQL syntax`{{LOAD MODULE name [WITH (‘type’=’xxx’,
'prop'='myProp', ...)] }}`
# Currently, load an already-loaded module will throw an exception. (See
[ModuleManager.java#L74|https://github.com/apache/flink/blob/master/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/module/ModuleManager.java#L74])
h4. My question
Based on this implementation, if module {{X}} has been loaded before(say define
in the YAML file), should `{{LOAD MODULE X}}` throw an exception, too? If so,
it makes `{{LOAD MODULE name}}` less meaningful because <1> if the module is
loaded before, an exception will be thrown. <2> if it's a new module, then
without specifying properties the TableFactoryService cannot find the suitable
factory. Thus the optional `{{WITH}}` may be necessary instead? (Or throw an
exception is ok?) On the other hand, if we want to describe an operation
performed on module name only, is it possible to`{{RELOAD MODULE X}}` instead
of `{{LOAD MODULE X}}`, which describes changing the resolution order of a
loaded module {{X}}. (It may beyond the scope of this improvement, I'm just
wondering).
Another question is currently {{SHOW MODULES}} is not supported by
{{FlinkSqlParserImpl}} (FLINK-17396 tracks this), do we have plans to support
it?
Best, Jane
was (Author: qingyue):
Hi [~jark], [~nicholasjiang], I apologized for the late reply.
Before creating subtasks, I'd like to clarify this improvement's goal and raise
some questions on non-trivial implementation details to ensure we're on the
same page. Please correct me if I'm wrong.
h4. The goal of this improvement
Support the SQL syntax on module operation. To be specific, support `{{LOAD
MODULE name [WITH (‘type’=’xxx’, 'prop'='myProp', ...)]`}} which corresponds to
{{tableEnv.loadModule(name, module)}} and `{{UNLOAD MODULE name`}} which
corresponds to {{tableEnv.unloadModule(name)}}.
Although
{quote}Objects in modules are loaded on demand instead of eagerly
{quote}
it requires all dependency jars to appear under classpath, and there's nothing
to do with dynamic module jar loading.
h4. The ground truth
# FLIP-68 proposed the SQL syntax`{{LOAD MODULE name [WITH (‘type’=’xxx’,
'prop'='myProp', ...)] }}`
# Currently, load an already-loaded module will throw an exception. (See
[ModuleManager.java#L74|https://github.com/apache/flink/blob/master/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/module/ModuleManager.java#L74])
h4. My question
Based on this implementation, if module {{X}} has been loaded before(say define
in the YAML file), should `{{LOAD MODULE X}}` throw an exception, too? If so,
it makes `{{LOAD MODULE name}}` less meaningful because the only feasible
operation for `{{LOAD}}` is to load a new module. Thus the optional `{{WITH}}`
may be necessary instead? (Or throw an exception is ok?) On the other hand, if
we want to describe an operation performed on module name only, is it possible
to`{{RELOAD MODULE X}}` instead of `{{LOAD MODULE X}}`, which describes
changing the resolution order of a loaded module {{X}}. (It may beyond the
scope of this improvement, I'm just wondering).
Another question is currently {{SHOW MODULES}} is not supported by
{{FlinkSqlParserImpl}} (FLINK-17396 tracks this), do we have plans to support
it?
Best, Jane
> Support 'load module' and 'unload module' SQL syntax
>
>
> Key: FLINK-21045
> URL: https://issues.apache.org/jira/browse/FLINK-21045
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Planner
>Affects Versions: 1.13.0
>Reporter: Nicholas Jiang
>Assignee: Jane Chan
>Priority: Major
> Fix For: 1.13.0
>
>
> At present, Fli
[GitHub] [flink] flinkbot edited a comment on pull request #14812: [hotfix][javadoc] fix grammar error for PlannerContext.java: from "be overwrite" to "be overwritten"
flinkbot edited a comment on pull request #14812: URL: https://github.com/apache/flink/pull/14812#issuecomment-769855928 ## CI report: * 878183c7ec59129e62b269e1958e1945f3ddb9aa Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12665) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14807: [FLINK-20654][FLINK-21104][network] Fix two bugs in the handling of UnalignedCheckpoints
flinkbot edited a comment on pull request #14807: URL: https://github.com/apache/flink/pull/14807#issuecomment-769784165 ## CI report: * fb5b2ecf70c62048073ba8677fe27baa02a63e7c UNKNOWN * f5901e6ca1e67fd29980bac95986ff5640a8976c UNKNOWN * 3a9e5890fe04ff480f5623bd752ed363a573d58a UNKNOWN * 4c752cedc9e008227243ce184670b4f130e98601 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12677) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14797: [FLINK-21104][network] Various fixes for UC
flinkbot edited a comment on pull request #14797: URL: https://github.com/apache/flink/pull/14797#issuecomment-769223743 ## CI report: * 6410a58539ad960900653721fff977c2566f4d8f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12671) * 181d2e3246c40cb0344726d4e4eb3d4f1dd80353 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14810: [FLINK-9844][client] PackagedProgram#close() closes ClassLoader
flinkbot edited a comment on pull request #14810: URL: https://github.com/apache/flink/pull/14810#issuecomment-769830895 ## CI report: * d666d443ee90e4a5ce6eccf1874f498b9f2744a8 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12663) * 0dee1f6c07a80da9abc5690b2a87442dc1725fd9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12679) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rkhachatryan commented on a change in pull request #14799: [WIP] Wrapped StateBackend to forward state changes to StateChangleLog
rkhachatryan commented on a change in pull request #14799:
URL: https://github.com/apache/flink/pull/14799#discussion_r567014439
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/state/proxy/ProxyKeyedStateBackend.java
##
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state.proxy;
+
+import org.apache.flink.api.common.state.AggregatingStateDescriptor;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ReducingStateDescriptor;
+import org.apache.flink.api.common.state.State;
+import org.apache.flink.api.common.state.StateDescriptor;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.runtime.checkpoint.CheckpointOptions;
+import org.apache.flink.runtime.state.AbstractKeyedStateBackend;
+import org.apache.flink.runtime.state.CheckpointStreamFactory;
+import org.apache.flink.runtime.state.KeyGroupedInternalPriorityQueue;
+import org.apache.flink.runtime.state.Keyed;
+import org.apache.flink.runtime.state.KeyedStateFunction;
+import org.apache.flink.runtime.state.KeyedStateHandle;
+import org.apache.flink.runtime.state.PriorityComparable;
+import org.apache.flink.runtime.state.SnapshotResult;
+import org.apache.flink.runtime.state.StateSnapshotTransformer;
+import org.apache.flink.runtime.state.heap.HeapKeyedStateBackend;
+import org.apache.flink.runtime.state.heap.HeapPriorityQueueElement;
+import org.apache.flink.runtime.state.internal.InternalKvState;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.flink.util.IOUtils;
+
+import javax.annotation.Nonnull;
+
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.RunnableFuture;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/** */
+public class ProxyKeyedStateBackend extends AbstractKeyedStateBackend {
+// wrapped keyed state backend, either HeapKeyedStateBackend or
RocksDBKeyedStateBackend
+AbstractKeyedStateBackend keyedStateBackend;
+
+private static final Map, StateFactory>
STATE_FACTORIES =
+Stream.of(
+Tuple2.of(
+ValueStateDescriptor.class,
+(StateFactory) ProxyValueState::create),
+Tuple2.of(
+ListStateDescriptor.class,
+(StateFactory) ProxyListState::create),
+Tuple2.of(
+ReducingStateDescriptor.class,
+(StateFactory) ProxyReducingState::create),
+Tuple2.of(
+AggregatingStateDescriptor.class,
+(StateFactory)
ProxyAggregatingState::create),
+Tuple2.of(
+MapStateDescriptor.class, (StateFactory)
ProxyMapState::create))
+.collect(Collectors.toMap(t -> t.f0, t -> t.f1));
+
+public ProxyKeyedStateBackend(AbstractKeyedStateBackend
keyedStateBackend) {
+super(
+keyedStateBackend.kvStateRegistry,
+keyedStateBackend.keySerializer,
+keyedStateBackend.userCodeClassLoader,
+keyedStateBackend.executionConfig,
+keyedStateBackend.ttlTimeProvider,
+keyedStateBackend.cancelStreamRegistry,
+keyedStateBackend.keyGroupCompressionDecorator,
+keyedStateBackend.keyContext);
+this.keyedStateBackend = keyedStateBackend;
Review comment:
In the end, we end up with 4 backends:
```
Proxy extends Abstract (has it's own fields AND refers to passed fields)
Proxied extends Abstract (has it's own fields AND refers to passed
fields)
```
Right?
I see the following issues with this:
1. Double caching and lookup in `keyValue
[GitHub] [flink] flinkbot edited a comment on pull request #14815: [WIP] [FLINK-21132][runtime] Don't end input on stop with savepoint
flinkbot edited a comment on pull request #14815: URL: https://github.com/apache/flink/pull/14815#issuecomment-769949393 ## CI report: * 5edf84e272426dcaae4e9d114d5ce9ea5d766346 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12674) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14810: [FLINK-9844][client] PackagedProgram#close() closes ClassLoader
flinkbot edited a comment on pull request #14810: URL: https://github.com/apache/flink/pull/14810#issuecomment-769830895 ## CI report: * d666d443ee90e4a5ce6eccf1874f498b9f2744a8 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12663) * 0dee1f6c07a80da9abc5690b2a87442dc1725fd9 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14807: [FLINK-20654][FLINK-21104][network] Fix two bugs in the handling of UnalignedCheckpoints
flinkbot edited a comment on pull request #14807: URL: https://github.com/apache/flink/pull/14807#issuecomment-769784165 ## CI report: * fb5b2ecf70c62048073ba8677fe27baa02a63e7c UNKNOWN * f5901e6ca1e67fd29980bac95986ff5640a8976c UNKNOWN * 3a9e5890fe04ff480f5623bd752ed363a573d58a UNKNOWN * 037bbc7022faf40d87a9cb48b0dbd79587befdc7 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12656) * 4c752cedc9e008227243ce184670b4f130e98601 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12677) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13912: [FLINK-19466][FLINK-19467][runtime / state backends] Add Flip-142 public interfaces and methods
flinkbot edited a comment on pull request #13912: URL: https://github.com/apache/flink/pull/13912#issuecomment-721398037 ## CI report: * fd87002c34c28b30a5cd7297c585592fe8333d80 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12675) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-21210) ApplicationClusterEntryPoints should explicitly close PackagedProgram
[
https://issues.apache.org/jira/browse/FLINK-21210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler updated FLINK-21210:
-
Labels: starter (was: )
> ApplicationClusterEntryPoints should explicitly close PackagedProgram
> -
>
> Key: FLINK-21210
> URL: https://issues.apache.org/jira/browse/FLINK-21210
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Reporter: Chesnay Schepler
>Priority: Major
> Labels: starter
> Fix For: 1.11.4, 1.12.2, 1.13.0
>
>
> FLINK-21164 introduced {{PackagedProgram#close}} to clean up temporary jars,
> and with FLINK-9844 the contained classloader will also be closed beforehand.
> This allows these files to be cleaned up properly.
> To make it obvious that this also happens in the application cluster
> entrypoints (and to reduce the chances of this potentially leaking code being
> copied elsewhere) we should adjust the entrypoints to explicitly close the
> \{{PackagedProgram}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-21210) ApplicationClusterEntryPoints should explicitly close PackagedProgram
Chesnay Schepler created FLINK-21210:
Summary: ApplicationClusterEntryPoints should explicitly close
PackagedProgram
Key: FLINK-21210
URL: https://issues.apache.org/jira/browse/FLINK-21210
Project: Flink
Issue Type: Improvement
Components: Runtime / Coordination
Reporter: Chesnay Schepler
Fix For: 1.11.4, 1.12.2, 1.13.0
FLINK-21164 introduced {{PackagedProgram#close}} to clean up temporary jars,
and with FLINK-9844 the contained classloader will also be closed beforehand.
This allows these files to be cleaned up properly.
To make it obvious that this also happens in the application cluster
entrypoints (and to reduce the chances of this potentially leaking code being
copied elsewhere) we should adjust the entrypoints to explicitly close the
\{{PackagedProgram}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] rkhachatryan commented on a change in pull request #14799: [WIP] Wrapped StateBackend to forward state changes to StateChangleLog
rkhachatryan commented on a change in pull request #14799:
URL: https://github.com/apache/flink/pull/14799#discussion_r566996438
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/state/proxy/ProxyKeyedStateBackend.java
##
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state.proxy;
+
+import org.apache.flink.api.common.state.AggregatingStateDescriptor;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.ReducingStateDescriptor;
+import org.apache.flink.api.common.state.State;
+import org.apache.flink.api.common.state.StateDescriptor;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.runtime.checkpoint.CheckpointOptions;
+import org.apache.flink.runtime.state.AbstractKeyedStateBackend;
+import org.apache.flink.runtime.state.CheckpointStreamFactory;
+import org.apache.flink.runtime.state.KeyGroupedInternalPriorityQueue;
+import org.apache.flink.runtime.state.Keyed;
+import org.apache.flink.runtime.state.KeyedStateFunction;
+import org.apache.flink.runtime.state.KeyedStateHandle;
+import org.apache.flink.runtime.state.PriorityComparable;
+import org.apache.flink.runtime.state.SnapshotResult;
+import org.apache.flink.runtime.state.StateSnapshotTransformer;
+import org.apache.flink.runtime.state.heap.HeapKeyedStateBackend;
+import org.apache.flink.runtime.state.heap.HeapPriorityQueueElement;
+import org.apache.flink.runtime.state.internal.InternalKvState;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.flink.util.IOUtils;
+
+import javax.annotation.Nonnull;
+
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.RunnableFuture;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/** */
+public class ProxyKeyedStateBackend extends AbstractKeyedStateBackend {
+// wrapped keyed state backend, either HeapKeyedStateBackend or
RocksDBKeyedStateBackend
+AbstractKeyedStateBackend keyedStateBackend;
+
+private static final Map, StateFactory>
STATE_FACTORIES =
+Stream.of(
+Tuple2.of(
+ValueStateDescriptor.class,
+(StateFactory) ProxyValueState::create),
+Tuple2.of(
+ListStateDescriptor.class,
Review comment:
I would prefer a code like this:
```
InternalKvState nested =
keyedStateBackend.createInternalState(
namespaceSerializer, stateDesc, snapshotTransformFactory);
if (stateDesc instanceof ValueStateDescriptor) {
return new ProxyValueState((InternalValueState) nested);
} else if (stateDesc instanceof ListStateDescriptor) {
return ...
}
```
It would be simpler and shorter.
I think that `Map` can be useful when the mappings can be updated
dynamically or programmatically. Neither is the case.
And then factory interface can be dropped as it's only needed to work with
map IIUC.
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/state/proxy/ProxyKeyedStateBackend.java
##
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permiss
[jira] [Created] (FLINK-21209) Update stackbrew maintainer field
Chesnay Schepler created FLINK-21209: Summary: Update stackbrew maintainer field Key: FLINK-21209 URL: https://issues.apache.org/jira/browse/FLINK-21209 Project: Flink Issue Type: Improvement Components: Documentation Reporter: Chesnay Schepler Fix For: 1.13.0 The Flink stackbrew file in the official-images repo lists the original maintainers as maintainers of the current images, where ideally it should now just list the Flink project with the [d...@flink.apache.org|mailto:d...@flink.apache.org] mailing address. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink-docker] zentol commented on a change in pull request #63: Adjust dev-1.12 branch for building 1.12.0
zentol commented on a change in pull request #63:
URL: https://github.com/apache/flink-docker/pull/63#discussion_r567015505
##
File path: add-version.sh
##
@@ -112,9 +112,9 @@ for source_variant in "${SOURCE_VARIANTS[@]}"; do
flink_url_file_path=flink/flink-${flink_version}/flink-${flink_version}-bin-scala_${scala_version}.tgz
-
flink_tgz_url="https://www.apache.org/dyn/closer.cgi?action=download&filename=${flink_url_file_path}";
+
flink_tgz_url="https://archive.apache.org/dist/${flink_url_file_path}";
Review comment:
But why change it in the first place? Have we seen issues anywhere with
downloads not working?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-21164) Jar handlers don't cleanup temporarily extracted jars
[
https://issues.apache.org/jira/browse/FLINK-21164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17275242#comment-17275242
]
Chesnay Schepler commented on FLINK-21164:
--
master: 30f6964a9b72c79925c62f952a4559f2e48bdf06
1.12: 43661b5ad6344fab221540622555e8e2ae81019e
> Jar handlers don't cleanup temporarily extracted jars
> -
>
> Key: FLINK-21164
> URL: https://issues.apache.org/jira/browse/FLINK-21164
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / REST
>Affects Versions: 1.11.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.4, 1.12.2, 1.13.0
>
>
> The jar handlers create a PackagedProgram when running a job or retrieving
> the execution plan. When a PackagedProgram is created and jars contained
> within a jar are automatically extracted to a temporary directory.
> This directory is not automatically cleaned up; this must be done manually.
> The CliFrontend does that within {{#run}}, but the jar handlers do not,
> leading to jars piling up in the filesystem.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14814: [WIP] (1.11) [FLINK-21132][runtime] Don't end input on stop with savepoint
flinkbot edited a comment on pull request #14814: URL: https://github.com/apache/flink/pull/14814#issuecomment-769949284 ## CI report: * cad0310547eafdfd2eee8dddb14813ce940d3abd Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12673) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14808: [FLINK-21204][coordination] Remove LogicalSlot#getSlotSharingGroupId
flinkbot edited a comment on pull request #14808: URL: https://github.com/apache/flink/pull/14808#issuecomment-769820261 ## CI report: * 232c04267560ca5716a027237ba5f803a036f1cc Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12661) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol merged pull request #14777: [FLINK-21164][rest] Delete temporary jars
zentol merged pull request #14777: URL: https://github.com/apache/flink/pull/14777 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13912: [FLINK-19466][FLINK-19467][runtime / state backends] Add Flip-142 public interfaces and methods
flinkbot edited a comment on pull request #13912: URL: https://github.com/apache/flink/pull/13912#issuecomment-721398037 ## CI report: * 3885663038156d52aec21062a9743009fb875cc3 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12535) * fd87002c34c28b30a5cd7297c585592fe8333d80 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-21132) BoundedOneInput.endInput is called when taking synchronous savepoint
[
https://issues.apache.org/jira/browse/FLINK-21132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17275234#comment-17275234
]
Kezhu Wang commented on FLINK-21132:
[~aljoscha] [~pnowojski] [~roman_khachatryan] We all actually are on the same
page. So, all above approaches are feasible to me as long as there is no api
level bug-specific solution.
But I am kind of preferring to not sending {{EndOfPartitionEvent}}, so I would
like to present my [local
work|https://github.com/apache/flink/commit/d6c6837cd724913716007265f03a50098bee985e]
for another approach. Here is a summary for the changes:
# {{StopTaskException}} to throw from {{AbstractInvokable.invoke}}.
# Upon receiving savepoint completion, stop task with savepoint. This stop
phase is same for mailbox model, so FLINK-21133 does not exist anymore.
# After run out of mailbox loop, check whether it is stopped by a savepoint.
If it is, throws {{StopTaskException}}.
# If {{StopTaskException}} happens, {{Task.doRun}} will not finish result
partition(hence no EndOfPartitionEvent), but still transit task to finished
state(I am not sure it is conflict with FLIP-147 or not as I saw "Reuse Tasks'
FINISH status").
# Other works: generalize {{SteamTask.finishTask}} to all stream task but not
only source tasks; generalize {{StreamTask.cancelTask}} to mailbox model.
The benefits of this approach is that:
# It does not sending {{EndOfPartitionEvent}} anymore. There will be only true
end of input.
# If finished status caused by stop-with-savepoint is conflict with FLIP-147,
then it is easy to change to new status.
# FLINK-21133 solved with no extra work.
There is one user-notable breaking change: {{SreamOperator.close}} will not be
called in case of stop-with-savepoint.
Last, feel free to choice either approach, it is just my personal preference:P,
and I don't take any compatibility/invasiveness into account.
> BoundedOneInput.endInput is called when taking synchronous savepoint
>
>
> Key: FLINK-21132
> URL: https://issues.apache.org/jira/browse/FLINK-21132
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.10.2, 1.10.3, 1.11.3, 1.12.1
>Reporter: Kezhu Wang
>Assignee: Roman Khachatryan
>Priority: Major
> Labels: pull-request-available
>
> [~elkhand](?) reported on project iceberg that {{BoundedOneInput.endInput}}
> was
> [called|https://github.com/apache/iceberg/issues/2033#issuecomment-765864038]
> when [stopping job with
> savepoint|https://github.com/apache/iceberg/issues/2033#issuecomment-765557995].
> I think it is a bug of Flink and was introduced in FLINK-14230. The
> [changes|https://github.com/apache/flink/pull/9854/files#diff-0c5fe245445b932fa83fdaf5c4802dbe671b73e8d76f39058c6eaaaffd9639faL577]
> rely on {{StreamTask.afterInvoke}} and {{OperatorChain.closeOperators}} will
> only be invoked after *end of input*. But that is not true long before after
> [FLIP-34: Terminate/Suspend Job with
> Savepoint|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103090212].
> Task could enter state called
> [*finished*|https://github.com/apache/flink/blob/3a8e06cd16480eacbbf0c10f36b8c79a6f741814/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java#L467]
> after synchronous savepoint, that is an expected job suspension and stopping.
> [~sunhaibotb] [~pnowojski] [~roman_khachatryan] Could you help confirm this ?
> For full context, see
> [apache/iceberg#2033|https://github.com/apache/iceberg/issues/2033]. I have
> pushed branch
> [synchronous-savepoint-conflict-with-bounded-end-input-case|https://github.com/kezhuw/flink/commits/synchronous-savepoint-conflict-with-bounded-end-input-case]
> in my repository. Test case
> {{SavepointITCase.testStopSavepointWithBoundedInput}} failed due to
> {{BoundedOneInput.endInput}} called.
> I am also aware of [FLIP-147: Support Checkpoints After Tasks
> Finished|https://cwiki.apache.org/confluence/x/mw-ZCQ], maybe the three
> should align on what *finished* means exactly. [~kkl0u] [~chesnay]
> [~gaoyunhaii]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14814: [WIP] (1.11) [FLINK-21132][runtime] Don't end input on stop with savepoint
flinkbot edited a comment on pull request #14814: URL: https://github.com/apache/flink/pull/14814#issuecomment-769949284 ## CI report: * cad0310547eafdfd2eee8dddb14813ce940d3abd Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12673) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14815: [WIP] [FLINK-21132][runtime] Don't end input on stop with savepoint
flinkbot edited a comment on pull request #14815: URL: https://github.com/apache/flink/pull/14815#issuecomment-769949393 ## CI report: * 5edf84e272426dcaae4e9d114d5ce9ea5d766346 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12674) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14777: [FLINK-21164][rest] Delete temporary jars
flinkbot edited a comment on pull request #14777: URL: https://github.com/apache/flink/pull/14777#issuecomment-768256830 ## CI report: * ce3c983720499212e587319cd6fdef3ccc6b97b5 UNKNOWN * c833d995b48a47ca6a8af7ae7c5eae5fd9d2c7ba Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12659) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-21208) pyarrow exception when using window with pandas udaf
YufeiLiu created FLINK-21208:
Summary: pyarrow exception when using window with pandas udaf
Key: FLINK-21208
URL: https://issues.apache.org/jira/browse/FLINK-21208
Project: Flink
Issue Type: Improvement
Components: API / Python
Affects Versions: 1.12.0
Reporter: YufeiLiu
I write a pyflink demo and execute in local environment, the logic is
simple:generate records and aggerate in 100s tumle window, using a pandas udaf.
But the job failed after several minutes, I don't think it's a resource problem
because the amount of data is small, here is the error trace.
{code:java}
Caused by: org.apache.flink.streaming.runtime.tasks.AsynchronousException:
Caught exception while processing timer.
at
org.apache.flink.streaming.runtime.tasks.StreamTask$StreamTaskAsyncExceptionHandler.handleAsyncException(StreamTask.java:1108)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.handleAsyncException(StreamTask.java:1082)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invokeProcessingTimeCallback(StreamTask.java:1213)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$null$17(StreamTask.java:1202)
at
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:47)
at
org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:78)
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:302)
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:184)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:575)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:539)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:722)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:547)
at java.lang.Thread.run(Thread.java:748)
Caused by: TimerException{java.lang.RuntimeException: Failed to close remote
bundle}
... 11 more
Caused by: java.lang.RuntimeException: Failed to close remote bundle
at
org.apache.flink.streaming.api.runners.python.beam.BeamPythonFunctionRunner.finishBundle(BeamPythonFunctionRunner.java:371)
at
org.apache.flink.streaming.api.runners.python.beam.BeamPythonFunctionRunner.flush(BeamPythonFunctionRunner.java:325)
at
org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator.invokeFinishBundle(AbstractPythonFunctionOperator.java:291)
at
org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator.checkInvokeFinishBundleByTime(AbstractPythonFunctionOperator.java:285)
at
org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator.lambda$open$0(AbstractPythonFunctionOperator.java:134)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invokeProcessingTimeCallback(StreamTask.java:1211)
... 10 more
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException:
Error received from SDK harness for instruction 3: Traceback (most recent call
last):
File
"/Users/lyf/Library/Python/3.7/lib/python/site-packages/apache_beam/runners/worker/sdk_worker.py",
line 253, in _execute
response = task()
File
"/Users/lyf/Library/Python/3.7/lib/python/site-packages/apache_beam/runners/worker/sdk_worker.py",
line 310, in
lambda: self.create_worker().do_instruction(request), request)
File
"/Users/lyf/Library/Python/3.7/lib/python/site-packages/apache_beam/runners/worker/sdk_worker.py",
line 480, in do_instruction
getattr(request, request_type), request.instruction_id)
File
"/Users/lyf/Library/Python/3.7/lib/python/site-packages/apache_beam/runners/worker/sdk_worker.py",
line 515, in process_bundle
bundle_processor.process_bundle(instruction_id))
File
"/Users/lyf/Library/Python/3.7/lib/python/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 978, in process_bundle
element.data)
File
"/Users/lyf/Library/Python/3.7/lib/python/site-packages/apache_beam/runners/worker/bundle_processor.py",
line 218, in process_encoded
self.output(decoded_value)
File "apache_beam/runners/worker/operations.py", line 330, in
apache_beam.runners.worker.operations.Operation.output
File "apache_beam/runners/worker/operations.py", line 332, in
apache_beam.runners.worker.operations.Operation.output
File "apache_beam/runners/worker/operations.py", line 195, in
apache_beam.runners.worker.operations.SingletonConsumerSet.receive
File "apache_beam/runners/worker/operations.py", line 292, in
apache_beam.runners.worker.operations.Operation.process
File
"/Users/lyf/Library/Python/3.7/lib/python/site-packages/pyflink/fn_execution/beam/beam_operations_slow.py",
lin
[GitHub] [flink] flinkbot commented on pull request #14815: [WIP] [FLINK-21132][runtime] Don't end input on stop with savepoint
flinkbot commented on pull request #14815: URL: https://github.com/apache/flink/pull/14815#issuecomment-769949393 ## CI report: * 5edf84e272426dcaae4e9d114d5ce9ea5d766346 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14814: [WIP] (1.11) [FLINK-21132][runtime] Don't end input on stop with savepoint
flinkbot commented on pull request #14814: URL: https://github.com/apache/flink/pull/14814#issuecomment-769949284 ## CI report: * cad0310547eafdfd2eee8dddb14813ce940d3abd UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14807: [FLINK-20654][FLINK-21104][network] Fix two bugs in the handling of UnalignedCheckpoints
flinkbot edited a comment on pull request #14807: URL: https://github.com/apache/flink/pull/14807#issuecomment-769784165 ## CI report: * fb5b2ecf70c62048073ba8677fe27baa02a63e7c UNKNOWN * f5901e6ca1e67fd29980bac95986ff5640a8976c UNKNOWN * 3a9e5890fe04ff480f5623bd752ed363a573d58a UNKNOWN * 037bbc7022faf40d87a9cb48b0dbd79587befdc7 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12656) * 4c752cedc9e008227243ce184670b4f130e98601 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-21045) Support 'load module' and 'unload module' SQL syntax
[
https://issues.apache.org/jira/browse/FLINK-21045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17275216#comment-17275216
]
Jane Chan commented on FLINK-21045:
---
Hi everyone, thanks for the discussion and the opinions, and I'm sorry for the
late reply.
IIUC, we seem to have three core unanswered questions
# Do we need to provide a mechanism for users to change resolution order
conveniently? E.g .`{{USE MODULE}}` syntax is proposed.
# SQL syntax upon `{{CREATE/DROP MODULE}}` v.s. `{{LOAD/UNLOAD MODULE}}`.
# Do we need to keep the module instance's uniqueness, or is it ok to map
modules purely by name? (Btw this question seems to be discussed before, starts
from [here|https://www.mail-archive.com/dev@flink.apache.org/msg30589.html],
the received comments are "_Jark has a good point. However, I think validation
logic can put in place to restrict one instance per type. Maybe the doc needs
to be specific on this._" and "_So at least, we can have a validation to
restrict one module instance per type in the first version._") However, I
cannot find this validation on the impl side. Currently, the module name is an
alias to the module instance.
E.g.
{code:scala}
@Test
def testLoadModuleInstanceTwice(): Unit = {
val moduleManager = new ModuleManager // this will load CoreModule by
default
val factory = new CoreModuleFactory
moduleManager.loadModule("CORE", factory.createModule(
Map[String, String]("type" -> "core").asJava))
assert(moduleManager.listModules().containsAll(List("core",
"CORE").asJava))
}
{code}
IMO,
# It's not very user-friendly to manually unload/load modules to shift the
order. For the syntax `{{USE MODULE}}`, I think `{{USE}}` sounds like
expressing a mutual-exclusive concept, E.g., `{{USE CATALOG}}`. While Module's
functions don't have namespaces, it's a little bit hard to express a kind of
"sth is specified" meaning. I would like to use `{{RELOAD MODULE}}` to
expressing upgrading the priority of a loaded module.
# I don't have a strong opinion on `{{CREATE/DROP MODULE}}` v.s.
`{{LOAD/UNLOAD MODULE}}`. But once we choose to use `{{CREATE/DROP MODULE}}`,
we'd better keep API consistent with the syntax.
I think it's time to involve more thoughts and suggestions. I'll start a
mailing discussion.
Best, Jane
> Support 'load module' and 'unload module' SQL syntax
>
>
> Key: FLINK-21045
> URL: https://issues.apache.org/jira/browse/FLINK-21045
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Planner
>Affects Versions: 1.13.0
>Reporter: Nicholas Jiang
>Assignee: Jane Chan
>Priority: Major
> Fix For: 1.13.0
>
>
> At present, Flink SQL doesn't support the 'load module' and 'unload module'
> SQL syntax. It's necessary for uses in the situation that users load and
> unload user-defined module through table api or sql client.
> SQL syntax has been proposed in FLIP-68:
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-68%3A+Extend+Core+Table+System+with+Pluggable+Modules
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] tillrohrmann commented on a change in pull request #14777: [FLINK-21164][rest] Delete temporary jars
tillrohrmann commented on a change in pull request #14777:
URL: https://github.com/apache/flink/pull/14777#discussion_r566981325
##
File path:
flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/handlers/JarRunHandler.java
##
@@ -103,6 +103,11 @@ public JarRunHandler(
executor)
.handle(
(jobIds, throwable) -> {
+try {
+program.deleteExtractedLibraries();
Review comment:
Alrighty. Thanks for the update.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14815: [WIP] [FLINK-21132][runtime] Don't end input on stop with savepoint
flinkbot commented on pull request #14815: URL: https://github.com/apache/flink/pull/14815#issuecomment-769942605 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 5edf84e272426dcaae4e9d114d5ce9ea5d766346 (Fri Jan 29 17:30:50 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14814: [WIP] (1.11) [FLINK-21132][runtime] Don't end input on stop with savepoint
flinkbot commented on pull request #14814: URL: https://github.com/apache/flink/pull/14814#issuecomment-769941674 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit cad0310547eafdfd2eee8dddb14813ce940d3abd (Fri Jan 29 17:29:00 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rkhachatryan opened a new pull request #14815: [WIP] [FLINK-21132][runtime] Don't end input on stop with savepoint
rkhachatryan opened a new pull request #14815: URL: https://github.com/apache/flink/pull/14815 ## What is the purpose of the change ``` EndOfInput was used to handle any stopping of the job. When stopping with savepoint the input is not actually ended. This causes issues with some sinks (e.g. Iceberg). With this change, endInput is not call for stop-with-savepoint. To differentiate stop-with-savepoint from other cases only checkpoint (RPC/barriers) are considered and not network EOP. That's enough because EOP is only injected after the CP completion (i.e. when the downstream is also notified by sync savepoint by CP barriers). ``` ## Verifying this change Added `SavepointITCase.testStopSavepointWithBoundedInput`. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: **yes** (`endInput` is not called anymore for stop-with-savepoint) - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? no This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-21132) BoundedOneInput.endInput is called when taking synchronous savepoint
[
https://issues.apache.org/jira/browse/FLINK-21132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-21132:
---
Labels: pull-request-available (was: )
> BoundedOneInput.endInput is called when taking synchronous savepoint
>
>
> Key: FLINK-21132
> URL: https://issues.apache.org/jira/browse/FLINK-21132
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.10.2, 1.10.3, 1.11.3, 1.12.1
>Reporter: Kezhu Wang
>Assignee: Roman Khachatryan
>Priority: Major
> Labels: pull-request-available
>
> [~elkhand](?) reported on project iceberg that {{BoundedOneInput.endInput}}
> was
> [called|https://github.com/apache/iceberg/issues/2033#issuecomment-765864038]
> when [stopping job with
> savepoint|https://github.com/apache/iceberg/issues/2033#issuecomment-765557995].
> I think it is a bug of Flink and was introduced in FLINK-14230. The
> [changes|https://github.com/apache/flink/pull/9854/files#diff-0c5fe245445b932fa83fdaf5c4802dbe671b73e8d76f39058c6eaaaffd9639faL577]
> rely on {{StreamTask.afterInvoke}} and {{OperatorChain.closeOperators}} will
> only be invoked after *end of input*. But that is not true long before after
> [FLIP-34: Terminate/Suspend Job with
> Savepoint|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103090212].
> Task could enter state called
> [*finished*|https://github.com/apache/flink/blob/3a8e06cd16480eacbbf0c10f36b8c79a6f741814/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java#L467]
> after synchronous savepoint, that is an expected job suspension and stopping.
> [~sunhaibotb] [~pnowojski] [~roman_khachatryan] Could you help confirm this ?
> For full context, see
> [apache/iceberg#2033|https://github.com/apache/iceberg/issues/2033]. I have
> pushed branch
> [synchronous-savepoint-conflict-with-bounded-end-input-case|https://github.com/kezhuw/flink/commits/synchronous-savepoint-conflict-with-bounded-end-input-case]
> in my repository. Test case
> {{SavepointITCase.testStopSavepointWithBoundedInput}} failed due to
> {{BoundedOneInput.endInput}} called.
> I am also aware of [FLIP-147: Support Checkpoints After Tasks
> Finished|https://cwiki.apache.org/confluence/x/mw-ZCQ], maybe the three
> should align on what *finished* means exactly. [~kkl0u] [~chesnay]
> [~gaoyunhaii]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] rkhachatryan opened a new pull request #14814: [WIP] (1.11) [FLINK-21132][runtime] Don't end input on stop with savepoint
rkhachatryan opened a new pull request #14814: URL: https://github.com/apache/flink/pull/14814 ## What is the purpose of the change (1.11 backport) ``` EndOfInput was used to handle any stopping of the job. When stopping with savepoint the input is not actually ended. This causes issues with some sinks (e.g. Iceberg). With this change, endInput is not call for stop-with-savepoint. To differentiate stop-with-savepoint from other cases only checkpoint (RPC/barriers) are considered and not network EOP. That's enough because EOP is only injected after the CP completion (i.e. when the downstream is also notified by sync savepoint by CP barriers). ``` ## Verifying this change Added `SavepointITCase.testStopSavepointWithBoundedInput`. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: **yes** (`endInput` is not called anymore for stop-with-savepoint) - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? no This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] pnowojski commented on a change in pull request #14797: [FLINK-21104][network] Various fixes for UC
pnowojski commented on a change in pull request #14797:
URL: https://github.com/apache/flink/pull/14797#discussion_r566976700
##
File path:
flink-tests/src/test/java/org/apache/flink/test/checkpointing/UnalignedCheckpointTestBase.java
##
@@ -221,15 +228,17 @@ public InputStatus pollNext(ReaderOutput output)
throws InterruptedExcepti
output.collect(withHeader(split.nextNumber), split.nextNumber);
split.nextNumber += split.increment;
-if (throttle) {
-// throttle source as long as sink is not backpressuring
(which it does only
-// after full recovery)
-Thread.sleep(1);
+switch (pollingState) {
+case FINISHING:
+return InputStatus.END_OF_INPUT;
+case THROTTLING:
+// throttle source as long as sink is not
backpressuring (which it does only
+// after full recovery)
+Thread.sleep(1);
+return InputStatus.MORE_AVAILABLE;
+default:
+return InputStatus.MORE_AVAILABLE;
Review comment:
nitty nit: add `case PUMPING` and make `default: throw new
UnssuportedOperationException(pollingState);`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #14798: [FLINK-21187] Provide exception history for root causes
tillrohrmann commented on a change in pull request #14798: URL: https://github.com/apache/flink/pull/14798#discussion_r566976287 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/SchedulerBase.java ## @@ -164,6 +168,8 @@ private final Map coordinatorMap; private final ComponentMainThreadExecutor mainThreadExecutor; +private final Map localTaskFailures = new HashMap<>(); +private final Map, ErrorInfo> globalTaskFailures = new HashMap<>(); Review comment: For what do we need the maps here? Wouldn't a single `List` be good enough here? Is this kind of a preparation for supporting to capture the exceptions of multiple `Executions`? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #14798: [FLINK-21187] Provide exception history for root causes
tillrohrmann commented on a change in pull request #14798:
URL: https://github.com/apache/flink/pull/14798#discussion_r566868391
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/AccessExecutionVertex.java
##
@@ -64,9 +64,10 @@
* Returns the exception that caused the job to fail. This is the first
root exception that was
* not recoverable and triggered job failure.
*
- * @return failure exception as a string, or {@code "(null)"}
+ * @return failure exception wrapped in a {@link ErrorInfo}, or {@code
null} if no exception was
+ * caught.
*/
-String getFailureCauseAsString();
+ErrorInfo getFailureInfo();
Review comment:
Same here.
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/AccessExecution.java
##
@@ -64,9 +65,11 @@
* Returns the exception that caused the job to fail. This is the first
root exception that was
* not recoverable and triggered job failure.
*
- * @return failure exception as a string, or {@code "(null)"}
+ * @return {@link ErrorInfo} containing the {@code Throwable} wrapped in a
{@link
+ * SerializedThrowable} and the time it was registered, or {@code
null} if no exception was
+ * caught.
*/
-String getFailureCauseAsString();
+ErrorInfo getFailureInfo();
Review comment:
If this method returns `null`, then I would suggest to return an
`Optional`. This makes the contract clearer.
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/ArchivedExecution.java
##
@@ -70,14 +69,14 @@ public ArchivedExecution(
ExecutionAttemptID attemptId,
int attemptNumber,
ExecutionState state,
-String failureCause,
+ErrorInfo failureCause,
Review comment:
`@Nullable` missing.
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/ArchivedExecution.java
##
@@ -37,7 +36,7 @@
private final ExecutionState state;
-private final String failureCause; // once assigned, never changes
+private final ErrorInfo failureInfo; // once assigned, never changes
Review comment:
I guess that this field is `@Nullable`, right?
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/failover/flip1/ExecutionFailureHandler.java
##
@@ -99,23 +103,31 @@ public FailureHandlingResult
getGlobalFailureHandlingResult(final Throwable caus
}
private FailureHandlingResult handleFailure(
+final ExecutionVertexID failingExecutionVertexId,
Review comment:
I think `@Nullable` annotation is missing here.
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/Execution.java
##
@@ -322,8 +322,11 @@ public Throwable getFailureCause() {
}
@Override
-public String getFailureCauseAsString() {
-return ExceptionUtils.stringifyException(getFailureCause());
+public ErrorInfo getFailureInfo() {
+return getFailureCause() == null
+? null
+: new ErrorInfo(
+new SerializedThrowable(getFailureCause()),
getStateTimestamp(FAILED));
Review comment:
I think wrapping `getFailureCause` in a `SerializedThrowable` is not
necessary because that's what `ErrorInfo` does in the constructor.
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ArchivedExecutionGraphTest.java
##
@@ -351,7 +351,19 @@ private static void compareExecutionVertex(
runtimeVertex.getStateTimestamp(ExecutionState.FAILED),
archivedVertex.getStateTimestamp(ExecutionState.FAILED));
assertEquals(
-runtimeVertex.getFailureCauseAsString(),
archivedVertex.getFailureCauseAsString());
+runtimeVertex.getFailureInfo() == null
+? null
+:
runtimeVertex.getFailureInfo().getExceptionAsString(),
+archivedVertex.getFailureInfo() == null
+? null
+:
archivedVertex.getFailureInfo().getExceptionAsString());
Review comment:
with `Optional` this might look a bit nicer.
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/failover/flip1/FailureHandlingResult.java
##
@@ -100,17 +116,22 @@ public long getRestartDelayMS() {
}
}
+/**
+ * Returns the {@link ExecutionVertexID} of the task causing this failure.
+ *
+ * @return The {@code ExecutionVertexID} or {@code null} if it's a global
failure.
+ */
+public ExecutionVertexID getExecutionVertexIdOfFailedTask() {
Review comment:
`@Nullable` is missing.
##
File path:
flink-runtime/src/main/java/org/apache/flink/r
[GitHub] [flink] pnowojski commented on a change in pull request #14807: [FLINK-20654][FLINK-21104][network] Fix two bugs in the handling of UnalignedCheckpoints
pnowojski commented on a change in pull request #14807:
URL: https://github.com/apache/flink/pull/14807#discussion_r566975267
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/ChannelStatePersister.java
##
@@ -67,16 +68,27 @@
this.channelInfo = checkNotNull(channelInfo);
}
-protected void startPersisting(long barrierId, List knownBuffers) {
+protected void startPersisting(long barrierId, List knownBuffers)
+throws CheckpointException {
logEvent("startPersisting", barrierId);
-if (checkpointStatus != CheckpointStatus.BARRIER_RECEIVED &&
lastSeenBarrier < barrierId) {
+if (checkpointStatus == CheckpointStatus.BARRIER_RECEIVED &&
lastSeenBarrier > barrierId) {
+throw new CheckpointException(
Review comment:
As discussed off-line, it's cancelling the currently triggered exception
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14797: [FLINK-21104][network] Various fixes for UC
flinkbot edited a comment on pull request #14797: URL: https://github.com/apache/flink/pull/14797#issuecomment-769223743 ## CI report: * 6410a58539ad960900653721fff977c2566f4d8f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12671) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14648: [FLINK-20978] Implement HeapSavepointRestoreOperation
flinkbot edited a comment on pull request #14648: URL: https://github.com/apache/flink/pull/14648#issuecomment-760154698 ## CI report: * 3cdc18cdbe7166b7993c98787dc1ec2f27fad896 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12672) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #14807: [FLINK-20654][FLINK-21104][network] Fix two bugs in the handling of UnalignedCheckpoints
AHeise commented on a change in pull request #14807:
URL: https://github.com/apache/flink/pull/14807#discussion_r566953027
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/ChannelStatePersisterTest.java
##
@@ -65,38 +83,58 @@ public void testNewBarrierNotOverwrittenByCheckForBarrier()
throws IOException {
}
@Test
-public void testLateBarrierOnCancelledCheckpoint() throws IOException {
-ChannelStatePersister persister =
-new ChannelStatePersister(ChannelStateWriter.NO_OP, new
InputChannelInfo(0, 0));
+public void testLateBarrierOnStartedAndCancelledCheckpoint() throws
Exception {
+testLateBarrier(true, true);
+}
-persister.startPersisting(1L, Collections.emptyList());
-// checkpoint aborted
-persister.stopPersisting(1L);
+@Test
+public void testLateBarrierOnCancelledCheckpoint() throws Exception {
+testLateBarrier(false, true);
+}
-// late barrier
-persister.checkForBarrier(barrier(1L));
+@Test
+public void testLateBarrierOnNotYetCancelledCheckpoint() throws Exception {
+testLateBarrier(false, false);
+}
-persister.startPersisting(2L, Collections.emptyList());
-persister.checkForBarrier(barrier(2L));
+public void testLateBarrier(
Review comment:
nit: private
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannel.java
##
@@ -638,6 +637,11 @@ private void notifyPriorityEventForce() {
CheckpointFailureReason
.CHECKPOINT_SUBSUMED); // currently, at most one
active unaligned
// checkpoint is possible
+} else if (checkpointId > lastBarrierId) {
+// This channel has received some obsolete barrier, older compared
to the checkpointId
+// which we are processing right now, and we should ignore that
obsoleted checkpoint
+// barrier sequence number.
+resetLastBarrier();
Review comment:
I wouldn't have had a side-effect here and just returned an empty list
(which I'd also return with side-effect).
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/ChannelStatePersister.java
##
@@ -67,16 +68,27 @@
this.channelInfo = checkNotNull(channelInfo);
}
-protected void startPersisting(long barrierId, List knownBuffers) {
+protected void startPersisting(long barrierId, List knownBuffers)
+throws CheckpointException {
logEvent("startPersisting", barrierId);
-if (checkpointStatus != CheckpointStatus.BARRIER_RECEIVED &&
lastSeenBarrier < barrierId) {
+if (checkpointStatus == CheckpointStatus.BARRIER_RECEIVED &&
lastSeenBarrier > barrierId) {
+throw new CheckpointException(
Review comment:
How does it guarantee that this doesn't cancel a newer checkpoint? I
don't see any checkpoint id being bound to the exception.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14806: [FLINK-10520][rest] Properly handle optional savepoint parameters
flinkbot edited a comment on pull request #14806: URL: https://github.com/apache/flink/pull/14806#issuecomment-769773030 ## CI report: * 80df70fdb86585fef239e1e323abdf2848f34d64 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12655) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14797: [FLINK-21104][network] Various fixes for UC
flinkbot edited a comment on pull request #14797: URL: https://github.com/apache/flink/pull/14797#issuecomment-769223743 ## CI report: * 94ea4acd0878f020530fb04fa38502a4087f2f19 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12643) * 4123c72616577254f795bede9fd2022830297b53 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12658) * 6410a58539ad960900653721fff977c2566f4d8f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12671) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14648: [FLINK-20978] Implement HeapSavepointRestoreOperation
flinkbot edited a comment on pull request #14648: URL: https://github.com/apache/flink/pull/14648#issuecomment-760154698 ## CI report: * 6179b4646f99bc39047ddc3820a14df8f41f1489 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12055) * 3cdc18cdbe7166b7993c98787dc1ec2f27fad896 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-statefun] igalshilman commented on a change in pull request #195: [FLINK-21171] Wire in TypedValue throughout the runtime as state values and message payloads
igalshilman commented on a change in pull request #195: URL: https://github.com/apache/flink-statefun/pull/195#discussion_r566851179 ## File path: statefun-flink/statefun-flink-common/pom.xml ## @@ -84,10 +88,63 @@ under the License. +
[GitHub] [flink] flinkbot edited a comment on pull request #14797: [FLINK-21104][network] Various fixes for UC
flinkbot edited a comment on pull request #14797: URL: https://github.com/apache/flink/pull/14797#issuecomment-769223743 ## CI report: * 94ea4acd0878f020530fb04fa38502a4087f2f19 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12643) * 4123c72616577254f795bede9fd2022830297b53 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12658) * 6410a58539ad960900653721fff977c2566f4d8f UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14734: [FLINK-21066][runtime][checkpoint] Refactor CheckpointCoordinator to compute tasks to trigger/ack/commit dynamically
flinkbot edited a comment on pull request #14734: URL: https://github.com/apache/flink/pull/14734#issuecomment-766092219 ## CI report: * 0c94f50f2a8e9a5753479fce0cb18e02a1931c16 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12476) * 39b84b60a20f67d813599bcea74ebbc28e9be573 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12669) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14591: FLINK-20359 Added Owner Reference to Job Manager in native kubernetes
flinkbot edited a comment on pull request #14591: URL: https://github.com/apache/flink/pull/14591#issuecomment-756825239 ## CI report: * 35947579b06f3b4d67e151e2fc38565813076008 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12488) * 402056b79cfdcfeb7c718ca58ec2a96eed1554d0 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12668) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-21108) Flink runtime rest server and history server webmonitor do not require authentication.
[ https://issues.apache.org/jira/browse/FLINK-21108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17274547#comment-17274547 ] Xiaoguang Sun commented on FLINK-21108: --- Of cause all authentication methods work at HTTP protocol level, just implement it according to standards will work. > Flink runtime rest server and history server webmonitor do not require > authentication. > -- > > Key: FLINK-21108 > URL: https://issues.apache.org/jira/browse/FLINK-21108 > Project: Flink > Issue Type: New Feature > Components: Runtime / REST, Runtime / Web Frontend >Reporter: Xiaoguang Sun >Priority: Major > Labels: pull-request-available > > Flink runtime rest server and history server webmonitor do not require > authentication. At certain scenarios, prohibiting unauthorized access is > desired. Http basic authentication can be used here. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-21132) BoundedOneInput.endInput is called when taking synchronous savepoint
[
https://issues.apache.org/jira/browse/FLINK-21132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17274538#comment-17274538
]
Roman Khachatryan commented on FLINK-21132:
---
Thanks for your feedback [~aljoscha] and [~kezhuw] . I've also discussed it
offline with [~pnowojski].
We agreed that endInpt should NOT be called (no operators should rely on it, if
any do they should be fixed).
(so no StopWithSavepointAware needed)
As for the detection (stop-with-savepoint vs other cases), we agree the
proposed solution isn't ideal. However, it's much less invasive and we'd like
to provide fix ASAP so we lean toward it.
I'm adjusting it to all cases of chaining (these are different code paths).
> BoundedOneInput.endInput is called when taking synchronous savepoint
>
>
> Key: FLINK-21132
> URL: https://issues.apache.org/jira/browse/FLINK-21132
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.10.2, 1.10.3, 1.11.3, 1.12.1
>Reporter: Kezhu Wang
>Assignee: Roman Khachatryan
>Priority: Major
>
> [~elkhand](?) reported on project iceberg that {{BoundedOneInput.endInput}}
> was
> [called|https://github.com/apache/iceberg/issues/2033#issuecomment-765864038]
> when [stopping job with
> savepoint|https://github.com/apache/iceberg/issues/2033#issuecomment-765557995].
> I think it is a bug of Flink and was introduced in FLINK-14230. The
> [changes|https://github.com/apache/flink/pull/9854/files#diff-0c5fe245445b932fa83fdaf5c4802dbe671b73e8d76f39058c6eaaaffd9639faL577]
> rely on {{StreamTask.afterInvoke}} and {{OperatorChain.closeOperators}} will
> only be invoked after *end of input*. But that is not true long before after
> [FLIP-34: Terminate/Suspend Job with
> Savepoint|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103090212].
> Task could enter state called
> [*finished*|https://github.com/apache/flink/blob/3a8e06cd16480eacbbf0c10f36b8c79a6f741814/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java#L467]
> after synchronous savepoint, that is an expected job suspension and stopping.
> [~sunhaibotb] [~pnowojski] [~roman_khachatryan] Could you help confirm this ?
> For full context, see
> [apache/iceberg#2033|https://github.com/apache/iceberg/issues/2033]. I have
> pushed branch
> [synchronous-savepoint-conflict-with-bounded-end-input-case|https://github.com/kezhuw/flink/commits/synchronous-savepoint-conflict-with-bounded-end-input-case]
> in my repository. Test case
> {{SavepointITCase.testStopSavepointWithBoundedInput}} failed due to
> {{BoundedOneInput.endInput}} called.
> I am also aware of [FLIP-147: Support Checkpoints After Tasks
> Finished|https://cwiki.apache.org/confluence/x/mw-ZCQ], maybe the three
> should align on what *finished* means exactly. [~kkl0u] [~chesnay]
> [~gaoyunhaii]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14734: [FLINK-21066][runtime][checkpoint] Refactor CheckpointCoordinator to compute tasks to trigger/ack/commit dynamically
flinkbot edited a comment on pull request #14734: URL: https://github.com/apache/flink/pull/14734#issuecomment-766092219 ## CI report: * 0c94f50f2a8e9a5753479fce0cb18e02a1931c16 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12476) * 39b84b60a20f67d813599bcea74ebbc28e9be573 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14591: FLINK-20359 Added Owner Reference to Job Manager in native kubernetes
flinkbot edited a comment on pull request #14591: URL: https://github.com/apache/flink/pull/14591#issuecomment-756825239 ## CI report: * 35947579b06f3b4d67e151e2fc38565813076008 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12488) * 402056b79cfdcfeb7c718ca58ec2a96eed1554d0 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] gaoyunhaii commented on a change in pull request #14734: [FLINK-21066][runtime][checkpoint] Refactor CheckpointCoordinator to compute tasks to trigger/ack/commit dynamically
gaoyunhaii commented on a change in pull request #14734:
URL: https://github.com/apache/flink/pull/14734#discussion_r566910553
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointStatsTracker.java
##
@@ -183,23 +166,27 @@ public CheckpointStatsSnapshot createSnapshot() {
/**
* Creates a new pending checkpoint tracker.
*
+ * @param tasksToAck Tasks to acknowledge in this checkpoint.
* @param checkpointId ID of the checkpoint.
* @param triggerTimestamp Trigger timestamp of the checkpoint.
* @param props The checkpoint properties.
* @return Tracker for statistics gathering.
*/
PendingCheckpointStats reportPendingCheckpoint(
-long checkpointId, long triggerTimestamp, CheckpointProperties
props) {
+List tasksToAck,
Review comment:
The method has been evolved, thus we do not need to modify this for now.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] gaoyunhaii commented on a change in pull request #14734: [FLINK-21066][runtime][checkpoint] Refactor CheckpointCoordinator to compute tasks to trigger/ack/commit dynamically
gaoyunhaii commented on a change in pull request #14734:
URL: https://github.com/apache/flink/pull/14734#discussion_r566909909
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java
##
@@ -730,7 +715,8 @@ private PendingCheckpoint createPendingCheckpoint(
if (statsTracker != null) {
PendingCheckpointStats callback =
-statsTracker.reportPendingCheckpoint(checkpointID,
timestamp, props);
+statsTracker.reportPendingCheckpoint(
Review comment:
Got it, I also agree with this. Since the master has been evolved to use
the `tasksToWaitFor`, so I just kept the latest behavior in this PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] gaoyunhaii commented on a change in pull request #14734: [FLINK-21066][runtime][checkpoint] Refactor CheckpointCoordinator to compute tasks to trigger/ack/commit dynamically
gaoyunhaii commented on a change in pull request #14734:
URL: https://github.com/apache/flink/pull/14734#discussion_r566909909
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java
##
@@ -730,7 +715,8 @@ private PendingCheckpoint createPendingCheckpoint(
if (statsTracker != null) {
PendingCheckpointStats callback =
-statsTracker.reportPendingCheckpoint(checkpointID,
timestamp, props);
+statsTracker.reportPendingCheckpoint(
Review comment:
Got it, I agree with this. Since the master has been evolved to use the
`tasksToWaitFor`, so I just kept the latest behavior in this PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14813: [FLINK-21207] Fix 'csv.disable-quote-character' can not take effect during deserial…
flinkbot edited a comment on pull request #14813: URL: https://github.com/apache/flink/pull/14813#issuecomment-769867480 ## CI report: * ab61d899b8b08c6c4707ba65f922c4a838bdbb6e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12667) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] blublinsky commented on pull request #14591: FLINK-20359 Added Owner Reference to Job Manager in native kubernetes
blublinsky commented on pull request #14591: URL: https://github.com/apache/flink/pull/14591#issuecomment-769877297 @xintongsong , done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-21132) BoundedOneInput.endInput is called when taking synchronous savepoint
[
https://issues.apache.org/jira/browse/FLINK-21132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17274517#comment-17274517
]
Piotr Nowojski commented on FLINK-21132:
I agree with {{endOfInput}} should NOT be called in case of stop with
savepoint. IMO adding another interface {{StopWithSavepointAware}} for the sake
of backward compatibility would unnecessarily complicate our APIs.
After offline discussion with [~roman_khachatryan], probably the easiest way to
do, would be to just suppress {{endOfInput}} calls somewhere in the
{{OperatorChain}} or {{StreamOperatorWrapper}} if we know we are waiting for
the stop with savepoint. It's not ideal, but adding an extra even to
distinguish between {{END_OF_INPUT}} and {{STOP_WITH_SAVEPOINT}} would be quite
invasive change.
> BoundedOneInput.endInput is called when taking synchronous savepoint
>
>
> Key: FLINK-21132
> URL: https://issues.apache.org/jira/browse/FLINK-21132
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.10.2, 1.10.3, 1.11.3, 1.12.1
>Reporter: Kezhu Wang
>Assignee: Roman Khachatryan
>Priority: Major
>
> [~elkhand](?) reported on project iceberg that {{BoundedOneInput.endInput}}
> was
> [called|https://github.com/apache/iceberg/issues/2033#issuecomment-765864038]
> when [stopping job with
> savepoint|https://github.com/apache/iceberg/issues/2033#issuecomment-765557995].
> I think it is a bug of Flink and was introduced in FLINK-14230. The
> [changes|https://github.com/apache/flink/pull/9854/files#diff-0c5fe245445b932fa83fdaf5c4802dbe671b73e8d76f39058c6eaaaffd9639faL577]
> rely on {{StreamTask.afterInvoke}} and {{OperatorChain.closeOperators}} will
> only be invoked after *end of input*. But that is not true long before after
> [FLIP-34: Terminate/Suspend Job with
> Savepoint|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103090212].
> Task could enter state called
> [*finished*|https://github.com/apache/flink/blob/3a8e06cd16480eacbbf0c10f36b8c79a6f741814/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java#L467]
> after synchronous savepoint, that is an expected job suspension and stopping.
> [~sunhaibotb] [~pnowojski] [~roman_khachatryan] Could you help confirm this ?
> For full context, see
> [apache/iceberg#2033|https://github.com/apache/iceberg/issues/2033]. I have
> pushed branch
> [synchronous-savepoint-conflict-with-bounded-end-input-case|https://github.com/kezhuw/flink/commits/synchronous-savepoint-conflict-with-bounded-end-input-case]
> in my repository. Test case
> {{SavepointITCase.testStopSavepointWithBoundedInput}} failed due to
> {{BoundedOneInput.endInput}} called.
> I am also aware of [FLIP-147: Support Checkpoints After Tasks
> Finished|https://cwiki.apache.org/confluence/x/mw-ZCQ], maybe the three
> should align on what *finished* means exactly. [~kkl0u] [~chesnay]
> [~gaoyunhaii]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #14813: [FLINK-21207] Fix 'csv.disable-quote-character' can not take effect during deserial…
flinkbot commented on pull request #14813: URL: https://github.com/apache/flink/pull/14813#issuecomment-769867480 ## CI report: * ab61d899b8b08c6c4707ba65f922c4a838bdbb6e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
