[jira] [Comment Edited] (FLINK-19940) Task names on web UI should change when an operator chain is chained with sources
[ https://issues.apache.org/jira/browse/FLINK-19940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235968#comment-17235968 ] Caizhi Weng edited comment on FLINK-19940 at 11/20/20, 7:58 AM: [~roman_khachatryan] It's not truncated. It's just that the TableSourceScan part does not appear. The task name in the detail tab on the right will not be truncated, however the TableSourceScan part still doesn't appear. The #2 TableSourceScan indicates that its an input of the multiple input operator and is included in the name of the MultipleInputOperator (we display all inputs of the multiple input operator in its name), which is not that obvious for the user to tell that there is a chained source. Indeed, one can argue that source chaining is currently only implemented for multiple input tasks, so it seems to be sufficient that chained sources are hinted in the multiple input operators. However source chaining is related more to operator chaining rather than an operator in the table API. If we implement source chaining for two-input operators in the future, or if the users write an multiple input transformation themselves, then users will have no way to tell whether there is a chained source or not. was (Author: tsreaper): [~roman_khachatryan] It's not truncated. It's just that the TableSourceScan part does not appear. The task name in the detail tab on the right will not be truncated, however the TableSourceScan part still doesn't appear. The #2 TableSourceScan indicates that its an input of the multiple input operator and is included in the name of the MultipleInputOperator (we display all inputs of the multiple input operator in its name), which is not that obvious for the user to tell that there is a chained source. Indeed, one can argue that source chaining is currently only implemented for multiple input tasks, so it seems to be sufficient that chained sources are hinted in the multiple input operators. However source chaining is related more to operator chaining rather than an operator in the table API. If we implement source chaining for two-input operators in the future, then users will have no way to tell whether there is a chained source or not. > Task names on web UI should change when an operator chain is chained with > sources > - > > Key: FLINK-19940 > URL: https://issues.apache.org/jira/browse/FLINK-19940 > Project: Flink > Issue Type: Bug > Components: Runtime / Web Frontend >Reporter: Caizhi Weng >Assignee: Roman Khachatryan >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > Attachments: multi-web-ui.png > > > Currently task names on web UI doesn't change even if the operator chain is > chained with sources. We should change its name to show that the sources are > chained. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #8532: [FLINK-12541][REST] Support to submit Python Table API jobs via REST API
flinkbot edited a comment on pull request #8532: URL: https://github.com/apache/flink/pull/8532#issuecomment-730962739 ## CI report: * d24c4f7af0dbf8890d87f040a932121ef9e8d869 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/202984747) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-19940) Task names on web UI should change when an operator chain is chained with sources
[ https://issues.apache.org/jira/browse/FLINK-19940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235968#comment-17235968 ] Caizhi Weng edited comment on FLINK-19940 at 11/20/20, 7:57 AM: [~roman_khachatryan] It's not truncated. It's just that the TableSourceScan part does not appear. The task name in the detail tab on the right will not be truncated, however the TableSourceScan part still doesn't appear. The #2 TableSourceScan indicates that its an input of the multiple input operator and is included in the name of the MultipleInputOperator (we display all inputs of the multiple input operator in its name), which is not that obvious for the user to tell that there is a chained source. Indeed, one can argue that source chaining is currently only implemented for multiple input tasks, so it seems to be sufficient that chained sources are hinted in the multiple input operators. However source chaining is related more to operator chaining rather than an operator in the table API. If we implement source chaining for two-input operators in the future, then users will have no way to tell whether there is a chained source or not. was (Author: tsreaper): [~roman_khachatryan] It's not truncated. It's just that the TableSourceScan part does not appear. The task name in the detail tab on the right will not be truncated, however the TableSourceScan part still doesn't appear. The #2 TableSourceScan indicates that its an input of the multiple input operator and is included in the name of the MultipleInputOperator (we display all inputs of the multiple input operator in its name), which is not that obvious for the user to tell that there is a chained source. If we implement source chaining for two-input operators in the future, then users will have no way to tell whether there is a chained source or not. > Task names on web UI should change when an operator chain is chained with > sources > - > > Key: FLINK-19940 > URL: https://issues.apache.org/jira/browse/FLINK-19940 > Project: Flink > Issue Type: Bug > Components: Runtime / Web Frontend >Reporter: Caizhi Weng >Assignee: Roman Khachatryan >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > Attachments: multi-web-ui.png > > > Currently task names on web UI doesn't change even if the operator chain is > chained with sources. We should change its name to show that the sources are > chained. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-19940) Task names on web UI should change when an operator chain is chained with sources
[ https://issues.apache.org/jira/browse/FLINK-19940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235968#comment-17235968 ] Caizhi Weng edited comment on FLINK-19940 at 11/20/20, 7:54 AM: [~roman_khachatryan] It's not truncated. It's just that the TableSourceScan part does not appear. The task name in the detail tab on the right will not be truncated, however the TableSourceScan part still doesn't appear. The #2 TableSourceScan indicates that its an input of the multiple input operator and is included in the name of the MultipleInputOperator (we display all inputs of the multiple input operator in its name), which is not that obvious for the user to tell that there is a chained source. If we implement source chaining for two input operators in the future, then users will have no way to tell whether there is a chained source or not. was (Author: tsreaper): [~roman_khachatryan] It's not truncated. It's just that the TableSourceScan part does not appear. The task name in the detail tab on the right will not be truncated, however the TableSourceScan part still doesn't appear. The [#2] TableSourceScan is included in the name of the MultipleInputOperator (we display all inputs of the multiple input operator in its name), which is not that obvious for the user to tell that there is a chained source. If we implement source chaining for two input operators in the future, then users will have no way to tell whether there is a chained source or not. > Task names on web UI should change when an operator chain is chained with > sources > - > > Key: FLINK-19940 > URL: https://issues.apache.org/jira/browse/FLINK-19940 > Project: Flink > Issue Type: Bug > Components: Runtime / Web Frontend >Reporter: Caizhi Weng >Assignee: Roman Khachatryan >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > Attachments: multi-web-ui.png > > > Currently task names on web UI doesn't change even if the operator chain is > chained with sources. We should change its name to show that the sources are > chained. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-19940) Task names on web UI should change when an operator chain is chained with sources
[ https://issues.apache.org/jira/browse/FLINK-19940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235968#comment-17235968 ] Caizhi Weng edited comment on FLINK-19940 at 11/20/20, 7:54 AM: [~roman_khachatryan] It's not truncated. It's just that the TableSourceScan part does not appear. The task name in the detail tab on the right will not be truncated, however the TableSourceScan part still doesn't appear. The #2 TableSourceScan indicates that its an input of the multiple input operator and is included in the name of the MultipleInputOperator (we display all inputs of the multiple input operator in its name), which is not that obvious for the user to tell that there is a chained source. If we implement source chaining for two-input operators in the future, then users will have no way to tell whether there is a chained source or not. was (Author: tsreaper): [~roman_khachatryan] It's not truncated. It's just that the TableSourceScan part does not appear. The task name in the detail tab on the right will not be truncated, however the TableSourceScan part still doesn't appear. The #2 TableSourceScan indicates that its an input of the multiple input operator and is included in the name of the MultipleInputOperator (we display all inputs of the multiple input operator in its name), which is not that obvious for the user to tell that there is a chained source. If we implement source chaining for two input operators in the future, then users will have no way to tell whether there is a chained source or not. > Task names on web UI should change when an operator chain is chained with > sources > - > > Key: FLINK-19940 > URL: https://issues.apache.org/jira/browse/FLINK-19940 > Project: Flink > Issue Type: Bug > Components: Runtime / Web Frontend >Reporter: Caizhi Weng >Assignee: Roman Khachatryan >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > Attachments: multi-web-ui.png > > > Currently task names on web UI doesn't change even if the operator chain is > chained with sources. We should change its name to show that the sources are > chained. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-19940) Task names on web UI should change when an operator chain is chained with sources
[ https://issues.apache.org/jira/browse/FLINK-19940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235968#comment-17235968 ] Caizhi Weng commented on FLINK-19940: - [~roman_khachatryan] It's not truncated. It's just that the TableSourceScan part does not appear. The task name in the detail tab on the right will not be truncated, however the TableSourceScan part still doesn't appear. The [#2] TableSourceScan is included in the name of the MultipleInputOperator (we display all inputs of the multiple input operator in its name), which is not that obvious for the user to tell that there is a chained source. If we implement source chaining for two input operators in the future, then users will have no way to tell whether there is a chained source or not. > Task names on web UI should change when an operator chain is chained with > sources > - > > Key: FLINK-19940 > URL: https://issues.apache.org/jira/browse/FLINK-19940 > Project: Flink > Issue Type: Bug > Components: Runtime / Web Frontend >Reporter: Caizhi Weng >Assignee: Roman Khachatryan >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > Attachments: multi-web-ui.png > > > Currently task names on web UI doesn't change even if the operator chain is > chained with sources. We should change its name to show that the sources are > chained. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-19940) Task names on web UI should change when an operator chain is chained with sources
[ https://issues.apache.org/jira/browse/FLINK-19940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235968#comment-17235968 ] Caizhi Weng edited comment on FLINK-19940 at 11/20/20, 7:53 AM: [~roman_khachatryan] It's not truncated. It's just that the TableSourceScan part does not appear. The task name in the detail tab on the right will not be truncated, however the TableSourceScan part still doesn't appear. The [#2] TableSourceScan is included in the name of the MultipleInputOperator (we display all inputs of the multiple input operator in its name), which is not that obvious for the user to tell that there is a chained source. If we implement source chaining for two input operators in the future, then users will have no way to tell whether there is a chained source or not. was (Author: tsreaper): [~roman_khachatryan] It's not truncated. It's just that the TableSourceScan part does not appear. The task name in the detail tab on the right will not be truncated, however the TableSourceScan part still doesn't appear. The [#2] TableSourceScan is included in the name of the MultipleInputOperator (we display all inputs of the multiple input operator in its name), which is not that obvious for the user to tell that there is a chained source. If we implement source chaining for two input operators in the future, then users will have no way to tell whether there is a chained source or not. > Task names on web UI should change when an operator chain is chained with > sources > - > > Key: FLINK-19940 > URL: https://issues.apache.org/jira/browse/FLINK-19940 > Project: Flink > Issue Type: Bug > Components: Runtime / Web Frontend >Reporter: Caizhi Weng >Assignee: Roman Khachatryan >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > Attachments: multi-web-ui.png > > > Currently task names on web UI doesn't change even if the operator chain is > chained with sources. We should change its name to show that the sources are > chained. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] rkhachatryan commented on a change in pull request #14139: [FLINK-20247][config] Hide unused unaligned checkpoint configuration options
rkhachatryan commented on a change in pull request #14139:
URL: https://github.com/apache/flink/pull/14139#discussion_r527497144
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/ExecutionCheckpointingOptions.java
##
@@ -162,10 +162,11 @@
"will timeout and checkpoint barrier
will start working as unaligned checkpoint.")
.build());
+ @Documentation.ExcludeFromDocumentation("Do not advertise this option
until rescaling of unaligned checkpoint is completed.")
public static final ConfigOption FORCE_UNALIGNED =
ConfigOptions.key("execution.checkpointing.unaligned.forced")
.booleanType()
- .defaultValue(false)
+ .defaultValue(true)
Review comment:
I see, thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14141: [FLINK-20145][checkpointing] Fix priority event handling.
flinkbot edited a comment on pull request #14141: URL: https://github.com/apache/flink/pull/14141#issuecomment-730461048 ## CI report: * 0126e8f9cb660e5205c32e080fbe8c20481fc476 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9834) * e67078cc99525b0cd2ed6fb23c1eac9063600191 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14139: [FLINK-20247][config] Hide unused unaligned checkpoint configuration options
flinkbot edited a comment on pull request #14139: URL: https://github.com/apache/flink/pull/14139#issuecomment-730377256 ## CI report: * e7f30f23c854949e8af6b0465631ee214839597a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9831) * a693a0a58d3d2fd48b5c5eef18074b1c66f0 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14111: [FLINK-20191][document] Add documents for FLIP-95 ability interfaces
flinkbot edited a comment on pull request #14111: URL: https://github.com/apache/flink/pull/14111#issuecomment-729488072 ## CI report: * ec3b7418bcca38b4b60ad5491a8f802244b6227a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9812) * 50cd38257dc8850b4c4e51346ceccdce01018251 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14150: [FLINK-18452][table-runtime-blink] Fix StateMigrationException because RetractableTopNFunction#ComparatorWrapper might be incompatible
flinkbot commented on pull request #14150: URL: https://github.com/apache/flink/pull/14150#issuecomment-730968094 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 0c7283425c0ff6d780355415d5f4db340ed0aeea (Fri Nov 20 07:41:25 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wuchong commented on pull request #14150: [FLINK-18452][table-runtime-blink] Fix StateMigrationException because RetractableTopNFunction#ComparatorWrapper might be incompatible
wuchong commented on pull request #14150: URL: https://github.com/apache/flink/pull/14150#issuecomment-730965603 @leonardBang , could you help to review this? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-18452) Flaws in RetractableTopNFunction.ComparatorWrapper#equals method prevent state access after restoration
[
https://issues.apache.org/jira/browse/FLINK-18452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-18452:
---
Labels: pull-request-available (was: )
> Flaws in RetractableTopNFunction.ComparatorWrapper#equals method prevent
> state access after restoration
> ---
>
> Key: FLINK-18452
> URL: https://issues.apache.org/jira/browse/FLINK-18452
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.10.0, 1.10.1, 1.11.0
>Reporter: Weike Dong
>Assignee: Jark Wu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0
>
> Attachments:
> c2ebeac8aadebad0dffa5cc255d45190594c5b2a84bda020dd30bf24b9169702.png
>
>
> We found that in SQL jobs using "Top-N" functionality provided by the blink
> planner, the job state cannot be retrieved because of "incompatible" state
> serializers (in fact they are compatible).
> The error log is displayed like below
> {panel:title=taskmanager.log}
> 2020-06-30 09:19:32.089 [Rank(strategy=[RetractStrategy],
> rankType=[ROW_NUMBER], rankRange=[rankStart=1, rankEnd=100],
> partitionBy=[appkey, serverid], orderBy=[quantity DESC], select=[appkey,
> serverid, quantity]) (1/1)] INFO org.apache.flink.runtime.taskmanager.Task
> - Rank(strategy=[RetractStrategy], rankType=[ROW_NUMBER],
> rankRange=[rankStart=1, rankEnd=100], partitionBy=[appkey, serverid],
> orderBy=[quantity DESC], select=[appkey, serverid, oid, quantity]) (1/1)
> (bd4d2e4327efac57dc70e220b8de460b) switched from RUNNING to FAILED.
> java.lang.RuntimeException: Error while getting state
> at
> org.apache.flink.runtime.state.DefaultKeyedStateStore.getState(DefaultKeyedStateStore.java:62)
> at
> org.apache.flink.streaming.api.operators.StreamingRuntimeContext.getState(StreamingRuntimeContext.java:144)
> at
> org.apache.flink.table.runtime.operators.rank.RetractableTopNFunction.open(RetractableTopNFunction.java:115)
> at
> org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
> at
> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102)
> at
> org.apache.flink.streaming.api.operators.KeyedProcessOperator.open(KeyedProcessOperator.java:57)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.initializeStateAndOpen(StreamTask.java:990)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:453)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:94)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:448)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:460)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:708)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:533)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.flink.util.StateMigrationException: The new state
> serializer cannot be incompatible.
> at
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.updateRestoredStateMetaInfo(RocksDBKeyedStateBackend.java:543)
> at
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.tryRegisterKvStateInformation(RocksDBKeyedStateBackend.java:491)
> at
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.createInternalState(RocksDBKeyedStateBackend.java:652)
> at
> org.apache.flink.runtime.state.KeyedStateFactory.createInternalState(KeyedStateFactory.java:47)
> at
> org.apache.flink.runtime.state.ttl.TtlStateFactory.createStateAndWrapWithTtlIfEnabled(TtlStateFactory.java:72)
> at
> org.apache.flink.runtime.state.AbstractKeyedStateBackend.getOrCreateKeyedState(AbstractKeyedStateBackend.java:279)
> at
> org.apache.flink.runtime.state.AbstractKeyedStateBackend.getPartitionedState(AbstractKeyedStateBackend.java:328)
> at
> org.apache.flink.runtime.state.DefaultKeyedStateStore.getPartitionedState(DefaultKeyedStateStore.java:124)
> at
> org.apache.flink.runtime.state.DefaultKeyedStateStore.getState(DefaultKeyedStateStore.java:60)
> ... 13 more{panel}
>
> After careful debugging, it is found to be an issue with the compatibility
> check of type serializers.
>
> In short, during checkpointing, Flink serializes _SortedMapSerializer_ by
> creating a _SortedMapSerializerSnapshot_ object, and the original comparator
> is encapsulated within the object (here we call
[GitHub] [flink] wuchong opened a new pull request #14150: [FLINK-18452][table-runtime-blink] Fix StateMigrationException because RetractableTopNFunction#ComparatorWrapper might be incompatible
wuchong opened a new pull request #14150: URL: https://github.com/apache/flink/pull/14150 ## What is the purpose of the change TopN query may throw StateMigrationException because the `RetractableTopNFunction#ComparatorWrapper` in serializer might be incompatible. The reason lays in `ComparatorWrapper` compares the generated code, however code might be differnet when generated in different client. ## Brief change log - Fix `ComparatorWrapper` to compare meta information which is used to generate the code. And rename it to `ComparableRecordComparator`. ## Verifying this change The test is hard to add. We cover this by using `ComparableRecordComparator` in harness unit test. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #8532: [FLINK-12541][REST] Support to submit Python Table API jobs via REST API
flinkbot commented on pull request #8532: URL: https://github.com/apache/flink/pull/8532#issuecomment-730962739 ## CI report: * d24c4f7af0dbf8890d87f040a932121ef9e8d869 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-20221) DelimitedInputFormat does not restore compressed filesplits correctly leading to dataloss
[ https://issues.apache.org/jira/browse/FLINK-20221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235955#comment-17235955 ] Robert Metzger commented on FLINK-20221: [~gyfora] What's the plan with merging this? do you want to create a PR for the change? it would be nice to include this into the next release candidate for 1.12.0 which I would like to create on Monday > DelimitedInputFormat does not restore compressed filesplits correctly leading > to dataloss > - > > Key: FLINK-20221 > URL: https://issues.apache.org/jira/browse/FLINK-20221 > Project: Flink > Issue Type: Bug > Components: Connectors / FileSystem >Affects Versions: 1.10.2, 1.12.0, 1.11.2 >Reporter: Gyula Fora >Assignee: Gyula Fora >Priority: Blocker > Fix For: 1.12.0, 1.10.3, 1.11.3 > > > It seems that the delimited input format cannot correctly restore input > splits if they belong to compressed files. Basically when a compressed > filesplit is restored in the middle, it won't read it anymore leading to > dataloss. > The cause of the problem is that for compressed splits that use an inflater > stream, the splitlength is set to the magic number -1 which is ignored in the > reopen method and causes the split to go to `end` state immediately. > The problem and the fix is shown in this commit: > [https://github.com/gyfora/flink/commit/4adc8ba8d1989fff2db43881c9cb3799848c6e0d] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-17096) Mini-batch group aggregation doesn't expire state even if state ttl is enabled
[ https://issues.apache.org/jira/browse/FLINK-17096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu closed FLINK-17096. --- Resolution: Fixed Fixed in master (1.12.0): - d84ad5e585c6573635a388de4a5602fe4229fa06 - df8de4ac45884bd3134abaf87baeb65c31060b3a > Mini-batch group aggregation doesn't expire state even if state ttl is enabled > -- > > Key: FLINK-17096 > URL: https://issues.apache.org/jira/browse/FLINK-17096 > Project: Flink > Issue Type: Bug > Components: Table SQL / Runtime >Affects Versions: 1.9.0, 1.10.0, 1.11.0 >Reporter: dalongliu >Assignee: dalongliu >Priority: Critical > Labels: pull-request-available > Fix For: 1.12.0 > > > At the moment, MiniBatch Group Agg include Local/Global doesn`t support State > TTL, for streaming job, it will lead to OOM in long time running, so we need > to make state data expire after ttl, the solution is that use incremental > cleanup feature refer to FLINK-16581 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] wuchong closed pull request #11830: [FLINK-17096] [table] Mini-batch group aggregation doesn't expire state even if state ttl is enabled
wuchong closed pull request #11830: URL: https://github.com/apache/flink/pull/11830 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] twalthr commented on pull request #14103: [FLINK-18545] Introduce `pipeline.name` to allow users to specify job name by configuration
twalthr commented on pull request #14103: URL: https://github.com/apache/flink/pull/14103#issuecomment-730959217 I agree with @jark and @godfreyhe. The name is just used for logging and easier identification in UIs. The user can always change it before the next execution. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wuchong commented on pull request #11830: [FLINK-17096] [table] Mini-batch group aggregation doesn't expire state even if state ttl is enabled
wuchong commented on pull request #11830: URL: https://github.com/apache/flink/pull/11830#issuecomment-730959172 Build is passed: https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=9847=results This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14149: [FLINK-20035][tests] Fix wrong config key in JobGraphRunningUtil
flinkbot edited a comment on pull request #14149: URL: https://github.com/apache/flink/pull/14149#issuecomment-730947533 ## CI report: * c090c7ec80bef6783cdd667b0d15d1f0b5fd9b57 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9855) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-12541) Add deploy a Python Flink job and session cluster on Kubernetes support.
[ https://issues.apache.org/jira/browse/FLINK-12541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235951#comment-17235951 ] Eui Heo commented on FLINK-12541: - How is it going? I want to submit a python job inside application, and it would be great if REST API supports it. > Add deploy a Python Flink job and session cluster on Kubernetes support. > > > Key: FLINK-12541 > URL: https://issues.apache.org/jira/browse/FLINK-12541 > Project: Flink > Issue Type: Sub-task > Components: API / Python, Runtime / REST >Affects Versions: 1.9.0 >Reporter: sunjincheng >Assignee: Dian Fu >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > Add deploy a Python Flink job and session cluster on Kubernetes support. > We need to have the same deployment step as the Java job. Please see: > [https://ci.apache.org/projects/flink/flink-docs-stable/ops/deployment/kubernetes.html] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-18983) Job doesn't changed to failed if close function has blocked
[
https://issues.apache.org/jira/browse/FLINK-18983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
YufeiLiu updated FLINK-18983:
-
Description:
If a operator throw a exception, it will break process loop and dispose all
operator. But state will never switch to FAILED if block in Function.close, and
JobMaster can't know the final state and do restart.
Task have {{TaskCancelerWatchDog}} to kill process if cancellation timeout, but
it doesn't work for FAILED task.TAskThread will allways hang at:
org.apache.flink.streaming.runtime.tasks.StreamTask#cleanUpInvoke
Test case:
{code:java}
Configuration configuration = new Configuration();
configuration.set(TaskManagerOptions.TASK_CANCELLATION_TIMEOUT, 1L);
StreamExecutionEnvironment env =
StreamExecutionEnvironment.createLocalEnvironment(2, configuration);
env.addSource(...)
.process(new ProcessFunction() {
@Override
public void processElement(String value, Context ctx,
Collector out) throws Exception {
if (getRuntimeContext().getIndexOfThisSubtask() == 0) {
throw new RuntimeException();
}
}
@Override
public void close() throws Exception {
if (getRuntimeContext().getIndexOfThisSubtask() == 0) {
Thread.sleep(1000);
}
}
}).setParallelism(2)
.print();
{code}
In this case, job will block at close action and never change to FAILED.
If change thread which subtaskIndex == 1 to sleep, TM will exit after
TASK_CANCELLATION_TIMEOUT.
was:
If a operator throw a exception, it will break process loop and dispose all
operator. But state will never switch to FAILED if block in Function.close, and
JobMaster can't know the final state and do restart.
Task have {{TaskCancelerWatchDog}} to kill process if cancellation timeout, but
it doesn't work for FAILED task.
Can we just report final state and trigger clean up action by JM.
Test case:
{code:java}
Configuration configuration = new Configuration();
configuration.set(TaskManagerOptions.TASK_CANCELLATION_TIMEOUT, 1L);
StreamExecutionEnvironment env =
StreamExecutionEnvironment.createLocalEnvironment(2, configuration);
env.addSource(...)
.process(new ProcessFunction() {
@Override
public void processElement(String value, Context ctx,
Collector out) throws Exception {
if (getRuntimeContext().getIndexOfThisSubtask() == 0) {
throw new RuntimeException();
}
}
@Override
public void close() throws Exception {
if (getRuntimeContext().getIndexOfThisSubtask() == 0) {
Thread.sleep(1000);
}
}
}).setParallelism(2)
.print();
{code}
In this case, job will block at close action and never change to FAILED.
If change thread which subtaskIndex == 1 to sleep, TM will exit after
TASK_CANCELLATION_TIMEOUT.
> Job doesn't changed to failed if close function has blocked
> ---
>
> Key: FLINK-18983
> URL: https://issues.apache.org/jira/browse/FLINK-18983
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.11.0
>Reporter: YufeiLiu
>Priority: Major
>
> If a operator throw a exception, it will break process loop and dispose all
> operator. But state will never switch to FAILED if block in Function.close,
> and JobMaster can't know the final state and do restart.
> Task have {{TaskCancelerWatchDog}} to kill process if cancellation timeout,
> but it doesn't work for FAILED task.TAskThread will allways hang at:
> org.apache.flink.streaming.runtime.tasks.StreamTask#cleanUpInvoke
> Test case:
> {code:java}
> Configuration configuration = new Configuration();
> configuration.set(TaskManagerOptions.TASK_CANCELLATION_TIMEOUT, 1L);
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.createLocalEnvironment(2, configuration);
> env.addSource(...)
> .process(new ProcessFunction() {
> @Override
> public void processElement(String value, Context ctx,
> Collector out) throws Exception {
> if (getRuntimeContext().getIndexOfThisSubtask() == 0) {
> throw new RuntimeException();
> }
> }
> @Override
> public void close() throws Exception {
> if (getRuntimeContext().getIndexOfThisSubtask() == 0) {
> Thread.sleep(1000);
>
[jira] [Comment Edited] (FLINK-20118) Test New File Source API - Continuous Streaming Execution
[ https://issues.apache.org/jira/browse/FLINK-20118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235406#comment-17235406 ] Andrey Zagrebin edited comment on FLINK-20118 at 11/20/20, 7:20 AM: True, I looked into how TM failures could be done with MiniCluster. I only found TestingMiniCluster::terminateTaskExecutor. If I try it, it currently fails JVM for me abruptly. I can continue looking into this if there are no better ideas or did you mean some other approach for TM / JM failures? On the other hand my own manual test is a bit broader (higher parallelism, checks output order, standalone cluster). Having it as an e2e test does not probably add much more than the existing IT case (if with TM / JM failures) but it would also help to discover e.g. FLINK-20196. Another idea (from offline discussion with [~trohrmann]) is to replace the artificial source generator in the existing FileSink e2e test with the FileSource. The e2e test for FileSink also tests failover. The ITCases are indeed generally faster but we already have an e2e test for FileSink with failover tests. Extending this e2e test should not explode test time too much. Testing two things at once is generally not good but maybe it is ok for Source/Sink pair and sparing test time. Individual aspects could be covered by unit tests. Then we could remove ITCases and e2e test still covers a bit more overall. Hence, this idea looks better to me atm. What do you think about the last idea? was (Author: azagrebin): True, I looked into how TM failures could be done with MiniCluster. I only found TestingMiniCluster::terminateTaskExecutor. If I try it, it currently fails JVM for me abruptly. I can continue looking into this if there are no better ideas or did you mean some other approach for TM / JM failures? On the other hand my own manual test is a bit broader (higher parallelism, checks output order, standalone cluster). Having it as an e2e test does not probably add much more than the existing IT case (if with TM / JM failures) but it would also help to discover e.g. FLINK-20196. Another idea (from offline discussion with [~trohrmann]) is to replace the artificial source generator in the existing FileSink e2e test with the FileSource. The ITCases are indeed generally faster but we already have an e2e test for FileSink with failover tests. Extending this e2e test should not explode test time too much. Testing two things at once is generally not good but maybe it is ok for Source/Sink pair and sparing test time. Individual aspects could be covered by unit tests. Then we could remove ITCases and e2e test still covers a bit more overall. Hence, this idea looks better to me atm. What do you think about the last idea? > Test New File Source API - Continuous Streaming Execution > - > > Key: FLINK-20118 > URL: https://issues.apache.org/jira/browse/FLINK-20118 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Common >Affects Versions: 1.12.0 >Reporter: Robert Metzger >Assignee: Andrey Zagrebin >Priority: Critical > Fix For: 1.12.0 > > > > [General Information about the Flink 1.12 release > testing|https://cwiki.apache.org/confluence/display/FLINK/1.12+Release+-+Community+Testing] > When testing a feature, consider the following aspects: > - Is the documentation easy to understand > - Are the error messages, log messages, APIs etc. easy to understand > - Is the feature working as expected under normal conditions > - Is the feature working / failing as expected with invalid input, induced > errors etc. > If you find a problem during testing, please file a ticket > (Priority=Critical; Fix Version = 1.12.0), and link it in this testing ticket. > During the testing keep us updated on tests conducted, or please write a > short summary of all things you have tested in the end. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] godfreyhe commented on pull request #14103: [FLINK-18545] Introduce `pipeline.name` to allow users to specify job name by configuration
godfreyhe commented on pull request #14103: URL: https://github.com/apache/flink/pull/14103#issuecomment-730956823 @kl0u, different from job id, job name is not the key info of a StreamGraph, and `pipeline.name` is just used for overwriting the default job name. Just as @wuchong mentioned, we can also change the value of `pipeline.name` before submitting a job. And it's a very useful feature which could let sql client specify job name. cc @dawidwys @twalthr This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #14139: [FLINK-20247][config] Hide unused unaligned checkpoint configuration options
AHeise commented on a change in pull request #14139:
URL: https://github.com/apache/flink/pull/14139#discussion_r527485088
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/ExecutionCheckpointingOptions.java
##
@@ -162,10 +162,11 @@
"will timeout and checkpoint barrier
will start working as unaligned checkpoint.")
.build());
+ @Documentation.ExcludeFromDocumentation("Do not advertise this option
until rescaling of unaligned checkpoint is completed.")
public static final ConfigOption FORCE_UNALIGNED =
ConfigOptions.key("execution.checkpointing.unaligned.forced")
.booleanType()
- .defaultValue(false)
+ .defaultValue(true)
Review comment:
Without this change UC wouldn't work with iterations. Now that there is
no `EndOfChannelStateEvent` and with it no blocking, we can also support
iterations for the time being.
Alternatively, we could keep the force option and discourage using UC in
iterations already in 1.12.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] elanv commented on pull request #8532: [FLINK-12541][REST] Support to submit Python Table API jobs via REST API
elanv commented on pull request #8532: URL: https://github.com/apache/flink/pull/8532#issuecomment-730956677 How is it going? I want to submit a python job inside application, and it would be great if REST API supports it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-20030) Barrier announcement causes outdated RemoteInputChannel#numBuffersOvertaken
[
https://issues.apache.org/jira/browse/FLINK-20030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arvid Heise resolved FLINK-20030.
-
Resolution: Fixed
> Barrier announcement causes outdated RemoteInputChannel#numBuffersOvertaken
> ---
>
> Key: FLINK-20030
> URL: https://issues.apache.org/jira/browse/FLINK-20030
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Runtime / Network
>Affects Versions: 1.12.0
>Reporter: Arvid Heise
>Assignee: Roman Khachatryan
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0
>
>
> {{numBuffersOvertaken}} is set when the announcement is enqueued, but it can
> take quite a while until the checkpoint is actually started with quite a
> non-priority buffers being drained.
> We should move away from {{numBuffersOvertaken}} and use sequence numbers.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20030) Barrier announcement causes outdated RemoteInputChannel#numBuffersOvertaken
[
https://issues.apache.org/jira/browse/FLINK-20030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235948#comment-17235948
]
Arvid Heise commented on FLINK-20030:
-
Merged into master as a97fe1f3f24993ff8cce2dd008281f162022fa27.
> Barrier announcement causes outdated RemoteInputChannel#numBuffersOvertaken
> ---
>
> Key: FLINK-20030
> URL: https://issues.apache.org/jira/browse/FLINK-20030
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Runtime / Network
>Affects Versions: 1.12.0
>Reporter: Arvid Heise
>Assignee: Roman Khachatryan
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0
>
>
> {{numBuffersOvertaken}} is set when the announcement is enqueued, but it can
> take quite a while until the checkpoint is actually started with quite a
> non-priority buffers being drained.
> We should move away from {{numBuffersOvertaken}} and use sequence numbers.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-20118) Test New File Source API - Continuous Streaming Execution
[ https://issues.apache.org/jira/browse/FLINK-20118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235406#comment-17235406 ] Andrey Zagrebin edited comment on FLINK-20118 at 11/20/20, 7:15 AM: True, I looked into how TM failures could be done with MiniCluster. I only found TestingMiniCluster::terminateTaskExecutor. If I try it, it currently fails JVM for me abruptly. I can continue looking into this if there are no better ideas or did you mean some other approach for TM / JM failures? On the other hand my own manual test is a bit broader (higher parallelism, checks output order, standalone cluster). Having it as an e2e test does not probably add much more than the existing IT case (if with TM / JM failures) but it would also help to discover e.g. FLINK-20196. Another idea (from offline discussion with [~trohrmann]) is to replace the artificial source generator in the existing FileSink e2e test with the FileSource. The ITCases are indeed generally faster but we already have an e2e test for FileSink with failover tests. Extending this e2e test should not explode test time too much. Testing two things at once is generally not good but maybe it is ok for Source/Sink pair and sparing test time. Individual aspects could be covered by unit tests. Then we could remove ITCases and e2e test still covers a bit more overall. Hence, this idea looks better to me atm. What do you think about the last idea? was (Author: azagrebin): True, I looked into how TM failures could be done with MiniCluster. I only found TestingMiniCluster::terminateTaskExecutor. If I try it, it currently fails JVM for me abruptly. I can continue looking into this if there are no better ideas. On the other hand my own manual test is a bit broader (higher parallelism, checks output order, standalone cluster). Having it as an e2e test does not probably add much more than the existing IT case (if with TM / JM failures) but it would also help to discover e.g. FLINK-20196. Another idea (from offline discussion with [~trohrmann]) is to replace the artificial source generator in the existing FileSink e2e test with the FileSource. The ITCases are indeed generally faster but we already have an e2e test for FileSink with failover tests. Extending this e2e test should not explode test time too much. Testing two things at once is generally not good but maybe it is ok for Source/Sink pair and sparing test time. Individual aspects could be covered by unit tests. Then we could remove ITCases and e2e test still covers a bit more overall. Hence, this idea looks better to me atm. > Test New File Source API - Continuous Streaming Execution > - > > Key: FLINK-20118 > URL: https://issues.apache.org/jira/browse/FLINK-20118 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Common >Affects Versions: 1.12.0 >Reporter: Robert Metzger >Assignee: Andrey Zagrebin >Priority: Critical > Fix For: 1.12.0 > > > > [General Information about the Flink 1.12 release > testing|https://cwiki.apache.org/confluence/display/FLINK/1.12+Release+-+Community+Testing] > When testing a feature, consider the following aspects: > - Is the documentation easy to understand > - Are the error messages, log messages, APIs etc. easy to understand > - Is the feature working as expected under normal conditions > - Is the feature working / failing as expected with invalid input, induced > errors etc. > If you find a problem during testing, please file a ticket > (Priority=Critical; Fix Version = 1.12.0), and link it in this testing ticket. > During the testing keep us updated on tests conducted, or please write a > short summary of all things you have tested in the end. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] AHeise merged pull request #14052: [FLINK-20030][network] Rewrite RemoteInputChannel#getInflightBuffersUnsafe to use sequence numbers
AHeise merged pull request #14052: URL: https://github.com/apache/flink/pull/14052 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-20255) Nested decorrelate failed
godfrey he created FLINK-20255:
--
Summary: Nested decorrelate failed
Key: FLINK-20255
URL: https://issues.apache.org/jira/browse/FLINK-20255
Project: Flink
Issue Type: Bug
Components: Table SQL / API
Affects Versions: 1.11.0, 1.12.0
Reporter: godfrey he
This issue is from ML
https://www.mail-archive.com/user@flink.apache.org/msg37746.html
We can reproduce the issue through the following code
{code:java}
@FunctionHint(output = new DataTypeHint("ROW"))
class SplitStringToRows extends TableFunction[Row] {
def eval(str: String, separator: String = ";"): Unit = {
if (str != null) {
str.split(separator).foreach(s => collect(Row.of(s.trim(
}
}
}
object Job {
def main(args: Array[String]): Unit = {
val settings =
EnvironmentSettings.newInstance().useBlinkPlanner().inStreamingMode().build()
val streamEnv = StreamExecutionEnvironment.getExecutionEnvironment
val streamTableEnv = StreamTableEnvironment.create(streamEnv, settings)
streamTableEnv.createTemporarySystemFunction(
"SplitStringToRows",
classOf[SplitStringToRows]
) // Class defined in previous email
streamTableEnv.executeSql(
"""
CREATE TABLE table2 (
attr1 STRING,
attr2 STRING,
attr3 DECIMAL,
attr4 DATE
) WITH (
'connector' = 'datagen'
)""")
val q2 = streamTableEnv.sqlQuery(
"""
SELECT
a.attr1 AS attr1,
attr2,
attr3,
attr4
FROM table2 p, LATERAL TABLE(SplitStringToRows(p.attr1, ';')) AS
a(attr1)
""")
streamTableEnv.createTemporaryView("view2", q2)
val q3 =
"""
SELECT
w.attr1,
p.attr3
FROM table2 w
LEFT JOIN LATERAL (
SELECT
attr1,
attr3
FROM (
SELECT
attr1,
attr3,
ROW_NUMBER() OVER (
PARTITION BY attr1
ORDER BY
attr4 DESC NULLS LAST,
w.attr2 = attr2 DESC NULLS LAST
) AS row_num
FROM view2)
WHERE row_num = 1) p
ON (w.attr1 = p.attr1)
"""
println(streamTableEnv.explainSql(q3))
}
}
{code}
The reason is {{RelDecorrelator}} in Calcite can't handle such nested
decorrelate pattern now
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] wsry commented on pull request #14149: [FLINK-20035][tests] Fix wrong config key in JobGraphRunningUtil
wsry commented on pull request #14149: URL: https://github.com/apache/flink/pull/14149#issuecomment-730947969 @rmetzger I used wrong config key in the fix of FLINK-20035, we should use RestOptions.BIND_PORT, but I used RestOptions.BIND_ADDRESS by mistake. Really sorry for that. I open this PR to fix that, could you please take a look? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14149: [FLINK-20035][tests] Fix wrong config key in JobGraphRunningUtil
flinkbot commented on pull request #14149: URL: https://github.com/apache/flink/pull/14149#issuecomment-730947533 ## CI report: * c090c7ec80bef6783cdd667b0d15d1f0b5fd9b57 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14148: [hotfix][javadocs] Fix typo in RichFunction
flinkbot edited a comment on pull request #14148: URL: https://github.com/apache/flink/pull/14148#issuecomment-730904115 ## CI report: * 714c860e2d34e0a6f034dc360b342e59bdde46e1 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9854) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14149: [FLINK-20035][tests] Fix wrong config key in JobGraphRunningUtil
flinkbot commented on pull request #14149: URL: https://github.com/apache/flink/pull/14149#issuecomment-730925448 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 02a9df9f5fcc1922cc906598e117b317655af4aa (Fri Nov 20 06:46:39 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wsry opened a new pull request #14149: [FLINK-20035][tests] Fix wrong config key in JobGraphRunningUtil
wsry opened a new pull request #14149: URL: https://github.com/apache/flink/pull/14149 ## What is the purpose of the change Fix wrong config key in JobGraphRunningUtil, we should use RestOptions.BIND_PORT instead of RestOptions.BIND_ADDRESS. ## Brief change log - Fix wrong config key in JobGraphRunningUtil, use RestOptions.BIND_PORT instead of RestOptions.BIND_ADDRESS. ## Verifying this change This change is already covered by existing tests. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14148: [hotfix][javadocs] Fix typo in RichFunction
flinkbot commented on pull request #14148: URL: https://github.com/apache/flink/pull/14148#issuecomment-730904115 ## CI report: * 714c860e2d34e0a6f034dc360b342e59bdde46e1 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14129: [hotfix] Reduce logging verbosity from the Checkpoint-related REST handlers
flinkbot edited a comment on pull request #14129: URL: https://github.com/apache/flink/pull/14129#issuecomment-730245586 ## CI report: * 8773590fc0706ddd93ee910084e6151b3a581c44 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9853) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-17470) Flink task executor process permanently hangs on `flink-daemon.sh stop`, deletes PID file
[
https://issues.apache.org/jira/browse/FLINK-17470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-17470:
---
Priority: Blocker (was: Critical)
> Flink task executor process permanently hangs on `flink-daemon.sh stop`,
> deletes PID file
> -
>
> Key: FLINK-17470
> URL: https://issues.apache.org/jira/browse/FLINK-17470
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.10.0
> Environment:
> {code:java}
> $ uname -a
> Linux hostname.local 3.10.0-1062.9.1.el7.x86_64 #1 SMP Fri Dec 6 15:49:49 UTC
> 2019 x86_64 x86_64 x86_64 GNU/Linux
> $ lsb_release -a
> LSB Version: :core-4.1-amd64:core-4.1-noarch
> Distributor ID: CentOS
> Description: CentOS Linux release 7.7.1908 (Core)
> Release: 7.7.1908
> Codename: Core
> {code}
> Flink version 1.10
>
>Reporter: Hunter Herman
>Assignee: Robert Metzger
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.12.0
>
> Attachments: flink_jstack.log, flink_mixed_jstack.log
>
>
> Hi Flink team!
> We've attempted to upgrade our flink 1.9 cluster to 1.10, but are
> experiencing reproducible instability on shutdown. Speciically, it appears
> that the `kill` issued in the `stop` case of flink-daemon.sh is causing the
> task executor process to hang permanently. Specifically, the process seems to
> be hanging in the
> `org.apache.flink.runtime.util.JvmShutdownSafeguard$DelayedTerminator.run` in
> a `Thread.sleep()` call. I think this is a bizarre behavior. Also note that
> every thread in the process is BLOCKED. on a `pthread_cond_wait` call. Is
> this an OS level issue? Banging my head on a wall here. See attached stack
> traces for details.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-20254) HiveTableSourceITCase.testStreamPartitionReadByCreateTime times out
Robert Metzger created FLINK-20254:
--
Summary: HiveTableSourceITCase.testStreamPartitionReadByCreateTime
times out
Key: FLINK-20254
URL: https://issues.apache.org/jira/browse/FLINK-20254
Project: Flink
Issue Type: Bug
Components: Connectors / Hive
Affects Versions: 1.12.0
Reporter: Robert Metzger
Fix For: 1.12.0
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=9808=logs=fc5181b0-e452-5c8f-68de-1097947f6483=62110053-334f-5295-a0ab-80dd7e2babbf
{code}
2020-11-19T10:34:23.5591765Z [ERROR] Tests run: 18, Failures: 0, Errors: 1,
Skipped: 0, Time elapsed: 192.243 s <<< FAILURE! - in
org.apache.flink.connectors.hive.HiveTableSourceITCase
2020-11-19T10:34:23.5593193Z [ERROR]
testStreamPartitionReadByCreateTime(org.apache.flink.connectors.hive.HiveTableSourceITCase)
Time elapsed: 120.075 s <<< ERROR!
2020-11-19T10:34:23.5593929Z org.junit.runners.model.TestTimedOutException:
test timed out after 12 milliseconds
2020-11-19T10:34:23.5594321Zat java.lang.Thread.sleep(Native Method)
2020-11-19T10:34:23.5594777Zat
org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.sleepBeforeRetry(CollectResultFetcher.java:231)
2020-11-19T10:34:23.5595378Zat
org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.next(CollectResultFetcher.java:119)
2020-11-19T10:34:23.5596001Zat
org.apache.flink.streaming.api.operators.collect.CollectResultIterator.nextResultFromFetcher(CollectResultIterator.java:103)
2020-11-19T10:34:23.5596610Zat
org.apache.flink.streaming.api.operators.collect.CollectResultIterator.hasNext(CollectResultIterator.java:77)
2020-11-19T10:34:23.5597218Zat
org.apache.flink.table.planner.sinks.SelectTableSinkBase$RowIteratorWrapper.hasNext(SelectTableSinkBase.java:115)
2020-11-19T10:34:23.5597811Zat
org.apache.flink.table.api.internal.TableResultImpl$CloseableRowIteratorWrapper.hasNext(TableResultImpl.java:355)
2020-11-19T10:34:23.5598555Zat
org.apache.flink.connectors.hive.HiveTableSourceITCase.fetchRows(HiveTableSourceITCase.java:653)
2020-11-19T10:34:23.5599407Zat
org.apache.flink.connectors.hive.HiveTableSourceITCase.testStreamPartitionReadByCreateTime(HiveTableSourceITCase.java:594)
2020-11-19T10:34:23.5599982Zat
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2020-11-19T10:34:23.5600393Zat
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2020-11-19T10:34:23.5600865Zat
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2020-11-19T10:34:23.5601300Zat
java.lang.reflect.Method.invoke(Method.java:498)
2020-11-19T10:34:23.5601713Zat
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
2020-11-19T10:34:23.5602211Zat
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
2020-11-19T10:34:23.5602688Zat
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
2020-11-19T10:34:23.5603181Zat
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
2020-11-19T10:34:23.5603753Zat
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
2020-11-19T10:34:23.5604308Zat
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
2020-11-19T10:34:23.5604780Zat
java.util.concurrent.FutureTask.run(FutureTask.java:266)
2020-11-19T10:34:23.5605114Zat java.lang.Thread.run(Thread.java:748)
2020-11-19T10:34:23.5605299Z
2020-11-19T10:34:24.4180149Z [INFO] Running
org.apache.flink.connectors.hive.TableEnvHiveConnectorITCase
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #14148: [hotfix][javadocs] Fix typo in RichFunction
flinkbot commented on pull request #14148: URL: https://github.com/apache/flink/pull/14148#issuecomment-730879019 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 714c860e2d34e0a6f034dc360b342e59bdde46e1 (Fri Nov 20 06:19:46 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14129: [hotfix] Reduce logging verbosity from the Checkpoint-related REST handlers
flinkbot edited a comment on pull request #14129: URL: https://github.com/apache/flink/pull/14129#issuecomment-730245586 ## CI report: * 89144ae4ffb688a9ca039b311c0f2a9434a71be6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9809) * 8773590fc0706ddd93ee910084e6151b3a581c44 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] dongxuwang opened a new pull request #14148: [hotfix][javadocs] Fix typo in RichFunction
dongxuwang opened a new pull request #14148: URL: https://github.com/apache/flink/pull/14148 Just remove the duplicated two words. `of the` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #14129: [hotfix] Reduce logging verbosity from the Checkpoint-related REST handlers
rmetzger commented on pull request #14129: URL: https://github.com/apache/flink/pull/14129#issuecomment-730869724 Thanks a lot for your review. I addressed your comment, once CI is green, I'll merge it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-statefun] klion26 commented on pull request #131: [FLINK-18968] Translate README.md to Chinese
klion26 commented on pull request #131: URL: https://github.com/apache/flink-statefun/pull/131#issuecomment-730868665 @wangzzu is there any update? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-web] klion26 closed pull request #395: [FLINK-17491] Translate Training page on project website
klion26 closed pull request #395: URL: https://github.com/apache/flink-web/pull/395 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-20110) Support 'merge' method for first_value and last_value UDAF
[ https://issues.apache.org/jira/browse/FLINK-20110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu reassigned FLINK-20110: --- Assignee: hailong wang > Support 'merge' method for first_value and last_value UDAF > -- > > Key: FLINK-20110 > URL: https://issues.apache.org/jira/browse/FLINK-20110 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Planner, Table SQL / Runtime >Affects Versions: 1.12.0 >Reporter: hailong wang >Assignee: hailong wang >Priority: Major > Fix For: 1.13.0 > > > From the user-zh email, when use first_value function in hop window, It > throws the exception because first_vaue does not implement the merge method. > We can support 'merge' method for first_value and last_value UDAF -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14147: [FLINK-20059][table-common] Update Javadoc for "merge" method of AggregateFunction
flinkbot edited a comment on pull request #14147: URL: https://github.com/apache/flink/pull/14147#issuecomment-730851946 ## CI report: * 68525d793b79e971b6d53ac654d738f16ab11d04 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9852) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14147: [FLINK-20059][table-common] Update Javadoc for "merge" method of AggregateFunction
flinkbot commented on pull request #14147: URL: https://github.com/apache/flink/pull/14147#issuecomment-730851946 ## CI report: * 68525d793b79e971b6d53ac654d738f16ab11d04 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14147: [FLINK-20059][table-common] Update Javadoc for "merge" method of AggregateFunction
flinkbot commented on pull request #14147: URL: https://github.com/apache/flink/pull/14147#issuecomment-730847049 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 68525d793b79e971b6d53ac654d738f16ab11d04 (Fri Nov 20 04:43:45 UTC 2020) ✅no warnings Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-20059) Outdated SQL docs on aggregate functions' merge
[
https://issues.apache.org/jira/browse/FLINK-20059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-20059:
---
Labels: pull-request-available (was: )
> Outdated SQL docs on aggregate functions' merge
> ---
>
> Key: FLINK-20059
> URL: https://issues.apache.org/jira/browse/FLINK-20059
> Project: Flink
> Issue Type: Bug
> Components: Documentation, Table SQL / API
>Affects Versions: 1.12.0, 1.11.2
>Reporter: Nico Kruber
>Assignee: Jark Wu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0
>
>
> In the java docs as well as the user docs, the {{merge}} method of an
> aggregation UDF is described as optional, e.g.
> {quote}Merges a group of accumulator instances into one accumulator instance.
> This function must be implemented for data stream session window grouping
> aggregates and data set grouping aggregates.{quote}
> However, it seems that nowadays this method is required in more cases (I
> stumbled on this for a HOP window in streaming):
> {code}
> StreamExecGlobalGroupAggregate.scala
> .needMerge(mergedAccOffset, mergedAccOnHeap, mergedAccExternalTypes)
> StreamExecGroupWindowAggregateBase.scala
> generator.needMerge(mergedAccOffset = 0, mergedAccOnHeap = false)
> StreamExecIncrementalGroupAggregate.scala
> .needMerge(mergedAccOffset, mergedAccOnHeap = true,
> mergedAccExternalTypes)
> StreamExecLocalGroupAggregate.scala
> .needMerge(mergedAccOffset = 0, mergedAccOnHeap = true)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] wuchong opened a new pull request #14147: [FLINK-20059][table-common] Update Javadoc for "merge" method of AggregateFunction
wuchong opened a new pull request #14147: URL: https://github.com/apache/flink/pull/14147 ## What is the purpose of the change In the java docs as well as the user docs, the merge method of an aggregation UDF is described as optional, e.g. ``` Merges a group of accumulator instances into one accumulator instance. This function must be implemented for data stream session window grouping aggregates and data set grouping aggregates. ``` However, it seems that nowadays this method is required in more cases (e.g. HOP window in streaming). ## Brief change log - Update the javadoc of "merge" method. - Update documentations. ## Verifying this change This change is a trivial rework without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-20110) Support 'merge' method for first_value and last_value UDAF
[ https://issues.apache.org/jira/browse/FLINK-20110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hailong wang updated FLINK-20110: - Fix Version/s: 1.13.0 > Support 'merge' method for first_value and last_value UDAF > -- > > Key: FLINK-20110 > URL: https://issues.apache.org/jira/browse/FLINK-20110 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Planner, Table SQL / Runtime >Affects Versions: 1.12.0 >Reporter: hailong wang >Priority: Major > Fix For: 1.13.0 > > > From the user-zh email, when use first_value function in hop window, It > throws the exception because first_vaue does not implement the merge method. > We can support 'merge' method for first_value and last_value UDAF -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20110) Support 'merge' method for first_value and last_value UDAF
[ https://issues.apache.org/jira/browse/FLINK-20110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235895#comment-17235895 ] hailong wang commented on FLINK-20110: -- [~jark] Yes, I can take it, thanks for assigning this to me. > Support 'merge' method for first_value and last_value UDAF > -- > > Key: FLINK-20110 > URL: https://issues.apache.org/jira/browse/FLINK-20110 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Planner, Table SQL / Runtime >Affects Versions: 1.12.0 >Reporter: hailong wang >Priority: Major > > From the user-zh email, when use first_value function in hop window, It > throws the exception because first_vaue does not implement the merge method. > We can support 'merge' method for first_value and last_value UDAF -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20110) Support 'merge' method for first_value and last_value UDAF
[ https://issues.apache.org/jira/browse/FLINK-20110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-20110: Component/s: Table SQL / Runtime > Support 'merge' method for first_value and last_value UDAF > -- > > Key: FLINK-20110 > URL: https://issues.apache.org/jira/browse/FLINK-20110 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Planner, Table SQL / Runtime >Affects Versions: 1.12.0 >Reporter: hailong wang >Priority: Major > > From the user-zh email, when use first_value function in hop window, It > throws the exception because first_vaue does not implement the merge method. > We can support 'merge' method for first_value and last_value UDAF -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20110) Support 'merge' method for first_value and last_value UDAF
[ https://issues.apache.org/jira/browse/FLINK-20110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235892#comment-17235892 ] Jark Wu commented on FLINK-20110: - Hi [~hailong wang], do you want to try this? > Support 'merge' method for first_value and last_value UDAF > -- > > Key: FLINK-20110 > URL: https://issues.apache.org/jira/browse/FLINK-20110 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Planner >Affects Versions: 1.12.0 >Reporter: hailong wang >Priority: Major > > From the user-zh email, when use first_value function in hop window, It > throws the exception because first_vaue does not implement the merge method. > We can support 'merge' method for first_value and last_value UDAF -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-20243) Remove useless words in documents
[ https://issues.apache.org/jira/browse/FLINK-20243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu closed FLINK-20243. --- Fix Version/s: 1.11.3 1.12.0 Assignee: xiaozilong Resolution: Fixed Fixed in - master (1.12.0): 5c7ed8e1c87b985ea80bb47ce8de36c234b13828 - 1.11.3: 13940b606fcf5aa4f26d172aaa86d82054ecd301 > Remove useless words in documents > - > > Key: FLINK-20243 > URL: https://issues.apache.org/jira/browse/FLINK-20243 > Project: Flink > Issue Type: Bug > Components: Documentation >Affects Versions: 1.11.0 >Reporter: xiaozilong >Assignee: xiaozilong >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0, 1.11.3 > > Attachments: image-2020-11-19-20-05-05-752.png > > > !image-2020-11-19-20-05-05-752.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] wuchong merged pull request #14138: [FLINK-20243][docs]Remove useless words in documents
wuchong merged pull request #14138: URL: https://github.com/apache/flink/pull/14138 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-20204) Translate page 'upsert-kafka Connector' into Chinese
[ https://issues.apache.org/jira/browse/FLINK-20204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu closed FLINK-20204. --- Resolution: Fixed Fixed in master (1.12.0): 5b9625ef84c198b84b2116e10ef2e2915c560fd6 > Translate page 'upsert-kafka Connector' into Chinese > > > Key: FLINK-20204 > URL: https://issues.apache.org/jira/browse/FLINK-20204 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Connectors / Kafka, Table SQL / > Ecosystem >Reporter: CaoZhen >Assignee: CaoZhen >Priority: Minor > Labels: pull-request-available > Fix For: 1.12.0 > > > The doc located in "flink/docs/dev/table/connectors/upsert-kafka.zh.md" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] wuchong closed pull request #14126: [FLINK-20204][docs-zh] Translate page 'upsert-kafka Connector' into Chinese
wuchong closed pull request #14126: URL: https://github.com/apache/flink/pull/14126 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] V1ncentzzZ commented on a change in pull request #14138: [FLINK-20243][docs]Remove useless words in documents
V1ncentzzZ commented on a change in pull request #14138: URL: https://github.com/apache/flink/pull/14138#discussion_r527381858 ## File path: docs/ops/cli.zh.md ## @@ -36,7 +36,7 @@ The command line can be used to - cancel a running job, - provide information about a job, - list running and waiting jobs, -- trigger and dispose savepoints, and +- trigger and dispose savepoints, Review comment: Done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-20253) `JsonRowDeserializationSchema` can not deserialize non utf-8 json string
[
https://issues.apache.org/jira/browse/FLINK-20253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zouyunhe updated FLINK-20253:
-
Description:
We run a flink job and the input data is json format, and found the exception
will throw while non utf8 string flows in, the exception as below
{code:java}
2020-06-15 20:55:48,777 WARN
org.apache.flink.formats.json.JsonRowDeserializationSchema- Failed to
deserialize JSON object: {"rip":"127.0.0.1","rtime":"2020-06-15
16:43:46","rchannel":"http_server","be_time
":1592210626,"be_ip":"127.0.0.1","uid":0,"activity_id":"{\"page\":\"share-video\",\"event\":\"testBClickVideo\",\"click_post_id\":\"6817589584893303314\",\"owner\":\"others\",\"currentUrl\":\"https:\/\/l.c.cc\/v\/IWB\",\"af_b1\":\"20459_ref\",\"af_sub2\":\"1_2971\",\"msg_text\":\"1\",\"share_text1\":\"100
cel✌❤\",\"share_text2\":\"5
teenal\",\"share_test\":2,\"e\":\"detail\",\"videoType\":\"playing\",\"req_id\":\"4C3BCE7A-AEE4-11EA-83BC-8397E4C235C9\",\"networkType\":\"3g\"}","country_code":"ID","os":"Mozilla\/5.0
(Li
nux; Android 7.0; L52F) AppleWebKit\/537.36 (KHTML, like Gecko)
Chrome\/79.0.3945.93 Mobile
Safari\/537.36","recv_time":1592210626,"remark":"","client_ip":"127.0.0.1"}
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonParseException:
Invalid UTF-8 start byte 0xbe
at [Source: (byte[])"{"rip":"127.0.0.1","rtime":"2020-06-15
16:43:46","rchannel":"http_server","be_time":1592210626,"be_ip":"127.0.0.1","uid":0,"activity_id":"{\"page\":\"share-video\",\"event\":\"testBClic
kVideo\",\"click_post_id\":\"6817589584893303314\",\"owner\":\"others\",\"currentUrl\":\"https:\/\/l.c.cc\/v\/IWB\",\"af_sub1\":\"20459_ref\",\"af_sub2\":\"1_2971\",\"msg_text\":\"1\",\"share_text1\":\
"100 cel✌❤\",\"share_text2\":\"5 teenal"[truncated 397 bytes]; line: 1, column:
406]
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1840)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:712)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidInitial(UTF8StreamJsonParser.java:3569)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidChar(UTF8StreamJsonParser.java:3565)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishString2(UTF8StreamJsonParser.java:2511)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishAndReturnString(UTF8StreamJsonParser.java:2437)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.json.UTF8StreamJsonParser.getText(UTF8StreamJsonParser.java:293)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:267)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:68)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:15)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper._readTreeAndClose(ObjectMapper.java:4254)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:2725)
at
org.apache.flink.formats.json.JsonRowDeserializationSchema.deserialize(JsonRowDeserializationSchema.java:166)
{code}
was:
We run a flink job and the input data is json format, and found the exception
will throw while non utf8 string flows in, the exception as below
{code:java}
2020-06-15 20:55:48,777 WARN
org.apache.flink.formats.json.JsonRowDeserializationSchema- Failed to
deserialize JSON object: {"rip":"127.0.0.1","rtime":"2020-06-15
16:43:46","rchannel":"http_server","be_time
":1592210626,"be_ip":"127.0.0.1","uid":0,"activity_id":"{\"page\":\"share-video\",\"event\":\"testBClickVideo\",\"click_post_id\":\"6817589584893303314\",\"owner\":\"others\",\"currentUrl\":\"https:\/\/l.lik
ee.video\/v\/IysQWB\",\"af_sub1\":\"20459_ref\",\"af_sub2\":\"1_2971\",\"msg_text\":\"1\",\"share_text1\":\"100
ner yg gak cek Nyesel✌❤\",\"share_text2\":\"Ayok Guys Ikuti syarat nya dan tag
5 temen kalian semog
a kalian Jadi Likers
terkenal\",\"share_test\":2,\"e\":\"detail\",\"videoType\":\"playing\",\"req_id\":\"4C3BCE7A-AEE4-11EA-83BC-8397E4C235C9\",\"networkType\":\"3g\"}","country_code":"ID","os":"Mozilla\/5.0
(Li
nux; Android 7.0; L52F) AppleWebKit\/537.36 (KHTML, like Gecko)
Chrome\/79.0.3945.93 Mobile
Safari\/537.36","recv_time":1592210626,"remark":"","client_ip":"127.0.0.1"}
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonParseException:
Invalid
[jira] [Comment Edited] (FLINK-20252) Manual Test for Approximate Local Recovery
[
https://issues.apache.org/jira/browse/FLINK-20252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235884#comment-17235884
]
Yuan Mei edited comment on FLINK-20252 at 11/20/20, 4:04 AM:
-
sink output:
{code:java}
index:146; key:0; selectedChannel:0; longOrShortString:I am a very long string
to test partial records hohoho hahaha
index:147; key:0; selectedChannel:0; longOrShortString:I am a very long string
to test partial records hohoho hahaha
index:148; key:0; selectedChannel:0; longOrShortString:I am a very long string
to test partial records hohoho hahaha
index:2658; key:0; selectedChannel:0; longOrShortString:I am a very long string
to test partial records hohoho hahaha
index:2659; key:0; selectedChannel:0; longOrShortString:I am a very long string
to test partial records hohoho hahaha
index:2660; key:0; selectedChannel:0; longOrShortString:I am a very long string
to test partial records hohoho hahaha{code}
was (Author: ym):
sink output:
{code:java}
index:146; key:0; selectedChannel:0; longOrShortString:I am a very long string
to test partial records hohoho hahaha index:147; key:0; selectedChannel:0;
longOrShortString:I am a very long string to test partial records hohoho hahaha
index:148; key:0; selectedChannel:0; longOrShortString:I am a very long string
to test partial records hohoho hahaha index:2658; key:0; selectedChannel:0;
longOrShortString:I am a very long string to test partial records hohoho hahaha
index:2659; key:0; selectedChannel:0; longOrShortString:I am a very long string
to test partial records hohoho hahaha index:2660; key:0; selectedChannel:0;
longOrShortString:I am a very long string to test partial records hohoho
hahaha{code}
> Manual Test for Approximate Local Recovery
> --
>
> Key: FLINK-20252
> URL: https://issues.apache.org/jira/browse/FLINK-20252
> Project: Flink
> Issue Type: Sub-task
>Reporter: Yuan Mei
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.12.0
>
>
> This manual test is to verify approximate failover works as expected in
> clusters.
>
> *Test Job Set up:*
> source -> mapper -> printer
> mapper fails after a certain number of records are received.
> link: [https://github.com/apache/flink/pull/14146]
> *Cluster Set up:*
> Test on Yarn clustersm
> add this to flink-conf.yaml
> jobmanager.scheduler.scheduling-strategy: legacy
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20252) Manual Test for Approximate Local Recovery
[
https://issues.apache.org/jira/browse/FLINK-20252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235884#comment-17235884
]
Yuan Mei commented on FLINK-20252:
--
sink output:
{code:java}
index:146; key:0; selectedChannel:0; longOrShortString:I am a very long string
to test partial records hohoho hahaha index:147; key:0; selectedChannel:0;
longOrShortString:I am a very long string to test partial records hohoho hahaha
index:148; key:0; selectedChannel:0; longOrShortString:I am a very long string
to test partial records hohoho hahaha index:2658; key:0; selectedChannel:0;
longOrShortString:I am a very long string to test partial records hohoho hahaha
index:2659; key:0; selectedChannel:0; longOrShortString:I am a very long string
to test partial records hohoho hahaha index:2660; key:0; selectedChannel:0;
longOrShortString:I am a very long string to test partial records hohoho
hahaha{code}
> Manual Test for Approximate Local Recovery
> --
>
> Key: FLINK-20252
> URL: https://issues.apache.org/jira/browse/FLINK-20252
> Project: Flink
> Issue Type: Sub-task
>Reporter: Yuan Mei
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.12.0
>
>
> This manual test is to verify approximate failover works as expected in
> clusters.
>
> *Test Job Set up:*
> source -> mapper -> printer
> mapper fails after a certain number of records are received.
> link: [https://github.com/apache/flink/pull/14146]
> *Cluster Set up:*
> Test on Yarn clustersm
> add this to flink-conf.yaml
> jobmanager.scheduler.scheduling-strategy: legacy
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] wuchong commented on a change in pull request #14138: [FLINK-20243][docs]Remove useless words in documents
wuchong commented on a change in pull request #14138: URL: https://github.com/apache/flink/pull/14138#discussion_r527379788 ## File path: docs/ops/cli.zh.md ## @@ -36,7 +36,7 @@ The command line can be used to - cancel a running job, - provide information about a job, - list running and waiting jobs, -- trigger and dispose savepoints, and +- trigger and dispose savepoints, Review comment: Please remove the ending comma too. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-19345) In Table File Sink, introduce streaming sink compaction
[ https://issues.apache.org/jira/browse/FLINK-19345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235882#comment-17235882 ] Jingsong Lee commented on FLINK-19345: -- Thank you for your in-depth exploration, yes, for big files: * HDFS: Just rename file. * S3 (Object Store): Copy bytes to new file from temp file. > In Table File Sink, introduce streaming sink compaction > --- > > Key: FLINK-19345 > URL: https://issues.apache.org/jira/browse/FLINK-19345 > Project: Flink > Issue Type: New Feature > Components: Table SQL / Runtime >Reporter: Jingsong Lee >Assignee: Jingsong Lee >Priority: Major > Fix For: 1.12.0 > > > Users often complain that many small files are written out. Small files will > affect the performance of file reading and the DFS system, and even the > stability of the DFS system. > Target: > * Compact all files generated by this job in a single checkpoint. > * With compaction, Users can have smaller checkpoint interval, even to > seconds. > Document: > https://docs.google.com/document/d/1cdlyoqgBq9yJEiHFBziimIoKHapQiEY2-0Tn8IF6G-c/edit?usp=sharing -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-19345) In Table File Sink, introduce streaming sink compaction
[ https://issues.apache.org/jira/browse/FLINK-19345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235882#comment-17235882 ] Jingsong Lee edited comment on FLINK-19345 at 11/20/20, 4:00 AM: - Thank you for your in-depth exploration, yes, for big files: * HDFS: Just rename file. * S3 (Object Store): Copy bytes to new file from temp file. was (Author: lzljs3620320): Thank you for your in-depth exploration, yes, for big files: * HDFS: Just rename file. * S3 (Object Store): Copy bytes to new file from temp file. > In Table File Sink, introduce streaming sink compaction > --- > > Key: FLINK-19345 > URL: https://issues.apache.org/jira/browse/FLINK-19345 > Project: Flink > Issue Type: New Feature > Components: Table SQL / Runtime >Reporter: Jingsong Lee >Assignee: Jingsong Lee >Priority: Major > Fix For: 1.12.0 > > > Users often complain that many small files are written out. Small files will > affect the performance of file reading and the DFS system, and even the > stability of the DFS system. > Target: > * Compact all files generated by this job in a single checkpoint. > * With compaction, Users can have smaller checkpoint interval, even to > seconds. > Document: > https://docs.google.com/document/d/1cdlyoqgBq9yJEiHFBziimIoKHapQiEY2-0Tn8IF6G-c/edit?usp=sharing -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20253) `JsonRowDeserializationSchema` can not deserialize non utf-8 json string
[
https://issues.apache.org/jira/browse/FLINK-20253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zouyunhe updated FLINK-20253:
-
Description:
We run a flink job and the input data is json format, and found the exception
will throw while non utf8 string flows in, the exception as below
{code:java}
2020-06-15 20:55:48,777 WARN
org.apache.flink.formats.json.JsonRowDeserializationSchema- Failed to
deserialize JSON object: {"rip":"127.0.0.1","rtime":"2020-06-15
16:43:46","rchannel":"http_server","be_time
":1592210626,"be_ip":"127.0.0.1","uid":0,"activity_id":"{\"page\":\"share-video\",\"event\":\"testBClickVideo\",\"click_post_id\":\"6817589584893303314\",\"owner\":\"others\",\"currentUrl\":\"https:\/\/l.lik
ee.video\/v\/IysQWB\",\"af_sub1\":\"20459_ref\",\"af_sub2\":\"1_2971\",\"msg_text\":\"1\",\"share_text1\":\"100
ner yg gak cek Nyesel✌❤\",\"share_text2\":\"Ayok Guys Ikuti syarat nya dan tag
5 temen kalian semog
a kalian Jadi Likers
terkenal\",\"share_test\":2,\"e\":\"detail\",\"videoType\":\"playing\",\"req_id\":\"4C3BCE7A-AEE4-11EA-83BC-8397E4C235C9\",\"networkType\":\"3g\"}","country_code":"ID","os":"Mozilla\/5.0
(Li
nux; Android 7.0; L52F) AppleWebKit\/537.36 (KHTML, like Gecko)
Chrome\/79.0.3945.93 Mobile
Safari\/537.36","recv_time":1592210626,"remark":"","client_ip":"127.0.0.1"}
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonParseException:
Invalid UTF-8 start byte 0xbe
at [Source: (byte[])"{"rip":"127.0.0.1","rtime":"2020-06-15
16:43:46","rchannel":"http_server","be_time":1592210626,"be_ip":"127.0.0.1","uid":0,"activity_id":"{\"page\":\"share-video\",\"event\":\"testBClic
kVideo\",\"click_post_id\":\"6817589584893303314\",\"owner\":\"others\",\"currentUrl\":\"https:\/\/l.likee.video\/v\/IysQWB\",\"af_sub1\":\"20459_ref\",\"af_sub2\":\"1_2971\",\"msg_text\":\"1\",\"share_text1\":\
"100 ner yg gak cek Nyesel✌❤\",\"share_text2\":\"Ayok Guys Ikuti syarat nya dan
tag 5 teme"[truncated 397 bytes]; line: 1, column: 406]
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1840)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:712)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidInitial(UTF8StreamJsonParser.java:3569)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidChar(UTF8StreamJsonParser.java:3565)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishString2(UTF8StreamJsonParser.java:2511)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishAndReturnString(UTF8StreamJsonParser.java:2437)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.json.UTF8StreamJsonParser.getText(UTF8StreamJsonParser.java:293)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:267)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:68)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:15)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper._readTreeAndClose(ObjectMapper.java:4254)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:2725)
at
org.apache.flink.formats.json.JsonRowDeserializationSchema.deserialize(JsonRowDeserializationSchema.java:166)
{code}
was:
We run a flink job and the input data is json format, and found the exception
will throw while non utf8 string flows in, the exception as below
{code:java}
2020-06-15 20:55:48,777 WARN
org.apache.flink.formats.json.JsonRowDeserializationSchema- Failed to
deserialize JSON object: {"rip":"127.0.0.1","rtime":"2020-06-15
16:43:46","rchannel":"http_server","be_time
":1592210626,"be_ip":"146.196.79.92","uid":0,"activity_id":"{\"page\":\"share-video\",\"event\":\"testBClickVideo\",\"click_post_id\":\"6817589584893303314\",\"owner\":\"others\",\"currentUrl\":\"https:\/\/l.lik
ee.video\/v\/IysQWB\",\"af_sub1\":\"20459_ref\",\"af_sub2\":\"1_2971\",\"msg_text\":\"1\",\"share_text1\":\"100
ner yg gak cek Nyesel✌❤\",\"share_text2\":\"Ayok Guys Ikuti syarat nya dan tag
5 temen kalian semog
a kalian Jadi Likers
terkenal\",\"share_test\":2,\"e\":\"detail\",\"videoType\":\"playing\",\"req_id\":\"4C3BCE7A-AEE4-11EA-83BC-8397E4C235C9\",\"networkType\":\"3g\"}","country_code":"ID","os":"Mozilla\/5.0
(Li
nux; Android 7.0; L52F) AppleWebKit\/537.36 (KHTML, like Gecko)
Chrome\/79.0.3945.93
[jira] [Comment Edited] (FLINK-20252) Manual Test for Approximate Local Recovery
[
https://issues.apache.org/jira/browse/FLINK-20252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235878#comment-17235878
]
Yuan Mei edited comment on FLINK-20252 at 11/20/20, 3:59 AM:
-
JM Log:
Failed mapper and sink are restarted.
{code:java}
2020-11-20 11:05:40,914 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: Custom
Source (1/1) (a431944bc4634099f59062c0e929fa9e) switched from SCHEDULED to
DEPLOYING. 2020-11-20 11:05:40,914 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source:
Custom Source (1/1) (attempt #0) with attempt id
a431944bc4634099f59062c0e929fa9e to container_e17_1602580065114_0666_01_02
@ i22a12256.sqa.eu95.tbsite.net (dataPort=57573) with allocation id
16cd53e0dba4d01daabfc95e2d794a38 2020-11-20 11:05:40,918 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Map (1/1)
(c33ea0a9306f7559f1c714f0527b30e4) switched from SCHEDULED to DEPLOYING.
2020-11-20 11:05:40,918 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Map (1/1)
(attempt #0) with attempt id c33ea0a9306f7559f1c714f0527b30e4 to
container_e17_1602580065114_0666_01_03 @ i22a09265.sqa.eu95.tbsite.net
(dataPort=33785) with allocation id c1d9691579e29b35c2ed54d9721a29f1 2020-11-20
11:05:40,920 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] -
Sink: Print to Std. Out (1/1) (1f3db4f707fd453ceef6aa72c2003f43) switched from
SCHEDULED to DEPLOYING. 2020-11-20 11:05:40,920 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Sink:
Print to Std. Out (1/1) (attempt #0) with attempt id
1f3db4f707fd453ceef6aa72c2003f43 to container_e17_1602580065114_0666_01_03
@ i22a09265.sqa.eu95.tbsite.net (dataPort=33785) with allocation id
c1d9691579e29b35c2ed54d9721a29f1 2020-11-20 11:05:41,023 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: Custom
Source (1/1) (a431944bc4634099f59062c0e929fa9e) switched from DEPLOYING to
RUNNING. 2020-11-20 11:05:41,093 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Map (1/1)
(c33ea0a9306f7559f1c714f0527b30e4) switched from DEPLOYING to RUNNING.
2020-11-20 11:05:41,093 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Sink: Print to Std.
Out (1/1) (1f3db4f707fd453ceef6aa72c2003f43) switched from DEPLOYING to
RUNNING. 2020-11-20 11:05:42,900 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Map (1/1)
(c33ea0a9306f7559f1c714f0527b30e4) switched from RUNNING to FAILED on
org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot@5f54bf85.
java.lang.Exception: Artificial Test Failure at
org.apache.flink.streaming.examples.wordcount.ApproximateFailover$FailingMapper.map(ApproximateFailover.java:139)
~[ApproximateFailover.jar:1.12-SNAPSHOT] at
org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at
org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:193)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at
org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.processElement(StreamTaskNetworkInput.java:179)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at
org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.emitNext(StreamTaskNetworkInput.java:152)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at
org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:67)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at
org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:372)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:186)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:574)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:538)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at
org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:722)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at
org.apache.flink.runtime.taskmanager.Task.run(Task.java:547)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at
java.lang.Thread.run(Thread.java:834) ~[?:1.8.0_102] 2020-11-20 11:05:42,920
INFO
org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy
[] - Calculating tasks to restart to recover the failed task
0a448493b4782967b150582570326227_0. 2020-11-20 11:05:42,920 INFO
org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy
[] - 2 tasks should be restarted to
[GitHub] [flink] flinkbot edited a comment on pull request #14126: [FLINK-20204][docs-zh] Translate page 'upsert-kafka Connector' into Chinese
flinkbot edited a comment on pull request #14126: URL: https://github.com/apache/flink/pull/14126#issuecomment-730132466 ## CI report: * a9fd72f26e6c70e68a6136890f155cd8b1aa024b Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9828) * ff0736d0f86a3da958f05e16f3348da1085b309b UNKNOWN * a863c0a71120216aef8f641a2acc60d0405f90c8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9848) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-20252) Manual Test for Approximate Local Recovery
[
https://issues.apache.org/jira/browse/FLINK-20252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235878#comment-17235878
]
Yuan Mei edited comment on FLINK-20252 at 11/20/20, 3:59 AM:
-
JM Log:
failed mapper and sink are restarted.
{code:java}
2020-11-20 11:05:40,914 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: Custom
Source (1/1) (a431944bc4634099f59062c0e929fa9e) switched from SCHEDULED to
DEPLOYING. 2020-11-20 11:05:40,914 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source:
Custom Source (1/1) (attempt #0) with attempt id
a431944bc4634099f59062c0e929fa9e to container_e17_1602580065114_0666_01_02
@ i22a12256.sqa.eu95.tbsite.net (dataPort=57573) with allocation id
16cd53e0dba4d01daabfc95e2d794a38 2020-11-20 11:05:40,918 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Map (1/1)
(c33ea0a9306f7559f1c714f0527b30e4) switched from SCHEDULED to DEPLOYING.
2020-11-20 11:05:40,918 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Map (1/1)
(attempt #0) with attempt id c33ea0a9306f7559f1c714f0527b30e4 to
container_e17_1602580065114_0666_01_03 @ i22a09265.sqa.eu95.tbsite.net
(dataPort=33785) with allocation id c1d9691579e29b35c2ed54d9721a29f1 2020-11-20
11:05:40,920 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] -
Sink: Print to Std. Out (1/1) (1f3db4f707fd453ceef6aa72c2003f43) switched from
SCHEDULED to DEPLOYING. 2020-11-20 11:05:40,920 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Sink:
Print to Std. Out (1/1) (attempt #0) with attempt id
1f3db4f707fd453ceef6aa72c2003f43 to container_e17_1602580065114_0666_01_03
@ i22a09265.sqa.eu95.tbsite.net (dataPort=33785) with allocation id
c1d9691579e29b35c2ed54d9721a29f1 2020-11-20 11:05:41,023 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: Custom
Source (1/1) (a431944bc4634099f59062c0e929fa9e) switched from DEPLOYING to
RUNNING. 2020-11-20 11:05:41,093 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Map (1/1)
(c33ea0a9306f7559f1c714f0527b30e4) switched from DEPLOYING to RUNNING.
2020-11-20 11:05:41,093 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Sink: Print to Std.
Out (1/1) (1f3db4f707fd453ceef6aa72c2003f43) switched from DEPLOYING to
RUNNING. 2020-11-20 11:05:42,900 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Map (1/1)
(c33ea0a9306f7559f1c714f0527b30e4) switched from RUNNING to FAILED on
org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot@5f54bf85.
java.lang.Exception: Artificial Test Failure at
org.apache.flink.streaming.examples.wordcount.ApproximateFailover$FailingMapper.map(ApproximateFailover.java:139)
~[ApproximateFailover.jar:1.12-SNAPSHOT] at
org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at
org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:193)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at
org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.processElement(StreamTaskNetworkInput.java:179)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at
org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.emitNext(StreamTaskNetworkInput.java:152)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at
org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:67)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at
org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:372)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:186)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:574)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:538)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at
org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:722)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at
org.apache.flink.runtime.taskmanager.Task.run(Task.java:547)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at
java.lang.Thread.run(Thread.java:834) ~[?:1.8.0_102] 2020-11-20 11:05:42,920
INFO
org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy
[] - Calculating tasks to restart to recover the failed task
0a448493b4782967b150582570326227_0. 2020-11-20 11:05:42,920 INFO
org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy
[] - 2 tasks should be restarted to
[jira] [Commented] (FLINK-20230) INSERT INTO EMPTY VALUES, THROW FileNotFoundException
[
https://issues.apache.org/jira/browse/FLINK-20230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235879#comment-17235879
]
Jingsong Lee commented on FLINK-20230:
--
Looks like this is a bug in Filesystem **batch** sink.
> INSERT INTO EMPTY VALUES, THROW FileNotFoundException
> --
>
> Key: FLINK-20230
> URL: https://issues.apache.org/jira/browse/FLINK-20230
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Table SQL / Ecosystem
>Affects Versions: 1.11.0, 1.11.1, 1.11.2
>Reporter: liu
>Priority: Major
> Fix For: 1.12.0
>
>
> table USE SQL "INSERT INTO VALUES", WHEN THE VALUES IS NULL, AND IT WILL
> THROW FileNotFoundException;
> TABLE DDL:
> {code:sql}
> CREATE TABLE IF NOT EXISTS test1 (
> uid string,
> truename string,
> dt string
> )
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY ''
> STORED AS TEXTFILE;
> CREATE TABLE IF NOT EXISTS test2(
> uid string COMMENT 'ID',
> truename string
> )
> PARTITIONED BY ( `dt` string )
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY ''
> STORED AS TEXTFILE;
> # CODE IS:
> EnvironmentSettings settings = EnvironmentSettings.newInstance()
> .useBlinkPlanner()
> .inBatchMode()
> .build();
> TableEnvironment tableEnv = TableEnvironment.create(settings);
> String nowDt = "20201";
> Table table = tableEnv.sqlQuery(String.format("select \n" +
> "uid,\n" +
> "truename,\n"
> "'%s' as dt\n" +
> "from test1 " +
> "where dt = '%s'", nowDt, nowDt
> ));
> table.executeInsert("test2");
> {code}
>
> *Exception:*
> {code:java}
> Caused by: java.lang.Exception: Failed to finalize execution on masterCaused
> by: java.lang.Exception: Failed to finalize execution on master ... 34
> moreCaused by: org.apache.flink.table.api.TableException: Exception in
> finalizeGlobal at
> org.apache.flink.table.filesystem.FileSystemOutputFormat.finalizeGlobal(FileSystemOutputFormat.java:97)
> at
> org.apache.flink.runtime.jobgraph.InputOutputFormatVertex.finalizeOnMaster(InputOutputFormatVertex.java:132)
> at
> org.apache.flink.runtime.executiongraph.ExecutionGraph.vertexFinished(ExecutionGraph.java:1286)
> ... 33 moreCaused by: java.io.FileNotFoundException: File
> hdfs://nameservice1/user/hive/warehouse/test.db/test2/.staging_1605749732741
> does not exist. at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:901)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:112)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:961)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:958)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:958)
> at
> org.apache.flink.hive.shaded.fs.hdfs.HadoopFileSystem.listStatus(HadoopFileSystem.java:157)
> at
> org.apache.flink.table.filesystem.PartitionTempFileManager.headCheckpoints(PartitionTempFileManager.java:140)
> at
> org.apache.flink.table.filesystem.FileSystemCommitter.commitUpToCheckpoint(FileSystemCommitter.java:98)
> at
> org.apache.flink.table.filesystem.FileSystemOutputFormat.finalizeGlobal(FileSystemOutputFormat.java:95)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20252) Manual Test for Approximate Local Recovery
[
https://issues.apache.org/jira/browse/FLINK-20252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235878#comment-17235878
]
Yuan Mei commented on FLINK-20252:
--
JM Log:
{code:java}
2020-11-20 11:05:40,914 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: Custom
Source (1/1) (a431944bc4634099f59062c0e929fa9e) switched from SCHEDULED to
DEPLOYING. 2020-11-20 11:05:40,914 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Source:
Custom Source (1/1) (attempt #0) with attempt id
a431944bc4634099f59062c0e929fa9e to container_e17_1602580065114_0666_01_02
@ i22a12256.sqa.eu95.tbsite.net (dataPort=57573) with allocation id
16cd53e0dba4d01daabfc95e2d794a38 2020-11-20 11:05:40,918 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Map (1/1)
(c33ea0a9306f7559f1c714f0527b30e4) switched from SCHEDULED to DEPLOYING.
2020-11-20 11:05:40,918 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Map (1/1)
(attempt #0) with attempt id c33ea0a9306f7559f1c714f0527b30e4 to
container_e17_1602580065114_0666_01_03 @ i22a09265.sqa.eu95.tbsite.net
(dataPort=33785) with allocation id c1d9691579e29b35c2ed54d9721a29f1 2020-11-20
11:05:40,920 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] -
Sink: Print to Std. Out (1/1) (1f3db4f707fd453ceef6aa72c2003f43) switched from
SCHEDULED to DEPLOYING. 2020-11-20 11:05:40,920 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying Sink:
Print to Std. Out (1/1) (attempt #0) with attempt id
1f3db4f707fd453ceef6aa72c2003f43 to container_e17_1602580065114_0666_01_03
@ i22a09265.sqa.eu95.tbsite.net (dataPort=33785) with allocation id
c1d9691579e29b35c2ed54d9721a29f1 2020-11-20 11:05:41,023 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: Custom
Source (1/1) (a431944bc4634099f59062c0e929fa9e) switched from DEPLOYING to
RUNNING. 2020-11-20 11:05:41,093 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Map (1/1)
(c33ea0a9306f7559f1c714f0527b30e4) switched from DEPLOYING to RUNNING.
2020-11-20 11:05:41,093 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Sink: Print to Std.
Out (1/1) (1f3db4f707fd453ceef6aa72c2003f43) switched from DEPLOYING to
RUNNING. 2020-11-20 11:05:42,900 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Map (1/1)
(c33ea0a9306f7559f1c714f0527b30e4) switched from RUNNING to FAILED on
org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot@5f54bf85.
java.lang.Exception: Artificial Test Failure at
org.apache.flink.streaming.examples.wordcount.ApproximateFailover$FailingMapper.map(ApproximateFailover.java:139)
~[ApproximateFailover.jar:1.12-SNAPSHOT] at
org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at
org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:193)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at
org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.processElement(StreamTaskNetworkInput.java:179)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at
org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.emitNext(StreamTaskNetworkInput.java:152)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at
org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:67)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at
org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:372)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:186)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:574)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:538)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at
org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:722)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at
org.apache.flink.runtime.taskmanager.Task.run(Task.java:547)
~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at
java.lang.Thread.run(Thread.java:834) ~[?:1.8.0_102] 2020-11-20 11:05:42,920
INFO
org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy
[] - Calculating tasks to restart to recover the failed task
0a448493b4782967b150582570326227_0. 2020-11-20 11:05:42,920 INFO
org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy
[] - 2 tasks should be restarted to recover the failed task
0a448493b4782967b150582570326227_0. 2020-11-20 11:05:42,921 INFO
[jira] [Updated] (FLINK-20253) `JsonRowDeserializationSchema` can not deserialize non utf-8 json string
[
https://issues.apache.org/jira/browse/FLINK-20253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-20253:
Component/s: Table SQL / Ecosystem
> `JsonRowDeserializationSchema` can not deserialize non utf-8 json string
>
>
> Key: FLINK-20253
> URL: https://issues.apache.org/jira/browse/FLINK-20253
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table
> SQL / Ecosystem
>Affects Versions: 1.11.0
>Reporter: zouyunhe
>Priority: Major
>
> We run a flink job and the input data is json format, and found the exception
> will throw while non utf8 string flows in, the exception as below
> {code:java}
> 2020-06-15 20:55:48,777 WARN
> org.apache.flink.formats.json.JsonRowDeserializationSchema- Failed to
> deserialize JSON object: {"rip":"127.0.0.1","rtime":"2020-06-15
> 16:43:46","rchannel":"http_server","be_time
> ":1592210626,"be_ip":"146.196.79.92","uid":0,"activity_id":"{\"page\":\"share-video\",\"event\":\"testBClickVideo\",\"click_post_id\":\"6817589584893303314\",\"owner\":\"others\",\"currentUrl\":\"https:\/\/l.lik
> ee.video\/v\/IysQWB\",\"af_sub1\":\"20459_ref\",\"af_sub2\":\"1_2971\",\"msg_text\":\"1\",\"share_text1\":\"100
> ner yg gak cek Nyesel✌❤\",\"share_text2\":\"Ayok Guys Ikuti syarat nya dan
> tag 5 temen kalian semog
> a kalian Jadi Likers
> terkenal\",\"share_test\":2,\"e\":\"detail\",\"videoType\":\"playing\",\"req_id\":\"4C3BCE7A-AEE4-11EA-83BC-8397E4C235C9\",\"networkType\":\"3g\"}","country_code":"ID","os":"Mozilla\/5.0
> (Li
> nux; Android 7.0; L52F) AppleWebKit\/537.36 (KHTML, like Gecko)
> Chrome\/79.0.3945.93 Mobile
> Safari\/537.36","recv_time":1592210626,"remark":"","client_ip":"120.188.73.136"}
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonParseException:
> Invalid UTF-8 start byte 0xbe
> at [Source: (byte[])"{"rip":"127.0.0.1","rtime":"2020-06-15
> 16:43:46","rchannel":"http_server","be_time":1592210626,"be_ip":"146.196.79.92","uid":0,"activity_id":"{\"page\":\"share-video\",\"event\":\"testBClic
> kVideo\",\"click_post_id\":\"6817589584893303314\",\"owner\":\"others\",\"currentUrl\":\"https:\/\/l.likee.video\/v\/IysQWB\",\"af_sub1\":\"20459_ref\",\"af_sub2\":\"1_2971\",\"msg_text\":\"1\",\"share_text1\":\
> "100 ner yg gak cek Nyesel✌❤\",\"share_text2\":\"Ayok Guys Ikuti syarat nya
> dan tag 5 teme"[truncated 397 bytes]; line: 1, column: 406]
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1840)
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:712)
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidInitial(UTF8StreamJsonParser.java:3569)
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidChar(UTF8StreamJsonParser.java:3565)
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishString2(UTF8StreamJsonParser.java:2511)
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishAndReturnString(UTF8StreamJsonParser.java:2437)
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.json.UTF8StreamJsonParser.getText(UTF8StreamJsonParser.java:293)
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:267)
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:68)
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:15)
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper._readTreeAndClose(ObjectMapper.java:4254)
> at
> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:2725)
> at
> org.apache.flink.formats.json.JsonRowDeserializationSchema.deserialize(JsonRowDeserializationSchema.java:166)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-20252) Manual Test for Approximate Local Recovery
[ https://issues.apache.org/jira/browse/FLINK-20252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu updated FLINK-20252: Fix Version/s: 1.12.0 > Manual Test for Approximate Local Recovery > -- > > Key: FLINK-20252 > URL: https://issues.apache.org/jira/browse/FLINK-20252 > Project: Flink > Issue Type: Sub-task >Reporter: Yuan Mei >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > > This manual test is to verify approximate failover works as expected in > clusters. > > *Test Job Set up:* > source -> mapper -> printer > mapper fails after a certain number of records are received. > link: [https://github.com/apache/flink/pull/14146] > *Cluster Set up:* > Test on Yarn clustersm > add this to flink-conf.yaml > jobmanager.scheduler.scheduling-strategy: legacy > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20252) Manual Test for Approximate Local Recovery
[ https://issues.apache.org/jira/browse/FLINK-20252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu updated FLINK-20252: Priority: Critical (was: Major) > Manual Test for Approximate Local Recovery > -- > > Key: FLINK-20252 > URL: https://issues.apache.org/jira/browse/FLINK-20252 > Project: Flink > Issue Type: Sub-task >Reporter: Yuan Mei >Priority: Critical > Labels: pull-request-available > Fix For: 1.12.0 > > > This manual test is to verify approximate failover works as expected in > clusters. > > *Test Job Set up:* > source -> mapper -> printer > mapper fails after a certain number of records are received. > link: [https://github.com/apache/flink/pull/14146] > *Cluster Set up:* > Test on Yarn clustersm > add this to flink-conf.yaml > jobmanager.scheduler.scheduling-strategy: legacy > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20252) Manual Test for Approximate Local Recovery
[ https://issues.apache.org/jira/browse/FLINK-20252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu updated FLINK-20252: Parent: FLINK-20112 Issue Type: Sub-task (was: Bug) > Manual Test for Approximate Local Recovery > -- > > Key: FLINK-20252 > URL: https://issues.apache.org/jira/browse/FLINK-20252 > Project: Flink > Issue Type: Sub-task >Reporter: Yuan Mei >Priority: Major > Labels: pull-request-available > > This manual test is to verify approximate failover works as expected in > clusters. > > *Test Job Set up:* > source -> mapper -> printer > mapper fails after a certain number of records are received. > link: [https://github.com/apache/flink/pull/14146] > *Cluster Set up:* > Test on Yarn clustersm > add this to flink-conf.yaml > jobmanager.scheduler.scheduling-strategy: legacy > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20253) `JsonRowDeserializationSchema` can not deserialize non utf-8 json string
[
https://issues.apache.org/jira/browse/FLINK-20253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zouyunhe updated FLINK-20253:
-
Description:
We run a flink job and the input data is json format, and found the exception
will throw while non utf8 string flows in, the exception as below
{code:java}
2020-06-15 20:55:48,777 WARN
org.apache.flink.formats.json.JsonRowDeserializationSchema- Failed to
deserialize JSON object: {"rip":"127.0.0.1","rtime":"2020-06-15
16:43:46","rchannel":"http_server","be_time
":1592210626,"be_ip":"146.196.79.92","uid":0,"activity_id":"{\"page\":\"share-video\",\"event\":\"testBClickVideo\",\"click_post_id\":\"6817589584893303314\",\"owner\":\"others\",\"currentUrl\":\"https:\/\/l.lik
ee.video\/v\/IysQWB\",\"af_sub1\":\"20459_ref\",\"af_sub2\":\"1_2971\",\"msg_text\":\"1\",\"share_text1\":\"100
ner yg gak cek Nyesel✌❤\",\"share_text2\":\"Ayok Guys Ikuti syarat nya dan tag
5 temen kalian semog
a kalian Jadi Likers
terkenal\",\"share_test\":2,\"e\":\"detail\",\"videoType\":\"playing\",\"req_id\":\"4C3BCE7A-AEE4-11EA-83BC-8397E4C235C9\",\"networkType\":\"3g\"}","country_code":"ID","os":"Mozilla\/5.0
(Li
nux; Android 7.0; L52F) AppleWebKit\/537.36 (KHTML, like Gecko)
Chrome\/79.0.3945.93 Mobile
Safari\/537.36","recv_time":1592210626,"remark":"","client_ip":"120.188.73.136"}
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonParseException:
Invalid UTF-8 start byte 0xbe
at [Source: (byte[])"{"rip":"127.0.0.1","rtime":"2020-06-15
16:43:46","rchannel":"http_server","be_time":1592210626,"be_ip":"146.196.79.92","uid":0,"activity_id":"{\"page\":\"share-video\",\"event\":\"testBClic
kVideo\",\"click_post_id\":\"6817589584893303314\",\"owner\":\"others\",\"currentUrl\":\"https:\/\/l.likee.video\/v\/IysQWB\",\"af_sub1\":\"20459_ref\",\"af_sub2\":\"1_2971\",\"msg_text\":\"1\",\"share_text1\":\
"100 ner yg gak cek Nyesel✌❤\",\"share_text2\":\"Ayok Guys Ikuti syarat nya dan
tag 5 teme"[truncated 397 bytes]; line: 1, column: 406]
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1840)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:712)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidInitial(UTF8StreamJsonParser.java:3569)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidChar(UTF8StreamJsonParser.java:3565)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishString2(UTF8StreamJsonParser.java:2511)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishAndReturnString(UTF8StreamJsonParser.java:2437)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.json.UTF8StreamJsonParser.getText(UTF8StreamJsonParser.java:293)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:267)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:68)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:15)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper._readTreeAndClose(ObjectMapper.java:4254)
at
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:2725)
at
org.apache.flink.formats.json.JsonRowDeserializationSchema.deserialize(JsonRowDeserializationSchema.java:166)
{code}
was:We run a flink job and the input data is json format, and found the
exception will throw while non utf8 string flows in, the exception as below
> `JsonRowDeserializationSchema` can not deserialize non utf-8 json string
>
>
> Key: FLINK-20253
> URL: https://issues.apache.org/jira/browse/FLINK-20253
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>Affects Versions: 1.11.0
>Reporter: zouyunhe
>Priority: Major
>
> We run a flink job and the input data is json format, and found the exception
> will throw while non utf8 string flows in, the exception as below
> {code:java}
> 2020-06-15 20:55:48,777 WARN
> org.apache.flink.formats.json.JsonRowDeserializationSchema- Failed to
> deserialize JSON object: {"rip":"127.0.0.1","rtime":"2020-06-15
> 16:43:46","rchannel":"http_server","be_time
>
[jira] [Commented] (FLINK-20230) INSERT INTO EMPTY VALUES, THROW FileNotFoundException
[
https://issues.apache.org/jira/browse/FLINK-20230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235874#comment-17235874
]
Jark Wu commented on FLINK-20230:
-
Could you help to have a look at this? [~lzljs3620320]
> INSERT INTO EMPTY VALUES, THROW FileNotFoundException
> --
>
> Key: FLINK-20230
> URL: https://issues.apache.org/jira/browse/FLINK-20230
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Table SQL / Ecosystem
>Affects Versions: 1.11.0, 1.11.1, 1.11.2
>Reporter: liu
>Priority: Major
> Fix For: 1.12.0
>
>
> table USE SQL "INSERT INTO VALUES", WHEN THE VALUES IS NULL, AND IT WILL
> THROW FileNotFoundException;
> TABLE DDL:
> {code:sql}
> CREATE TABLE IF NOT EXISTS test1 (
> uid string,
> truename string,
> dt string
> )
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY ''
> STORED AS TEXTFILE;
> CREATE TABLE IF NOT EXISTS test2(
> uid string COMMENT 'ID',
> truename string
> )
> PARTITIONED BY ( `dt` string )
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY ''
> STORED AS TEXTFILE;
> # CODE IS:
> EnvironmentSettings settings = EnvironmentSettings.newInstance()
> .useBlinkPlanner()
> .inBatchMode()
> .build();
> TableEnvironment tableEnv = TableEnvironment.create(settings);
> String nowDt = "20201";
> Table table = tableEnv.sqlQuery(String.format("select \n" +
> "uid,\n" +
> "truename,\n"
> "'%s' as dt\n" +
> "from test1 " +
> "where dt = '%s'", nowDt, nowDt
> ));
> table.executeInsert("test2");
> {code}
>
> *Exception:*
> {code:java}
> Caused by: java.lang.Exception: Failed to finalize execution on masterCaused
> by: java.lang.Exception: Failed to finalize execution on master ... 34
> moreCaused by: org.apache.flink.table.api.TableException: Exception in
> finalizeGlobal at
> org.apache.flink.table.filesystem.FileSystemOutputFormat.finalizeGlobal(FileSystemOutputFormat.java:97)
> at
> org.apache.flink.runtime.jobgraph.InputOutputFormatVertex.finalizeOnMaster(InputOutputFormatVertex.java:132)
> at
> org.apache.flink.runtime.executiongraph.ExecutionGraph.vertexFinished(ExecutionGraph.java:1286)
> ... 33 moreCaused by: java.io.FileNotFoundException: File
> hdfs://nameservice1/user/hive/warehouse/test.db/test2/.staging_1605749732741
> does not exist. at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:901)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:112)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:961)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:958)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:958)
> at
> org.apache.flink.hive.shaded.fs.hdfs.HadoopFileSystem.listStatus(HadoopFileSystem.java:157)
> at
> org.apache.flink.table.filesystem.PartitionTempFileManager.headCheckpoints(PartitionTempFileManager.java:140)
> at
> org.apache.flink.table.filesystem.FileSystemCommitter.commitUpToCheckpoint(FileSystemCommitter.java:98)
> at
> org.apache.flink.table.filesystem.FileSystemOutputFormat.finalizeGlobal(FileSystemOutputFormat.java:95)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-20253) `JsonRowDeserializationSchema` can not deserialize non utf-8 json string
zouyunhe created FLINK-20253: Summary: `JsonRowDeserializationSchema` can not deserialize non utf-8 json string Key: FLINK-20253 URL: https://issues.apache.org/jira/browse/FLINK-20253 Project: Flink Issue Type: Bug Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) Affects Versions: 1.11.0 Reporter: zouyunhe We run a flink job and the input data is json format, and found the exception will throw while non utf8 string flows in, the exception as below -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20230) INSERT INTO EMPTY VALUES, THROW FileNotFoundException
[
https://issues.apache.org/jira/browse/FLINK-20230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-20230:
Description:
table USE SQL "INSERT INTO VALUES", WHEN THE VALUES IS NULL, AND IT WILL THROW
FileNotFoundException;
TABLE DDL:
{code:sql}
CREATE TABLE IF NOT EXISTS test1 (
uid string,
truename string,
dt string
)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ''
STORED AS TEXTFILE;
CREATE TABLE IF NOT EXISTS test2(
uid string COMMENT 'ID',
truename string
)
PARTITIONED BY ( `dt` string )
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ''
STORED AS TEXTFILE;
# CODE IS:
EnvironmentSettings settings = EnvironmentSettings.newInstance()
.useBlinkPlanner()
.inBatchMode()
.build();
TableEnvironment tableEnv = TableEnvironment.create(settings);
String nowDt = "20201";
Table table = tableEnv.sqlQuery(String.format("select \n" +
"uid,\n" +
"truename,\n"
"'%s' as dt\n" +
"from test1 " +
"where dt = '%s'", nowDt, nowDt
));
table.executeInsert("test2");
{code}
*Exception:*
{code:java}
Caused by: java.lang.Exception: Failed to finalize execution on masterCaused
by: java.lang.Exception: Failed to finalize execution on master ... 34
moreCaused by: org.apache.flink.table.api.TableException: Exception in
finalizeGlobal at
org.apache.flink.table.filesystem.FileSystemOutputFormat.finalizeGlobal(FileSystemOutputFormat.java:97)
at
org.apache.flink.runtime.jobgraph.InputOutputFormatVertex.finalizeOnMaster(InputOutputFormatVertex.java:132)
at
org.apache.flink.runtime.executiongraph.ExecutionGraph.vertexFinished(ExecutionGraph.java:1286)
... 33 moreCaused by: java.io.FileNotFoundException: File
hdfs://nameservice1/user/hive/warehouse/test.db/test2/.staging_1605749732741
does not exist. at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:901)
at
org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:112)
at
org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:961)
at
org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:958)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:958)
at
org.apache.flink.hive.shaded.fs.hdfs.HadoopFileSystem.listStatus(HadoopFileSystem.java:157)
at
org.apache.flink.table.filesystem.PartitionTempFileManager.headCheckpoints(PartitionTempFileManager.java:140)
at
org.apache.flink.table.filesystem.FileSystemCommitter.commitUpToCheckpoint(FileSystemCommitter.java:98)
at
org.apache.flink.table.filesystem.FileSystemOutputFormat.finalizeGlobal(FileSystemOutputFormat.java:95)
{code}
was:
table USE SQL "INSERT INTO VALUES", WHEN THE VALUES IS NULL, AND IT WILL THROW
FileNotFoundException;
*Exception:*
Caused by: java.lang.Exception: Failed to finalize execution on masterCaused
by: java.lang.Exception: Failed to finalize execution on master ... 34
moreCaused by: org.apache.flink.table.api.TableException: Exception in
finalizeGlobal at
org.apache.flink.table.filesystem.FileSystemOutputFormat.finalizeGlobal(FileSystemOutputFormat.java:97)
at
org.apache.flink.runtime.jobgraph.InputOutputFormatVertex.finalizeOnMaster(InputOutputFormatVertex.java:132)
at
org.apache.flink.runtime.executiongraph.ExecutionGraph.vertexFinished(ExecutionGraph.java:1286)
... 33 moreCaused by: java.io.FileNotFoundException: File
hdfs://nameservice1/user/hive/warehouse/test.db/test2/.staging_1605749732741
does not exist. at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:901)
at
org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:112)
at
org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:961)
at
org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:958)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:958)
at
org.apache.flink.hive.shaded.fs.hdfs.HadoopFileSystem.listStatus(HadoopFileSystem.java:157)
at
org.apache.flink.table.filesystem.PartitionTempFileManager.headCheckpoints(PartitionTempFileManager.java:140)
at
org.apache.flink.table.filesystem.FileSystemCommitter.commitUpToCheckpoint(FileSystemCommitter.java:98)
at
org.apache.flink.table.filesystem.FileSystemOutputFormat.finalizeGlobal(FileSystemOutputFormat.java:95)
> INSERT INTO EMPTY VALUES, THROW FileNotFoundException
> --
>
> Key: FLINK-20230
> URL: https://issues.apache.org/jira/browse/FLINK-20230
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Table SQL /
[jira] [Updated] (FLINK-20230) INSERT INTO EMPTY VALUES, THROW FileNotFoundException
[
https://issues.apache.org/jira/browse/FLINK-20230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-20230:
Environment: (was: TABLE DDL:
CREATE TABLE IF NOT EXISTS test1 (
uid string,
truename string,
dt string
)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ''
STORED AS TEXTFILE;
CREATE TABLE IF NOT EXISTS test2(
uid string COMMENT 'ID',
truename string
)
PARTITIONED BY ( `dt` string )
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ''
STORED AS TEXTFILE;
# CODE IS:
EnvironmentSettings settings = EnvironmentSettings.newInstance()
.useBlinkPlanner()
.inBatchMode()
.build();
TableEnvironment tableEnv = TableEnvironment.create(settings);
String nowDt = "20201";
Table table = tableEnv.sqlQuery(String.format("select \n" +
"uid,\n" +
"truename,\n"
"'%s' as dt\n" +
"from test1 " +
"where dt = '%s'", nowDt, nowDt
));
table.executeInsert("test2");)
> INSERT INTO EMPTY VALUES, THROW FileNotFoundException
> --
>
> Key: FLINK-20230
> URL: https://issues.apache.org/jira/browse/FLINK-20230
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Table SQL / Ecosystem
>Affects Versions: 1.11.0, 1.11.1, 1.11.2
>Reporter: liu
>Priority: Major
> Fix For: 1.12.0
>
>
> table USE SQL "INSERT INTO VALUES", WHEN THE VALUES IS NULL, AND IT WILL
> THROW FileNotFoundException;
>
> *Exception:*
> Caused by: java.lang.Exception: Failed to finalize execution on masterCaused
> by: java.lang.Exception: Failed to finalize execution on master ... 34
> moreCaused by: org.apache.flink.table.api.TableException: Exception in
> finalizeGlobal at
> org.apache.flink.table.filesystem.FileSystemOutputFormat.finalizeGlobal(FileSystemOutputFormat.java:97)
> at
> org.apache.flink.runtime.jobgraph.InputOutputFormatVertex.finalizeOnMaster(InputOutputFormatVertex.java:132)
> at
> org.apache.flink.runtime.executiongraph.ExecutionGraph.vertexFinished(ExecutionGraph.java:1286)
> ... 33 moreCaused by: java.io.FileNotFoundException: File
> hdfs://nameservice1/user/hive/warehouse/test.db/test2/.staging_1605749732741
> does not exist. at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:901)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:112)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:961)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:958)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:958)
> at
> org.apache.flink.hive.shaded.fs.hdfs.HadoopFileSystem.listStatus(HadoopFileSystem.java:157)
> at
> org.apache.flink.table.filesystem.PartitionTempFileManager.headCheckpoints(PartitionTempFileManager.java:140)
> at
> org.apache.flink.table.filesystem.FileSystemCommitter.commitUpToCheckpoint(FileSystemCommitter.java:98)
> at
> org.apache.flink.table.filesystem.FileSystemOutputFormat.finalizeGlobal(FileSystemOutputFormat.java:95)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] wuchong commented on pull request #14103: [FLINK-18545] Introduce `pipeline.name` to allow users to specify job name by configuration
wuchong commented on pull request #14103:
URL: https://github.com/apache/flink/pull/14103#issuecomment-730823946
Hi @kl0u , I don't think the job name configuration conflicts with
application mode. I think multi-execute applications may al so have same job
name, e.g. use the default job name when submitting using `env.execute()`.
With the job name configuration, users can still change the job name before
submitting, for example:
```java
Configuration conf = new Configuration();
conf.setString("pipeline.name", "job1");
env.configure(conf, Thread.currentThread().getContextClassLoader);
env.execute(); // submit job1
conf.setString("pipeline.name", "job2");
env.configure(conf, Thread.currentThread().getContextClassLoader);
env.execute(); // submit job2
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-20252) Manual Test for Approximate Local Recovery
[ https://issues.apache.org/jira/browse/FLINK-20252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-20252: - Description: This manual test is to verify approximate failover works as expected in clusters. *Test Job Set up:* source -> mapper -> printer mapper fails after a certain number of records are received. link: [https://github.com/apache/flink/pull/14146] *Cluster Set up:* Test on Yarn clustersm add this to flink-conf.yaml jobmanager.scheduler.scheduling-strategy: legacy was: This manual test is to verify approximate failover works as expected in clusters. Test Job Set up: source -> mapper -> printer mapper fails after a certain number of records are received. link: > Manual Test for Approximate Local Recovery > -- > > Key: FLINK-20252 > URL: https://issues.apache.org/jira/browse/FLINK-20252 > Project: Flink > Issue Type: Bug >Reporter: Yuan Mei >Priority: Major > Labels: pull-request-available > > This manual test is to verify approximate failover works as expected in > clusters. > > *Test Job Set up:* > source -> mapper -> printer > mapper fails after a certain number of records are received. > link: [https://github.com/apache/flink/pull/14146] > *Cluster Set up:* > Test on Yarn clustersm > add this to flink-conf.yaml > jobmanager.scheduler.scheduling-strategy: legacy > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] curcur closed pull request #14146: [FLINK-20252] Test Job for Approximate Local Recovery
curcur closed pull request #14146: URL: https://github.com/apache/flink/pull/14146 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14126: [FLINK-20204][docs-zh] Translate page 'upsert-kafka Connector' into Chinese
flinkbot edited a comment on pull request #14126: URL: https://github.com/apache/flink/pull/14126#issuecomment-730132466 ## CI report: * a9fd72f26e6c70e68a6136890f155cd8b1aa024b Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9828) * ff0736d0f86a3da958f05e16f3348da1085b309b UNKNOWN * a863c0a71120216aef8f641a2acc60d0405f90c8 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] lirui-apache commented on a change in pull request #14145: [FLINK-20239][docs] Confusing pages: Hive Read & Write and Hive Strea…
lirui-apache commented on a change in pull request #14145:
URL: https://github.com/apache/flink/pull/14145#discussion_r527368062
##
File path: docs/dev/table/hive/hive_read_write.md
##
@@ -22,119 +22,199 @@ specific language governing permissions and limitations
under the License.
-->
-Using the `HiveCatalog` and Flink's connector to Hive, Flink can read and
write from Hive data as an alternative to Hive's batch engine.
-Be sure to follow the instructions to include the correct [dependencies]({{
site.baseurl }}/dev/table/hive/#depedencies) in your application.
-And please also note that Hive connector only works with blink planner.
+Using the HiveCatalog, Apache Flink can be used for unified `BATCH` and STREAM
processing of Apache
+Hive Tables. This means Flink can be used as a more performant alternative to
Hive’s batch engine,
+or to continuously read and write data into and out of Hive tables to power
real-time data
+warehousing applications.
+
+
+ IMPORTANT: Reading and writing to and from Apache Hive is only
supported by the Blink table planner.
+
* This will be replaced by the TOC
{:toc}
-## Reading From Hive
+## Reading
-Assume Hive contains a single table in its `default` database, named people
that contains several rows.
+Flink supports reading data from Hive in both `BATCH` and `STREAMING` modes.
When run as a `BATCH`
+application, Flink will execute its query over the state of the table at the
point in time when the
+query is executed. `STREAMING` reads will continuously monitor the table and
incrementally fetch
+new data as it is made available. Flink will read tables as bounded by default.
+
+`STREAMING` reads support consuming both partitioned and non-partitioned
tables.
+For partitioned tables, Flink will monitor the generation of new partitions,
and read
+them incrementally when available. For non-partitioned tables, Flink will
monitor the generation
+of new files in the folder and read new files incrementally.
+
+
+
+
+Key
+Default
+Type
+Description
+
+
+
+
+streaming-source.enable
+false
+Boolean
+Enable streaming source or not. NOTES: Please make sure that each
partition/file should be written atomically, otherwise the reader may get
incomplete data.
+
+
+streaming-source.monitor-interval
+1 m
+Duration
+Time interval for consecutively monitoring partition/file.
+
+
+streaming-source.consume-order
+create-time
+String
+The consume order of streaming source, support create-time and
partition-time. create-time compare partition/file creation time, this is not
the partition create time in Hive metaStore, but the folder/file modification
time in filesystem; partition-time compare time represented by partition name,
if the partition folder somehow gets updated, e.g. add new file into folder, it
can affect how the data is consumed. For non-partition table, this value should
always be 'create-time'.
+
+
+streaming-source.consume-start-offset
+1970-00-00
+String
+Start offset for streaming consuming. How to parse and compare
offsets depends on your order. For create-time and partition-time, should be a
timestamp string (-[m]m-[d]d [hh:mm:ss]). For partition-time, will use
partition time extractor to extract time from partition.
+
+
+
+
+[SQL Hints]({% link dev/table/sql/hints.md %}) can be used to apply
configurations to a Hive table
+without changing its definition in the Hive metastore.
+
+{% highlight sql %}
+
+SELECT *
+FROM hive_table
+/*+ OPTIONS('streaming-source.enable'='true',
'streaming-source.consume-start-offset'='2020-05-20') */;
-{% highlight bash %}
-hive> show databases;
-OK
-default
-Time taken: 0.841 seconds, Fetched: 1 row(s)
-
-hive> show tables;
-OK
-Time taken: 0.087 seconds
-
-hive> CREATE TABLE mytable(name string, value double);
-OK
-Time taken: 0.127 seconds
-
-hive> SELECT * FROM mytable;
-OK
-Tom 4.72
-John 8.0
-Tom 24.2
-Bob 3.14
-Bob 4.72
-Tom 34.9
-Mary 4.79
-Tiff 2.72
-Bill 4.33
-Mary 77.7
-Time taken: 0.097 seconds, Fetched: 10 row(s)
{% endhighlight %}
-With the data ready your can connect to Hive [connect to an existing Hive
installation]({{ site.baseurl }}/dev/table/hive/#connecting-to-hive) and begin
querying.
+**Notes**
-{% highlight bash %}
+- Monitor strategy is to scan all directories/files currently in the location
path. Many partitions may cause performance degradation.
+- Streaming reads for non-partitioned tables requires that each file be
written atomically into the target directory.
+- Streaming reading for partitioned tables requires that each partition should
be added atomically in the view of hive metastore. If not, new data added to an
existing partition will be consumed.
+- Streaming reads do not support watermark grammar in Flink DDL. These
[GitHub] [flink] flinkbot commented on pull request #14146: [FLINK-20252] Test Job for Approximate Local Recovery
flinkbot commented on pull request #14146: URL: https://github.com/apache/flink/pull/14146#issuecomment-730822458 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit b4e2216ddf2eb8fbadf9687e1b3bb7b43c70717f (Fri Nov 20 03:30:10 UTC 2020) **Warnings:** * **1 pom.xml files were touched**: Check for build and licensing issues. * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-20252).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-20252) Manual Test for Approximate Local Recovery
[ https://issues.apache.org/jira/browse/FLINK-20252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-20252: --- Labels: pull-request-available (was: ) > Manual Test for Approximate Local Recovery > -- > > Key: FLINK-20252 > URL: https://issues.apache.org/jira/browse/FLINK-20252 > Project: Flink > Issue Type: Bug >Reporter: Yuan Mei >Priority: Major > Labels: pull-request-available > > This manual test is to verify approximate failover works as expected in > clusters. > > Test Job Set up: source -> mapper -> printer > mapper fails after a certain number of records are received. > link: -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #11830: [FLINK-17096] [table] Mini-batch group aggregation doesn't expire state even if state ttl is enabled
flinkbot edited a comment on pull request #11830: URL: https://github.com/apache/flink/pull/11830#issuecomment-616574921 ## CI report: * aa11b396cf32dc372ebab32ec177172fb5388a62 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9833) * d9f38096571c19304a7fefcee3c188aa7cf315cf Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9847) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] curcur opened a new pull request #14146: [FLINK-20252] Test Job for Approximate Local Recovery
curcur opened a new pull request #14146: URL: https://github.com/apache/flink/pull/14146 ## What is the purpose of the change *(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)* ## Brief change log *(for example:)* - *The TaskInfo is stored in the blob store on job creation time as a persistent artifact* - *Deployments RPC transmits only the blob storage reference* - *TaskManagers retrieve the TaskInfo from the blob cache* ## Verifying this change *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (100MB)* - *Extended integration test for recovery after master (JobManager) failure* - *Added test that validates that TaskInfo is transferred only once across recoveries* - *Manually verified the change by running a 4 node cluser with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) - The serializers: (yes / no / don't know) - The runtime per-record code paths (performance sensitive): (yes / no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / no / don't know) - The S3 file system connector: (yes / no / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-20252) Manual Test for Approximate Local Recovery
[ https://issues.apache.org/jira/browse/FLINK-20252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-20252: - Description: This manual test is to verify approximate failover works as expected in clusters. Test Job Set up: source -> mapper -> printer mapper fails after a certain number of records are received. link: was:This > Manual Test for Approximate Local Recovery > -- > > Key: FLINK-20252 > URL: https://issues.apache.org/jira/browse/FLINK-20252 > Project: Flink > Issue Type: Bug >Reporter: Yuan Mei >Priority: Major > > This manual test is to verify approximate failover works as expected in > clusters. > > Test Job Set up: source -> mapper -> printer > mapper fails after a certain number of records are received. > link: -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20246) Add documentation on Python worker memory tuning in the memory tuning page
[ https://issues.apache.org/jira/browse/FLINK-20246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235865#comment-17235865 ] Zhu Zhu commented on FLINK-20246: - Thanks for firing and taking this task! [~dian.fu] Link it to FLINK-20116 to get it tracked. > Add documentation on Python worker memory tuning in the memory tuning page > -- > > Key: FLINK-20246 > URL: https://issues.apache.org/jira/browse/FLINK-20246 > Project: Flink > Issue Type: Improvement > Components: API / Python, Documentation >Reporter: Dian Fu >Assignee: Dian Fu >Priority: Major > Fix For: 1.12.0 > > > Per the discussion in FLINK-19177, we need to add some documentation on > Python worker memory tuning in the memory tuning page. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-19775) SystemProcessingTimeServiceTest.testImmediateShutdown is instable
[
https://issues.apache.org/jira/browse/FLINK-19775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235861#comment-17235861
]
jiawen xiao commented on FLINK-19775:
-
hi, [~dian.fu],[~trohrmann],After my recent research，this question is similar
to https://issues.apache.org/jira/browse/FLINK-6571
I summarized everyone's description and my own thoughts. First of all, this is
a type of problem, and there will be instability problems. Perhaps it is time
to find the reason why the main thread is interrupted by other threads.
Secondly, make an assumption that when the main thread is lock.wait(), the
child thread where latch.trigger() is located is not scheduled immediately by
the cpu, so triggered=false is unchanged. If the main thread is interrupted by
other threads, it will cause a hot loop problem in the await() method and
continuously throw InterruptedException exceptions. Finally, I think that
simple catch exceptions cannot solve the instability problem. We should change
the loop flag to false while catching the exception. Do you have any
suggestions?
> SystemProcessingTimeServiceTest.testImmediateShutdown is instable
> -
>
> Key: FLINK-19775
> URL: https://issues.apache.org/jira/browse/FLINK-19775
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.11.0
>Reporter: Dian Fu
>Assignee: jiawen xiao
>Priority: Major
> Labels: pull-request-available, test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=8131=logs=d89de3df-4600-5585-dadc-9bbc9a5e661c=66b5c59a-0094-561d-0e44-b149dfdd586d
> {code}
> 2020-10-22T21:12:54.9462382Z [ERROR]
> testImmediateShutdown(org.apache.flink.streaming.runtime.tasks.SystemProcessingTimeServiceTest)
> Time elapsed: 0.009 s <<< ERROR!
> 2020-10-22T21:12:54.9463024Z java.lang.InterruptedException
> 2020-10-22T21:12:54.9463331Z at java.lang.Object.wait(Native Method)
> 2020-10-22T21:12:54.9463766Z at java.lang.Object.wait(Object.java:502)
> 2020-10-22T21:12:54.9464140Z at
> org.apache.flink.core.testutils.OneShotLatch.await(OneShotLatch.java:63)
> 2020-10-22T21:12:54.9466014Z at
> org.apache.flink.streaming.runtime.tasks.SystemProcessingTimeServiceTest.testImmediateShutdown(SystemProcessingTimeServiceTest.java:154)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14126: [FLINK-20204][docs-zh] Translate page 'upsert-kafka Connector' into Chinese
flinkbot edited a comment on pull request #14126: URL: https://github.com/apache/flink/pull/14126#issuecomment-730132466 ## CI report: * a9fd72f26e6c70e68a6136890f155cd8b1aa024b Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9828) * ff0736d0f86a3da958f05e16f3348da1085b309b UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-20252) Manual Test for Approximate Local Recovery
[ https://issues.apache.org/jira/browse/FLINK-20252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-20252: - Description: This > Manual Test for Approximate Local Recovery > -- > > Key: FLINK-20252 > URL: https://issues.apache.org/jira/browse/FLINK-20252 > Project: Flink > Issue Type: Bug >Reporter: Yuan Mei >Priority: Major > > This -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-20252) Manual Test for Approximate Local Recovery
Yuan Mei created FLINK-20252: Summary: Manual Test for Approximate Local Recovery Key: FLINK-20252 URL: https://issues.apache.org/jira/browse/FLINK-20252 Project: Flink Issue Type: Bug Reporter: Yuan Mei -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #11830: [FLINK-17096] [table] Mini-batch group aggregation doesn't expire state even if state ttl is enabled
flinkbot edited a comment on pull request #11830: URL: https://github.com/apache/flink/pull/11830#issuecomment-616574921 ## CI report: * aa11b396cf32dc372ebab32ec177172fb5388a62 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9833) * d9f38096571c19304a7fefcee3c188aa7cf315cf UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
