[jira] [Commented] (FLINK-20431) KafkaSourceReaderTest.testCommitOffsetsWithoutAliveFetchers:133->lambda$testCommitOffsetsWithoutAliveFetchers$3:134 expected:<10> but was:<1>
[ https://issues.apache.org/jira/browse/FLINK-20431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17407121#comment-17407121 ] Xintong Song commented on FLINK-20431: -- https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=23144&view=logs&j=1fc6e7bf-633c-5081-c32a-9dea24b05730&t=576aba0a-d787-51b6-6a92-cf233f360582&l=7328 > KafkaSourceReaderTest.testCommitOffsetsWithoutAliveFetchers:133->lambda$testCommitOffsetsWithoutAliveFetchers$3:134 > expected:<10> but was:<1> > - > > Key: FLINK-20431 > URL: https://issues.apache.org/jira/browse/FLINK-20431 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka >Affects Versions: 1.12.2, 1.13.0 >Reporter: Huang Xingbo >Priority: Major > Labels: auto-unassigned, pull-request-available, test-stability > Fix For: 1.12.6, 1.13.3 > > > [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=10351&view=logs&j=c5f0071e-1851-543e-9a45-9ac140befc32&t=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5] > [ERROR] Failures: > [ERROR] > KafkaSourceReaderTest.testCommitOffsetsWithoutAliveFetchers:133->lambda$testCommitOffsetsWithoutAliveFetchers$3:134 > expected:<10> but was:<1> > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-23493) python tests hang on Azure
[ https://issues.apache.org/jira/browse/FLINK-23493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17407120#comment-17407120 ] Dian Fu edited comment on FLINK-23493 at 8/31/21, 6:51 AM: --- [~xtsong] I'd like to downgrade the issue as just as [~hxbks2ks] investigated, it's strange that the failed test cases only occurred in stage "cron_azure" and so I also suspect it's more like an environment issue. Besides, it only failed once in the last week. Personally I think this should not be a release blocker for 1.14. Do you think it makes sense to downgrade it to critical? (PS: We will continue to investigate this issue) was (Author: dianfu): [~xtsong] [~hxbks2ks] I'd like to downgrade the issue as just as [~hxbks2ks] investigated, it's strange that the failed test cases only occurred in stage "cron_azure" and so I also suspect it's more like an environment issue. Besides, it only failed once in the last week. Personally I think this should not be a release blocker for 1.14. Do you think it makes sense to downgrade it to critical? (PS: We will continue to investigate this issue) > python tests hang on Azure > -- > > Key: FLINK-23493 > URL: https://issues.apache.org/jira/browse/FLINK-23493 > Project: Flink > Issue Type: Bug > Components: API / Python >Affects Versions: 1.14.0, 1.13.1, 1.12.4 >Reporter: Dawid Wysakowicz >Assignee: Huang Xingbo >Priority: Blocker > Labels: test-stability > Fix For: 1.14.0 > > > https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=20898&view=logs&j=821b528f-1eed-5598-a3b4-7f748b13f261&t=4fad9527-b9a5-5015-1b70-8356e5c91490&l=22829 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #17066: FLINK-24066
flinkbot commented on pull request #17066: URL: https://github.com/apache/flink/pull/17066#issuecomment-908955355 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit c4c3e02c2c2122c90f71548caebfac6602592a1d (Tue Aug 31 06:59:56 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-24066).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] AHeise opened a new pull request #17067: [FLINK-23817][docs] Add metric description to Kafka source and File sink.
AHeise opened a new pull request #17067: URL: https://github.com/apache/flink/pull/17067 Add metric description to Kafka source and File sink (see FLIP-33, FLIP-179). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-23817) Write documentation for standardized operator metrics
[ https://issues.apache.org/jira/browse/FLINK-23817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-23817: --- Labels: pull-request-available (was: ) > Write documentation for standardized operator metrics > - > > Key: FLINK-23817 > URL: https://issues.apache.org/jira/browse/FLINK-23817 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Common, Documentation >Reporter: Arvid Heise >Assignee: Arvid Heise >Priority: Blocker > Labels: pull-request-available > Fix For: 1.14.0 > > > Incorporate metrics in connector page. Use > [data-templates|https://gohugo.io/templates/data-templates/] for common > metrics. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #17067: [FLINK-23817][docs] Add metric description to Kafka source and File sink.
flinkbot commented on pull request #17067: URL: https://github.com/apache/flink/pull/17067#issuecomment-908956953 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit ac9ad3d39442f0479d1f74d4664c3351fc311a41 (Tue Aug 31 07:02:24 UTC 2021) **Warnings:** * Documentation files were touched, but no `docs/content.zh/` files: Update Chinese documentation or file Jira ticket. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-24035) Fix the deadlock issue caused by buffer listeners may not be notified
[
https://issues.apache.org/jira/browse/FLINK-24035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17407129#comment-17407129
]
Piotr Nowojski commented on FLINK-24035:
Solution here is to always request at least a single buffer.
This is not an issue on the output side, because there are no
{{BufferListeners}} in that case. Also if buffer is requested on the output
side, it will be used and eventually flushed.
On the input on the other hand, with exclusive buffers > 0, we are already
requesting exclusive buffers in a blocking way with a timeout (FLINK-12852), so
we know that task will be able to make progress regardless if we notify about
more buffers or not. With exclusive buffers = 0, this solution requests a
single floating buffer, so we will also be able to make a progress. Once data
starts flowing/this single buffer will be recycled, listeners would be notified
and more buffers would be requested.
> Fix the deadlock issue caused by buffer listeners may not be notified
> -
>
> Key: FLINK-24035
> URL: https://issues.apache.org/jira/browse/FLINK-24035
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
>Affects Versions: 1.14.0
>Reporter: Yingjie Cao
>Assignee: Yingjie Cao
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.14.0
>
>
> The buffer listeners are not notified when the the local buffer pool receives
> available notification from the global pool. This may cause potential
> deadlock issue:
> # A LocalBufferPool is created, but there is no available buffers in the
> global NetworkBufferPool.
> # The LocalBufferPool registers an available buffer listener to the global
> NetworkBufferPool.
> # The BufferManager requests buffers from the LocalBufferPool but no buffer
> is available. As a result, it registers an available buffer listener to the
> LocalBufferPool.
> # A buffer is recycled to the global pool and the local buffer pool is
> notified about the available buffer.
> # The local buffer pool requests the available buffer from the global pool
> but the registered available buffer listener of BufferManager is not notified
> and it can never get a chance to be notified so deadlock occurs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-24038) DispatcherResourceManagerComponent fails to deregister application if no leading ResourceManager
[
https://issues.apache.org/jira/browse/FLINK-24038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17407130#comment-17407130
]
Yang Wang commented on FLINK-24038:
---
[~trohrmann] Using a K8s job could help a bit when doing the deregistration
failed. However, we still have the residual TaskManager pods and flink
ConfigMaps. Maybe let the JobManager relaunched and recover the finished or
failed jobs, then the dispatcher will deregister the application again. It is
more reasonable.
For Yarn, I am afraid it is in the same situation. Even though the
JobManager(application master) exit with zero code, it will be launched again
when deregistering failed.
For the new options #1 [~xtsong] listed, do you mean let the leader dispatcher
do the deregistration? If it is, what will happen without leader.
> DispatcherResourceManagerComponent fails to deregister application if no
> leading ResourceManager
>
>
> Key: FLINK-24038
> URL: https://issues.apache.org/jira/browse/FLINK-24038
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.14.0
>Reporter: Till Rohrmann
>Priority: Critical
> Fix For: 1.14.0
>
>
> With FLINK-21667 we introduced a change that can cause the
> {{DispatcherResourceManagerComponent}} to fail when trying to stop the
> application. The problem is that the {{DispatcherResourceManagerComponent}}
> needs a leading {{ResourceManager}} to successfully execute the
> stop/deregister application call. If this is not the case, then it will fail
> fatally. In the case of multiple standby JobManager processes it can happen
> that the leading {{ResourceManager}} runs somewhere else.
> I do see two possible solutions:
> 1. Run the leader election process for the whole JobManager process
> 2. Move the registration/deregistration of the application out of the
> {{ResourceManager}} so that it can be executed w/o a leader
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-21589) Document Table/SQL API limitations regarding upgrades with savepoints
[ https://issues.apache.org/jira/browse/FLINK-21589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xintong Song reassigned FLINK-21589: Assignee: Timo Walther > Document Table/SQL API limitations regarding upgrades with savepoints > - > > Key: FLINK-21589 > URL: https://issues.apache.org/jira/browse/FLINK-21589 > Project: Flink > Issue Type: Improvement > Components: Documentation, Table SQL / API >Reporter: Chesnay Schepler >Assignee: Timo Walther >Priority: Critical > Labels: auto-deprioritized-major > Fix For: 1.14.0, 1.11.5, 1.12.6 > > > We don't appear to mention anywhere that users cannot upgrade Table/SQL API > applications with savepoints (or it is well hidden). > This should be mentioned in the API docs (presumably under streaming > concepts) and the application upgrading documentation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-24068) CheckpointBarrierHandler may skip the markAlignmentStart for alignment -with-timeout checkpoint
[
https://issues.apache.org/jira/browse/FLINK-24068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski updated FLINK-24068:
---
Priority: Critical (was: Major)
> CheckpointBarrierHandler may skip the markAlignmentStart for alignment
> -with-timeout checkpoint
> ---
>
> Key: FLINK-24068
> URL: https://issues.apache.org/jira/browse/FLINK-24068
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Checkpointing
>Affects Versions: 1.14.0
>Reporter: Yun Gao
>Priority: Critical
> Fix For: 1.14.0
>
>
> {code:java}
> 04:51:11,727 [Flat Map -> Sink: Unnamed (10/12)#1] WARN
> org.apache.flink.runtime.taskmanager.Task [] - Flat Map -> Sink: Unnamed
> (10/12)#1 (0d965fd3c0de11dc7fb6793435cda8ba) switched from RUNNING to FAILED
> with failure cause: java.lang.IllegalStateException: Alignment time is less
> than zero({}). Is the time monotonic? [-9223369873401849363]
> at org.apache.flink.util.Preconditions.checkState(Preconditions.java:215)
> at
> org.apache.flink.streaming.runtime.io.checkpointing.CheckpointBarrierHandler.markAlignmentEnd(CheckpointBarrierHandler.java:181)
> at
> org.apache.flink.streaming.runtime.io.checkpointing.CheckpointBarrierHandler.markAlignmentEnd(CheckpointBarrierHandler.java:177)
> at
> org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.checkCheckpointAlignedAndTransformState(SingleCheckpointBarrierHandler.java:253)
> at
> org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.processBarrier(SingleCheckpointBarrierHandler.java:240)
> at
> org.apache.flink.streaming.runtime.io.checkpointing.CheckpointedInputGate.handleEvent(CheckpointedInputGate.java:181)
> at
> org.apache.flink.streaming.runtime.io.checkpointing.CheckpointedInputGate.pollNext(CheckpointedInputGate.java:159)
> at
> org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:110)
> at
> org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:490)
> {code}
> This is caused by:
> # SingleCheckpointBarrierHandler first receive a BarrierAnnouncement, which
> would call checkNewCheckpoint to reset the currentCheckpointId and the
> alignedChannels.
> # SingleCheckpointBarrierHandler then received an EndOfPartition, which
> would add the channel to the alignedChannels.
> # SingleCheckpointBarrierHandler then received a barrier, which would found
> that the alignedChannels is already not empty, then it would skip the
> markAlignmentStart.
> We might change 3 to judge if this is the first barrier to receive.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-24068) CheckpointBarrierHandler may skip the markAlignmentStart for alignment -with-timeout checkpoint
[
https://issues.apache.org/jira/browse/FLINK-24068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski updated FLINK-24068:
---
Fix Version/s: 1.14.0
> CheckpointBarrierHandler may skip the markAlignmentStart for alignment
> -with-timeout checkpoint
> ---
>
> Key: FLINK-24068
> URL: https://issues.apache.org/jira/browse/FLINK-24068
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Checkpointing
>Affects Versions: 1.14.0
>Reporter: Yun Gao
>Priority: Major
> Fix For: 1.14.0
>
>
> {code:java}
> 04:51:11,727 [Flat Map -> Sink: Unnamed (10/12)#1] WARN
> org.apache.flink.runtime.taskmanager.Task [] - Flat Map -> Sink: Unnamed
> (10/12)#1 (0d965fd3c0de11dc7fb6793435cda8ba) switched from RUNNING to FAILED
> with failure cause: java.lang.IllegalStateException: Alignment time is less
> than zero({}). Is the time monotonic? [-9223369873401849363]
> at org.apache.flink.util.Preconditions.checkState(Preconditions.java:215)
> at
> org.apache.flink.streaming.runtime.io.checkpointing.CheckpointBarrierHandler.markAlignmentEnd(CheckpointBarrierHandler.java:181)
> at
> org.apache.flink.streaming.runtime.io.checkpointing.CheckpointBarrierHandler.markAlignmentEnd(CheckpointBarrierHandler.java:177)
> at
> org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.checkCheckpointAlignedAndTransformState(SingleCheckpointBarrierHandler.java:253)
> at
> org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.processBarrier(SingleCheckpointBarrierHandler.java:240)
> at
> org.apache.flink.streaming.runtime.io.checkpointing.CheckpointedInputGate.handleEvent(CheckpointedInputGate.java:181)
> at
> org.apache.flink.streaming.runtime.io.checkpointing.CheckpointedInputGate.pollNext(CheckpointedInputGate.java:159)
> at
> org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:110)
> at
> org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:490)
> {code}
> This is caused by:
> # SingleCheckpointBarrierHandler first receive a BarrierAnnouncement, which
> would call checkNewCheckpoint to reset the currentCheckpointId and the
> alignedChannels.
> # SingleCheckpointBarrierHandler then received an EndOfPartition, which
> would add the channel to the alignedChannels.
> # SingleCheckpointBarrierHandler then received a barrier, which would found
> that the alignedChannels is already not empty, then it would skip the
> markAlignmentStart.
> We might change 3 to judge if this is the first barrier to receive.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-24065) Upgrade the TwoPhaseCommitSink to support empty transaction after finished
[ https://issues.apache.org/jira/browse/FLINK-24065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Piotr Nowojski updated FLINK-24065: --- Priority: Critical (was: Major) > Upgrade the TwoPhaseCommitSink to support empty transaction after finished > -- > > Key: FLINK-24065 > URL: https://issues.apache.org/jira/browse/FLINK-24065 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Common, Runtime / Checkpointing >Affects Versions: 1.14.0 >Reporter: Yun Gao >Assignee: Yun Gao >Priority: Critical > > In https://issues.apache.org/jira/browse/FLINK-23473 for the > TwoPhaseCommitSink, we would not create new transactions after finished to > avoid we have transactions left after job finished. However, since with the > current implementation of the TwoPhaseCommitSink, we would have to write the > transactions into the state for each checkpoint, and the state does not > support null transaction now, thus there would be NullPointerException in > this case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-24065) Upgrade the TwoPhaseCommitSink to support empty transaction after finished
[ https://issues.apache.org/jira/browse/FLINK-24065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Piotr Nowojski updated FLINK-24065: --- Fix Version/s: 1.14.0 > Upgrade the TwoPhaseCommitSink to support empty transaction after finished > -- > > Key: FLINK-24065 > URL: https://issues.apache.org/jira/browse/FLINK-24065 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Common, Runtime / Checkpointing >Affects Versions: 1.14.0 >Reporter: Yun Gao >Assignee: Yun Gao >Priority: Critical > Fix For: 1.14.0 > > > In https://issues.apache.org/jira/browse/FLINK-23473 for the > TwoPhaseCommitSink, we would not create new transactions after finished to > avoid we have transactions left after job finished. However, since with the > current implementation of the TwoPhaseCommitSink, we would have to write the > transactions into the state for each checkpoint, and the state does not > support null transaction now, thus there would be NullPointerException in > this case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (FLINK-24035) Fix the deadlock issue caused by buffer listeners may not be notified
[ https://issues.apache.org/jira/browse/FLINK-24035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Piotr Nowojski reopened FLINK-24035: Re-opening as we want to refactor the code that we have just merged a bit. > Fix the deadlock issue caused by buffer listeners may not be notified > - > > Key: FLINK-24035 > URL: https://issues.apache.org/jira/browse/FLINK-24035 > Project: Flink > Issue Type: Bug > Components: Runtime / Network >Affects Versions: 1.14.0 >Reporter: Yingjie Cao >Assignee: Yingjie Cao >Priority: Critical > Labels: pull-request-available > Fix For: 1.14.0 > > > The buffer listeners are not notified when the the local buffer pool receives > available notification from the global pool. This may cause potential > deadlock issue: > # A LocalBufferPool is created, but there is no available buffers in the > global NetworkBufferPool. > # The LocalBufferPool registers an available buffer listener to the global > NetworkBufferPool. > # The BufferManager requests buffers from the LocalBufferPool but no buffer > is available. As a result, it registers an available buffer listener to the > LocalBufferPool. > # A buffer is recycled to the global pool and the local buffer pool is > notified about the available buffer. > # The local buffer pool requests the available buffer from the global pool > but the registered available buffer listener of BufferManager is not notified > and it can never get a chance to be notified so deadlock occurs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-24035) Fix the deadlock issue caused by buffer listeners may not be notified
[ https://issues.apache.org/jira/browse/FLINK-24035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17407139#comment-17407139 ] Piotr Nowojski commented on FLINK-24035: This is a similar problem as in FLINK-13203. We need to setup the input gates in a such way, that we are sure that they can always make progress. > Fix the deadlock issue caused by buffer listeners may not be notified > - > > Key: FLINK-24035 > URL: https://issues.apache.org/jira/browse/FLINK-24035 > Project: Flink > Issue Type: Bug > Components: Runtime / Network >Affects Versions: 1.14.0 >Reporter: Yingjie Cao >Assignee: Yingjie Cao >Priority: Critical > Labels: pull-request-available > Fix For: 1.14.0 > > > The buffer listeners are not notified when the the local buffer pool receives > available notification from the global pool. This may cause potential > deadlock issue: > # A LocalBufferPool is created, but there is no available buffers in the > global NetworkBufferPool. > # The LocalBufferPool registers an available buffer listener to the global > NetworkBufferPool. > # The BufferManager requests buffers from the LocalBufferPool but no buffer > is available. As a result, it registers an available buffer listener to the > LocalBufferPool. > # A buffer is recycled to the global pool and the local buffer pool is > notified about the available buffer. > # The local buffer pool requests the available buffer from the global pool > but the registered available buffer listener of BufferManager is not notified > and it can never get a chance to be notified so deadlock occurs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-24024) Fix syntax mistake in session Window TVF
[
https://issues.apache.org/jira/browse/FLINK-24024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-24024:
Fix Version/s: 1.15.0
1.14.0
> Fix syntax mistake in session Window TVF
> -
>
> Key: FLINK-24024
> URL: https://issues.apache.org/jira/browse/FLINK-24024
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.14.0
>Reporter: JING ZHANG
>Assignee: JING ZHANG
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.14.0, 1.15.0
>
>
> There is a syntax mistake in session Window TVF in FLINK-23543.
> For example, the following SQL has syntax mistake.
>
> {code:java}
> """
> |SELECT
> | a,
> | window_start,
> | window_end,
> | count(*),
> | sum(d),
> | max(d) filter (where b > 1000),
> | count(distinct c) AS uv
> |FROM TABLE(
> | SESSION(
> |TABLE MyTable,
> |DESCRIPTOR(proctime),
> |INTERVAL '5' MINUTE))
> |GROUP BY a, window_start, window_end
> """.stripMargin
> {code}
>
> It should updated to the following SQL, while partition key (a) should be
> moved into SESSION
> Window TVF based on Calcite [SESSION window
> TVF|https://calcite.apache.org/docs/reference.html#session].
>
> {code:java}
> val sql =
> """
> |SELECT
> | a,
> | window_start,
> | window_end,
> | count(*),
> | sum(d),
> | max(d) filter (where b > 1000),
> | count(distinct c) AS uv
> |FROM TABLE(
> | SESSION(
> |TABLE MyTable,
> |DESCRIPTOR(proctime),
> |DESCRIPTOR(a)
> |INTERVAL '5' MINUTE))
> |GROUP BY a, window_start, window_end
> """.stripMargin{code}
>
> To fix the bug, we only need update Session Window TVF syntax, we don't need
> update the operator part.
> Besides, we should check group keys of window aggregate should only contain
> window_start, window_end, partition_key. group keys could not contain other
> fields.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-23993) Describe eventually-consistency of materializing upserts
[ https://issues.apache.org/jira/browse/FLINK-23993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17407142#comment-17407142 ] Till Rohrmann commented on FLINK-23993: --- cc [~lzljs3620320] [~twalthr]. Is this something we want to do for the 1.14 release? > Describe eventually-consistency of materializing upserts > > > Key: FLINK-23993 > URL: https://issues.apache.org/jira/browse/FLINK-23993 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Table SQL / Ecosystem >Affects Versions: 1.14.0 >Reporter: Nico Kruber >Priority: Major > > FLINK-20374 added an upsert materialization operator which fixes the order of > shuffled streams. The results of this operator are actually _eventually > consistent_ (it collects the latest value it has seen and redacts older > versions when these are not valid anymore). You could see a result stream > like this, based on the order the materialization receives events: > +I10, -I10, +I5, -I5, +I10, -I10, +I3, -I3, +I10 > Each time, the value stored in Kafka would change until the "final" result is > in. > > It may be acceptable for upsert sinks, but should be documented (or > changed/fixed) nonetheless. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-23993) Describe eventually-consistency of materializing upserts
[ https://issues.apache.org/jira/browse/FLINK-23993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17407143#comment-17407143 ] Timo Walther commented on FLINK-23993: -- This issue is the result of a private discussion I had with Nico last week. In general, the entire changelog processor documentation needs a big overhaul. > Describe eventually-consistency of materializing upserts > > > Key: FLINK-23993 > URL: https://issues.apache.org/jira/browse/FLINK-23993 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Table SQL / Ecosystem >Affects Versions: 1.14.0 >Reporter: Nico Kruber >Priority: Major > > FLINK-20374 added an upsert materialization operator which fixes the order of > shuffled streams. The results of this operator are actually _eventually > consistent_ (it collects the latest value it has seen and redacts older > versions when these are not valid anymore). You could see a result stream > like this, based on the order the materialization receives events: > +I10, -I10, +I5, -I5, +I10, -I10, +I3, -I3, +I10 > Each time, the value stored in Kafka would change until the "final" result is > in. > > It may be acceptable for upsert sinks, but should be documented (or > changed/fixed) nonetheless. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #16773: [FLINK-21090][tests] Add IT case for stop-with-savepoint and final checkpoint
flinkbot edited a comment on pull request #16773: URL: https://github.com/apache/flink/pull/16773#issuecomment-896196766 ## CI report: * f11f93a01d1d0e769d04eb0b66a34914526096cc UNKNOWN * 61cc9c310c02dea756bc02436d7fab4654caaa1a UNKNOWN * 043df6eaf3e6e69b3871edea72291a9a2266bf3a UNKNOWN * 68c122bbf36516048c5b9a31759ba331c340b738 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23155) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-24038) DispatcherResourceManagerComponent fails to deregister application if no leading ResourceManager
[
https://issues.apache.org/jira/browse/FLINK-24038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17407145#comment-17407145
]
Xintong Song commented on FLINK-24038:
--
bq. For the new options #1 Xintong Song listed, do you mean let the leader
dispatcher do the deregistration? If it is, what will happen without leader.
Yes, I meant let the leading dispatcher to do the deregistration. According to
[~trohrmann], the decision whether to shut down the cluster or not is currently
made by the Dispatcher. IIUC, that means there should not be a shutdown without
a leading dispatcher, because there won't be any dispatcher to make that
decision without obtaining leadership.
> DispatcherResourceManagerComponent fails to deregister application if no
> leading ResourceManager
>
>
> Key: FLINK-24038
> URL: https://issues.apache.org/jira/browse/FLINK-24038
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.14.0
>Reporter: Till Rohrmann
>Priority: Critical
> Fix For: 1.14.0
>
>
> With FLINK-21667 we introduced a change that can cause the
> {{DispatcherResourceManagerComponent}} to fail when trying to stop the
> application. The problem is that the {{DispatcherResourceManagerComponent}}
> needs a leading {{ResourceManager}} to successfully execute the
> stop/deregister application call. If this is not the case, then it will fail
> fatally. In the case of multiple standby JobManager processes it can happen
> that the leading {{ResourceManager}} runs somewhere else.
> I do see two possible solutions:
> 1. Run the leader election process for the whole JobManager process
> 2. Move the registration/deregistration of the application out of the
> {{ResourceManager}} so that it can be executed w/o a leader
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #16966: [FLINK-23875][connectors/kafka] Snapshot reduceBuffer of ReducingUpse…
flinkbot edited a comment on pull request #16966: URL: https://github.com/apache/flink/pull/16966#issuecomment-904741081 ## CI report: * 0b2d69e248d26d8c96e93909b121c1dace533d3b Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23122) * 0b725628d29c9cab3dd35929bc3e10e0231920fd UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17000: [FLINK-23946][clients] Application mode should shutdown gracefully on…
flinkbot edited a comment on pull request #17000: URL: https://github.com/apache/flink/pull/17000#issuecomment-906394364 ## CI report: * d632973fab7c7235a18bece90835c3c469130b7c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23147) * 371e8744569d180669b538e27bcdd355914571a6 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17007: [FLINK-17495][metrics] Add additional variables config for metric reporters
flinkbot edited a comment on pull request #17007: URL: https://github.com/apache/flink/pull/17007#issuecomment-906871869 ## CI report: * cfa922e3bb72d929517b396481693aa1a01c7fe6 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22935) * ed8cc744f1416145c6dd46dd43edcf90e972fdf7 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17049: [FLINK-23971][tests] fix connector testing framework error when compare records in different splits
flinkbot edited a comment on pull request #17049: URL: https://github.com/apache/flink/pull/17049#issuecomment-908270735 ## CI report: * Unknown: [CANCELED](TBD) * 8e565dae5256ed75289f4ad53c11840889fdc5e4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23176) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17053: [FLINK-24021][HA]Handle curator framework start error by register UnhandledErrorListener before start zk client
flinkbot edited a comment on pull request #17053: URL: https://github.com/apache/flink/pull/17053#issuecomment-908359102 ## CI report: * da1ab056d42b37d5e36a3121a4827fc7200a6164 UNKNOWN * 2a176d1b90dc2c547287471d40d89c3f4000c7cf Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23157) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17062: [FLINK-21090][tests] Add IT case for stop-with-savepoint and final checkpoint (1.14)
flinkbot edited a comment on pull request #17062: URL: https://github.com/apache/flink/pull/17062#issuecomment-908868024 ## CI report: * 5f49d4c824acc8c2170abceeef45bd918f674b2c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23168) Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23158) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17063: [FLINK-24060][Tests]Move ZooKeeperUtilTest to right class
flinkbot edited a comment on pull request #17063: URL: https://github.com/apache/flink/pull/17063#issuecomment-908868069 ## CI report: * 07d4586f7d2427b0bb6181ca5b6122c0c44a4791 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23159) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17066: FLINK-24066
flinkbot commented on pull request #17066: URL: https://github.com/apache/flink/pull/17066#issuecomment-908973411 ## CI report: * c4c3e02c2c2122c90f71548caebfac6602592a1d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zhwangke commented on pull request #17025: [FLINK-24029][docs] Update doc example sql field in release-1.13.2 to dept_id
zhwangke commented on pull request #17025: URL: https://github.com/apache/flink/pull/17025#issuecomment-908973841 > Could you fix `DeptId` in line 108 as well? @wuchong done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-24070) Support different over window aggregates in streaming queries
zl created FLINK-24070:
--
Summary: Support different over window aggregates in streaming
queries
Key: FLINK-24070
URL: https://issues.apache.org/jira/browse/FLINK-24070
Project: Flink
Issue Type: Improvement
Components: Table SQL / Planner
Reporter: zl
Currently, Flink does not support using different over window aggregates in a
query statement, such as:
{code:java}
// code placeholder
select
f1, f2, f3,
sum(f3) over (PARTITION BY f1 ORDER BY f2 ROWS BETWEEN 2 PRECEDING AND
CURRENT ROW) as sum_1,
sum(f3) over (PARTITION BY f1 ORDER BY f2 ROWS BETWEEN 5 PRECEDING AND
CURRENT ROW) as sum_2
from table_a;{code}
However, this feature is commonly used in feature engineering for generating
some new features, do we have plan to support this?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-23764) add RuntimeContext to SourceReaderContext
[ https://issues.apache.org/jira/browse/FLINK-23764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17407150#comment-17407150 ] Hang Ruan commented on FLINK-23764: --- I think it could resolve this problem, and I don't have other information to add to _SourceReaderContext_ by now. > add RuntimeContext to SourceReaderContext > - > > Key: FLINK-23764 > URL: https://issues.apache.org/jira/browse/FLINK-23764 > Project: Flink > Issue Type: Improvement > Components: Connectors / Common >Reporter: Hang Ruan >Priority: Major > Labels: connector > > RuntimeContext is the important information for sourceReader. Not only the > subtask index, we sometimes need to get other information in RuntimeContext > like the operator id. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-24066) Provides a new stop entry for Kubernetes session mode
[
https://issues.apache.org/jira/browse/FLINK-24066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17407151#comment-17407151
]
Yang Wang commented on FLINK-24066:
---
Hmm. Thanks for creating this ticket.
AFAIK, the Yarn session cluster has the same behavior. And I do not feel any
concrete differences between them except for personal taste.
> Provides a new stop entry for Kubernetes session mode
> -
>
> Key: FLINK-24066
> URL: https://issues.apache.org/jira/browse/FLINK-24066
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / Kubernetes
>Reporter: liuzhuo
>Priority: Minor
> Labels: pull-request-available
>
> For the current Native Kubernetes session mode, the way to stop a session is：
> {code:java}
> # (3) Stop Kubernetes session by deleting cluster deployment
> $ kubectl delete deployment/my-first-flink-cluster
> {code}
> or
> {code:java}
> $ echo 'stop' | ./bin/kubernetes-session.sh \
> -Dkubernetes.cluster-id=my-first-flink-cluster \
> -Dexecution.attached=true
> {code}
> I think a more friendly interface should be added to stop the session mode,
> such as:
> {code:java}
> $ ./bin/kubernetes-session.sh stop
> -Dkubernetes.cluster-id=my-first-flink-cluster
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] Airblader commented on a change in pull request #17060: [FLINK-23895][table] Upsert materializer is not inserted for all sink providers
Airblader commented on a change in pull request #17060:
URL: https://github.com/apache/flink/pull/17060#discussion_r699048585
##
File path:
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/common/CommonExecSink.java
##
@@ -260,76 +191,135 @@ public DynamicTableSinkSpec getTableSinkSpec() {
* the parallelism is provided, otherwise it uses parallelism of input
transformation.
*/
private int deriveSinkParallelism(
-ParallelismProvider parallelismProvider, int inputParallelism) {
-final Optional parallelismOptional =
parallelismProvider.getParallelism();
-if (parallelismOptional.isPresent()) {
-int sinkParallelism = parallelismOptional.get();
-if (sinkParallelism <= 0) {
-throw new TableException(
-String.format(
-"Table: %s configured sink parallelism: "
-+ "%s should not be less than zero or
equal to zero",
-
tableSinkSpec.getObjectIdentifier().asSummaryString(),
-sinkParallelism));
-}
-return sinkParallelism;
-} else {
-// use input parallelism if not specified
+Transformation inputTransform, SinkRuntimeProvider
runtimeProvider) {
+final int inputParallelism = inputTransform.getParallelism();
+if (!(runtimeProvider instanceof ParallelismProvider)) {
return inputParallelism;
}
+final ParallelismProvider parallelismProvider = (ParallelismProvider)
runtimeProvider;
+return parallelismProvider
+.getParallelism()
+.map(
+sinkParallelism -> {
+if (sinkParallelism <= 0) {
+throw new TableException(
+String.format(
+"Invalid configured
parallelism %s for table '%s'.",
+sinkParallelism,
+tableSinkSpec
+.getObjectIdentifier()
+.asSummaryString()));
Review comment:
This is a complete nit, but I don't really understand why we're going
through the trouble of calling a method (rather than relying on toString) to
print a potentially incorrect representation rather than a properly escaped one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] lirui-apache closed pull request #16778: [FLINK-23715][parquet] Support for reading fields that do not exist in Parquet files
lirui-apache closed pull request #16778: URL: https://github.com/apache/flink/pull/16778 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-23715) Support for reading fields that do not exist in Parquet files
[ https://issues.apache.org/jira/browse/FLINK-23715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li resolved FLINK-23715. Fix Version/s: 1.15.0 Resolution: Fixed Pushed to master: 8abf8c627ca5ceab7a7e4f5b40f89b7ccceb9ceb > Support for reading fields that do not exist in Parquet files > - > > Key: FLINK-23715 > URL: https://issues.apache.org/jira/browse/FLINK-23715 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Reporter: tartarus >Assignee: tartarus >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > > In a production environment, it is often encountered that users add fields to > the hive table, but do not refresh the data of the historical partition. > Therefore, if the new field is not in the historical partition file, an error > will be reported when reading the historical partition. > General users would expect that if there is no such field, then fill in null > and return. > The current flink Parquet format does not support this function. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-23715) Support for reading fields that do not exist in Parquet files
[ https://issues.apache.org/jira/browse/FLINK-23715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li reassigned FLINK-23715: -- Assignee: tartarus > Support for reading fields that do not exist in Parquet files > - > > Key: FLINK-23715 > URL: https://issues.apache.org/jira/browse/FLINK-23715 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Reporter: tartarus >Assignee: tartarus >Priority: Major > Labels: pull-request-available > > In a production environment, it is often encountered that users add fields to > the hive table, but do not refresh the data of the historical partition. > Therefore, if the new field is not in the historical partition file, an error > will be reported when reading the historical partition. > General users would expect that if there is no such field, then fill in null > and return. > The current flink Parquet format does not support this function. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink-web] joemoe closed pull request #464: announcing the rc0 for 1.14. and asking for help
joemoe closed pull request #464: URL: https://github.com/apache/flink-web/pull/464 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16966: [FLINK-23875][connectors/kafka] Snapshot reduceBuffer of ReducingUpse…
flinkbot edited a comment on pull request #16966: URL: https://github.com/apache/flink/pull/16966#issuecomment-904741081 ## CI report: * 0b2d69e248d26d8c96e93909b121c1dace533d3b Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23122) * 0b725628d29c9cab3dd35929bc3e10e0231920fd Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23179) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17007: [FLINK-17495][metrics] Add additional variables config for metric reporters
flinkbot edited a comment on pull request #17007: URL: https://github.com/apache/flink/pull/17007#issuecomment-906871869 ## CI report: * cfa922e3bb72d929517b396481693aa1a01c7fe6 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22935) * ed8cc744f1416145c6dd46dd43edcf90e972fdf7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23181) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17000: [FLINK-23946][clients] Application mode should shutdown gracefully on…
flinkbot edited a comment on pull request #17000: URL: https://github.com/apache/flink/pull/17000#issuecomment-906394364 ## CI report: * d632973fab7c7235a18bece90835c3c469130b7c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23147) * 371e8744569d180669b538e27bcdd355914571a6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23180) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17025: [FLINK-24029][docs] Update doc example sql field in release-1.13.2 to dept_id
flinkbot edited a comment on pull request #17025: URL: https://github.com/apache/flink/pull/17025#issuecomment-907189639 ## CI report: * b86319824d4e8f11d8ad66658eb8297b3c12debf Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22967) * 20abcd8ee08baa3acb52d606bf1db76df6e9769e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17058: [FLINK-23664][docs] Add docs for KafkaSink
flinkbot edited a comment on pull request #17058: URL: https://github.com/apache/flink/pull/17058#issuecomment-908449514 ## CI report: * 6071b8b511f1a462cdf6d22bb45a2a8bf083cf18 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23131) * 28bf74c268622468560e20f5edcfdc728c1fef18 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17067: [FLINK-23817][docs] Add metric description to Kafka source and File sink.
flinkbot commented on pull request #17067: URL: https://github.com/apache/flink/pull/17067#issuecomment-908989247 ## CI report: * ac9ad3d39442f0479d1f74d4664c3351fc311a41 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17066: FLINK-24066
flinkbot edited a comment on pull request #17066: URL: https://github.com/apache/flink/pull/17066#issuecomment-908973411 ## CI report: * c4c3e02c2c2122c90f71548caebfac6602592a1d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23182) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-web] infoverload commented on a change in pull request #465: announcing the 1.14.0 release candidate
infoverload commented on a change in pull request #465: URL: https://github.com/apache/flink-web/pull/465#discussion_r699076774 ## File path: _posts/2021-08-31-release-1.14.0-rc0.md ## @@ -0,0 +1,39 @@ +--- +layout: post +title: "Help us stabilize Apache Flink 1.14.0 RC0" +date: 2021-08-31 00:00:00 +categories: news +authors: +- joemoe: + name: "Johannes Moser" + +--- + +Dear Flink Community, + +we are excited to announce the first release candidate of Apache Flink 1.14. 🎉 + +There are a lot of features and fixes - improving +the unified batch and streaming experience, increased the fault tolerance by +reducing the in-flight data, worked on connectors, as well as tidied some +components, just to mention some - that went into this release and it wouldn't +have been possible without your help. :) + +[Around 211 people]((https://github.com/apache/flink/graphs/contributors?from=2021-03-31&to=2021-08-31&type=c) ) have made contributions! +Two weeks ago (August 16th) we created a feature freeze. This means that only a Review comment: ```suggestion Two weeks ago (August 16th) we created a feature freeze. This means that only a ``` ## File path: _posts/2021-08-31-release-1.14.0-rc0.md ## @@ -0,0 +1,39 @@ +--- +layout: post +title: "Help us stabilize Apache Flink 1.14.0 RC0" +date: 2021-08-31 00:00:00 +categories: news +authors: +- joemoe: + name: "Johannes Moser" + +--- + +Dear Flink Community, + +we are excited to announce the first release candidate of Apache Flink 1.14. 🎉 + +There are a lot of features and fixes - improving +the unified batch and streaming experience, increased the fault tolerance by +reducing the in-flight data, worked on connectors, as well as tidied some +components, just to mention some - that went into this release and it wouldn't +have been possible without your help. :) Review comment: ```suggestion A lot of features and fixes went into this release, including improvements to the unified batch and streaming experience, an increase in fault tolerance by reducing in-flight data, more developments on connectors and components, and more! It wouldn't have been possible without your help. :) [Around 211 people]((https://github.com/apache/flink/graphs/contributors?from=2021-03-31&to=2021-08-31&type=c) ) have made contributions! ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-24071) Flink 1.12 HDFS sink have a problem,reported as OOM
zhuzhengjun created FLINK-24071: --- Summary: Flink 1.12 HDFS sink have a problem,reported as OOM Key: FLINK-24071 URL: https://issues.apache.org/jira/browse/FLINK-24071 Project: Flink Issue Type: Bug Affects Versions: 1.12.0 Reporter: zhuzhengjun org.apache.flink.runtime.JobException: Recovery is suppressed by NoRestartBackoffTimeStrategy at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:118) at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:80) at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:233) at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:224) at org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:215) at org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:669) at org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:89) at org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:449) at sun.reflect.GeneratedMethodAccessor144.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:305) at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:212) at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:77) at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:158) at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) at akka.actor.Actor$class.aroundReceive(Actor.scala:517) at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) at akka.actor.ActorCell.invoke(ActorCell.scala:561) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) at akka.dispatch.Mailbox.run(Mailbox.scala:225) at akka.dispatch.Mailbox.exec(Mailbox.scala:235) at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Caused by: java.lang.OutOfMemoryError: Java heap space at java.util.HashMap.entrySet(HashMap.java:1007) at com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:80) at com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:21) at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523) at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61) at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495) at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:599) at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.serialize(KryoSerializer.java:316) at org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer.serialize(StreamElementSerializer.java:168) at org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer.serialize(StreamElementSerializer.java:46) at org.apache.flink.runtime.plugable.SerializationDelegate.write(SerializationDelegate.java:54) at org.apache.flink.runtime.io.network.api.writer.RecordWriter.serializeRecord(RecordWriter.java:130) at org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:104) at org.apache.flink.runtime.io.network.api.writer.ChannelSelectorRecordWriter.emit(ChannelSelectorRecordWriter.java:54) at org.apache.flink.streaming.runtime.io.RecordWriterOutput.pushToRecordWriter(RecordWriterOutput.java:101) at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:87) at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:43) at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:50) at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:28) at org.apache.fl
[GitHub] [flink] ruanhang1993 commented on pull request #17049: [FLINK-23971][tests] fix connector testing framework error when compare records in different splits
ruanhang1993 commented on pull request #17049: URL: https://github.com/apache/flink/pull/17049#issuecomment-908991907 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-web] infoverload commented on a change in pull request #465: announcing the 1.14.0 release candidate
infoverload commented on a change in pull request #465: URL: https://github.com/apache/flink-web/pull/465#discussion_r699076319 ## File path: _posts/2021-08-31-release-1.14.0-rc0.md ## @@ -0,0 +1,39 @@ +--- +layout: post +title: "Help us stabilize Apache Flink 1.14.0 RC0" +date: 2021-08-31 00:00:00 +categories: news +authors: +- joemoe: + name: "Johannes Moser" + +--- + +Dear Flink Community, + +we are excited to announce the first release candidate of Apache Flink 1.14. 🎉 + +There are a lot of features and fixes - improving +the unified batch and streaming experience, increased the fault tolerance by +reducing the in-flight data, worked on connectors, as well as tidied some +components, just to mention some - that went into this release and it wouldn't +have been possible without your help. :) Review comment: ```suggestion A lot of features and fixes went into this release, including improvements to the unified batch and streaming experience, an increase in fault tolerance by reducing in-flight data, more developments on connectors and components. It wouldn't have been possible without your help. :) [Around 211 people]((https://github.com/apache/flink/graphs/contributors?from=2021-03-31&to=2021-08-31&type=c) ) have made contributions! ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-web] infoverload commented on a change in pull request #465: announcing the 1.14.0 release candidate
infoverload commented on a change in pull request #465: URL: https://github.com/apache/flink-web/pull/465#discussion_r699076319 ## File path: _posts/2021-08-31-release-1.14.0-rc0.md ## @@ -0,0 +1,39 @@ +--- +layout: post +title: "Help us stabilize Apache Flink 1.14.0 RC0" +date: 2021-08-31 00:00:00 +categories: news +authors: +- joemoe: + name: "Johannes Moser" + +--- + +Dear Flink Community, + +we are excited to announce the first release candidate of Apache Flink 1.14. 🎉 + +There are a lot of features and fixes - improving +the unified batch and streaming experience, increased the fault tolerance by +reducing the in-flight data, worked on connectors, as well as tidied some +components, just to mention some - that went into this release and it wouldn't +have been possible without your help. :) Review comment: ```suggestion A lot of features and fixes went into this release, including improvements to the unified batch and streaming experience, an increase in fault tolerance by reducing in-flight data, and more developments on connectors and components. It wouldn't have been possible without your help. :) [Around 211 people]((https://github.com/apache/flink/graphs/contributors?from=2021-03-31&to=2021-08-31&type=c) ) have made contributions! ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] twalthr closed pull request #17011: [FLINK-23663][table-planner] Reduce state size of ChangelogNormalize
twalthr closed pull request #17011: URL: https://github.com/apache/flink/pull/17011 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-24072) Add support for setting default headers in elasticsearch connector
jinfeng created FLINK-24072:
---
Summary: Add support for setting default headers in elasticsearch
connector
Key: FLINK-24072
URL: https://issues.apache.org/jira/browse/FLINK-24072
Project: Flink
Issue Type: Improvement
Components: Connectors / ElasticSearch
Reporter: jinfeng
If we add support for setting default headers , we can add some head options in
sql options.
The ddl would be like this.
{code:sql}
// Some comments here
create table es-sink (
a varchar,
b varchar
) with (
'connector' = 'elasticsearch-7',
'connection.default-headers' = 'Authorization:xxx'
);
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-23820) Test profiling for Python user-defined functions
[ https://issues.apache.org/jira/browse/FLINK-23820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu reassigned FLINK-23820: --- Assignee: JING ZHANG (was: Jing Zhang) > Test profiling for Python user-defined functions > > > Key: FLINK-23820 > URL: https://issues.apache.org/jira/browse/FLINK-23820 > Project: Flink > Issue Type: Improvement > Components: API / Python >Reporter: Huang Xingbo >Assignee: JING ZHANG >Priority: Blocker > Labels: release-testing > Fix For: 1.14.0 > > > The newly introduced profiling feature allows users to profiling their python > user-defined functions. > In order to test this new feature I recommend to follow the documentation[1] > [1] > https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/python/debugging/#profiling-python-udfs -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-24054) Let SinkUpsertMaterializer emit +U instead of only +I
[
https://issues.apache.org/jira/browse/FLINK-24054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17407171#comment-17407171
]
Kurt Young commented on FLINK-24054:
In most cases, +I and -D are easier to handle, and are equivalant with +U and -U
Is it worth to treat this as blocker bug? I'm not even sure if this is a bug.
> Let SinkUpsertMaterializer emit +U instead of only +I
> -
>
> Key: FLINK-24054
> URL: https://issues.apache.org/jira/browse/FLINK-24054
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Reporter: Timo Walther
>Assignee: Timo Walther
>Priority: Blocker
> Fix For: 1.14.0
>
>
> Currently, {{SinkUpsertMaterializer}} is not able to emit +U's but will
> always emit +I's. Thus, resulting changelogs are incorrect strictly speaking
> and only valid when treating +U and +I as similar changes in downstream
> operators.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] curcur commented on a change in pull request #16606: [FLINK-21357][runtime/statebackend]Periodic materialization for generalized incremental checkpoints
curcur commented on a change in pull request #16606:
URL: https://github.com/apache/flink/pull/16606#discussion_r699097517
##
File path:
flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/PeriodicMaterializer.java
##
@@ -0,0 +1,260 @@
+package org.apache.flink.state.changelog;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.core.fs.FileSystemSafetyNet;
+import org.apache.flink.runtime.checkpoint.CheckpointOptions;
+import org.apache.flink.runtime.checkpoint.CheckpointType;
+import org.apache.flink.runtime.mailbox.MailboxExecutor;
+import org.apache.flink.runtime.state.CheckpointStorageLocationReference;
+import org.apache.flink.runtime.state.CheckpointStorageWorkerView;
+import org.apache.flink.runtime.state.CheckpointStreamFactory;
+import org.apache.flink.runtime.state.KeyedStateHandle;
+import org.apache.flink.runtime.state.SnapshotResult;
+import org.apache.flink.runtime.state.StateObject;
+import org.apache.flink.runtime.state.changelog.SequenceNumber;
+import org.apache.flink.runtime.taskmanager.AsyncExceptionHandler;
+import org.apache.flink.util.concurrent.ExecutorThreadFactory;
+import org.apache.flink.util.concurrent.FutureUtils;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnull;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.RunnableFuture;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static java.util.Collections.emptyList;
+import static java.util.Collections.singletonList;
+
+public class PeriodicMaterializer {
+private static final Logger LOG =
LoggerFactory.getLogger(PeriodicMaterializer.class);
+
+/**
+ * ChangelogStateBackend only supports CheckpointType.CHECKPOINT; The rest
of information in
+ * CheckpointOptions is not used in Snapshotable#snapshot(). More details
in FLINK-23441.
+ */
+private static final CheckpointOptions CHECKPOINT_OPTIONS =
+new CheckpointOptions(
+CheckpointType.CHECKPOINT,
CheckpointStorageLocationReference.getDefault());
+
+/** task mailbox executor, execute from Task Thread. */
+private final MailboxExecutor mailboxExecutor;
+
+/** Async thread pool, to complete async phase of materialization. */
+private final ExecutorService asyncOperationsThreadPool;
+
+/** scheduled executor, periodically trigger materialization. */
+private final ScheduledExecutorService periodicExecutor;
+
+private final CheckpointStreamFactory streamFactory;
+
+private final AsyncExceptionHandler asyncExceptionHandler;
+
+private final String subtaskName;
+
+private final long periodicMaterializeDelay;
+
+/** Allowed number of consecutive materialization failures. */
+private final int allowedNumberOfFailures;
+
+/** Number of consecutive materialization failures. */
+private final AtomicInteger numberOfConsecutiveFailures;
+
+private long materializedId;
+
+PeriodicMaterializer(
+MailboxExecutor mailboxExecutor,
+ExecutorService asyncOperationsThreadPool,
+CheckpointStorageWorkerView checkpointStorageWorkerView,
+String subtaskName,
+AsyncExceptionHandler asyncExceptionHandler,
+long periodicMaterializeDelay,
+int allowedNumberOfFailures) {
+this.mailboxExecutor = mailboxExecutor;
+this.asyncOperationsThreadPool = asyncOperationsThreadPool;
+
+this.subtaskName = subtaskName;
+this.periodicMaterializeDelay = periodicMaterializeDelay;
+this.asyncExceptionHandler = asyncExceptionHandler;
+this.periodicExecutor =
+Executors.newSingleThreadScheduledExecutor(
+new ExecutorThreadFactory(
+"periodic-materialization-scheduler-" +
subtaskName));
+this.streamFactory = shared ->
checkpointStorageWorkerView.createTaskOwnedStateStream();
+this.allowedNumberOfFailures = allowedNumberOfFailures;
+this.numberOfConsecutiveFailures = new AtomicInteger(0);
+
+this.materializedId = 0;
+
+scheduleNextMaterialization();
+}
+
+@VisibleForTesting
+public void triggerMaterialization() {
+mailboxExecutor.execute(
+() -> {
+// synchronize phase
+SequenceNumber upTo =
+
stateChangelogWriter.lastAppendedSequenceNumber().next();
+
+if
(upTo.equals(changelogStateBackendState.lastMaterializedTo())) {
+
+scheduleNextMaterialization();
+
+LOG.info(
+"Task {} has no state updates since
[GitHub] [flink] fapaul commented on a change in pull request #17019: [FLINK-23854][connectors/Kafka] Ensure lingering Kafka transactions are aborted reliably.
fapaul commented on a change in pull request #17019:
URL: https://github.com/apache/flink/pull/17019#discussion_r699070362
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/PrioritizedOperatorSubtaskState.java
##
@@ -192,7 +196,17 @@
* snapshots.
*/
public boolean isRestored() {
-return restored;
+return restoredCheckpointId != null;
+}
+
+/**
+ * Returns non-empty if this was created for a restored operator, false
otherwise. Restored
Review comment:
Update the comment because it is not optional anymore?
##
File path:
flink-connectors/flink-connector-kafka/src/test/java/org/apache/flink/connector/kafka/sink/KafkaSinkITCase.java
##
@@ -334,9 +348,9 @@ private void writeRecordsToKafka(
drainAllRecordsFromTopic(topic);
final long recordsCount = expectedRecords.get().get();
assertEquals(collectedRecords.size(), recordsCount);
-assertEquals(
+assertThat(
deserializeValues(collectedRecords),
-LongStream.range(1, recordsCount +
1).boxed().collect(Collectors.toList()));
+containsInAnyOrder(LongStream.range(1, recordsCount +
1).boxed().toArray()));
Review comment:
order?
##
File path:
flink-connectors/flink-connector-kafka/src/test/java/org/apache/flink/connector/kafka/sink/KafkaSinkITCase.java
##
@@ -226,11 +236,12 @@ public void
testAbortTransactionsOfPendingCheckpointsAfterFailure() throws Excep
new FailingCheckpointMapper(failed, lastCheckpointedRecord),
config, "newPrefix");
final List> collectedRecords =
drainAllRecordsFromTopic(topic);
-assertEquals(
+assertThat(
deserializeValues(collectedRecords),
-LongStream.range(1, lastCheckpointedRecord.get().get() + 1)
-.boxed()
-.collect(Collectors.toList()));
+containsInAnyOrder(
Review comment:
Shouldn't the order be guaranteed?
##
File path:
flink-connectors/flink-connector-kafka/src/test/java/org/apache/flink/connector/kafka/sink/KafkaSinkITCase.java
##
@@ -254,11 +265,12 @@ public void
testAbortTransactionsAfterScaleInBeforeFirstCheckpoint() throws Exce
new FailingCheckpointMapper(failed, lastCheckpointedRecord),
config, null);
final List> collectedRecords =
drainAllRecordsFromTopic(topic);
-assertEquals(
+assertThat(
deserializeValues(collectedRecords),
-LongStream.range(1, lastCheckpointedRecord.get().get() + 1)
-.boxed()
-.collect(Collectors.toList()));
+containsInAnyOrder(
Review comment:
Same here what about the order?
##
File path:
flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/sink/TransactionAborter.java
##
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.kafka.sink;
+
+import javax.annotation.Nullable;
+
+import java.util.List;
+import java.util.function.Function;
+
+/**
+ * Aborts lingering transactions on restart.
+ *
+ * Transactions are lingering if they are not tracked anywhere. For
example, if a job is started
+ * transactions are opened. A restart without checkpoint would not allow Flink
to abort old
+ * transactions. Since Kafka's transactions are sequential, newly produced
data does not become
+ * visible for read_committed consumers. However, Kafka has no API for
querying open transactions,
+ * so they become lingering.
+ *
+ * Flink solves this by assuming consecutive transaction ids. On restart of
checkpoint C on
+ * subtask S, it will sequentially cancel transaction C+1, C+2, ... of S until
it finds the first
+ * unused transaction.
+ *
+ * Additionally, to cover for weird downscaling cases without checkpoints,
it also checks for
+ * transactions of subtask S+P where P is the current parallelism until it
finds a subtask without
+ * transactions.
+ */
+class TransactionAborter {
+
[GitHub] [flink] flinkbot edited a comment on pull request #16606: [FLINK-21357][runtime/statebackend]Periodic materialization for generalized incremental checkpoints
flinkbot edited a comment on pull request #16606: URL: https://github.com/apache/flink/pull/16606#issuecomment-887431748 ## CI report: * 264be5cc6a0485171413099e8b64b9e917d06e85 UNKNOWN * 1b7da8565a2ab9560f1aad65007930c91945087f UNKNOWN * 5df5c7c97639f3c6d2e52a1b182253e4c32d8373 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22448) * 3421b81c2502f61112bd131a7336c16e3dd30925 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16988: [FLINK-23458][docs] Added the network buffer documentation along wit…
flinkbot edited a comment on pull request #16988: URL: https://github.com/apache/flink/pull/16988#issuecomment-905730008 ## CI report: * ed7683ef12ccef4aa05782a762ff5c4a61535053 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23139) * 680f8172bea8a9b7396604e8f0c5dcaf4efff5f9 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17025: [FLINK-24029][docs] Update doc example sql field in release-1.13.2 to dept_id
flinkbot edited a comment on pull request #17025: URL: https://github.com/apache/flink/pull/17025#issuecomment-907189639 ## CI report: * 20abcd8ee08baa3acb52d606bf1db76df6e9769e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23184) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17049: [FLINK-23971][tests] fix connector testing framework error when compare records in different splits
flinkbot edited a comment on pull request #17049: URL: https://github.com/apache/flink/pull/17049#issuecomment-908270735 ## CI report: * Unknown: [CANCELED](TBD) * 8e565dae5256ed75289f4ad53c11840889fdc5e4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23176) * daad66852e5f50e47b36652f0207a63f3096c125 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17067: [FLINK-23817][docs] Add metric description to Kafka source and File sink.
flinkbot edited a comment on pull request #17067: URL: https://github.com/apache/flink/pull/17067#issuecomment-908989247 ## CI report: * ac9ad3d39442f0479d1f74d4664c3351fc311a41 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23186) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17058: [FLINK-23664][docs] Add docs for KafkaSink
flinkbot edited a comment on pull request #17058: URL: https://github.com/apache/flink/pull/17058#issuecomment-908449514 ## CI report: * 6071b8b511f1a462cdf6d22bb45a2a8bf083cf18 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23131) * 28bf74c268622468560e20f5edcfdc728c1fef18 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23185) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] Aitozi commented on a change in pull request #17053: [FLINK-24021][HA]Handle curator framework start error by register UnhandledErrorListener before start zk client
Aitozi commented on a change in pull request #17053:
URL: https://github.com/apache/flink/pull/17053#discussion_r699104914
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/util/ZooKeeperUtils.java
##
@@ -226,9 +230,21 @@ public static CuratorFramework
startCuratorFramework(Configuration configuration
}
CuratorFramework cf = curatorFrameworkBuilder.build();
-
+// This handler is only used to handle the error during start phase,
and should be
+// removed after start curator success.
+UnhandledErrorListener unhandledErrorListener =
+(message, throwable) -> {
+LOG.error(
+"Exiting process for unhandled error in start
curator framework, "
++ "error message: {}, exiting code: {}",
+message,
+ZOOKEEPER_FAILURE_EXIT_CODE,
+throwable);
+System.exit(ZOOKEEPER_FAILURE_EXIT_CODE);
Review comment:
Now I just call `System.exit()` here. Another option I think we can
throw exception here, let it be handled in entryPoint. What's your idea?
@tillrohrmann
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-23544) Window TVF Supports session window
[ https://issues.apache.org/jira/browse/FLINK-23544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17407174#comment-17407174 ] Jark Wu commented on FLINK-23544: - I guess the reason maybe PARTITIONED BY clause of TVF syntax is not supported in Calcite parser. If we want to support syntax mentioned in FLIP, we need to wait for CALCITE-4337, but this ticket doesn't have much progress in Calcite community and this means it may takes long to deliver SESSION Window TVF feature for Flink users. But I agree [~twalthr] the API must be consistent with FLIP, otherwise, there should be another public discussion, and we need to come up with a solution for API compatibility/migration. > Window TVF Supports session window > -- > > Key: FLINK-23544 > URL: https://issues.apache.org/jira/browse/FLINK-23544 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Reporter: JING ZHANG >Assignee: JING ZHANG >Priority: Minor > Labels: pull-request-available > Fix For: 1.14.0 > > > Window TVF would support SESSION window in the following case: > # SESSION Window TVF followed by Window Aggregate, in this case SESSION > window TVF would be pulled up into WindowAggregate, so Window assigner would > happen in WindowAggrgeate > > *Note, SESSION window TVF only works in limited cases currently, the > following user cases is not supported yet:* > # *SESSION WINDOW TVF followed by Window JOIN* > # *SESSION WINDOW TVF followed by Window RANK*** > *BESIDES, SESSION window Aggregate does not support the following performance > improvement yet:* > 1. Split Distinct Aggregation > 2. Local-global Aggregation > 3. Mini-batch Aggregate -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] pnowojski commented on a change in pull request #16988: [FLINK-23458][docs] Added the network buffer documentation along wit…
pnowojski commented on a change in pull request #16988:
URL: https://github.com/apache/flink/pull/16988#discussion_r699022660
##
File path: docs/content/docs/deployment/network_buffer.md
##
@@ -0,0 +1,125 @@
+---
+title: "Network Tuning"
+weight: 100
+type: docs
+aliases:
+ - /deployment/network_buffer.html
+---
+
+
+# Network buffer
+
+## Overview
+
+Each record in flink is sent to the next subtask not individually but
compounded in Network buffer,
+the smallest unit for communication between subtasks. Also, in order to keep
consistent high throughput,
+Flink uses the network buffer queues (so called in-flight data) both on the
output as well as on the input side.
+In the result each subtask have an input queue waiting for the consumption and
an output queue
+waiting for sending to the next subtask. Having a larger amount of the
in-flight data means Flink can provide a
+higher throughput that's more resilient to small hiccups in the pipeline but
it has negative effect for the
+checkpoint time. The long checkpoint time issue can be caused by many things,
one of those is checkpoint barriers
Review comment:
nit: brake a paragraph before "The long checkpoint time"?
##
File path: docs/content/docs/deployment/network_buffer.md
##
@@ -0,0 +1,125 @@
+---
+title: "Network Tuning"
+weight: 100
+type: docs
+aliases:
+ - /deployment/network_buffer.html
+---
+
+
+# Network buffer
+
+## Overview
+
+Each record in flink is sent to the next subtask not individually but
compounded in Network buffer,
+the smallest unit for communication between subtasks. Also, in order to keep
consistent high throughput,
+Flink uses the network buffer queues (so called in-flight data) both on the
output as well as on the input side.
+In the result each subtask have an input queue waiting for the consumption and
an output queue
+waiting for sending to the next subtask. Having a larger amount of the
in-flight data means Flink can provide a
+higher throughput that's more resilient to small hiccups in the pipeline but
it has negative effect for the
+checkpoint time. The long checkpoint time issue can be caused by many things,
one of those is checkpoint barriers
+propagation time. Checkpoint in Flink can finish only once all subtask
receives all injected checkpoint
+barriers. In aligned checkpoints([see]{{< ref
"docs/concepts/stateful-stream-processing" >}}#checkpointing)
+those checkpoint barriers are traveling throughout the job graph long along
+the network buffers and the larger amount of in-flight data the longer the
checkpoint barrier propagation
+time. In unaligned checkpoints(([see]{{< ref
"docs/concepts/stateful-stream-processing" >}}#unaligned-checkpointing)) on the
other hand, the more in-flight data, the larger the checkpoint size as
+all of the captured in-flight data has to be persisted as part of the
checkpoint.
+
+## Buffer debloat
+
+Historically the only way to configure the amount of in-flight data was to
specify both amount and the size
+of the buffers. However ideal values for those numbers are hard to pick, as
they are different for every
+deployment. The buffer debloating mechanism added in Flink 1.14 attempts to
address this issue.
+It tries to automatically adjust the amount of in-flight data in order to a
reasonable values.
+More precisely, the buffer debloating calculate the maximum possible throughput
+(the maximum throughput which would be if the subtask was always busy)
+for the subtask and adjusts the amount of in-flight data in such a way that
the time for consumption of those in-flight data will be equal to the
configured value.
+
+The most useful settings:
Review comment:
```suggestion
The most important settings:
```
##
File path: docs/content/docs/deployment/network_buffer.md
##
@@ -0,0 +1,125 @@
+---
+title: "Network Tuning"
+weight: 100
+type: docs
+aliases:
+ - /deployment/network_buffer.html
+---
+
+
+# Network buffer
+
+## Overview
+
+Each record in flink is sent to the next subtask not individually but
compounded in Network buffer,
+the smallest unit for communication between subtasks. Also, in order to keep
consistent high throughput,
+Flink uses the network buffer queues (so called in-flight data) both on the
output as well as on the input side.
+In the result each subtask have an input queue waiting for the consumption and
an output queue
+waiting for sending to the next subtask. Having a larger amount of the
in-flight data means Flink can provide a
+higher throughput that's more resilient to small hiccups in the pipeline but
it has negative effect for the
+checkpoint time. The long checkpoint time issue can be caused by many things,
one of those is checkpoint barriers
+propagation time. Checkpoint in Flink can finish only once all subtask
receives all injected checkpoint
+barriers. In aligned checkpoints([see]{{< ref
"docs/concepts/stateful-stream-processing" >}}#checkpointing)
+thos
[jira] [Closed] (FLINK-23663) Reduce state size in ChangelogNormalize through filter push down
[
https://issues.apache.org/jira/browse/FLINK-23663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timo Walther closed FLINK-23663.
Fix Version/s: 1.15.0
1.14.0
Resolution: Fixed
Fixed in master:
commit c1bf5da8696b0369a3475d9cda355d149c652cff
[table-planner] Push primary key filters through ChangelogNormalize
commit bc69c1c77ec94a57b3603c5eefcbddb8ce83ea64
[table-planner] Introduce TableFactoryHarness
Fixed in 1.14:
commit 5c96e5b985ecce6b39bb3913470f11a8e3a2ad83
[table-planner] Push primary key filters through ChangelogNormalize
commit d49eb8c22be4084fea318d1701abbbc23db72c4d
[table-planner] Introduce TableFactoryHarness
> Reduce state size in ChangelogNormalize through filter push down
>
>
> Key: FLINK-23663
> URL: https://issues.apache.org/jira/browse/FLINK-23663
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Planner
>Reporter: Timo Walther
>Assignee: Ingo Bürk
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.14.0, 1.15.0
>
>
> {{ChangelogNormalize}} is an expensive stateful operation as it stores data
> for each key.
> Filters are generally not pushed through a ChangelogNormalize node which
> means that users have no possibility to at least limit the key space. Pushing
> filters like {{a < 10}} into a source like {{upsert-kafka}} that is emitting
> {{+I[key1, a=9]}} and {{-D[key1, a=10]}}, is problematic as the deletion will
> be filtered and leads to wrong results. But limiting the filter push down to
> key space should be safe.
> Furthermore, it seems the current implementation is also wrong as it pushes
> any kind of filter through {{ChangelogNormalize}} but only if the source
> implements filter push down.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-24071) Flink 1.12 HDFS sink have a problem,reported as OOM
[
https://issues.apache.org/jira/browse/FLINK-24071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-24071:
Description:
{code}
org.apache.flink.runtime.JobException: Recovery is suppressed by
NoRestartBackoffTimeStrategy at
org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:118)
at
org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:80)
at
org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:233)
at
org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:224)
at
org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:215)
at
org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:669)
at
org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:89)
at
org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:449)
at sun.reflect.GeneratedMethodAccessor144.invoke(Unknown Source) at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:305)
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:212)
at
org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:77)
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:158)
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) at
akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) at
scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) at
akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) at
scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) at
scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) at
scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) at
akka.actor.Actor$class.aroundReceive(Actor.scala:517) at
akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) at
akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) at
akka.actor.ActorCell.invoke(ActorCell.scala:561) at
akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) at
akka.dispatch.Mailbox.run(Mailbox.scala:225) at
akka.dispatch.Mailbox.exec(Mailbox.scala:235) at
akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at
akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at
akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.OutOfMemoryError: Java heap space at
java.util.HashMap.entrySet(HashMap.java:1007) at
com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:80)
at
com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:21)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523) at
com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61) at
com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:599) at
org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.serialize(KryoSerializer.java:316)
at
org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer.serialize(StreamElementSerializer.java:168)
at
org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer.serialize(StreamElementSerializer.java:46)
at
org.apache.flink.runtime.plugable.SerializationDelegate.write(SerializationDelegate.java:54)
at
org.apache.flink.runtime.io.network.api.writer.RecordWriter.serializeRecord(RecordWriter.java:130)
at
org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:104)
at
org.apache.flink.runtime.io.network.api.writer.ChannelSelectorRecordWriter.emit(ChannelSelectorRecordWriter.java:54)
at
org.apache.flink.streaming.runtime.io.RecordWriterOutput.pushToRecordWriter(RecordWriterOutput.java:101)
at
org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:87)
at
org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:43)
at
org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:50)
at
org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:28)
at
org.apache.flink.streaming.runtime.operators.TimestampsAndWatermarksOperator.processElement(TimestampsAndWatermarksOperator.java:104)
at
org.apache.flink.streaming.runti
[jira] [Updated] (FLINK-24071) Flink 1.12 HDFS sink have a problem,reported as OOM
[
https://issues.apache.org/jira/browse/FLINK-24071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-24071:
Component/s: Connectors / FileSystem
> Flink 1.12 HDFS sink have a problem,reported as OOM
> --
>
> Key: FLINK-24071
> URL: https://issues.apache.org/jira/browse/FLINK-24071
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem
>Affects Versions: 1.12.0
>Reporter: zhuzhengjun
>Priority: Major
>
> {code}
> org.apache.flink.runtime.JobException: Recovery is suppressed by
> NoRestartBackoffTimeStrategy at
> org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:118)
> at
> org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:80)
> at
> org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:233)
> at
> org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:224)
> at
> org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:215)
> at
> org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:669)
> at
> org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:89)
> at
> org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:449)
> at sun.reflect.GeneratedMethodAccessor144.invoke(Unknown Source) at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:305)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:212)
> at
> org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:77)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:158)
> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) at
> akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) at
> scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) at
> akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) at
> scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) at
> scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) at
> scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) at
> akka.actor.Actor$class.aroundReceive(Actor.scala:517) at
> akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) at
> akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) at
> akka.actor.ActorCell.invoke(ActorCell.scala:561) at
> akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) at
> akka.dispatch.Mailbox.run(Mailbox.scala:225) at
> akka.dispatch.Mailbox.exec(Mailbox.scala:235) at
> akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at
> akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at
> akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> Caused by: java.lang.OutOfMemoryError: Java heap space at
> java.util.HashMap.entrySet(HashMap.java:1007) at
> com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:80)
> at
> com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:21)
> at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523) at
> com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
> at
> com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:599) at
> org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.serialize(KryoSerializer.java:316)
> at
> org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer.serialize(StreamElementSerializer.java:168)
> at
> org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer.serialize(StreamElementSerializer.java:46)
> at
> org.apache.flink.runtime.plugable.SerializationDelegate.write(SerializationDelegate.java:54)
> at
> org.apache.flink.runtime.io.network.api.writer.RecordWriter.serializeRecord(RecordWriter.java:130)
> at
> org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:104)
> at
> org.apache.flink.runtime.io.network.api.writer.ChannelSelectorRecordWriter.emit(ChannelSelectorRecordWriter.java:54)
> at
> org.apache.flink.streaming.runtime.io.RecordWriterOutput.pushToRecordWrite
[jira] [Commented] (FLINK-23993) Describe eventually-consistency of materializing upserts
[ https://issues.apache.org/jira/browse/FLINK-23993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17407180#comment-17407180 ] Jingsong Lee commented on FLINK-23993: -- I think we can not finish this in 1.14 release. Actually, we can't solve all the disorder problems. Our sink can freely define Primary key. There are many cases, the data seen by sink are disordered in a sense, some are acceptable and some are unacceptable. This is a complex mechanism (It may also involve the concept of dynamic table), and we need to create a detailed explanation documentation. (Actually, FLINK-20374 just fixed the more common cases, for upsert sink, sink just do what it should do) > Describe eventually-consistency of materializing upserts > > > Key: FLINK-23993 > URL: https://issues.apache.org/jira/browse/FLINK-23993 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Table SQL / Ecosystem >Affects Versions: 1.14.0 >Reporter: Nico Kruber >Priority: Major > > FLINK-20374 added an upsert materialization operator which fixes the order of > shuffled streams. The results of this operator are actually _eventually > consistent_ (it collects the latest value it has seen and redacts older > versions when these are not valid anymore). You could see a result stream > like this, based on the order the materialization receives events: > +I10, -I10, +I5, -I5, +I10, -I10, +I3, -I3, +I10 > Each time, the value stored in Kafka would change until the "final" result is > in. > > It may be acceptable for upsert sinks, but should be documented (or > changed/fixed) nonetheless. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-24073) Statefun 2.2 documentation build fails on buildbot
Chesnay Schepler created FLINK-24073:
Summary: Statefun 2.2 documentation build fails on buildbot
Key: FLINK-24073
URL: https://issues.apache.org/jira/browse/FLINK-24073
Project: Flink
Issue Type: Bug
Components: Documentation, Stateful Functions
Reporter: Chesnay Schepler
https://ci2.apache.org/#/builders/10/builds/51
{{/bin/sh: 14: rvm: not found}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-24074) Check 1.14 licensing
Xintong Song created FLINK-24074: Summary: Check 1.14 licensing Key: FLINK-24074 URL: https://issues.apache.org/jira/browse/FLINK-24074 Project: Flink Issue Type: Technical Debt Reporter: Xintong Song Assignee: Xintong Song Fix For: 1.14.0 Check the licensing of the 1.14 release. https://cwiki.apache.org/confluence/display/FLINK/Licensing -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #16606: [FLINK-21357][runtime/statebackend]Periodic materialization for generalized incremental checkpoints
flinkbot edited a comment on pull request #16606: URL: https://github.com/apache/flink/pull/16606#issuecomment-887431748 ## CI report: * 264be5cc6a0485171413099e8b64b9e917d06e85 UNKNOWN * 1b7da8565a2ab9560f1aad65007930c91945087f UNKNOWN * 5df5c7c97639f3c6d2e52a1b182253e4c32d8373 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22448) * 3421b81c2502f61112bd131a7336c16e3dd30925 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23189) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16723: [FLINK-22885][table] Supports 'SHOW COLUMNS' syntax.
flinkbot edited a comment on pull request #16723: URL: https://github.com/apache/flink/pull/16723#issuecomment-893364145 ## CI report: * 1de6e0b83ee64c22c861a9d8e639483aaaba9e7f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23078) * ce83c52813ef9b754a0aded512e80503507b1547 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16988: [FLINK-23458][docs] Added the network buffer documentation along wit…
flinkbot edited a comment on pull request #16988: URL: https://github.com/apache/flink/pull/16988#issuecomment-905730008 ## CI report: * ed7683ef12ccef4aa05782a762ff5c4a61535053 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23139) * 680f8172bea8a9b7396604e8f0c5dcaf4efff5f9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23190) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17049: [FLINK-23971][tests] fix connector testing framework error when compare records in different splits
flinkbot edited a comment on pull request #17049: URL: https://github.com/apache/flink/pull/17049#issuecomment-908270735 ## CI report: * Unknown: [CANCELED](TBD) * 8e565dae5256ed75289f4ad53c11840889fdc5e4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23176) * daad66852e5f50e47b36652f0207a63f3096c125 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23191) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17064: [FLINK-24059][Connectors/Common] Allow SourceReaderTestBase.NUM_SPLITS to be overridden
flinkbot edited a comment on pull request #17064: URL: https://github.com/apache/flink/pull/17064#issuecomment-908893945 ## CI report: * 167a600021ec50d0fa82e8806593391fe57a545a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23166) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17067: [FLINK-23817][docs] Add metric description to Kafka source and File sink.
flinkbot edited a comment on pull request #17067: URL: https://github.com/apache/flink/pull/17067#issuecomment-908989247 ## CI report: * ac9ad3d39442f0479d1f74d4664c3351fc311a41 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23186) * b673222ab13a56c49d31c804d3bde5f8f69c440c UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-24075) Apache Flink 1.13.1 Docker images are published for linux/arm
[
https://issues.apache.org/jira/browse/FLINK-24075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler reassigned FLINK-24075:
Assignee: Chesnay Schepler
> Apache Flink 1.13.1 Docker images are published for linux/arm
> -
>
> Key: FLINK-24075
> URL: https://issues.apache.org/jira/browse/FLINK-24075
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.13.1
>Reporter: Matthias
>Assignee: Chesnay Schepler
>Priority: Major
>
> The [Apache Flink 1.13.1 Docker
> images|https://hub.docker.com/r/apache/flink/tags?page=1&ordering=last_updated&name=1.13.1]
> are published for {{linux/arm}} instead of {{linux/amd64}}. This causes
> issues for users pulling these images resulting in a warning:
> {code}
> WARNING: The requested image's platform (linux/arm64/v8) does not match the
> detected host platform (linux/amd64) and no specific platform was requested
> ERROR: ld.so: object '/usr/lib/x86_64-linux-gnu/libjemalloc.so' from
> LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
> [...]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-24075) Apache Flink 1.13.1 Docker images are published for linux/arm
Matthias created FLINK-24075:
Summary: Apache Flink 1.13.1 Docker images are published for
linux/arm
Key: FLINK-24075
URL: https://issues.apache.org/jira/browse/FLINK-24075
Project: Flink
Issue Type: Bug
Affects Versions: 1.13.1
Reporter: Matthias
The [Apache Flink 1.13.1 Docker
images|https://hub.docker.com/r/apache/flink/tags?page=1&ordering=last_updated&name=1.13.1]
are published for {{linux/arm}} instead of {{linux/amd64}}. This causes issues
for users pulling these images resulting in a warning:
{code}
WARNING: The requested image's platform (linux/arm64/v8) does not match the
detected host platform (linux/amd64) and no specific platform was requested
ERROR: ld.so: object '/usr/lib/x86_64-linux-gnu/libjemalloc.so' from LD_PRELOAD
cannot be preloaded (cannot open shared object file): ignored.
[...]
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] twalthr commented on a change in pull request #17060: [FLINK-23895][table] Upsert materializer is not inserted for all sink providers
twalthr commented on a change in pull request #17060:
URL: https://github.com/apache/flink/pull/17060#discussion_r699129141
##
File path:
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/common/CommonExecSink.java
##
@@ -260,76 +191,135 @@ public DynamicTableSinkSpec getTableSinkSpec() {
* the parallelism is provided, otherwise it uses parallelism of input
transformation.
*/
private int deriveSinkParallelism(
-ParallelismProvider parallelismProvider, int inputParallelism) {
-final Optional parallelismOptional =
parallelismProvider.getParallelism();
-if (parallelismOptional.isPresent()) {
-int sinkParallelism = parallelismOptional.get();
-if (sinkParallelism <= 0) {
-throw new TableException(
-String.format(
-"Table: %s configured sink parallelism: "
-+ "%s should not be less than zero or
equal to zero",
-
tableSinkSpec.getObjectIdentifier().asSummaryString(),
-sinkParallelism));
-}
-return sinkParallelism;
-} else {
-// use input parallelism if not specified
+Transformation inputTransform, SinkRuntimeProvider
runtimeProvider) {
+final int inputParallelism = inputTransform.getParallelism();
+if (!(runtimeProvider instanceof ParallelismProvider)) {
return inputParallelism;
}
+final ParallelismProvider parallelismProvider = (ParallelismProvider)
runtimeProvider;
+return parallelismProvider
+.getParallelism()
+.map(
+sinkParallelism -> {
+if (sinkParallelism <= 0) {
+throw new TableException(
+String.format(
+"Invalid configured
parallelism %s for table '%s'.",
+sinkParallelism,
+tableSinkSpec
+.getObjectIdentifier()
+.asSummaryString()));
Review comment:
Let's not start this discussion now. It seems the backticks are hard to
read in strings. We could have used `asSummaryString` in `toString` but now it
is kind of too late I guess.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-22889) JdbcExactlyOnceSinkE2eTest.testInsert hangs on azure
[ https://issues.apache.org/jira/browse/FLINK-22889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17407191#comment-17407191 ] Xintong Song commented on FLINK-22889: -- Reached out to [~roman] offline. We are downgrading this to Critical for the moment. As the Postgres issue being fixed this is expected to fail less frequently (due to MySQL issues). We can upgrade this again if more instances are reported. > JdbcExactlyOnceSinkE2eTest.testInsert hangs on azure > > > Key: FLINK-22889 > URL: https://issues.apache.org/jira/browse/FLINK-22889 > Project: Flink > Issue Type: Bug > Components: Connectors / JDBC >Affects Versions: 1.14.0, 1.13.1 >Reporter: Dawid Wysakowicz >Assignee: Roman Khachatryan >Priority: Critical > Labels: pull-request-available, test-stability > Fix For: 1.14.0 > > > https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=18690&view=logs&j=ba53eb01-1462-56a3-8e98-0dd97fbcaab5&t=bfbc6239-57a0-5db0-63f3-41551b4f7d51&l=16658 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-22889) JdbcExactlyOnceSinkE2eTest.testInsert hangs on azure
[ https://issues.apache.org/jira/browse/FLINK-22889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xintong Song updated FLINK-22889: - Priority: Critical (was: Blocker) > JdbcExactlyOnceSinkE2eTest.testInsert hangs on azure > > > Key: FLINK-22889 > URL: https://issues.apache.org/jira/browse/FLINK-22889 > Project: Flink > Issue Type: Bug > Components: Connectors / JDBC >Affects Versions: 1.14.0, 1.13.1 >Reporter: Dawid Wysakowicz >Assignee: Roman Khachatryan >Priority: Critical > Labels: pull-request-available, test-stability > Fix For: 1.14.0 > > > https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=18690&view=logs&j=ba53eb01-1462-56a3-8e98-0dd97fbcaab5&t=bfbc6239-57a0-5db0-63f3-41551b4f7d51&l=16658 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] fapaul commented on a change in pull request #17067: [FLINK-23817][docs] Add metric description to Kafka source and File sink.
fapaul commented on a change in pull request #17067:
URL: https://github.com/apache/flink/pull/17067#discussion_r699130795
##
File path: docs/content/docs/connectors/datastream/streamfile_sink.md
##
@@ -29,6 +29,10 @@ under the License.
This connector provides a Sink that writes partitioned files to filesystems
supported by the [Flink `FileSystem` abstraction]({{< ref
"docs/deployment/filesystems/overview" >}}).
+{{< hint info >}}
Review comment:
Nit: Make the info to a warning?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Comment Edited] (FLINK-23993) Describe eventually-consistency of materializing upserts
[ https://issues.apache.org/jira/browse/FLINK-23993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17407180#comment-17407180 ] Jingsong Lee edited comment on FLINK-23993 at 8/31/21, 9:07 AM: I think we can not finish this in 1.14 release. Actually, we can't solve all the disorder problems. Our sink can freely define Primary key. There are many cases, the data seen by sink are disordered in a sense, some are acceptable and some are unacceptable. This is a complex mechanism (It may also involve the concept of dynamic table), and we need to create a detailed explanation documentation. (Actually, FLINK-20374 just fixed the more common cases, for upsert sink, sink just do what it should do) I tried to write a version, but others said it was not easy to understand :(. was (Author: lzljs3620320): I think we can not finish this in 1.14 release. Actually, we can't solve all the disorder problems. Our sink can freely define Primary key. There are many cases, the data seen by sink are disordered in a sense, some are acceptable and some are unacceptable. This is a complex mechanism (It may also involve the concept of dynamic table), and we need to create a detailed explanation documentation. (Actually, FLINK-20374 just fixed the more common cases, for upsert sink, sink just do what it should do) > Describe eventually-consistency of materializing upserts > > > Key: FLINK-23993 > URL: https://issues.apache.org/jira/browse/FLINK-23993 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Table SQL / Ecosystem >Affects Versions: 1.14.0 >Reporter: Nico Kruber >Priority: Major > > FLINK-20374 added an upsert materialization operator which fixes the order of > shuffled streams. The results of this operator are actually _eventually > consistent_ (it collects the latest value it has seen and redacts older > versions when these are not valid anymore). You could see a result stream > like this, based on the order the materialization receives events: > +I10, -I10, +I5, -I5, +I10, -I10, +I3, -I3, +I10 > Each time, the value stored in Kafka would change until the "final" result is > in. > > It may be acceptable for upsert sinks, but should be documented (or > changed/fixed) nonetheless. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink-web] xintongsong commented on a change in pull request #465: announcing the 1.14.0 release candidate
xintongsong commented on a change in pull request #465: URL: https://github.com/apache/flink-web/pull/465#discussion_r699131230 ## File path: _posts/2021-08-31-release-1.14.0-rc0.md ## @@ -0,0 +1,39 @@ +--- +layout: post +title: "Help us stabilize Apache Flink 1.14.0 RC0" +date: 2021-08-31 00:00:00 +categories: news +authors: +- joemoe: + name: "Johannes Moser" + +--- + +Dear Flink Community, + +we are excited to announce the first release candidate of Apache Flink 1.14. 🎉 + +A lot of features and fixes went into this release, including improvements to the +unified batch and streaming experience, an increase in fault tolerance by reducing +in-flight data, and more developments on connectors and components. +It wouldn't have been possible without your help. +[Around 211 people](https://github.com/apache/flink/graphs/contributors?from=2021-03-31&to=2021-08-31&type=c) Review comment: I'd suggest to remove this link, as it confuses people with an inconsistent number. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-24076) Implement Table API walkthrough in flink-playground
Matthias created FLINK-24076:
Summary: Implement Table API walkthrough in flink-playground
Key: FLINK-24076
URL: https://issues.apache.org/jira/browse/FLINK-24076
Project: Flink
Issue Type: Sub-task
Components: Documentation, Table SQL / Ecosystem
Affects Versions: 1.13.2, 1.12.5, 1.14.0
Reporter: Matthias
Currently, the {{flink-playground}} Table API walkthrough fails due to it only
providing a skeleton implementation. An {{UnimplementedException}} is thrown
when triggering the {{SpendReport}} job.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-24076) Implement Table API walkthrough in flink-playground
[
https://issues.apache.org/jira/browse/FLINK-24076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias updated FLINK-24076:
-
Description:
Currently, the {{flink-playground}} Table API walkthrough fails due to it only
providing a skeleton implementation. An {{UnimplementedException}} is thrown
when triggering the {{SpendReport}} job.
{code}
$ cd table-walkthrough
$ docker-compose up -d
$ docker logs docker logs table-walkthrough_jobmanager_1
# [...]
2021-08-31 07:47:42,465 WARN
org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap
[] - Application failed unexpectedly:
java.util.concurrent.CompletionException:
org.apache.flink.client.deployment.application.ApplicationExecutionException:
Could not execute application.
at
java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
~[?:1.8.0_275]
at
java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
~[?:1.8.0_275]
at
java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:957)
~[?:1.8.0_275]
at
java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:940)
~[?:1.8.0_275]
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
~[?:1.8.0_275]
at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
~[?:1.8.0_275]
at
org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.runApplicationEntryPoint(ApplicationDispatcherBootstrap.java:257)
~[flink-dist_2.11-1.12.1.jar:1.12.1]
at
org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.lambda$runApplicationAsync$1(ApplicationDispatcherBootstrap.java:212)
~[flink-dist_2.11-1.12.1.jar:1.12.1]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[?:1.8.0_275]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[?:1.8.0_275]
at
org.apache.flink.runtime.concurrent.akka.ActorSystemScheduledExecutorAdapter$ScheduledFutureTask.run(ActorSystemScheduledExecutorAdapter.java:159)
[flink-dist_2.11-1.12.1.jar:1.12.1]
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
[flink-dist_2.11-1.12.1.jar:1.12.1]
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
[flink-dist_2.11-1.12.1.jar:1.12.1]
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
[flink-dist_2.11-1.12.1.jar:1.12.1]
at
akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
[flink-dist_2.11-1.12.1.jar:1.12.1]
at
akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
[flink-dist_2.11-1.12.1.jar:1.12.1]
at
akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[flink-dist_2.11-1.12.1.jar:1.12.1]
Caused by:
org.apache.flink.client.deployment.application.ApplicationExecutionException:
Could not execute application.
... 11 more
Caused by: org.apache.flink.client.program.ProgramInvocationException: The main
method caused an error: This method has not yet been implemented
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:360)
~[flink-dist_2.11-1.12.1.jar:1.12.1]
at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:213)
~[flink-dist_2.11-1.12.1.jar:1.12.1]
at
org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
~[flink-dist_2.11-1.12.1.jar:1.12.1]
at
org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.runApplicationEntryPoint(ApplicationDispatcherBootstrap.java:242)
~[flink-dist_2.11-1.12.1.jar:1.12.1]
... 10 more
Caused by: org.apache.flink.playgrounds.spendreport.UnimplementedException:
This method has not yet been implemented
at
org.apache.flink.playgrounds.spendreport.SpendReport.report(SpendReport.java:32)
~[?:?]
at
org.apache.flink.playgrounds.spendreport.SpendReport.main(SpendReport.java:66)
~[?:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_275]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_275]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_275]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_275]
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:343)
~[flink-dist_2.11-1.12.1.jar:1.12.1]
at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:213)
~[flink-dist_2.11-1.12.1.jar:1.12.1]
at
org.apache.flink.client.ClientUtils.executeProgram
[GitHub] [flink] twalthr closed pull request #17060: [FLINK-23895][table] Upsert materializer is not inserted for all sink providers
twalthr closed pull request #17060: URL: https://github.com/apache/flink/pull/17060 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-web] xintongsong closed pull request #465: announcing the 1.14.0 release candidate
xintongsong closed pull request #465: URL: https://github.com/apache/flink-web/pull/465 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] CrynetLogistics opened a new pull request #17068: [FLINK-24041][connectors] Flip 171 - Abstract Async Sink Base
CrynetLogistics opened a new pull request #17068: URL: https://github.com/apache/flink/pull/17068 ## What is the purpose of the change "As a Sink *developer*, I would like to be able to inherit the AbstractAsyncSink class, and implement its abstract method submitRequestEntries(List requestEntries, ResultFuture requestResult) and be ready to use my new Sink e.g. Kinesis/DynamoDB Sink." "As a Sink *user*, I would like to tell an Async Sink how to convert an input element (e.g. a data item from a Kafka stream) into a request that the destination platform can understand by implementing the ElementConverter interface." ## Brief change log - *Added abstract async sink classes and interfaces* ## Verifying this change This change added tests and can be verified as follows: - *Added integration tests for successful/unsuccessful persistence of data in barebones implementation of the generic sink* - *Added unit tests for added code* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): yes - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? yes - If yes, how is the feature documented? JavaDocs -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-23895) Upsert materializer is not inserted for all sink providers
[
https://issues.apache.org/jira/browse/FLINK-23895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timo Walther closed FLINK-23895.
Fix Version/s: 1.15.0
Resolution: Fixed
Fixed in master: db5af67aee7fb449a9db8930213eec7dc925e58c
Fixed in 1.14: 6334323d486b413e91a04295516f2bbf898b653b
> Upsert materializer is not inserted for all sink providers
> --
>
> Key: FLINK-23895
> URL: https://issues.apache.org/jira/browse/FLINK-23895
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Reporter: Timo Walther
>Assignee: Timo Walther
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.14.0, 1.15.0
>
>
> The new {{SinkUpsertMaterializer}} is not inserted for
> {{TransformationSinkProvider}} or {{DataStreamSinkProvider}} which means that
> neither {{toChangelogStream}} not the current {{KafkaDynamicSink}} work
> correctly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-23850) Test Kafka table connector with new runtime provider
[ https://issues.apache.org/jira/browse/FLINK-23850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17407198#comment-17407198 ] Timo Walther commented on FLINK-23850: -- [~mapohl] you could implement the same pipeline as in https://github.com/apache/flink/pull/17060/files#diff-a08bcfb72b9ea9207dbc53c4149608589136f22c1f8b6acb1a84083708d18807 with Kafka upsert, it discovered already a couple of issues. > Test Kafka table connector with new runtime provider > > > Key: FLINK-23850 > URL: https://issues.apache.org/jira/browse/FLINK-23850 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Kafka >Affects Versions: 1.14.0 >Reporter: Qingsheng Ren >Assignee: Matthias >Priority: Blocker > Labels: release-testing > Fix For: 1.14.0 > > > The runtime provider of Kafka table connector has been replaced with new > KafkaSource and KafkaSink. The table connector requires to be tested to make > sure nothing is surprised to Table/SQL API users. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] AHeise commented on pull request #17067: [FLINK-23817][docs] Add metric description to Kafka source and File sink.
AHeise commented on pull request #17067: URL: https://github.com/apache/flink/pull/17067#issuecomment-909060257 > LGTM % do you know if the KafkaSource implements all the listed metrics? Thanks for reminding. I actually found 3 metrics that are currently not supported. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17068: [FLINK-24041][connectors] Flip 171 - Abstract Async Sink Base
flinkbot commented on pull request #17068: URL: https://github.com/apache/flink/pull/17068#issuecomment-909061122 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 28513efba235e7f4b923bcfd4a111eb3cf241d43 (Tue Aug 31 09:24:33 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-24041).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-23850) Test Kafka table connector with new runtime provider
[ https://issues.apache.org/jira/browse/FLINK-23850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17407198#comment-17407198 ] Timo Walther edited comment on FLINK-23850 at 8/31/21, 9:25 AM: [~mapohl] you could implement the same pipeline as in https://github.com/apache/flink/pull/17060/files#diff-a08bcfb72b9ea9207dbc53c4149608589136f22c1f8b6acb1a84083708d18807 with Kafka upsert, it discovered already a couple of issues. Please note that the Kafka connector was broken for some SQL queries before FLINK-23895. was (Author: twalthr): [~mapohl] you could implement the same pipeline as in https://github.com/apache/flink/pull/17060/files#diff-a08bcfb72b9ea9207dbc53c4149608589136f22c1f8b6acb1a84083708d18807 with Kafka upsert, it discovered already a couple of issues. > Test Kafka table connector with new runtime provider > > > Key: FLINK-23850 > URL: https://issues.apache.org/jira/browse/FLINK-23850 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Kafka >Affects Versions: 1.14.0 >Reporter: Qingsheng Ren >Assignee: Matthias >Priority: Blocker > Labels: release-testing > Fix For: 1.14.0 > > > The runtime provider of Kafka table connector has been replaced with new > KafkaSource and KafkaSink. The table connector requires to be tested to make > sure nothing is surprised to Table/SQL API users. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] AHeise merged pull request #17052: [FLINK-24051][connectors/kafka] Make groupId optional when constructi…
AHeise merged pull request #17052: URL: https://github.com/apache/flink/pull/17052 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-24051) Make consumer.group-id optional for KafkaSource
[ https://issues.apache.org/jira/browse/FLINK-24051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17407202#comment-17407202 ] Arvid Heise commented on FLINK-24051: - Merged into master as c10bd30e42d677447eaf39be1a1b48d0dcf8061d. > Make consumer.group-id optional for KafkaSource > --- > > Key: FLINK-24051 > URL: https://issues.apache.org/jira/browse/FLINK-24051 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka >Affects Versions: 1.14.0, 1.12.5, 1.13.2, 1.15.0 >Reporter: Fabian Paul >Assignee: Fabian Paul >Priority: Major > Labels: pull-request-available > > For most of the users it is not necessary to generate a group-id and the > source itself can provide a meaningful group-id during startup. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #16606: [FLINK-21357][runtime/statebackend]Periodic materialization for generalized incremental checkpoints
flinkbot edited a comment on pull request #16606: URL: https://github.com/apache/flink/pull/16606#issuecomment-887431748 ## CI report: * 264be5cc6a0485171413099e8b64b9e917d06e85 UNKNOWN * 1b7da8565a2ab9560f1aad65007930c91945087f UNKNOWN * 3421b81c2502f61112bd131a7336c16e3dd30925 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23189) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16723: [FLINK-22885][table] Supports 'SHOW COLUMNS' syntax.
flinkbot edited a comment on pull request #16723: URL: https://github.com/apache/flink/pull/16723#issuecomment-893364145 ## CI report: * 1de6e0b83ee64c22c861a9d8e639483aaaba9e7f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23078) * ce83c52813ef9b754a0aded512e80503507b1547 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23192) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17058: [FLINK-23664][docs] Add docs for KafkaSink
flinkbot edited a comment on pull request #17058: URL: https://github.com/apache/flink/pull/17058#issuecomment-908449514 ## CI report: * 6071b8b511f1a462cdf6d22bb45a2a8bf083cf18 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23131) * 28bf74c268622468560e20f5edcfdc728c1fef18 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23185) * 9c505bff67259f080b8724eff1d2e7659aa5e3a3 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17062: [FLINK-21090][tests] Add IT case for stop-with-savepoint and final checkpoint (1.14)
flinkbot edited a comment on pull request #17062: URL: https://github.com/apache/flink/pull/17062#issuecomment-908868024 ## CI report: * 5f49d4c824acc8c2170abceeef45bd918f674b2c Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23168) Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=23158) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
