Re: [DISCUSS] FLIP-319: Integrating with Kafka’s proper support for 2PC participation (KIP-939).

Thanks for taking a look at the proposals, Gyula and Alex! @Alexander Sorokoumov please see my responses below. I've tagged each bullet point to match the tag of each of your questions. ***KIP 939 specific questions:*** (1.) Your understanding is correct. The completeTransaction introduced by

Re: [DISCUSS] FLIP-357: Deprecate Iteration API of DataStream

Thanks Wencong for initiating the discussion. +1 for the proposal. On Fri, Sep 1, 2023 at 12:00 PM Wencong Liu wrote: > Hi devs, > > I would like to start a discussion on FLIP-357: Deprecate Iteration API of > DataStream [1]. > > Currently, the Iteration API of DataStream is incomplete. For

[DISCUSS] FLIP-357: Deprecate Iteration API of DataStream

Hi devs, I would like to start a discussion on FLIP-357: Deprecate Iteration API of DataStream [1]. Currently, the Iteration API of DataStream is incomplete. For instance, it lacks support for iteration in sync mode and exactly once semantics. Additionally, it does not offer the ability to

[jira] [Created] (FLINK-33014) flink jobmanager raise java.io.IOException: Connection reset by peer

zhu created FLINK-33014: --- Summary: flink jobmanager raise java.io.IOException: Connection reset by peer Key: FLINK-33014 URL: https://issues.apache.org/jira/browse/FLINK-33014 Project: Flink Issue

[jira] [Created] (FLINK-33013) Shade flink-connector-base info flink-sql-connector-connector

Jiabao Sun created FLINK-33013: -- Summary: Shade flink-connector-base info flink-sql-connector-connector Key: FLINK-33013 URL: https://issues.apache.org/jira/browse/FLINK-33013 Project: Flink

[DISCUSS] FLIP-331: Support EndOfStreamWindows and isOutputOnEOF operator attribute to optimize task deployment

Hi all, Jinhao (cc'ed) and I are opening this thread to discuss FLIP-331: Support EndOfStreamWindows and isOutputOnEOF operator attribute to optimize task deployment. The design doc can be found at

Re: [DISCUSS] FLIP-328: Allow source operators to determine isProcessingBacklog based on watermark lag

Hi Hang, Thanks for the review. This is a good question. It appears that this particular piece of information is absent from the FLIP document. If I understand correctly, the term "From the source" refers to a source that determines the backlog status based on its state. Both the backlog

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

Gentle ping on the vote for FLIP-356: Support Nested fields filter pushdown . Regards Venkata krishnan On Tue, Aug 29, 2023 at 9:18 PM Venkatakrishnan Sowrirajan wrote: > Sure, will reference this discussion to resume where we

Re: [DISCUSS] FLIP-319: Integrating with Kafka’s proper support for 2PC participation (KIP-939).

Hi Gordon! Thank you for publishing this FLIP! I would like to ask several questions and confirm my understanding of several aspects of this proposal. Even though this discussion is focused on FLIP-319, as it is based on top of KIP-939, my questions will also cover the KIP. 1. Why does

Re: Azure pipelines CI view permissions

Hi Ryan, It seems we are facing some issues on Azure side[1]. In the meantime, as a workaround, you can go to the "Runs" tab and find your build via filter. Best regards, Jing [1] https://developercommunity.visualstudio.com/t/Pipeline-runs-are-not-visible-for-non-ad/10455034 On Thu, Aug 31,

Re: [DISCUSS] FLIP-358: flink-avro enhancement and cleanup

Hi Becket, It is a very useful proposal, thanks for driving it. +1. I'd like to ask some questions to make sure I understand your thoughts correctly: 1. "For the batch cases, currently the BulkFormat for DataStream is missing" - true, and there is another option to leverage

Re: [DISCUSS] FLIP-327: Support stream-batch unified operator to improve job throughput when processing backlog data

Hi, some keywords in this triggered my attention, so sorry for late jumping in, but I'd like to comprehend the nature of the proposal. I'll try to summarize my understanding: The goal of the FLIP is to support automatic switching between streaming and batch processing, leveraging the fact

Re: [DISCUSS] FLIP-358: flink-avro enhancement and cleanup

Hi Ryan, thanks for the reply. Verifying the component with the schemas you have would be super helpful. I think enum is actually a type that is generally useful. Although it is not a part of ANSI SQL, MySQL and some other databases have this type. BTW, ENUM_STRING proposed in this FLIP is

Azure pipelines CI view permissions

Hey everyone, Has something changed in the configuration for the CI page? I can see the pipelines, but we can't see the list of recent / individual runs: https://dev.azure.com/apache-flink/apache-flink/_build That being said, we can still see the results of SUCCESS or FAILURE when a PR runs,

Re: [VOTE] FLIP-348: Make expanding behavior of virtual metadata columns configurable

+1 (binding) Best, Jark > 2023年8月31日 18:54，Jing Ge 写道： > > +1(binding) > > On Thu, Aug 31, 2023 at 11:22 AM Sergey Nuyanzin > wrote: > >> +1 (binding) >> >> On Thu, Aug 31, 2023 at 9:28 AM Benchao Li wrote: >> >>> +1 (binding) >>> >>> Martijn Visser 于2023年8月31日周四 15:24写道： +1

Re: [DISCUSS] FLIP-327: Support stream-batch unified operator to improve job throughput when processing backlog data

Hi all, I would like to share some updates on FLIP-327. Dong and I have had a series of discussions and have made several refinements to the FLIP. The major change to the FLIP is to allow the input of the one-input operator to be automatically sorted during backlog processing. When combined with

Re: [VOTE] FLIP-348: Make expanding behavior of virtual metadata columns configurable

+1(binding) On Thu, Aug 31, 2023 at 11:22 AM Sergey Nuyanzin wrote: > +1 (binding) > > On Thu, Aug 31, 2023 at 9:28 AM Benchao Li wrote: > > > +1 (binding) > > > > Martijn Visser 于2023年8月31日周四 15:24写道： > > > > > > +1 (binding) > > > > > > On Thu, Aug 31, 2023 at 9:09 AM Timo Walther > wrote:

[jira] [Created] (FLINK-33012) LeaderElectionTest.testHasLeadership fails on AZP

Sergey Nuyanzin created FLINK-33012: --- Summary: LeaderElectionTest.testHasLeadership fails on AZP Key: FLINK-33012 URL: https://issues.apache.org/jira/browse/FLINK-33012 Project: Flink

[jira] [Created] (FLINK-33011) Operator deletes HA data unexpectedly

Ruibin Xing created FLINK-33011: --- Summary: Operator deletes HA data unexpectedly Key: FLINK-33011 URL: https://issues.apache.org/jira/browse/FLINK-33011 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-33010) NPE when using GREATEST() in Flink SQL

Hector Rios created FLINK-33010: --- Summary: NPE when using GREATEST() in Flink SQL Key: FLINK-33010 URL: https://issues.apache.org/jira/browse/FLINK-33010 Project: Flink Issue Type: Bug

Re: [VOTE] FLIP-348: Make expanding behavior of virtual metadata columns configurable

+1 (binding) On Thu, Aug 31, 2023 at 9:28 AM Benchao Li wrote: > +1 (binding) > > Martijn Visser 于2023年8月31日周四 15:24写道： > > > > +1 (binding) > > > > On Thu, Aug 31, 2023 at 9:09 AM Timo Walther wrote: > > > > > Hi everyone, > > > > > > I'd like to start a vote on FLIP-348: Make expanding

Re: [DISCUSS] FLIP-358: flink-avro enhancement and cleanup

Hey -- I have a certain knowledge of Avro, and I'd be willing to help out with some of these enhancements, writing tests and reviewing. I have a *lot* of Avro schemas available for validation! The FLIP looks pretty good and covers the possible cases pretty rigorously. I wasn't aware of some of

[jira] [Created] (FLINK-33009) tools/release/update_japicmp_configuration.sh

Matthias Pohl created FLINK-33009: - Summary: tools/release/update_japicmp_configuration.sh Key: FLINK-33009 URL: https://issues.apache.org/jira/browse/FLINK-33009 Project: Flink Issue Type:

Re: [VOTE] FLIP-348: Make expanding behavior of virtual metadata columns configurable

+1 (binding) Martijn Visser 于2023年8月31日周四 15:24写道： > > +1 (binding) > > On Thu, Aug 31, 2023 at 9:09 AM Timo Walther wrote: > > > Hi everyone, > > > > I'd like to start a vote on FLIP-348: Make expanding behavior of virtual > > metadata columns configurable [1] which has been discussed in this

Re: [VOTE] FLIP-348: Make expanding behavior of virtual metadata columns configurable

+1 (binding) On Thu, Aug 31, 2023 at 9:09 AM Timo Walther wrote: > Hi everyone, > > I'd like to start a vote on FLIP-348: Make expanding behavior of virtual > metadata columns configurable [1] which has been discussed in this > thread [2]. > > The vote will be open for at least 72 hours unless

[jira] [Created] (FLINK-33008) Use KubernetesClient from JOSDK context

Gyula Fora created FLINK-33008: -- Summary: Use KubernetesClient from JOSDK context Key: FLINK-33008 URL: https://issues.apache.org/jira/browse/FLINK-33008 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-33007) Integrate autoscaler config validation into the general validator flow

Gyula Fora created FLINK-33007: -- Summary: Integrate autoscaler config validation into the general validator flow Key: FLINK-33007 URL: https://issues.apache.org/jira/browse/FLINK-33007 Project: Flink

[VOTE] FLIP-348: Make expanding behavior of virtual metadata columns configurable

Hi everyone, I'd like to start a vote on FLIP-348: Make expanding behavior of virtual metadata columns configurable [1] which has been discussed in this thread [2]. The vote will be open for at least 72 hours unless there is an objection or not enough votes. [1]

[jira] [Created] (FLINK-33006) Add e2e test for Kubernetes Operator HA

Gyula Fora created FLINK-33006: -- Summary: Add e2e test for Kubernetes Operator HA Key: FLINK-33006 URL: https://issues.apache.org/jira/browse/FLINK-33006 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-33005) Upgrade JOSDK to 4.4.2 and Fabric8 to 6.8.1

Gyula Fora created FLINK-33005: -- Summary: Upgrade JOSDK to 4.4.2 and Fabric8 to 6.8.1 Key: FLINK-33005 URL: https://issues.apache.org/jira/browse/FLINK-33005 Project: Flink Issue Type: