答复: [DISCUSS] Release 2.0 Work Items
@Xingtong I already know the modification of some api, but because there are many changes involved, I am afraid that the consideration is not comprehensive. I'm willing to do the work, but I haven't found a committer yet. Best, Zhiqiang 发件人: Xintong Song 日期: 星期四, 2023年7月13日 10:03 收件人: dev@flink.apache.org 主题: Re: [DISCUSS] Release 2.0 Work Items Thanks for the inputs, Zhiqiang and Jiabao. @Zhiqiang, The proposal sounds interesting. Do you already have an idea what API changes are needed in order to make the connectors pluggable? I think whether this should go into Flink 2.0 would significantly depend on what API changes are needed. Moreover, would you like to work on this effort or simply raise a need? And if you'd like to work on this, do you already find a committer who can help on this? @Jiabao, Thanks for the suggestions. I agree that it would be nice to improve the experiences in deploying Flink instances and submitting tasks. It would be helpful if you can point out the specific behaviors that make integrating Flink in your production difficult. Also, I'd like to understand how this topic is related to the Release 2.0 topic. Or asked differently, is this something that requires breaking changes that can only happen in major version bumps, or is it just improvement that can go into any minor version? Best, Xintong On Thu, Jul 13, 2023 at 12:49 AM Jiabao Sun wrote: > Thanks Xintong for driving the effort. > > > I’d add a +1 to improving out-of-box user experience, as suggested by > @Jark and @Chesnay. > For beginners, understanding complex configurations is a hard work. > > In addition, the deployment of a set of Flink runtime environment is also > a complex matter. > At present, there are still big differences in the submission tasks for > different computing resource. If users need time for their own data > development platform, they need to deeply understand these differences when > processing task submission and running status check. > > I'm glad to see features like flink-sql-gateway being implemented by the > community because it makes it easy for users to submit flink sql tasks. > Further more, can we provide more unified, out-of-the-box capabilities that > allow users to quickly pull up a production-ready Flink environment and > easily integrate Flink into their own data development platform? > > > Best, > Jiabao > > > > 2023年7月12日 下午8:16，zhiqiang li 写道： > > > > I have seen in [1] connectors and formats, and user code will be > pluggable. > > If the connectors are pluggable, the benefits are obvious, as the > conflicts > > between different jar package versions can be avoided. > > If you don't use classloader isolation, shade is needed to resolve > > conflicts. A lot of development time is wasted. > > I know that this change may involve a lot of API changes, so I would like > > to discuss in this email whether we can make changes in Flink 2.0. > > Plugins facilitate a strict separation of code through restricted > > classloaders. > > > > Plugins cannot access classes from other plugins or from Flink that have > >> not been specifically whitelisted. > >> This strict isolation allows plugins to contain conflicting versions of > >> the same library without the need to relocate classes or to converge to > >> common versions. > >> Currently, file systems and metric reporters are pluggable *but in the > >> future, connectors, formats, and even user code should also be > pluggable.* > >> > > > > [1] > > > https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/filesystems/overview/ > > > > Xintong Song 于2023年7月11日周二 18:50写道： > > > >>> > >>> What we might want to come up with is a summary with each 2.0.0 issue > on > >>> why it should be included or not. That summary is something the > community > >>> could vote on. WDYT? I'm happy to help here. > >>> > >> > >> That sounds great. Thanks for offering the help. I'll also try to go > >> through the issues, but TBH I'm quite overwhelmed and cannot promise to > get > >> this done very soon. Your help is very much needed. > >> > >> > >> Best, > >> > >> Xintong > >> > >> > >> > >> On Tue, Jul 11, 2023 at 6:08 PM Matthias Pohl > >> wrote: > >> > >>> @Xintong I guess it makes sense. I agree with your conclusions on the > >> four > >>> mentioned Jira issues. > >>> > >>> I just checked any issues that have fixVersion = 2.0.0 [1]. There are a > >> few > >>> more items that are not affiliated with FLINK-3957 [2]. I guess we > should > >>> find answers for these issues: Either closing them with a reason to > have > >> a > >>> consistent state in Jira or adding them to the feature list as part of > a > >>> separate voting thread (to leave the current vote untouched). > >>> > >>> What we might want to come up with is a summary with each 2.0.0 issue > on > >>> why it should be included or not. That summary is something the > community > >>> could vote on. WDYT? I'm happy to help here. > >>> > >>> Matthias > >>> > >>> [1] > >>> > >>> > >>
Re: [DISCUSS] Release 2.0 Work Items
Thanks for the inputs, Zhiqiang and Jiabao. @Zhiqiang, The proposal sounds interesting. Do you already have an idea what API changes are needed in order to make the connectors pluggable? I think whether this should go into Flink 2.0 would significantly depend on what API changes are needed. Moreover, would you like to work on this effort or simply raise a need? And if you'd like to work on this, do you already find a committer who can help on this? @Jiabao, Thanks for the suggestions. I agree that it would be nice to improve the experiences in deploying Flink instances and submitting tasks. It would be helpful if you can point out the specific behaviors that make integrating Flink in your production difficult. Also, I'd like to understand how this topic is related to the Release 2.0 topic. Or asked differently, is this something that requires breaking changes that can only happen in major version bumps, or is it just improvement that can go into any minor version? Best, Xintong On Thu, Jul 13, 2023 at 12:49 AM Jiabao Sun wrote: > Thanks Xintong for driving the effort. > > > I’d add a +1 to improving out-of-box user experience, as suggested by > @Jark and @Chesnay. > For beginners, understanding complex configurations is a hard work. > > In addition, the deployment of a set of Flink runtime environment is also > a complex matter. > At present, there are still big differences in the submission tasks for > different computing resource. If users need time for their own data > development platform, they need to deeply understand these differences when > processing task submission and running status check. > > I'm glad to see features like flink-sql-gateway being implemented by the > community because it makes it easy for users to submit flink sql tasks. > Further more, can we provide more unified, out-of-the-box capabilities that > allow users to quickly pull up a production-ready Flink environment and > easily integrate Flink into their own data development platform? > > > Best, > Jiabao > > > > 2023年7月12日 下午8:16，zhiqiang li 写道： > > > > I have seen in [1] connectors and formats, and user code will be > pluggable. > > If the connectors are pluggable, the benefits are obvious, as the > conflicts > > between different jar package versions can be avoided. > > If you don't use classloader isolation, shade is needed to resolve > > conflicts. A lot of development time is wasted. > > I know that this change may involve a lot of API changes, so I would like > > to discuss in this email whether we can make changes in Flink 2.0. > > Plugins facilitate a strict separation of code through restricted > > classloaders. > > > > Plugins cannot access classes from other plugins or from Flink that have > >> not been specifically whitelisted. > >> This strict isolation allows plugins to contain conflicting versions of > >> the same library without the need to relocate classes or to converge to > >> common versions. > >> Currently, file systems and metric reporters are pluggable *but in the > >> future, connectors, formats, and even user code should also be > pluggable.* > >> > > > > [1] > > > https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/filesystems/overview/ > > > > Xintong Song 于2023年7月11日周二 18:50写道： > > > >>> > >>> What we might want to come up with is a summary with each 2.0.0 issue > on > >>> why it should be included or not. That summary is something the > community > >>> could vote on. WDYT? I'm happy to help here. > >>> > >> > >> That sounds great. Thanks for offering the help. I'll also try to go > >> through the issues, but TBH I'm quite overwhelmed and cannot promise to > get > >> this done very soon. Your help is very much needed. > >> > >> > >> Best, > >> > >> Xintong > >> > >> > >> > >> On Tue, Jul 11, 2023 at 6:08 PM Matthias Pohl > >> wrote: > >> > >>> @Xintong I guess it makes sense. I agree with your conclusions on the > >> four > >>> mentioned Jira issues. > >>> > >>> I just checked any issues that have fixVersion = 2.0.0 [1]. There are a > >> few > >>> more items that are not affiliated with FLINK-3957 [2]. I guess we > should > >>> find answers for these issues: Either closing them with a reason to > have > >> a > >>> consistent state in Jira or adding them to the feature list as part of > a > >>> separate voting thread (to leave the current vote untouched). > >>> > >>> What we might want to come up with is a summary with each 2.0.0 issue > on > >>> why it should be included or not. That summary is something the > community > >>> could vote on. WDYT? I'm happy to help here. > >>> > >>> Matthias > >>> > >>> [1] > >>> > >>> > >> > https://issues.apache.org/jira/browse/FLINK-32437?jql=project%20%3D%20FLINK%20AND%20fixVersion%20%3D%202.0.0%20AND%20status%20NOT%20IN%20(Closed%2C%20Resolved)%20%20 > >>> [2] https://issues.apache.org/jira/browse/FLINK-3957 > >>> > >>> > >>> On Tue, Jul 11, 2023 at 5:01 AM Xintong Song > >>> wrote: > >>> > @Zhu, > As you are downgrading
Re: [DISCUSS] FLIP-330: Support specifying record timestamp requirement
Hi Yunfeng, Thanks for the proposal. It makes sense to offer the optimization. I got some NIT questions. 1. I guess you changed your thoughts while coding the POC, I found pipeline.enable-operator-timestamp in the code but is pipeline.force-timestamp-support defined in the FLIP 2. about the benchmark example, why did you enable object reuse? Since It is an optimization of serde, will the benchmark be better if it is disabled? Best regards, Jing On Mon, Jul 10, 2023 at 11:54 AM Yunfeng Zhou wrote: > Hi all, > > Dong(cc'ed) and I are opening this thread to discuss our proposal to > support optimizing StreamRecord's serialization performance. > > Currently, a StreamRecord would be converted into a 1-byte tag (+ > 8-byte timestamp) + N-byte serialized value during the serialization > process. In scenarios where timestamps and watermarks are not needed, > and latency tracking is enabled, this process would include > unnecessary information in the serialized byte array. This FLIP aims > to avoid such overhead and increases Flink job's performance during > serialization. > > Please refer to the FLIP document for more details about the proposed > design and implementation. We welcome any feedback and opinions on > this proposal. > > Best regards, Dong and Yunfeng > > [1] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-330%3A+Support+specifying+record+timestamp+requirement >
Re: [DISCUSS] FLIP 333 - Redesign Apache Flink website
Thanks for driving this Deepthi. Looks great to me; a huge improvement over the current website. We had agreed in the previous discussion [1] to keep the main website separate from the docs since the target audience is different. I am inclined to decouple the two and re-evaluate updating the docs in a separate FLIP later. Thanks, Danny [1] https://lists.apache.org/thread/c3pt00cf77lrtgt242p26lgp9l2z5yc8 On Tue, Jul 11, 2023 at 2:38 PM Jingsong Li wrote: > It's exciting to finally have someone to refactor the Flink website. > > Thanks Deepthi. > > To Xintong, > > > maintain two sets of website designs at the same time > > If our website is not so complex, and new webui can be full feature > coverage, there would be no need to maintain two sets of web. > > To Jing > > +1 to consider Flink doc too. > > Best, > Jingsong > > On Tue, Jul 11, 2023 at 8:16 PM Jing Ge > wrote: > > > > Hi, > > > > +1, the UI design looks good! > > > > Commonly speaking, there are two parts of the whole website: Flink Web > and > > Flink doc. Will the dart mode also cover Flink doc? > > > > Best regards, > > Jing > > > > On Tue, Jul 11, 2023 at 12:40 PM Matthias Pohl > > wrote: > > > > > I also like the proposed designs. Considering that you want to touch > > > individual subpages, there are also some subpages of Flink's website > not > > > being mentioned in the FLIP (e.g. roadmap [1]). What is the plan with > > > those? Are they covered by the "We recommend modifications only to the > > > design of the following pages" part but are not listed there? > > > > > > Additionally, it would be nice to get a bit more insight into the > feedback > > > from users as Xintong pointed out. It would be interesting to > understand > > > how the new design helps solving certain problems (besides having the > > > motivation to modernize the look and feel). > > > > > > I'm also wondering whether it's doable to do a discussion (FLIP?) per > > > subpage on the design (as proposed by Chesnay in the first discussion > on > > > this topic [2] to have smaller changes rather than a single big one). > But I > > > could imagine this being quite tedious because different people might > > > have different opinions on how something should be done. > > > > > > I don't have any experience with frontend design. I'm wondering how > much > > > time such a redesign takes. Could it be linked to the 2.0 release? > > > > > > Thanks for the FLIP (you might want to add [2] to the FLIP's header > for the > > > sake of transparency). > > > Matthias > > > > > > [1] https://flink.apache.org/roadmap/ > > > [2] https://lists.apache.org/thread/c3pt00cf77lrtgt242p26lgp9l2z5yc8 > > > > > > On Tue, Jul 11, 2023 at 11:39 AM Xintong Song > > > wrote: > > > > > > > +1 in general. > > > > > > > > Thanks for proposing this contribution, Deepthi. The new design looks > > > very > > > > cool. > > > > > > > > I have a few questions, which might be entry-level given that I > barely > > > know > > > > anything about the website design. > > > > - Do you think it's feasible to maintain two sets of website designs > at > > > the > > > > same time? E.g., adding a button "back to previous version". I'm > asking > > > > because, while the new UI might be more friendly to newcomers, the > > > original > > > > website might be more convenient for people who are already familiar > with > > > > it to find things. It would be nice if we can offer both options to > > > users. > > > > - For the documentation, I wonder if it makes sense to offer the same > > > color > > > > theme as the website, to keep the experience consistent. How much > effort > > > > does it require? > > > > - In the FLIP, you mentioned things like "there's a general > consensus" > > > and > > > > "feedback from customers". I'm curious where these come from. Have > you > > > > conducted some sort of survey? Would you mind sharing a bit more > about > > > > that? > > > > > > > > Best, > > > > > > > > Xintong > > > > > > > > > > > > > > > > On Tue, Jul 11, 2023 at 4:57 PM Feifan Wang > wrote: > > > > > > > > > +1 , the new design looks more attractive and is well organized > > > > > > > > > > | > > > > > | > > > > > Feifan Wang > > > > > | > > > > > | > > > > > zoltar9...@163.com > > > > > | > > > > > > > > > > > > > > > Replied Message > > > > > | From | Leonard Xu | > > > > > | Date | 07/11/2023 16:34 | > > > > > | To | dev | > > > > > | Subject | Re: [DISCUSS] FLIP 333 - Redesign Apache Flink website > | > > > > > +1 for the redesigning, the new website looks cool. > > > > > > > > > > > > > > > Best, > > > > > Leonard > > > > > > > > > > On Jul 11, 2023, at 7:55 AM, Mohan, Deepthi > > > > > > > > > wrote: > > > > > > > > > > Hi, > > > > > > > > > > I’m opening this thread to discuss a proposal to redesign the > Apache > > > > Flink > > > > > website: https://flink.apache.org. The approach and a few initial > > > > mockups > > > > > are included in FLIP 333 - Redesign Apache Flink website.< > > > > > > > > > > >
Re: [DISCUSS] Release 2.0 Work Items
Thanks Xintong for driving the effort. I’d add a +1 to improving out-of-box user experience, as suggested by @Jark and @Chesnay. For beginners, understanding complex configurations is a hard work. In addition, the deployment of a set of Flink runtime environment is also a complex matter. At present, there are still big differences in the submission tasks for different computing resource. If users need time for their own data development platform, they need to deeply understand these differences when processing task submission and running status check. I'm glad to see features like flink-sql-gateway being implemented by the community because it makes it easy for users to submit flink sql tasks. Further more, can we provide more unified, out-of-the-box capabilities that allow users to quickly pull up a production-ready Flink environment and easily integrate Flink into their own data development platform? Best, Jiabao > 2023年7月12日 下午8:16，zhiqiang li 写道： > > I have seen in [1] connectors and formats, and user code will be pluggable. > If the connectors are pluggable, the benefits are obvious, as the conflicts > between different jar package versions can be avoided. > If you don't use classloader isolation, shade is needed to resolve > conflicts. A lot of development time is wasted. > I know that this change may involve a lot of API changes, so I would like > to discuss in this email whether we can make changes in Flink 2.0. > Plugins facilitate a strict separation of code through restricted > classloaders. > > Plugins cannot access classes from other plugins or from Flink that have >> not been specifically whitelisted. >> This strict isolation allows plugins to contain conflicting versions of >> the same library without the need to relocate classes or to converge to >> common versions. >> Currently, file systems and metric reporters are pluggable *but in the >> future, connectors, formats, and even user code should also be pluggable.* >> > > [1] > https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/filesystems/overview/ > > Xintong Song 于2023年7月11日周二 18:50写道： > >>> >>> What we might want to come up with is a summary with each 2.0.0 issue on >>> why it should be included or not. That summary is something the community >>> could vote on. WDYT? I'm happy to help here. >>> >> >> That sounds great. Thanks for offering the help. I'll also try to go >> through the issues, but TBH I'm quite overwhelmed and cannot promise to get >> this done very soon. Your help is very much needed. >> >> >> Best, >> >> Xintong >> >> >> >> On Tue, Jul 11, 2023 at 6:08 PM Matthias Pohl >> wrote: >> >>> @Xintong I guess it makes sense. I agree with your conclusions on the >> four >>> mentioned Jira issues. >>> >>> I just checked any issues that have fixVersion = 2.0.0 [1]. There are a >> few >>> more items that are not affiliated with FLINK-3957 [2]. I guess we should >>> find answers for these issues: Either closing them with a reason to have >> a >>> consistent state in Jira or adding them to the feature list as part of a >>> separate voting thread (to leave the current vote untouched). >>> >>> What we might want to come up with is a summary with each 2.0.0 issue on >>> why it should be included or not. That summary is something the community >>> could vote on. WDYT? I'm happy to help here. >>> >>> Matthias >>> >>> [1] >>> >>> >> https://issues.apache.org/jira/browse/FLINK-32437?jql=project%20%3D%20FLINK%20AND%20fixVersion%20%3D%202.0.0%20AND%20status%20NOT%20IN%20(Closed%2C%20Resolved)%20%20 >>> [2] https://issues.apache.org/jira/browse/FLINK-3957 >>> >>> >>> On Tue, Jul 11, 2023 at 5:01 AM Xintong Song >>> wrote: >>> @Zhu, As you are downgrading "Clarify the scopes of configuration options" to nice-to-have priority, could you also bring that up in the vote >>> thread[1]? I'm asking because there are people who already voted on the original >>> list. I think restarting the vote is probably an overkill and unnecessary, >> but >>> we should at least bring this change to their attention. @Matthias, Thanks a lot for bringing this up. I wasn't aware of this early >>> umbrella. I haven't gone through everything in FLINK-3957 yet. I'll do it asap. Just quickly went through the 4 issues you mentioned. - FLINK-4675 & FLINK-14068: I'd be +1 to deprecate them in 1.18, as >> long >>> as the new APIs that we want users to migrate to are ready. For these 2 tickets, I think introduction of the updated APIs should be >>> straightforward and feasible for 1.18. - FLINK-13926: I'm not sure about this one. The two mentioned classes `ProcessingTimeSessionWindows` and `EventTimeSessionWindows` are not >> even marked as Public or PublicEvolving APIs. Moreover, I don't see a good >> way to smoothly replace the classes with a generic version. - FLINK-5126: This is a bit unclear to me. From
[jira] [Created] (FLINK-32585) Filter javax.xml.bind:jaxb-api false positive for Pulsar connector
Zili Chen created FLINK-32585: - Summary: Filter javax.xml.bind:jaxb-api false positive for Pulsar connector Key: FLINK-32585 URL: https://issues.apache.org/jira/browse/FLINK-32585 Project: Flink Issue Type: Improvement Components: Build System / CI Reporter: Zili Chen Assignee: Zili Chen -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [VOTE] FLIP-322 Cooldown period for adaptive scheduler. Second vote.
Hi all, I'm going on vacation tonight for 3 weeks. Even if the vote is not finished, as the implementation is rather quick and the design discussion had settled, I preferred I implementing FLIP-322 [1] to allow people to take a look while I'm off. [1] https://github.com/apache/flink/pull/22985 Best Etienne Le 12/07/2023 à 09:56, Etienne Chauchot a écrit : Hi all, Would you mind casting your vote to this second vote thread (opened after new discussions) so that the subject can move forward ? @David, @Chesnay, @Robert you took part to the discussions, can you please sent your vote ? Thank you very much Best Etienne Le 06/07/2023 à 13:02, Etienne Chauchot a écrit : Hi all, Thanks for your feedback about the FLIP-322: Cooldown period for adaptive scheduler [1]. This FLIP was discussed in [2]. I'd like to start a vote for it. The vote will be open for at least 72 hours (until July 9th 15:00 GMT) unless there is an objection or insufficient votes. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-322+Cooldown+period+for+adaptive+scheduler [2] https://lists.apache.org/thread/qvgxzhbp9rhlsqrybxdy51h05zwxfns6 Best, Etienne
[jira] [Created] (FLINK-32584) Make it possible to unset default catalog and/or database
Dawid Wysakowicz created FLINK-32584: Summary: Make it possible to unset default catalog and/or database Key: FLINK-32584 URL: https://issues.apache.org/jira/browse/FLINK-32584 Project: Flink Issue Type: New Feature Components: Table SQL / API Reporter: Dawid Wysakowicz Assignee: Dawid Wysakowicz Fix For: 1.18.0 In certain scenarios it might make sense to unset the default catalog and/or database. For example in a situation when there is no sane default one, but we want the user make that decision consciously. This change has a narrow scope and changes only some checks in the API surface. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [DISCUSS] Release 2.0 Work Items
I have seen in [1] connectors and formats, and user code will be pluggable. If the connectors are pluggable, the benefits are obvious, as the conflicts between different jar package versions can be avoided. If you don't use classloader isolation, shade is needed to resolve conflicts. A lot of development time is wasted. I know that this change may involve a lot of API changes, so I would like to discuss in this email whether we can make changes in Flink 2.0. Plugins facilitate a strict separation of code through restricted classloaders. Plugins cannot access classes from other plugins or from Flink that have > not been specifically whitelisted. > This strict isolation allows plugins to contain conflicting versions of > the same library without the need to relocate classes or to converge to > common versions. > Currently, file systems and metric reporters are pluggable *but in the > future, connectors, formats, and even user code should also be pluggable.* > [1] https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/filesystems/overview/ Xintong Song 于2023年7月11日周二 18:50写道： > > > > What we might want to come up with is a summary with each 2.0.0 issue on > > why it should be included or not. That summary is something the community > > could vote on. WDYT? I'm happy to help here. > > > > That sounds great. Thanks for offering the help. I'll also try to go > through the issues, but TBH I'm quite overwhelmed and cannot promise to get > this done very soon. Your help is very much needed. > > > Best, > > Xintong > > > > On Tue, Jul 11, 2023 at 6:08 PM Matthias Pohl > wrote: > > > @Xintong I guess it makes sense. I agree with your conclusions on the > four > > mentioned Jira issues. > > > > I just checked any issues that have fixVersion = 2.0.0 [1]. There are a > few > > more items that are not affiliated with FLINK-3957 [2]. I guess we should > > find answers for these issues: Either closing them with a reason to have > a > > consistent state in Jira or adding them to the feature list as part of a > > separate voting thread (to leave the current vote untouched). > > > > What we might want to come up with is a summary with each 2.0.0 issue on > > why it should be included or not. That summary is something the community > > could vote on. WDYT? I'm happy to help here. > > > > Matthias > > > > [1] > > > > > https://issues.apache.org/jira/browse/FLINK-32437?jql=project%20%3D%20FLINK%20AND%20fixVersion%20%3D%202.0.0%20AND%20status%20NOT%20IN%20(Closed%2C%20Resolved)%20%20 > > [2] https://issues.apache.org/jira/browse/FLINK-3957 > > > > > > On Tue, Jul 11, 2023 at 5:01 AM Xintong Song > > wrote: > > > > > @Zhu, > > > As you are downgrading "Clarify the scopes of configuration options" to > > > nice-to-have priority, could you also bring that up in the vote > > thread[1]? > > > I'm asking because there are people who already voted on the original > > list. > > > I think restarting the vote is probably an overkill and unnecessary, > but > > we > > > should at least bring this change to their attention. > > > > > > @Matthias, > > > Thanks a lot for bringing this up. I wasn't aware of this early > > umbrella. I > > > haven't gone through everything in FLINK-3957 yet. I'll do it asap. > > > > > > Just quickly went through the 4 issues you mentioned. > > > - FLINK-4675 & FLINK-14068: I'd be +1 to deprecate them in 1.18, as > long > > as > > > the new APIs that we want users to migrate to are ready. For these 2 > > > tickets, I think introduction of the updated APIs should be > > straightforward > > > and feasible for 1.18. > > > - FLINK-13926: I'm not sure about this one. The two mentioned classes > > > `ProcessingTimeSessionWindows` and `EventTimeSessionWindows` are not > even > > > marked as Public or PublicEvolving APIs. Moreover, I don't see a good > way > > > to smoothly replace the classes with a generic version. > > > - FLINK-5126: This is a bit unclear to me. From the description and > > > conversation on the ticket, I don't fully understand which concrete > APIs > > > the ticket is referring to. Or maybe it refers to all / most of the > APIs > > > that throws Exception / IOException in general. Moreover, I don't think > > > removing Exception / IOException from the API signature is a breaking > > > change. It requires no code changes on the caller side. > > > > > > WDYT? > > > > > > Best, > > > > > > Xintong > > > > > > > > > [1] https://lists.apache.org/thread/r0y9syc6k5nmcxvnd0hj33htdpdj9k6m > > > [2] https://issues.apache.org/jira/browse/FLINK-3957 > > > > > > On Mon, Jul 10, 2023 at 10:53 PM Matthias Pohl > > > wrote: > > > > > > > I brought it up in the deprecating APIs in 1.18 thread [1] already > but > > it > > > > feels misplaced there. I just wanted to ask whether someone did a > pass > > > over > > > > FLINK-3957 [2]. I came across it when going through the release 2.0 > > > feature > > > > list [3] as part of the vote. I have the feeling that there are some > > > valid > > > > action
[jira] [Created] (FLINK-32583) RestClient can deadlock if request made after Netty event executor terminated
Patrick Lucas created FLINK-32583:
-
Summary: RestClient can deadlock if request made after Netty event
executor terminated
Key: FLINK-32583
URL: https://issues.apache.org/jira/browse/FLINK-32583
Project: Flink
Issue Type: Bug
Components: Runtime / REST
Affects Versions: 1.18.0, 1.16.3, 1.17.2
Reporter: Patrick Lucas
The RestClient can deadlock if a request is made after the Netty event executor
has terminated.
This is due to the listener that would resolve the CompletableFuture that is
attached to the ChannelFuture returned by the call to Netty to connect not
being able to run because the executor to run it rejects the execution.
[RestClient.java|https://github.com/apache/flink/blob/release-1.17.1/flink-runtime/src/main/java/org/apache/flink/runtime/rest/RestClient.java#L471-L482]:
{code:java}
final ChannelFuture connectFuture = bootstrap.connect(targetAddress,
targetPort);
final CompletableFuture channelFuture = new CompletableFuture<>();
connectFuture.addListener(
(ChannelFuture future) -> {
if (future.isSuccess()) {
channelFuture.complete(future.channel());
} else {
channelFuture.completeExceptionally(future.cause());
}
});
{code}
In this code, the call to {{addListener()}} can fail silently (only logging to
the console), meaning any code waiting on the CompletableFuture returned by
this method will deadlock.
There was some work in Netty around this back in 2015, but it's unclear to me
how this situation is expected to be handled given the discussion and changes
from these issues:
* [https://github.com/netty/netty/issues/3449] (open)
* [https://github.com/netty/netty/pull/3483] (closed)
* [https://github.com/netty/netty/pull/3566] (closed)
* [https://github.com/netty/netty/pull/5087] (merged)
I think a reasonable fix for Flink would be to check the state of
{{connectFuture}} and {{channelFuture}} immediately after the call to
{{addListener()}}, resolving {{channelFuture}} with {{completeExceptionally()}}
if {{connectFuture}} is done and failed and {{channelFuture}} has not been
completed. In the possible race condition where the listener was attached
successfully and the connection fails instantly, the result is the same, as
calls to {{CompletableFuture#completeExceptionally()}} are idempotent.
A workaround for users of RestClient is to call {{CompletableFuture#get(long
timeout, TimeUnit unit)}} rather than {{#get()}} or {{#join()}} on the
CompletableFutures it returns. However, if the call throws TimeoutException,
the cause of the failure cannot easily be determined.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [DISCUSS] FLIP-327: Support stream-batch unified operator to improve job throughput when processing backlog data
Hi Dong, Thanks for your reply! Best regards, Jing On Wed, Jul 12, 2023 at 3:25 AM Dong Lin wrote: > Hi Jing, > > Thanks for the comments. Please see my reply inline. > > On Wed, Jul 12, 2023 at 5:04 AM Jing Ge > wrote: > > > Hi Dong, > > > > Thanks for the clarification. Now it is clear for me. I got additional > noob > > questions wrt the internal sorter. > > > > 1. when to call setter to set the internalSorterSupported to be true? > > > > Developer of the operator class (i.e. those classes which implements > `StreamOperator`) should override the `#getOperatorAttributes()` API to set > internalSorterSupported to true, if he/she decides to sort records > internally in the operator. > > > > 2 > > *"For those operators whose throughput can be considerably improved with > an > > internal sorter, update it to take advantage of the internal sorter when > > its input has isBacklog=true.* > > *Typically, operators that involve aggregation operation (e.g. join, > > cogroup, aggregate) on keyed inputs can benefit from using an internal > > sorter."* > > > > *"The operator that performs CoGroup operation will instantiate two > > internal sorter to sorts records from its two inputs separately. Then it > > can pull the sorted records from these two sorters. This can be done > > without wrapping input records with TaggedUnion<...>. In comparison, the > > existing DataStream#coGroup needs to wrap input records with > > TaggedUnion<...> before sorting them using one external sorter, which > > introduces higher overhead."* > > > > According to the performance test, it seems that internal sorter has > better > > performance than external sorter. Is it possible to make those operators > > that can benefit from it use internal sorter by default? > > > > Yes, it is possible. After this FLIP is done, users can use > DataStream#coGroup with EndOfStreamWindows as the window assigner to > co-group two streams in effectively the batch manner. An operator that uses > an internal sorter will be used to perform the co-group operation. There is > no need for users of the DataStream API to explicitly know or set the > internal sorter in anyway. > > In the future, we plan to incrementally optimize other aggregation > operation (e.g. aggregate) on the DataStream API when EndOfStreamWindows is > used as the window assigner. > > Best, > Dong > > > > > > Best regards, > > Jing > > > > > > On Tue, Jul 11, 2023 at 2:58 PM Dong Lin wrote: > > > > > Hi Jing, > > > > > > Thank you for the comments! Please see my reply inline. > > > > > > On Tue, Jul 11, 2023 at 5:41 AM Jing Ge > > > wrote: > > > > > > > Hi Dong, > > > > > > > > Thanks for the proposal! The FLIP is already in good shape. I got > some > > > NIT > > > > questions. > > > > > > > > 1. It is a little bit weird to write the hint right after the > > motivation > > > > that some features have been moved to FLIP-331, because at that time, > > > > readers don't know the context about what features does it mean. I > > would > > > > suggest moving the note to the beginning of "Public interfaces" > > sections. > > > > > > > > > > Given that the reviewer who commented on this email thread before I > > > refactored the FLIP (i.e. Piotr) has read FLP-331, I think it is > simpler > > to > > > just remove any mention of FLIP-331. I have updated the FLIP > accordingly. > > > > > > > > > > 2. It is also a little bit weird to describe all behaviour changes at > > > first > > > > but only focus on one single feature, i.e. how to implement > > > > internalSorterSupported. TBH, I was lost while I was reading the > Public > > > > interfaces. Maybe change the FLIP title? Another option could be to > > > write a > > > > short summary of all features and point out that this FLIP will only > > > focus > > > > on the internalSorterSupported feature. Others could be found in > > > FLIP-331. > > > > WDYT? > > > > > > > > > > Conceptually, the purpose of this FLIP is to allow a stream mode job to > > run > > > parts of the topology in batch mode so that it can apply > > > optimizations/computations that can not be used together with > > checkpointing > > > (and thus not usable in stream mode). Although internal sorter is the > > only > > > optimization immediately supported in this FLIP, this FLIP lays the > > > foundation to support other optimizations in the future, such as using > > GPU > > > to process a bounded stream of records. > > > > > > Therefore, I find it better to keep the current title rather than > > limiting > > > the scope to internal sorter. What do you think? > > > > > > > > > > > > > 3. There should be a typo at 4) Checkpoint and failover strategy -> > > Mixed > > > > mode -> > > > > > > > >- If any task fails when isBacklog=false true, this task is > > restarted > > > to > > > >re-process its input from the beginning. > > > > > > > > > > > Thank you for catching this issue. It is fixed now. > > > > > > Best, > > > Dong > > > > > > > > > > > > > > > > > > Best regards > > >
Re: [VOTE] FLIP-322 Cooldown period for adaptive scheduler. Second vote.
Hi all, Would you mind casting your vote to this second vote thread (opened after new discussions) so that the subject can move forward ? @David, @Chesnay, @Robert you took part to the discussions, can you please sent your vote ? Thank you very much Best Etienne Le 06/07/2023 à 13:02, Etienne Chauchot a écrit : Hi all, Thanks for your feedback about the FLIP-322: Cooldown period for adaptive scheduler [1]. This FLIP was discussed in [2]. I'd like to start a vote for it. The vote will be open for at least 72 hours (until July 9th 15:00 GMT) unless there is an objection or insufficient votes. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-322+Cooldown+period+for+adaptive+scheduler [2] https://lists.apache.org/thread/qvgxzhbp9rhlsqrybxdy51h05zwxfns6 Best, Etienne
[jira] [Created] (FLINK-32582) Move TypeSerializerUpgradeTestBase from Kafka connector into flink-connector-common
Matthias Pohl created FLINK-32582:
-
Summary: Move TypeSerializerUpgradeTestBase from Kafka connector
into flink-connector-common
Key: FLINK-32582
URL: https://issues.apache.org/jira/browse/FLINK-32582
Project: Flink
Issue Type: Improvement
Components: Connectors / Common
Reporter: Matthias Pohl
The externalization of connectors made caused problems with the Flink's test
data generation. The Kafka connector relied on TypeSerializerUpgradeTestBase
for some test cases which was fine prior to FLINK-27518 where the test data
generation was handled individually.
With FLINK-27518 the process was automated in Flink 1.18. For now, the
TypeSerializerUpgradeTestBase class was just copied over into the Kafka
connector since it was the only connector that would utilize this test base.
But we might want to provide a more generalized solution where the test base is
provided by {{flink-connector-common}} to offer a generalized approach for any
connector.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [VOTE] Release 2.0 must-have work items
Thanks Xintong for clarification. A guideline to help users migrating from DataSet to DataStream will definitely be helpful. Best regards, Yuxia - 原始邮件 - 发件人: "Xintong Song" 收件人: "dev" 发送时间: 星期三, 2023年 7 月 12日 上午 11:40:12 主题: Re: [VOTE] Release 2.0 must-have work items @Yuxia, We are aware of the issue that you mentioned. Actually, I don't think the DataStream API can cover everything in the DataSet API in exactly the same way, because the fundamental model, concepts and primitives of the two sets of APIs are completely different. Many of the DataSet APIs, especially those accessing the full data set at once, do not fit in the DataStream concepts at all. I think what's important is that users can achieve the same function, even if they may need to code in a different way. We have gone through all the existing DataSet APIs, and categorized them into 3 kinds: - APIs that are well supported by DataStream API as is. E.g., map, reduce on grouped dataset, etc. - APIs that can be achieved by DataStream API as is, but with a price (programming complexity, or computation efficiency). E.g., reduce on full dataset, sort partition, etc. Admittedly, there is room for improvement on these. We may keep improving these for the DataStream API, or we can concentrate on supporting them better in the new ProcessFunction API. Either way, I don't think we should block the retiring of DataSet API on them. - There are also a few APIs that cannot be supported by the DataStream API as is, unless users write their custom operators from the ground up. Only left/rightOuterJoin and combineGroup fall into this category. I think combinedGroup is probably not a problem, because this is more like a variant of reduceGroup that allows the framework to execute more efficiently. As for the outer joins, depending on how badly this is needed, it can be supported by emitting the non-joined entries upon triggering a window join. We are also planning to draft a guideline to help users migrating from DataSet to DataStream, which should demonstrate how users can achieve things like sort-partition with DataStream API. Last but not least, I'd like to point out that the decision to deprecate and eventually remove the DataSet API was approved in FLIP-131, and all the prerequisites mentioned in the FLIP have been completed. Best, Xintong [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158866741 On Wed, Jul 12, 2023 at 10:20 AM Jingsong Li wrote: > +1 to Leonard and Galen and Jing. > > About Source and Sink. > We're still missing quite a bit of work, including functionality, > including ease of use, including bug fixes, and I'm not sure we'll be > completely done by 2.0. > Until that's done, we won't be in a position to clean up the old APIs. > > Best, > Jingsong > > On Wed, Jul 12, 2023 at 9:41 AM yuxia wrote: > > > > Hi，Xintong. > > Sorry to disturb the voting. I just found an email[1] about DataSet API > from flink-user-zh channel. And I think it's not just a single case > according to my observation. > > > > Remove DataSet is a must have item in release-2.0. But as the user email > said, if we remove DataSet, how users can implement Sort/PartitionBy, etc > as they did with DataSet? > > Do we will also provide similar api in datastream or some other thing > before we remove DataSet? > > Btw, as far as I see, with regarding to replcaing DataSet with > Datastream, Datastream are missing many API. I think it may well take much > effort to fully cover the missing api. > > > > [1] https://lists.apache.org/thread/syjmt8f74gh8ok3z4lhgt95zl4dzn168 > > > > Best regards, > > Yuxia > > > > - 原始邮件 - > > 发件人: "Jing Ge" > > 收件人: "dev" > > 发送时间: 星期三, 2023年 7 月 12日 上午 1:23:40 > > 主题: Re: [VOTE] Release 2.0 must-have work items > > > > agree with what Leonard said. There are actually more issues wrt the new > > Source and SinkV2[1] > > > > Speaking of must-have vs nice-to-have, I think it depends on the > priority. > > If removing them has higher priority, we should keep related tasks as > > must-have and make sure enough effort will be put to solve those issues > and > > therefore be able to remove those APIs. > > > > Best regards, > > Jing > > > > [1] https://lists.apache.org/thread/90qc9nrlzf0vbvg92klzp9ftxxc43nbk > > > > On Tue, Jul 11, 2023 at 10:26 AM Leonard Xu wrote: > > > > > Thanks Xintong for driving this great work! But I’ve to give my > > > -1(binding) here： > > > > > > -1 to mark "deprecat SourceFunction/SinkFunction/Sinkv1" item as must > to > > > have for release 2.0. > > > > > > I do a lot of connector work in the community, and I have two insights > > > from past experience: > > > > > > 1. Many developers reported that it is very difficult to migrate from > > > SourceFunction to new Source [1]. The migration of existing conenctors > > > after deprecated SourceFunction is very difficult. Some developers > (Flavio > > > Pompermaier) reported that they gave up the migration because it was > too
