Re: [DISCUSS] FLIP-447: Upgrade FRocksDB from 6.20.3 to 8.10.0

[Congxian Qiu]

Thanks for driving this,  yue

We also observed significant performance improvements in some cases after
bumped the Rocksdb version, +1 for this work

Best,
Congxian


yue ma  于2024年4月24日周三 19:16写道：

> hi Yanfei,
>
> Thanks for your feedback and reminders I have updated related information.
> In fact, most of them use the default Configrations.
>
> Yanfei Lei  于2024年4月23日周二 12:51写道：
>
> > Hi Yue & Roman,
> >
> > Thanks for initiating this FLIP and all the efforts for the upgrade.
> >
> > 8.10.0 introduces some new features, making it possible for Flink to
> > implement some new exciting features, and the upgrade also makes
> > FRocksDB easier to maintain, +1 for upgrading.
> >
> > I read the FLIP and have a minor comment, it would be better to add
> > some description about the environment/configuration of the nexmark's
> > result.
> >
> > Roman Khachatryan  于2024年4月23日周二 12:07写道：
> >
> > >
> > > Hi,
> > >
> > > Thanks for writing the proposal and preparing the upgrade.
> > >
> > > FRocksDB  definitely needs to be kept in sync with the upstream and the
> > new
> > > APIs are necessary for faster rescaling.
> > > We're already using a similar version internally.
> > >
> > > I reviewed the FLIP and it looks good to me (disclaimer: I took part in
> > > some steps of this effort).
> > >
> > >
> > > Regards,
> > > Roman
> > >
> > > On Mon, Apr 22, 2024, 08:11 yue ma  wrote:
> > >
> > > > Hi Flink devs,
> > > >
> > > > I would like to start a discussion on FLIP-447: Upgrade FRocksDB from
> > > > 6.20.3 to 8.10.0
> > > >
> > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-447%3A+Upgrade+FRocksDB+from+6.20.3++to+8.10.0
> > > >
> > > > This FLIP proposes upgrading the version of FRocksDB in the Flink
> > Project
> > > > from 6.20.3 to 8.10.0.
> > > > The FLIP mainly introduces the main benefits of upgrading FRocksDB,
> > > > including the use of IngestDB which can improve Rescaling performance
> > by
> > > > more than 10 times in certain scenarios, as well as other potential
> > > > optimization points such as async_io, blob db, and tiered storage.The
> > > > FLIP also presented test results based on RocksDB 8.10, including
> > > > StateBenchmark and Nexmark tests.
> > > > Overall, upgrading FRocksDB may result in a small regression of write
> > > > performance( which is a very small part of the overall overhead), but
> > it
> > > > can bring many important performance benefits.
> > > > So we hope to upgrade the version of FRocksDB through this FLIP.
> > > >
> > > > Looking forward to everyone's feedback and suggestions. Thank you!
> > > > --
> > > > Best regards,
> > > > Yue
> > > >
> >
> >
> >
> > --
> > Best,
> > Yanfei
> >
>
>
> --
> Best,
> Yue
>

[jira] [Created] (FLINK-35144) Support multi source sync for FlinkCDC

[Congxian Qiu (Jira)]

Congxian Qiu created FLINK-35144:


 Summary: Support multi source sync for FlinkCDC
 Key: FLINK-35144
 URL: https://issues.apache.org/jira/browse/FLINK-35144
 Project: Flink
  Issue Type: Improvement
  Components: Flink CDC
Affects Versions: cdc-3.1.0
Reporter: Congxian Qiu


Currently, the FlinkCDC pipeline can only support a single source in one 
pipeline, we need to start multiple pipelines when there are various sources. 

For upstream which uses sharding, we need to sync multiple sources in one 
pipeline, the current pipeline can't do this because it can only support a 
single source.

This issue wants to support the sync of multiple sources in one pipeline.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New Apache Flink Committer - Zakelly Lan

[Congxian Qiu]

Congratulations!

Best,
Congxian


Ron liu  于2024年4月15日周一 11:00写道：

> Congratulations!
>
> Best,
> Ron
>
> Yuan Mei  于2024年4月15日周一 10:51写道：
>
> > Hi everyone,
> >
> > On behalf of the PMC, I'm happy to let you know that Zakelly Lan has
> become
> > a new Flink Committer!
> >
> > Zakelly has been continuously contributing to the Flink project since
> 2020,
> > with a focus area on Checkpointing, State as well as frocksdb (the
> default
> > on-disk state db).
> >
> > He leads several FLIPs to improve checkpoints and state APIs, including
> > File Merging for Checkpoints and configuration/API reorganizations. He is
> > also one of the main contributors to the recent efforts of "disaggregated
> > state management for Flink 2.0" and drives the entire discussion in the
> > mailing thread, demonstrating outstanding technical depth and breadth of
> > knowledge.
> >
> > Beyond his technical contributions, Zakelly is passionate about helping
> the
> > community in numerous ways. He spent quite some time setting up the Flink
> > Speed Center and rebuilding the benchmark pipeline after the original one
> > was out of lease. He helps build frocksdb and tests for the upcoming
> > frocksdb release (bump rocksdb from 6.20.3->8.10).
> >
> > Please join me in congratulating Zakelly for becoming an Apache Flink
> > committer!
> >
> > Best,
> > Yuan (on behalf of the Flink PMC)
> >
>

Re: Re: Re: [VOTE] Accept Flink CDC into Apache Flink

[Congxian Qiu]

+1 (non-binding)

Best,
Congxian


Gyula Fóra  于2024年1月15日周一 15:37写道：

> +1 (binding)
>
> On Sun, Jan 14, 2024 at 9:01 AM Yun Tang  wrote:
>
> > +1 (non-binding)
> >
> > On 2024/01/14 02:26:13 Yun Gao wrote:
> > > +1 (binding)
> > >
> > > On Sat, Jan 13, 2024 at 10:08 AM Rodrigo Meneses 
> > wrote:
> > > >
> > > > +1 (non binding)
> > > >
> > > > On Fri, Jan 12, 2024 at 5:45 PM Dong Lin 
> wrote:
> > > >
> > > > > +1 (binding)
> > > > >
> > > > > On Sat, Jan 13, 2024 at 6:04 AM Austin Bennett 
> > wrote:
> > > > >
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > On Fri, Jan 12, 2024 at 5:44 PM Becket Qin  >
> > wrote:
> > > > > >
> > > > > > > +1 (binding)
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jiangjie (Becket) Qin
> > > > > > >
> > > > > > > On Fri, Jan 12, 2024 at 5:58 AM Zhijiang <
> > wangzhijiang...@aliyun.com
> > > > > > > .invalid>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > +1 (binding)
> > > > > > > > Best,
> > > > > > > > Zhijiang
> > > > > > > >
> > --
> > > > > > > > From:Kurt Yang 
> > > > > > > > Send Time:2024年1月12日(星期五) 15:31
> > > > > > > > To:dev
> > > > > > > > Subject:Re: Re: Re: [VOTE] Accept Flink CDC into Apache Flink
> > > > > > > > +1 (binding)
> > > > > > > > Best,
> > > > > > > > Kurt
> > > > > > > > On Fri, Jan 12, 2024 at 2:21 PM Hequn Cheng <
> he...@apache.org>
> > > > > wrote:
> > > > > > > > > +1 (binding)
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Hequn
> > > > > > > > >
> > > > > > > > > On Fri, Jan 12, 2024 at 2:19 PM godfrey he <
> > godfre...@gmail.com>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > +1 (binding)
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Godfrey
> > > > > > > > > >
> > > > > > > > > > Zhu Zhu  于2024年1月12日周五 14:10写道：
> > > > > > > > > > >
> > > > > > > > > > > +1 (binding)
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Zhu
> > > > > > > > > > >
> > > > > > > > > > > Hangxiang Yu  于2024年1月11日周四
> > 14:26写道：
> > > > > > > > > > >
> > > > > > > > > > > > +1 (non-binding)
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Jan 11, 2024 at 11:19 AM Xuannan Su <
> > > > > > > suxuanna...@gmail.com
> > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > +1 (non-binding)
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best,
> > > > > > > > > > > > > Xuannan
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Jan 11, 2024 at 10:28 AM Xuyang <
> > > > > xyzhong...@163.com>
> > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > +1 (non-binding)--
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best！
> > > > > > > > > > > > > > Xuyang
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 在 2024-01-11 10:00:11，"Yang Wang" <
> > > > > wangyang0...@apache.org
> > > > > > >
> > > > > > > > 写道：
> > > > > > > > > > > > > > >+1 (binding)
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >Best,
> > > > > > > > > > > > > > >Yang
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >On Thu, Jan 11, 2024 at 9:53 AM liu ron <
> > > > > > ron9@gmail.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >> +1 non-binding
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Best
> > > > > > > > > > > > > > >> Ron
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Matthias Pohl  .invalid>
> > > > > > > > 于2024年1月10日周三
> > > > > > > > > > > > 23:05写道：
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> > +1 (binding)
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >> > On Wed, Jan 10, 2024 at 3:35 PM ConradJam <
> > > > > > > > > > jam.gz...@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >> > > +1 non-binding
> > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > >> > > Dawid Wysakowicz 
> > > > > > > 于2024年1月10日周三
> > > > > > > > > > > > 21:06写道：
> > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > >> > > > +1 (binding)
> > > > > > > > > > > > > > >> > > > Best,
> > > > > > > > > > > > > > >> > > > Dawid
> > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > >> > > > On Wed, 10 Jan 2024 at 11:54, Piotr
> > Nowojski <
> > > > > > > > > > > > > pnowoj...@apache.org>
> > > > > > > > > > > > > > >> > > wrote:
> > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > >> > > > > +1 (binding)
> > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > > >> > > > > śr., 10 sty 2024 o 11:25 Martijn
> Visser
> > <
> > > > > > > > > > > > > martijnvis...@apache.org>
> > > > > > > > > > > > > > >> > > > > napisał(a):
> > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > >

Re: [ANNOUNCE] Performance Daily Monitoring Moved from Ververica to Apache Flink Slack Channel

[Congxian Qiu]

Thanks for driving this and making the performance monitoring public,  this
can make us know and resolve the performance problem quickly.

Looking forward to the workflow and detailed descriptions fo
flink-dev-benchmarks.

Best,
Congxian


Yun Tang  于2022年10月27日周四 12:41写道：

> Thanks, Yanfei for driving this to monitor the performance in the Apache
> Flink Slack Channel.
>
> Look forward to the workflow and detailed descriptions of
> flink-dev-benchmarks.
>
> Best
> Yun Tang
> 
> From: Hangxiang Yu 
> Sent: Thursday, October 27, 2022 10:59
> To: dev@flink.apache.org 
> Subject: Re: [ANNOUNCE] Performance Daily Monitoring Moved from Ververica
> to Apache Flink Slack Channel
>
> Hi, Yanfei.
> Thanks for driving this.
> It could help us to detect and resolve the regression problem quickly and
> officially.
> I'd like to join as a maintainer.
> Looking forward to the workflow.
>
> On Wed, Oct 26, 2022 at 5:18 PM Yuan Mei  wrote:
>
> > Thanks, Yanfei, to drive this and make the performance monitoring
> publicly
> > available.
> >
> > Looking forward to seeing the workflow, and more details as Martijn
> > mentioned.
> >
> > Best
> > Yuan
> >
> > On Wed, Oct 26, 2022 at 2:59 PM Martijn Visser  >
> > wrote:
> >
> > > Hi Yanfei Lei,
> > >
> > > Thanks for setting this up! It would be interesting to also know which
> > > aspects of Flink are monitored for "performance". I'm assuming there
> are
> > > specific pieces of functionality that are performance tested, but it
> > would
> > > be great if this would be written down somewhere (next to a procedure
> how
> > > to detect a regression and what should be next steps).
> > >
> > > Best regards,
> > >
> > > Martijn
> > >
> > > On Wed, Oct 26, 2022 at 8:21 AM Zakelly Lan 
> > wrote:
> > >
> > > > Hi yanfei,
> > > >
> > > > Thanks for driving this! It's a great help.
> > > >
> > > > I would like to join as a maintainer.
> > > >
> > > > Best,
> > > > Zakelly
> > > >
> > > > On Wed, Oct 26, 2022 at 11:32 AM yanfei lei 
> > wrote:
> > > > >
> > > > > Hi everyone,
> > > > >
> > > > > As discussed earlier, we plan to create a benchmark channel in
> Apache
> > > > Flink
> > > > > slack[1], but the plan was shelved for a while[2]. So I went on
> with
> > > this
> > > > > work, and created the #flink-dev-benchmarks channel for performance
> > > > > regression notifications.
> > > > >
> > > > > We have a regression report script[3] that runs daily, and a
> > > notification
> > > > > would be sent to the slack channel when the last few benchmark
> > results
> > > > are
> > > > > significantly worse than the baseline.
> > > > > Note, regressions are detected by a simple script which may have
> > false
> > > > > positives and false negatives. And all benchmarks are executed on
> one
> > > > > physical machine[4] which is provided by Ververica(Alibaba)[5], it
> > > might
> > > > > happen that hardware issues affect performance, like "[FLINK-18614
> > > > > ] Performance
> > > > regression
> > > > > 2020.07.13"[6].
> > > > >
> > > > > After the migration, we need a procedure to watch over the entire
> > > > > performance of Flink code together. For example, if a regression
> > > > > occurs, investigating the cause and resolving the problem are
> needed.
> > > In
> > > > > the past, this procedure is maintained internally within Ververica,
> > but
> > > > we
> > > > > think making the procedure public would benefit all. I volunteer to
> > > serve
> > > > > as one of the initial maintainers, and would be glad if more
> > > contributors
> > > > > can join me. I'd also prepare some guidelines to help others get
> > > familiar
> > > > > with the workflow. I will start a new thread to discuss the
> workflow
> > > > soon.
> > > > >
> > > > >
> > > > > [1]
> https://www.mail-archive.com/dev@flink.apache.org/msg58666.html
> > > > > [2] https://issues.apache.org/jira/browse/FLINK-28468
> > > > > [3]
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink-benchmarks/blob/master/regression_report.py
> > > > > [4] http://codespeed.dak8s.net:8080
> > > > > [5]
> https://lists.apache.org/thread/jzljp4233799vwwqnr0vc9wgqs0xj1ro
> > > > >
> > > > > [6] https://issues.apache.org/jira/browse/FLINK-18614
> > > >
> > >
> >
>
>
> --
> Best,
> Hangxiang.
>

Re: [DISCUSS] Planning Flink 1.17

[Congxian Qiu]

Thanks for kicking off the 1.17 release, and volunteers as release managers.

+1 to the feature freeze date

Best,
Congxian


Yun Tang  于2022年10月21日周五 17:24写道：

> Thanks, Qingsheng and Leonard to kick off the release plan of flink-1.17,
> which targets a feature freeze date of January 17th 🙂.
>
> +1 to include Martijn and Matthias as the release managers.
>
> Best,
> Yun Tang
> 
> From: Piotr Nowojski 
> Sent: Friday, October 21, 2022 16:22
> To: dev@flink.apache.org 
> Subject: Re: [DISCUSS] Planning Flink 1.17
>
> Hi,
>
> Thanks for kicking this discussion off Qingsheng and Leonard. I like the
> January 17th, 2023 touch :)
>
> 1 for including Matthias Pohl and Martijn Visser as release managers.
>
> Best,
> Piotrek
> Always alright
>
> pt., 21 paź 2022 o 05:55 Yuan Mei  napisał(a):
>
> > Thanks, Qingsheng for the kicking-off efforts.
> >
> > 1. January 17th, 2023 as feature freeze data sounds reasonable to me.
> > 2. We will input our plan to the wiki link.
> >
> > Thanks
> >
> > Best
> > Yuan
> > Ververica (Alibaba)
> >
> >
> > On Fri, Oct 21, 2022 at 10:38 AM Xingbo Huang  wrote:
> >
> > > Thanks Qingsheng, Leonard and Martijn for starting the discussion and
> > > volunteering.
> > > The timeline proposal sounds reasonable :+1:
> > >
> > > Best,
> > > Xingbo
> > >
> > > Jark Wu  于2022年10月21日周五 00:17写道：
> > >
> > > > Thanks for kicking off the 1.17 release.
> > > >
> > > > Targeting feature freeze on 1/17 for 1.17 release sounds pretty good
> to
> > > me.
> > > > +1 for the volunteers as release managers.
> > > >
> > > > Best,
> > > > Jark
> > > > Ververica (Alibaba)
> > > >
> > > > On Thu, 20 Oct 2022 at 18:09, Matthias Pohl  > > > .invalid>
> > > > wrote:
> > > >
> > > > > Thanks for starting the discussion about Flink 1.17. I would be
> > > > interested
> > > > > in helping out around the release as well.
> > > > >
> > > > > Best,
> > > > > Matthias
> > > > >
> > > > > On Thu, Oct 20, 2022 at 12:07 PM Xintong Song <
> tonysong...@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Thanks for kicking this off.
> > > > > >
> > > > > > +1 for the proposed timeline.
> > > > > >
> > > > > > Also +1 for Qingsheng, Leonard and Martijn as the release
> managers.
> > > > > Thanks
> > > > > > for volunteering.
> > > > > >
> > > > > > Best,
> > > > > >
> > > > > > Xintong
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, Oct 20, 2022 at 3:59 PM Martijn Visser <
> > > > martijnvis...@apache.org
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Qingsheng,
> > > > > > >
> > > > > > > I'm definitely interested in participating as a release manager
> > > > again.
> > > > > > >
> > > > > > > Best regards,
> > > > > > >
> > > > > > > Martijn
> > > > > > >
> > > > > > > On Thu, Oct 20, 2022 at 9:47 AM Qingsheng Ren <
> re...@apache.org>
> > > > > wrote:
> > > > > > >
> > > > > > > > Hi everyone,
> > > > > > > >
> > > > > > > > As we are approaching the official release of Flink 1.16,
> it’s
> > a
> > > > good
> > > > > > > time
> > > > > > > > to kick off some discussions and march toward 1.17.
> > > > > > > >
> > > > > > > > - Release managers
> > > > > > > >
> > > > > > > > Leonard Xu and I would like to volunteer as release managers
> > for
> > > > > 1.17,
> > > > > > > and
> > > > > > > > it would be great to have someone else working together on
> this
> > > > > > release.
> > > > > > > > Please let us know if you have any interest!
> > > > > > > >
> > > > > > > > - Timeline
> > > > > > > >
> > > > > > > > Having 1.16 will be released in the next few days and the 4
> > > months
> > > > > > > release
> > > > > > > > cycle after that, we propose to set the feature freezing date
> > on
> > > > > > *January
> > > > > > > > 17th, 2023* (aligned with our version number 1.17 :-)), so
> that
> > > > > > everyone
> > > > > > > > could enjoy the holiday season and Chinese new year.
> > > > > > > >
> > > > > > > > - What we’ll be focusing
> > > > > > > >
> > > > > > > > Similar to our previous releases, we’d like to keep an eye on
> > the
> > > > > > > > timeline, CI stability, release testing, and any
> communication
> > > and
> > > > > > > > coordination across teams and developers. One thing we’d like
> > to
> > > > > > mention
> > > > > > > in
> > > > > > > > particular is compatibility, which is a frequent complaint
> from
> > > our
> > > > > > > > ecosystem developers and users. We encourage all committers
> to
> > do
> > > > an
> > > > > > > extra
> > > > > > > > manual check to see if any public interface is touched before
> > > > > merging a
> > > > > > > PR.
> > > > > > > > We could discuss details in another thread later and update
> the
> > > > > > > > contributing guidelines to list which should be treated as
> > public
> > > > > APIs.
> > > > > > > > Please feel free to raise any discussions if you have
> anything
> > > else
> > > > > to
> > > > > > > > emphasize specifically.
> > > > > > > >
> > > > > > > > - Collecting features
> > > > > > > >
> > > > > > > >

Re: [ANNOUNCE] New Apache Flink PMC Member - Danny Cranmer

[Congxian Qiu]

Congratulations Danny!

Best,
Congxian


Leonard Xu  于2022年10月11日周二 18:03写道：

> Congratulations Danny!
>
>
> Best,
> Leonard
>
>

Re: [VOTE] Release 1.16.0, release candidate #1

[Congxian Qiu]

+1 (non-binding)

- checked sha512 & gpg
- tag in the GitHub repo seems ok
- no binary files found in the source
- check all versions in the pom file nothing suspicious found
- compile from source ok
- add one comment about the release blog
- deploy standalone session cluster & application job, run streaming
example jobs, seems ok

Best,
Congxian


Teoh, Hong  于2022年10月9日周日 04:32写道：

>
> +1 (non-binding)
>
> * Hashes and Signatures look good
> * All required files on dist.apache.org
> * Tag is present in Github
> * Verified source archive does not contain any binary files
> * Source archive builds using maven
> * Deployed standalone session cluster and ran TopSpeedWindowing example in
> streaming. Looks ok
> * Added a couple of nits to the release PR
>
> Cheers,
> Hong
>
> On 04/10/2022, 12:56, "Matthias Pohl" 
> wrote:
>
> CAUTION: This email originated from outside of the organization. Do
> not click links or open attachments unless you can confirm the sender and
> know the content is safe.
>
>
>
> +1 (non-binding)
>
> * Downloaded artifacts
> * Verified checksums/GPG signatures
> * Compared checkout with provided sources
> * Verified pom file versions
> * Went over NOTICE file/pom files changes without finding anything
> suspicious
> * Build Flink from sources
> * Deployed standalone session cluster and ran WordCount example in
> batch
> and streaming: Nothing suspicious in log files found
>
> On Fri, Sep 30, 2022 at 3:56 PM Matthias Pohl 
> wrote:
>
> > Ok, I was wrong. The step is actually documented at the end of the
> Flink
> > release documentation [1] in item 15) in the "Checklist to declare
> the
> > process completed" subsection. I missed that one. Sorry for the
> confusion.
> >
> > I revoke my veto and close FLINK-29485 [2].
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/Creating+a+Flink+Release
> > [2] https://issues.apache.org/jira/browse/FLINK-29485
> >
> > On Fri, Sep 30, 2022 at 3:50 PM Matthias Pohl <
> matthias.p...@aiven.io>
> > wrote:
> >
> >> Probably, my expectations were wrong here: I expected that we want
> to
> >> test the compatibility in these tests between different major
> versions and
> >> that we would want to verify the current version of the release
> branch as
> >> well before releasing the artifacts. What's the rationale behind
> doing it
> >> after the artifacts are released?
> >>
> >> On Fri, Sep 30, 2022 at 3:37 PM Chesnay Schepler <
> ches...@apache.org>
> >> wrote:
> >>
> >>> Could you be more specific as to what you believe should be
> updated?
> >>>
> >>> IIRC the release-1.16 branch only gets updated ta all once the
> release
> >>> is out (e.g., mark docs as stable, update japicmp reference).
> >>>
> >>> On 30/09/2022 15:32, Matthias Pohl wrote:
> >>> > Looking into the git history, there are numerous locations that
> need
> >>> to be
> >>> > updated in the release-1.16 branch. Yun Gao did a few commits
> around
> >>> that
> >>> > topic (da9e6be..6f69f4e). But these changes were committed close
> to the
> >>> > actual release date rather than the release branch creation
> date. Is
> >>> this
> >>> > part of the release steps [1]. It feels like it should be part
> of the
> >>> > section where we create the release branch. But cannot find the
> >>> pointer.
> >>> >
> >>> > [1]
> >>> >
> >>>
> https://cwiki.apache.org/confluence/display/FLINK/Creating+a+Flink+Release
> >>> >
> >>> > On Fri, Sep 30, 2022 at 2:47 PM Matthias Pohl <
> matthias.p...@aiven.io>
> >>> > wrote:
> >>> >
> >>> >> -1 (non-binding)
> >>> >>
> >>> >> Hi Xingbo,
> >>> >> I just noticed that we haven't updated the current Flink
> version in
> >>> >> TypeSerializerUpgradeTest. It is missing in the release-1.16
> branch
> >>> and on
> >>> >> master. That means that the serialization tests are not
> executed for
> >>> Flink
> >>> >> 1.16. See FLINK-29485 [1].
> >>> >>
> >>> >> [1] https://issues.apache.org/jira/browse/FLINK-29485
> >>> >>
> >>> >> On Fri, Sep 30, 2022 at 11:28 AM Xingbo Huang 
> wrote:
> >>> >>
> >>> >>> Hi everyone,
> >>> >>>
> >>> >>> Please review and vote on the release candidate #1 for the
> version
> >>> 1.16.0,
> >>> >>> as follows:
> >>> >>> [ ] +1, Approve the release
> >>> >>> [ ] -1, Do not approve the release (please provide specific
> comments)
> >>> >>>
> >>> >>> The complete staging area is available for your review, which
> >>> includes:
> >>> >>> * JIRA release notes [1],
> >>> >>> * the official Apache source release and binary convenience
> releases
> >>> to be
> >>> >>> deployed to dist.apache.org [2], which are signed with the
> key with
> >>> >>> fingerprint 3C2C9FFB59DF9F3E [3],
> >>> >>> *

[jira] [Created] (FLINK-23346) RocksDBStateBackend may core dump in flink_compactionfilterjni.cc

[Congxian Qiu (Jira)]

Congxian Qiu created FLINK-23346:


 Summary: RocksDBStateBackend may core dump in 
flink_compactionfilterjni.cc
 Key: FLINK-23346
 URL: https://issues.apache.org/jira/browse/FLINK-23346
 Project: Flink
  Issue Type: Bug
  Components: Runtime / State Backends
Affects Versions: 1.12.4, 1.13.1, 1.14.0
Reporter: Congxian Qiu


The code in [flink_compactionfilte.cpp 
|https://github.com/ververica/frocksdb/blob/49bc897d5d768026f1eb816d960c1f2383396ef4/java/rocksjni/flink_compactionfilterjni.cc#L21]
{code:cpp}
inline void CheckAndRethrowException(JNIEnv* env) const {
if (env->ExceptionCheck()) {
  env->ExceptionDescribe();
  env->Throw(env->ExceptionOccurred());
}
{code}
may core dump in some sence, please see more information here[1][2][3]

We can fix it by changing this to
{code:cpp}
inline void CheckAndRethrowException(JNIEnv* env) const {
if (env->ExceptionCheck()) {
  env->Throw(env->ExceptionOccurred());
}
  }
{code}
or
{code:cpp}
   inline void CheckAndRethrowException(JNIEnv* env) const {
if (env->ExceptionCheck()) {
  jobject obj = env->ExceptionOccurred();
  env->ExceptionDescribe();
  env->Throw(obj);
}
  }
{code}
[1] 
[https://stackoverflow.com/questions/30971068/does-jniexceptiondescribe-implicitily-clear-the-exception-trace-of-the-jni-env]
 [2] [https://bugs.openjdk.java.net/browse/JDK-4067541]
 [3] [https://bugs.openjdk.java.net/browse/JDK-8051947]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] Welcome Roman Khachatryan a new Apache Flink Committer

[Congxian Qiu]

Congratulations, Roman
Best,
Congxian


Leonard Xu  于2021年2月18日周四 下午1:47写道：

> Congrats Roman!
>
> Best,
> Leonard
>
> > 在 2021年2月18日，11:10，Yu Li  写道：
> >
> > Congratulations, Roman!
> >
> > Best Regards,
> > Yu
> >
> >
> > On Thu, 18 Feb 2021 at 11:05, Xingbo Huang  wrote:
> >
> >> Congratulations Roman!
> >>
> >> Best,
> >> Xingbo
> >>
> >> Yang Wang  于2021年2月18日周四 上午10:29写道：
> >>
> >>> Congrats Roman!
> >>>
> >>> Best,
> >>> Yang
> >>>
> >>> Xintong Song  于2021年2月18日周四 上午10:00写道：
> >>>
>  Congratulations, Roman~!
> 
>  Thank you~
> 
>  Xintong Song
> 
> 
> 
>  On Thu, Feb 18, 2021 at 9:42 AM Dian Fu 
> wrote:
> 
> > Congratulations, Roman!
> >
> > Regards,
> > Dian
> >
> >> 在 2021年2月16日，下午5:56，Yuan Mei  写道：
> >>
> >> Well deserved! Congrats Roman!
> >>
> >> Best,
> >> Yuan
> >>
> >> On Tue, Feb 16, 2021 at 5:10 PM Guowei Ma 
>  wrote:
> >>
> >>> Congratulations Roman!
> >>> Best,
> >>> Guowei
> >>>
> >>>
> >>> On Thu, Feb 11, 2021 at 3:37 PM Yun Tang 
> >> wrote:
> >>>
>  Congratulations, Roman!
> 
>  Today is also the beginning of Chinese Spring Festival holiday,
> >> at
> > which
>  we Chinese celebrate across the world for the next lunar new
> >> year,
>  and
> >>> also
>  very happy to have you on board!
> 
>  Best
>  Yun Tang
>  
>  From: Roman Khachatryan 
>  Sent: Thursday, February 11, 2021 4:03
>  To: matth...@ververica.com 
>  Cc: dev 
>  Subject: Re: [ANNOUNCE] Welcome Roman Khachatryan a new Apache
> >>> Flink
>  Committer
> 
>  Many thanks to all of you!
> 
>  Regards,
>  Roman
> 
> 
>  On Wed, Feb 10, 2021 at 7:12 PM Matthias Pohl <
>  matth...@ververica.com>
>  wrote:
> 
> > Congratulations, Roman! :-)
> >
> > On Wed, Feb 10, 2021 at 3:23 PM Kezhu Wang 
>  wrote:
> >
> >> Congratulations!
> >>
> >> Best,
> >> Kezhu Wang
> >>
> >>
> >> On February 10, 2021 at 21:53:52, Dawid Wysakowicz (
> >> dwysakow...@apache.org)
> >> wrote:
> >>
> >> Congratulations Roman! Glad to have you on board!
> >>
> >> Best,
> >>
> >> Dawid
> >>
> >> On 10/02/2021 14:44, Igal Shilman wrote:
> >>> Welcome Roman!
> >>> Top-notch stuff! :)
> >>>
> >>> All the best,
> >>> Igal.
> >>>
> >>> On Wed, Feb 10, 2021 at 2:15 PM Kostas Kloudas <
>  kklou...@gmail.com>
> >> wrote:
> >>>
>  Congrats Roman!
> 
>  Kostas
> 
>  On Wed, Feb 10, 2021 at 2:08 PM Arvid Heise <
> >> ar...@apache.org>
>  wrote:
> > Congrats! Well deserved.
> >
> > On Wed, Feb 10, 2021 at 1:54 PM Yun Gao
>   >>>
> > wrote:
> >
> >> Congratulations Roman!
> >>
> >> Best,
> >> Yun
> >>
> >>
> >> --Original Mail --
> >> Sender:Till Rohrmann 
> >> Send Date:Wed Feb 10 20:53:21 2021
> >> Recipients:dev 
> >> CC:Khachatryan Roman , Roman
> >> Khachatryan
>  <
> >> ro...@apache.org>
> >> Subject:Re: [ANNOUNCE] Welcome Roman Khachatryan a new
> >> Apache
> >>> Flink
> >> Committer
> >> Congratulations Roman :-)
> >>
> >> Cheers,
> >> Till
> >>
> >> On Wed, Feb 10, 2021 at 1:01 PM Konstantin Knauf <
>  kna...@apache.org>
> >> wrote:
> >>
> >>> Congratulations Roman!
> >>>
> >>> On Wed, Feb 10, 2021 at 11:29 AM Piotr Nowojski <
>  pnowoj...@apache.org>
> >>> wrote:
> >>>
>  Hi everyone,
> 
>  I'm very happy to announce that Roman Khachatryan has
>  accepted
>  the
>  invitation to
>  become a Flink committer.
> 
>  Roman has been recently active in the runtime parts of
> >> the
> >>> Flink.
>  He is
> >>> one
>  of the main developers behind FLIP-76 Unaligned
> >>> Checkpoints,
>  FLIP-151
>  Incremental Heap/FS State Backend [3] and providing a
> >>> faster
> >>> checkpointing
>  mechanism in FLIP-158.
> 
>  Please join me in congratulating Roman for becoming a
> >

Re: Re: [ANNOUNCE] Welcome Guowei Ma as a new Apache Flink Committer

[Congxian Qiu]

Congrats Guowei!

Best,
Congxian


Danny Chan  于2021年1月20日周三 下午2:59写道：

> Congratulations Guowei!
>
> Best,
> Danny
>
> Jark Wu  于2021年1月20日周三 下午2:47写道：
>
> > Congratulations Guowei!
> >
> > Cheers,
> > Jark
> >
> > On Wed, 20 Jan 2021 at 14:36, SHI Xiaogang 
> wrote:
> >
> > > Congratulations MA!
> > >
> > > Regards,
> > > Xiaogang
> > >
> > > Yun Tang  于2021年1月20日周三 下午2:24写道：
> > >
> > > > Congratulations Guowei!
> > > >
> > > > Best
> > > > Yun Tang
> > > > 
> > > > From: Yang Wang 
> > > > Sent: Wednesday, January 20, 2021 13:59
> > > > To: dev 
> > > > Subject: Re: Re: [ANNOUNCE] Welcome Guowei Ma as a new Apache Flink
> > > > Committer
> > > >
> > > > Congratulations Guowei!
> > > >
> > > >
> > > > Best,
> > > > Yang
> > > >
> > > > Yun Gao  于2021年1月20日周三 下午1:52写道：
> > > >
> > > > > Congratulations Guowei!
> > > > >
> > > > > Best,
> > > > >
> > Yun--
> > > > > Sender:Yangze Guo
> > > > > Date:2021/01/20 13:48:52
> > > > > Recipient:dev
> > > > > Theme:Re: [ANNOUNCE] Welcome Guowei Ma as a new Apache Flink
> > Committer
> > > > >
> > > > > Congratulations, Guowei! Well deserved.
> > > > >
> > > > >
> > > > > Best,
> > > > > Yangze Guo
> > > > >
> > > > > On Wed, Jan 20, 2021 at 1:46 PM Xintong Song <
> tonysong...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > Congratulations, Guowei~!
> > > > > >
> > > > > >
> > > > > > Thank you~
> > > > > >
> > > > > > Xintong Song
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Wed, Jan 20, 2021 at 1:42 PM Yuan Mei  >
> > > > wrote:
> > > > > >
> > > > > > > Congrats Guowei :-)
> > > > > > >
> > > > > > > Best,
> > > > > > > Yuan
> > > > > > >
> > > > > > > On Wed, Jan 20, 2021 at 1:36 PM tison 
> > > wrote:
> > > > > > >
> > > > > > > > Congrats Guowei!
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > tison.
> > > > > > > >
> > > > > > > >
> > > > > > > > Kurt Young  于2021年1月20日周三 下午1:34写道：
> > > > > > > >
> > > > > > > > > Hi everyone,
> > > > > > > > >
> > > > > > > > > I'm very happy to announce that Guowei Ma has accepted the
> > > > > invitation
> > > > > > > to
> > > > > > > > > become a Flink committer.
> > > > > > > > >
> > > > > > > > > Guowei is a very long term Flink developer, he has been
> > > extremely
> > > > > > > helpful
> > > > > > > > > with
> > > > > > > > > some important runtime changes, and also been  active with
> > > > > answering
> > > > > > > user
> > > > > > > > > questions as well as discussing designs.
> > > > > > > > >
> > > > > > > > > Please join me in congratulating Guowei for becoming a
> Flink
> > > > > committer!
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Kurt
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] Apache Flink 1.12.0 released

[Congxian Qiu]

Thanks Dian and Robert for the great work as release manager, and everyone
who makes this release possible.

Best,
Congxian


Xintong Song  于2020年12月11日周五 下午1:49写道：

> Congratulations~! Thanks Robert, Dian and everyone.
>
> Thank you~
>
> Xintong Song
>
>
>
> On Thu, Dec 10, 2020 at 11:23 PM Robert Metzger 
> wrote:
>
> > Thanks Nick!
> >
> > This is blocked on the maintainers of the official images to merge this
> PR:
> > https://github.com/docker-library/official-images/pull/9249
> > I can not really make an estimate when they'll merge it.
> >
> > On Thu, Dec 10, 2020 at 4:20 PM nick toker 
> > wrote:
> >
> > > Hi
> > > first good job  and tank you
> > >
> > > i don't find in docker hub the new version 1.12
> > >
> > > when it will be there ?
> > >
> > > nick
> > >
> > > ‫בתאריך יום ה׳, 10 בדצמ׳ 2020 ב-14:17 מאת ‪Robert Metzger‬‏ <‪
> > > rmetz...@apache.org‬‏>:‬
> > >
> > >> The Apache Flink community is very happy to announce the release of
> > >> Apache Flink 1.12.0, which is the latest major release.
> > >>
> > >> Apache Flink® is an open-source stream processing framework for
> > >> distributed, high-performing, always-available, and accurate data
> > streaming
> > >> applications.
> > >>
> > >> The release is available for download at:
> > >> https://flink.apache.org/downloads.html
> > >>
> > >> Please check out the release blog post for an overview of the
> > >> improvements for this bugfix release:
> > >> https://flink.apache.org/news/2020/12/10/release-1.12.0.html
> > >>
> > >> The full release notes are available in Jira:
> > >>
> > >>
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12348263
> > >>
> > >> We would like to thank all contributors of the Apache Flink community
> > who
> > >> made this release possible!
> > >>
> > >> Regards,
> > >> Dian & Robert
> > >>
> > >>
> >
>

Re: [VOTE] Release 1.12.0, release candidate #3

[Congxian Qiu]

+1  (non-biding)

checklist
- checksum & gpg  ok
- build from source, ok
- all pom version point to 1.12.0
- run some program in local env, ok, and no strange log showed

Best,
Congxian


Dawid Wysakowicz  于2020年12月7日周一 下午5:52写道：

> +1 (binding)
>
> * Verified the checksums
> * Verified that the source archives do not contain any binaries
> * Built Flink from sources
> * Run a streaming WorldCount example in BATCH and STREAM mode
> * Run a slightly heavier WorldCount version in BATCH and STREAM mode
> * Verified licensing of Hbase connectors*** I found one issue that we do not 
> list protobuf-java-util in flink-sql-connector-hbase-2.2 NOTICE file, which 
> is pulled in through hbase-shaded-miscellaneous (there is a couple more 
> dependencies we do not list from it, but they are Apache licensed)*
> * Run some Table examples (I found two examples are not runnable[1]), but I 
> would not consider it a blocker
>
> Let me know what you think about the licensing issue.
>
> Best,
> Dawid
>
> [1]https://issues.apache.org/jira/browse/FLINK-20464
>
> On 07/12/2020 08:07, Leonard Xu wrote:
>
> +1 (non-binding)
>
> - checked/verified signatures and hashes
> - built from source code with scala 2.11 succeeded
> - checked that there are no missing artifacts
> - started a cluster, WebUI was accessible, submitted a wordcount job and ran 
> succeeded, no suspicious log output
> - tested using SQL Client to submit job and the query result is expected
> - tested read/write from/to sql kafka/upsert-kafka connector in SQL Client
> - tested read/write/join hvie table in SQL Client
>
> Best,
> Leonard Xu
>
>
>
> 在 2020年12月7日，11:17，Guowei Ma   写道：
>
> +1(non-binding)
> - build from source
> - build a docker image
> - start a session from local k8s cluster
> - submit a wordcount job in streaming mode.
> - submit a wordcount job in batch mode.
> Best,
> Guowei
>
>
> On Sat, Dec 5, 2020 at 3:13 PM Zhu Zhu  
>  wrote:
>
>
> +1 (binding)
>
> - verified signature and checksum
> - built from source
> - run testing jobs on yarn with manually triggered failures. checked logs
> and WebUI of those jobs
>  * DataStream job (paralelism=1000) with multiple disjoint pipelined
> regions
>  * DataSet job (paralelism=1000) with all edges blocking
>
> Thanks,
> Zhu
>
> Till Rohrmann   于2020年12月4日周五 
> 下午11:45写道：
>
>
> +1 (binding)
>
> * Verified the checksums
> * Ran RC on Minikube cluster
> ** Session mode
> ** Application mode
> * Built Flink from sources
>
> Cheers,
> Till
>
> On Fri, Dec 4, 2020 at 2:15 PM Wei Zhong  
>  wrote:
>
>
> +1 (non-binding)
>
> - verified checksums and signatures
> - build Flink with Scala 2.11
> - pip install pyflink on Windows python 3.7
> - run a python job with udfs on Windows
> - pyflink shell works well on local mode and remote mode
>
> Best,
> Wei
>
>
> 在 2020年12月4日，17:21，Yang Wang   
> 写道：
>
> +1 (non-binding)
>
> * Build from source
> * Deploy Flink cluster in following deployments with HA
>
> enabled(ZooKeeper
>
> and K8s), including kill JobManager and check failover
> * Native K8s Session
> * Native K8s Application
> * Yarn Session
> * Yarn Per-Job
> * Yarn Application
> * Check webUI and logs in different deployments especially via
>
> `kubectl
>
> logs` in K8s
>
> Best,
> Yang
>
> Xintong Song   于2020年12月4日周五 
> 下午3:00写道：
>
>
> +1 (non-binding)
>
>  - Verified checksums and signatures
>  - No binaries found in source archive
>  - Build from source
>  - Tried a couple of example jobs in various deployment mode
> - Local
> - Standalone
> - Native Kubernetes Application
> - Native Kubernetes Session
> - Yarn Job
> - Yarn Session
>  - Changing memory configurations, things work as expected
>  - UI looks good
>  - Logs look good
>
>
>
> Thank you~
>
> Xintong Song
>
>
>
> On Thu, Dec 3, 2020 at 9:18 PM Rui Li  
> 
>
> wrote:
>
> +1 (non-binding)
>
> Built from source and verified hive connector tests for different
>
> hive
>
> versions.
> Setup a cluster to connect to a real hive warehouse and run some
>
> queries
>
> successfully.
>
> On Thu, Dec 3, 2020 at 8:44 PM Xingbo Huang  
> 
>
> wrote:
>
> +1 (non-binding)
>
> Checks:
> 1. verified checksums and signatures
> 2. build Flink with Scala 2.11
> 3. pip install pyflink in MacOS/CentOS under py35,py36,py37,py38
> 4. test Pandas UDAF/General UDAF/Python DataStream MapFunction
> 5. start standalone cluster and submit a python udf job.
> 6. verified NOTICE/LICENSE files of some regular modules
>
> I observed that the NOTICE file of flink-sql-connector-hbase-2.2
>
> lists
>
> 3
>
> dependencies that are not bundled in:
> commons-lang:commons-lang:2.6
> org.apache.hbase:hbase-hadoop-compat:2.2.3
> org.apache.hbase:hbase-hadoop2-compat:2.2.3
>
> I guess listing more than dependencies with apache licensed
>
> shouldn't
>
> be
>
> a
>
> blocker issue. I have opened a PR[1] to fix it.
>
> [1] https://github.com/apache/flink/pull/14299
>
> Best,
> Xingbo
>
> Robert Metzger   于2020年12月3日周四 
> 下午5:36写道：
>
>
> There's now a pull request for the 

Re: [ANNOUNCE] New Apache Flink Committer - Congxian Qiu

[Congxian Qiu]

Thanks all for the support. It's a great honor for me.

Best,
Congxian


Paul Lam  于2020年10月30日周五 下午3:18写道：

> Congrats, Congxian! Well deserved!
>
> Best,
> Paul Lam
>
> > 2020年10月30日 15:12，Zhijiang  写道：
> >
> > Congrats, Congxian!
>
>

Re: Re: [VOTE] Add Translation Specification for Stateful Functions

[Congxian Qiu]

@Shawn Thanks for the notice, that's is not related to this thread, but I
think we should fix it, will update it accordingly.

Best,
Congxian


Shawn Huang  于2020年10月17日周六 上午11:27写道：

> Hi, Congxian.
> Thanks for your contribution.
> I noticed that in wiki, the link format is still like "{{ site.baseurl
> }}/.../xxx.html".
> But the "{% link %}" tag is recommended now [1].
> So is it necessary to update this in wiki? It seems some translators
> haven't noticed this.
>
> [1]
>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Reminder-Prefer-link-tag-in-documentation-td42362.html
>
> Best,
> Shawn Huang
>
>
> Congxian Qiu  于2020年10月16日周五 下午11:10写道：
>
> > FYI, I've added the Specification for Stateful Functions to the existing
> > wiki[1]
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Translation+Specifications
> >
> > Best,
> > Congxian
> >
> >
> > Congxian Qiu  于2020年10月15日周四 下午8:33写道：
> >
> > > Hi all
> > > Thanks everyone for the voting.
> > >
> > > The voting time for "Add Translation Specification for Stateful
> > > Functions" has passed, I'm closing the vote now.
> > >
> > > There were 7 votes, 4 of which are binding:
> > >- Yu Li (binding)
> > >- Jark Wu (binding)
> > >- Xintong Song (binding)
> > >- Smile
> > >- Dian Fu (binding)
> > >- Hailong Wang
> > >- Shawn Huang
> > >
> > > There were no -1 votes.
> > >
> > > Thus, changes have been accepted. I'll update the wiki accordingly.
> > >
> > > Best,
> > > Congxian
> > >
> > >
> > > Shawn Huang  于2020年10月13日周二 下午3:19写道：
> > >
> > >> +1
> > >>
> > >> Best,
> > >> Shawn Huang
> > >>
> > >>
> > >> hailongwang <18868816...@163.com> 于2020年10月12日周一 下午11:21写道：
> > >>
> > >> > +1
> > >> > Best,
> > >> > Hailong Wang
> > >> > At 2020-10-12 17:00:34, "Xintong Song" 
> wrote:
> > >> > >+1
> > >> > >
> > >> > >Thank you~
> > >> > >
> > >> > >Xintong Song
> > >> > >
> > >> > >
> > >> > >
> > >> > >On Mon, Oct 12, 2020 at 5:59 PM Jark Wu  wrote:
> > >> > >
> > >> > >> +1
> > >> > >>
> > >> > >> On Mon, 12 Oct 2020 at 17:14, Yu Li  wrote:
> > >> > >>
> > >> > >> > +1
> > >> > >> >
> > >> > >> > Best Regards,
> > >> > >> > Yu
> > >> > >> >
> > >> > >> >
> > >> > >> > On Mon, 12 Oct 2020 at 14:41, Congxian Qiu <
> > qcx978132...@gmail.com
> > >> >
> > >> > >> wrote:
> > >> > >> >
> > >> > >> > > I would like to start a voting thread for adding translation
> > >> > >> > specification
> > >> > >> > > for Stateful Functions, which we’ve reached consensus in [1].
> > >> > >> > >
> > >> > >> > >
> > >> > >> > > This voting will be open for a minimum 3 days till 3:00 pm
> UTC,
> > >> Oct
> > >> > 15.
> > >> > >> > >
> > >> > >> > >
> > >> > >> > > [1]
> > >> > >> > >
> > >> > >> > >
> > >> > >> >
> > >> > >>
> > >> >
> > >>
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Add-Translation-Specification-for-Stateful-Functions-td45531.html
> > >> > >> > >
> > >> > >> > >
> > >> > >> > > Best,
> > >> > >> > > Congxian
> > >> > >> > >
> > >> > >> >
> > >> > >>
> > >> >
> > >>
> > >
> >
>

Re: Re: [VOTE] Add Translation Specification for Stateful Functions

[Congxian Qiu]

FYI, I've added the Specification for Stateful Functions to the existing
wiki[1]

[1]
https://cwiki.apache.org/confluence/display/FLINK/Flink+Translation+Specifications

Best,
Congxian


Congxian Qiu  于2020年10月15日周四 下午8:33写道：

> Hi all
> Thanks everyone for the voting.
>
> The voting time for "Add Translation Specification for Stateful
> Functions" has passed, I'm closing the vote now.
>
> There were 7 votes, 4 of which are binding:
>- Yu Li (binding)
>- Jark Wu (binding)
>- Xintong Song (binding)
>- Smile
>- Dian Fu (binding)
>- Hailong Wang
>- Shawn Huang
>
> There were no -1 votes.
>
> Thus, changes have been accepted. I'll update the wiki accordingly.
>
> Best,
> Congxian
>
>
> Shawn Huang  于2020年10月13日周二 下午3:19写道：
>
>> +1
>>
>> Best,
>> Shawn Huang
>>
>>
>> hailongwang <18868816...@163.com> 于2020年10月12日周一 下午11:21写道：
>>
>> > +1
>> > Best,
>> > Hailong Wang
>> > At 2020-10-12 17:00:34, "Xintong Song"  wrote:
>> > >+1
>> > >
>> > >Thank you~
>> > >
>> > >Xintong Song
>> > >
>> > >
>> > >
>> > >On Mon, Oct 12, 2020 at 5:59 PM Jark Wu  wrote:
>> > >
>> > >> +1
>> > >>
>> > >> On Mon, 12 Oct 2020 at 17:14, Yu Li  wrote:
>> > >>
>> > >> > +1
>> > >> >
>> > >> > Best Regards,
>> > >> > Yu
>> > >> >
>> > >> >
>> > >> > On Mon, 12 Oct 2020 at 14:41, Congxian Qiu > >
>> > >> wrote:
>> > >> >
>> > >> > > I would like to start a voting thread for adding translation
>> > >> > specification
>> > >> > > for Stateful Functions, which we’ve reached consensus in [1].
>> > >> > >
>> > >> > >
>> > >> > > This voting will be open for a minimum 3 days till 3:00 pm UTC,
>> Oct
>> > 15.
>> > >> > >
>> > >> > >
>> > >> > > [1]
>> > >> > >
>> > >> > >
>> > >> >
>> > >>
>> >
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Add-Translation-Specification-for-Stateful-Functions-td45531.html
>> > >> > >
>> > >> > >
>> > >> > > Best,
>> > >> > > Congxian
>> > >> > >
>> > >> >
>> > >>
>> >
>>
>

Re: [ANNOUNCE] New PMC member: Zhu Zhu

[Congxian Qiu]

Congratulations, Zhu Zhu!

Best,
Congxian


godfrey he  于2020年10月16日周五 上午11:03写道：

> Congratulations, Zhu
>
> Godfrey,
> Best
>
> 刘 芃成  于2020年10月10日周六 下午1:03写道：
>
> > Congratulations, Zhu!
> >
> > Best,
> > Pengcheng
> >
> > 在 2020/10/10 上午11:14，“Kurt Young” 写入:
> >
> > Congratulations, Zhu Zhu!
> >
> > Best,
> > Kurt
> >
> >
> > On Sat, Oct 10, 2020 at 11:03 AM Yang Wang 
> > wrote:
> >
> > > Congratulations! Zhu Zhu.
> > >
> > > Best,
> > > Yang
> > >
> > > Xintong Song  于2020年10月9日周五 下午3:35写道：
> > >
> > > > Congratulations, Zhu~!
> > > >
> > > > Thank you~
> > > >
> > > > Xintong Song
> > > >
> > > >
> > > >
> > > > On Fri, Oct 9, 2020 at 3:17 PM Jingsong Li <
> jingsongl...@gmail.com
> > >
> > > wrote:
> > > >
> > > > > Congratulations, Zhu Zhu!
> > > > >
> > > > > On Fri, Oct 9, 2020 at 3:08 PM Zhijiang <
> > wangzhijiang...@aliyun.com
> > > > > .invalid>
> > > > > wrote:
> > > > >
> > > > > > Congratulations and welcome, Zhu Zhu!
> > > > > >
> > > > > > Best,
> > > > > > Zhijiang
> > > > > >
> > --
> > > > > > From:Yun Tang 
> > > > > > Send Time:2020年10月9日(星期五) 14:20
> > > > > > To:dev@flink.apache.org 
> > > > > > Subject:Re: [ANNOUNCE] New PMC member: Zhu Zhu
> > > > > >
> > > > > > Congratulations, Zhu！
> > > > > >
> > > > > > Best
> > > > > > Yun Tang
> > > > > > 
> > > > > > From: Danny Chan 
> > > > > > Sent: Friday, October 9, 2020 13:51
> > > > > > To: dev@flink.apache.org 
> > > > > > Subject: Re: [ANNOUNCE] New PMC member: Zhu Zhu
> > > > > >
> > > > > > Congrats, Zhu Zhu ~
> > > > > >
> > > > > > Best,
> > > > > > Danny Chan
> > > > > > 在 2020年10月9日 +0800 PM1:05，dev@flink.apache.org，写道：
> > > > > > >
> > > > > > > Congrats, Zhu Zhu
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > Best, Jingsong Lee
> > > > >
> > > >
> > >
> >
>

Re: Re: [VOTE] Add Translation Specification for Stateful Functions

[Congxian Qiu]

Hi all
Thanks everyone for the voting.

The voting time for "Add Translation Specification for Stateful
Functions" has passed, I'm closing the vote now.

There were 7 votes, 4 of which are binding:
   - Yu Li (binding)
   - Jark Wu (binding)
   - Xintong Song (binding)
   - Smile
   - Dian Fu (binding)
   - Hailong Wang
   - Shawn Huang

There were no -1 votes.

Thus, changes have been accepted. I'll update the wiki accordingly.

Best,
Congxian


Shawn Huang  于2020年10月13日周二 下午3:19写道：

> +1
>
> Best,
> Shawn Huang
>
>
> hailongwang <18868816...@163.com> 于2020年10月12日周一 下午11:21写道：
>
> > +1
> > Best,
> > Hailong Wang
> > At 2020-10-12 17:00:34, "Xintong Song"  wrote:
> > >+1
> > >
> > >Thank you~
> > >
> > >Xintong Song
> > >
> > >
> > >
> > >On Mon, Oct 12, 2020 at 5:59 PM Jark Wu  wrote:
> > >
> > >> +1
> > >>
> > >> On Mon, 12 Oct 2020 at 17:14, Yu Li  wrote:
> > >>
> > >> > +1
> > >> >
> > >> > Best Regards,
> > >> > Yu
> > >> >
> > >> >
> > >> > On Mon, 12 Oct 2020 at 14:41, Congxian Qiu 
> > >> wrote:
> > >> >
> > >> > > I would like to start a voting thread for adding translation
> > >> > specification
> > >> > > for Stateful Functions, which we’ve reached consensus in [1].
> > >> > >
> > >> > >
> > >> > > This voting will be open for a minimum 3 days till 3:00 pm UTC,
> Oct
> > 15.
> > >> > >
> > >> > >
> > >> > > [1]
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Add-Translation-Specification-for-Stateful-Functions-td45531.html
> > >> > >
> > >> > >
> > >> > > Best,
> > >> > > Congxian
> > >> > >
> > >> >
> > >>
> >
>

[VOTE] Add Translation Specification for Stateful Functions

[Congxian Qiu]

I would like to start a voting thread for adding translation specification
for Stateful Functions, which we’ve reached consensus in [1].


This voting will be open for a minimum 3 days till 3:00 pm UTC, Oct 15.


[1]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Add-Translation-Specification-for-Stateful-Functions-td45531.html


Best,
Congxian

Re: [DISCUSS] Add Translation Specification for Stateful Functions

[Congxian Qiu]

Thanks all for the positive feedback.

merging the proposed translation specification into the existing document
is better. In that form, people can find the specification on one page,
it's more convenient。

I'll start a separate email for the vote.

Best,
Congxian


Tzu-Li (Gordon) Tai  于2020年10月12日周一 下午12:57写道：

> +1 to merging the specification to the existing Flink translation
> specifications.
>
> We also have an open translation PR that is pending on an official
> translation specification for StateFun to be available:
> https://github.com/apache/flink-statefun/pull/131
>
> Cheers,
> Gordon
>
> On Fri, Oct 9, 2020 at 4:52 PM Jark Wu  wrote:
>
> > Thanks for driving the discussion Congxian!
> >
> > I'm +1 for adding translation specifications for stateful functions.
> >
> > What about merging the proposed translation specifications into the
> > existing document [1]?
> > It seems the new specifications only add some terminology translation for
> > the glossary.
> > Most of the content can be shared with the existing translation
> > specifications, e.g. the translation guidelines.
> >
> > [1]:
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Translation+Specifications
> >
> > On Fri, 9 Oct 2020 at 16:34, Yu Li  wrote:
> >
> > > +1 for adding the translation specification for stateful functions.
> > >
> > > Thanks for driving the discussion Congxian!
> > >
> > > Best Regards,
> > > Yu
> > >
> > >
> > > On Fri, 9 Oct 2020 at 15:55, Congxian Qiu 
> > wrote:
> > >
> > > > Hi all,
> > > >
> > > >
> > > > As Stateful Functions continues to evolve, and there have been many
> > > > attempts by Chinese users. I propose to add a translation
> specification
> > > for
> > > > Stateful Functions like the Translation Specification for Apache
> > > > Flink[1](the proposed translation specification[2] is written in
> > Chinese,
> > > > we can convert it into other languages if needed). The purpose is to
> > > > improve the translation quality, reduce the workload of the reviewer,
> > > keep
> > > > the style consistent, and make the reader’s reading experience
> better.
> > > >
> > > >
> > > > I would also propose to convert it into a WIKI page when it is
> > accepted,
> > > > and link it from the contribute documentation page[2]
> > > >
> > > >
> > > > Please feel free to join the discussion and give some feedback.
> > > >
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Translation+Specifications
> > > >
> > > > [2]
> > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1Fv56PG50pHlDxZfF49iRhv-8dY7ZaCSHUCvBM2Vy0ao
> > > >
> > > > [3] https://flink.apache.org/contributing/docs-style.html
> > > >
> > > >
> > > > Best,
> > > > Congxian
> > > >
> > >
> >
>

[DISCUSS] Add Translation Specification for Stateful Functions

[Congxian Qiu]

Hi all,


As Stateful Functions continues to evolve, and there have been many
attempts by Chinese users. I propose to add a translation specification for
Stateful Functions like the Translation Specification for Apache
Flink[1](the proposed translation specification[2] is written in Chinese,
we can convert it into other languages if needed). The purpose is to
improve the translation quality, reduce the workload of the reviewer, keep
the style consistent, and make the reader’s reading experience better.


I would also propose to convert it into a WIKI page when it is accepted,
and link it from the contribute documentation page[2]


Please feel free to join the discussion and give some feedback.


[1]
https://cwiki.apache.org/confluence/display/FLINK/Flink+Translation+Specifications

[2]
https://docs.google.com/document/d/1Fv56PG50pHlDxZfF49iRhv-8dY7ZaCSHUCvBM2Vy0ao

[3] https://flink.apache.org/contributing/docs-style.html


Best,
Congxian

[jira] [Created] (FLINK-19325) Optimize the consumed time for checkpoint completion

[Congxian Qiu(klion26) (Jira)]

Congxian Qiu(klion26) created FLINK-19325:
-

 Summary: Optimize the consumed time for checkpoint completion
 Key: FLINK-19325
 URL: https://issues.apache.org/jira/browse/FLINK-19325
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Checkpointing
Reporter: Congxian Qiu(klion26)


Currently when completing a checkpoint, we'll write out the state handle out in 
{{MetadataV2V3SerializerBase.java#serializeStreamStateHandle}}
{code:java}
static void serializeStreamStateHandle(StreamStateHandle stateHandle, 
DataOutputStream dos) throws IOException {
   if (stateHandle == null) {
  dos.writeByte(NULL_HANDLE);

   } else if (stateHandle instanceof RelativeFileStateHandle) {
  dos.writeByte(RELATIVE_STREAM_STATE_HANDLE);
  RelativeFileStateHandle relativeFileStateHandle = 
(RelativeFileStateHandle) stateHandle;
  dos.writeUTF(relativeFileStateHandle.getRelativePath());
  dos.writeLong(relativeFileStateHandle.getStateSize());
   } else if (stateHandle instanceof FileStateHandle) {
  dos.writeByte(FILE_STREAM_STATE_HANDLE);
  FileStateHandle fileStateHandle = (FileStateHandle) stateHandle;
  dos.writeLong(stateHandle.getStateSize());
  dos.writeUTF(fileStateHandle.getFilePath().toString());

   } else if (stateHandle instanceof ByteStreamStateHandle) {
  dos.writeByte(BYTE_STREAM_STATE_HANDLE);
  ByteStreamStateHandle byteStreamStateHandle = (ByteStreamStateHandle) 
stateHandle;
  dos.writeUTF(byteStreamStateHandle.getHandleName());
  byte[] internalData = byteStreamStateHandle.getData();
  dos.writeInt(internalData.length);
  dos.write(byteStreamStateHandle.getData());
   } else {
  throw new IOException("Unknown implementation of StreamStateHandle: " + 
stateHandle.getClass());
   }

   dos.flush();
}

{code}

We'll call {{dos.flush()}} after every state handle written out. But this may 
consume too much time and is not needed, because we'll close the outputstream 
after all things have been written out.

I propose to remove the {{dos.flush()}} here to optimize the consumed time for 
checkpoint completion



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] FLIP-142: Disentangle StateBackends from Checkpointing

[Congxian Qiu]

+1 (non-binding)

Best,
Congxian


David Anderson  于2020年9月15日周二 下午11:38写道：

> +1 (binding)
>
> David
>
> On Tue, Sep 15, 2020 at 10:25 AM Tzu-Li (Gordon) Tai 
> wrote:
>
> > +1 (binding)
> >
> > On Tue, Sep 15, 2020 at 3:26 PM Aljoscha Krettek 
> > wrote:
> >
> > > +1 (binding)
> > >
> > > Aljoscha
> > >
> >
>

Re: [DISCUSS] FLIP-142: Disentangle StateBackends from Checkpointing

[Congxian Qiu]

> > >> On 16.09.20 16:54, Seth Wiesman wrote:
> > >> > Hi Congxian,
> > >> >
> > >> > There is an allusion to those configs in the wiki but let me better
> > >> spell
> > >> > out my thinking. The flink-conf configurations will not change and I
> > >> > believe the java code switches should remain on the state backend
> > >> objects.
> > >> >
> > >> > We are of course not fully disentangling state backends from
> snapshots
> > >> and
> > >> > these configurations affect how your state backend runs in
> > production. I
> > >> > believe users would find it strange to have configurations like
> > >> > `state.backend.rocksdb.checkpoint.transfer.thred.num` not be part of
> > the
> > >> > EmbeddedRocksdbStateBackend but somewhere else. This then leads to
> the
> > >> > question, is it better to split configurations between multiple
> places
> > >> or
> > >> > not. Users appreciate consistency, and so having all the
> > configurations
> > >> on
> > >> > the state backend objects makes them more discoverable and your
> > >> application
> > >> > easier to reason about.
> > >> >
> > >> > Additionally, I view these as advanced configurations. My hope is
> most
> > >> > users can simply use the no-arg constructor for a state backend in
> > >> > production. If a user is changing the number of rocksdb transfer
> > >> threads or
> > >> > disabling async checkpoints, they likely know what they are doing.
> > >> >
> > >> > Please let me know if you have any concerns or would like to cancel
> > the
> > >> > vote.
> > >> >
> > >> > Seth
> > >> >
> > >> > On Wed, Sep 16, 2020 at 12:37 AM Congxian Qiu <
> qcx978132...@gmail.com
> > >
> > >> > wrote:
> > >> >
> > >> >> Sorry for jump late in.
> > >> >>
> > >> >> I like the separation here, this separation makes more user
> friendly
> > >> now.
> > >> >>
> > >> >> I just wonder how the configuration such as
> > >> 'state.backend.incremental',
> > >> >> 'state.backend.async' and
> > >> >> `state.backend.rocksdb.checkpoint.transfer.thred.num` will be
> > >> configured
> > >> >> after the separation, I think these configurations are more related
> > to
> > >> >> snapshots (maybe a little strange to configure these on
> statebackend
> > >> side).
> > >> >> did not see this on the FLIP wiki currently.
> > >> >>
> > >> >> Best,
> > >> >> Congxian
> > >> >>
> > >> >>
> > >> >> Seth Wiesman  于2020年9月15日周二 下午9:51写道：
> > >> >>
> > >> >>> Sounds good to me. I'll update the FLIP.
> > >> >>>
> > >> >>> On Tue, Sep 15, 2020 at 8:35 AM Dawid Wysakowicz <
> > >> dwysakow...@apache.org
> > >> >>>
> > >> >>> wrote:
> > >> >>>
> > >> >>>> There is a good number of precedents that introduced backwards
> > >> >>>> incompatible changes to that interface (which is PublicEvolving
> > btw).
> > >> >> We
> > >> >>>> introduced a couple of additional arguments to the
> > >> >>>> createKeyedStateBackend method and later on removed the methods
> > with
> > >> >>>> default implementation for backwards compatibility. I want to
> > >> introduce
> > >> >>>> a backward incompatible change in FLIP-140 (replace the
> > >> >>>> AbstractKeyedStateBackend with an interface). From my perspective
> > we
> > >> >>>> should just do these changes. The impact should be minimal.
> > >> >>>>
> > >> >>>> Best,
> > >> >>>>
> > >> >>>> Dawid
> > >> >>>>
> > >> >>>>
> > >> >>>> On 15/09/2020 15:20, Seth Wiesman wrote:
> > >> >>>>> Hey Dawid,
> > >> >>>>>
> > >> >>&g

Re: [DISCUSS] FLIP-142: Disentangle StateBackends from Checkpointing

[Congxian Qiu]

Sorry for jump late in.

I like the separation here, this separation makes more user friendly now.

I just wonder how the configuration such as 'state.backend.incremental',
'state.backend.async' and
`state.backend.rocksdb.checkpoint.transfer.thred.num` will be configured
after the separation, I think these configurations are more related to
snapshots (maybe a little strange to configure these on statebackend side).
did not see this on the FLIP wiki currently.

Best,
Congxian


Seth Wiesman  于2020年9月15日周二 下午9:51写道：

> Sounds good to me. I'll update the FLIP.
>
> On Tue, Sep 15, 2020 at 8:35 AM Dawid Wysakowicz 
> wrote:
>
> > There is a good number of precedents that introduced backwards
> > incompatible changes to that interface (which is PublicEvolving btw). We
> > introduced a couple of additional arguments to the
> > createKeyedStateBackend method and later on removed the methods with
> > default implementation for backwards compatibility. I want to introduce
> > a backward incompatible change in FLIP-140 (replace the
> > AbstractKeyedStateBackend with an interface). From my perspective we
> > should just do these changes. The impact should be minimal.
> >
> > Best,
> >
> > Dawid
> >
> >
> > On 15/09/2020 15:20, Seth Wiesman wrote:
> > > Hey Dawid,
> > >
> > > I didn't want to break compatibility but if there is precedent and
> > everyone
> > > is ok with it then I'm +1.
> > >
> > > Seth
> > >
> > > On Tue, Sep 15, 2020 at 2:22 AM Dawid Wysakowicz <
> dwysakow...@apache.org
> > >
> > > wrote:
> > >
> > >> Sorry for joining so late.
> > >>
> > >> Generally speaking I like this idea very much!
> > >>
> > >> I have one idea about the StateBackend interface. Could we instead of
> > >> adding a flag method boolean isLegacyStateBackend remove the
> > >> checkpointstorage related methods from StateBackend right away? The
> > >> old/legacy implementations could then implement both StateBackend and
> > >> SnapshotStorage. In turn in the method env.setStateBackend we could
> do:
> > >>
> > >> setStateBackend(StateBackend backend) {
> > >>
> > >> this.stateBackend = backend;
> > >>
> > >> if (backend instanceof SnapshotStorage) {
> > >>
> > >>  this.setSnapshotStorage(backend);
> > >>
> > >> }
> > >>
> > >> }
> > >>
> > >> This has the benefit that we could already get rid off the methods
> from
> > >> StateBackend which would be problematic in the new implementations
> (such
> > >> as e.g. HashMapStateBackend - what would you return there? null?). I
> > >> know this would break the interface, but StateBackend is actually
> quite
> > >> internal, we did it quite freely in the past, and I don't think there
> > >> are many custom state implementation in the wild. And even if there
> are
> > >> some the workaround is as easy as simply adding implements
> > SnapshotStorage.
> > >>
> > >> Best,
> > >>
> > >> Dawid
> > >>
> > >> On 11/09/2020 16:48, Aljoscha Krettek wrote:
> > >>> I could try and come up with a longer name if you need it ... ;-)
> > >>>
> > >>> Aljoscha
> > >>>
> > >>> On 11.09.20 16:25, Seth Wiesman wrote:
> >  Having thought about it more, HashMapStateBackend has won me over.
> > I'll
> >  update the FLIP. If there aren't any more comments I'll open it up
> for
> >  voting on monday.
> > 
> >  Seth
> > 
> >  On Wed, Sep 9, 2020 at 9:09 AM Seth Wiesman 
> > >> wrote:
> > > @Yun yes, this is really about making CheckpointStorage an
> orthogonal
> > > concept. I think we can remain pragmatic and keep state-backend
> > > specific
> > > configurations (async, incremental, etc) in the state backend
> > > themselves. I
> > > view these as more advanced configurations and by the time someone
> is
> > > changing the defaults they likely understand what is going on. My
> > > goal is
> > > to help on-board users and so long as each state backend has a
> no-arg
> > > default constructor that works for many users I think we've
> achieved
> > > that
> > > goal.
> > >
> > > Regarding the checkpoint coordinator, that makes sense but I will
> > > consider
> > > out of the scope of this FLIP. I want to focus on simplifying APIs.
> > >
> > > @Aljoscha Krettek 
> > >
> > > My feeling is that state backends and checkpointing are going to be
> > > integral to Flink for many years, regardless or other enhancements
> > > so this
> > > change is still valuable.
> > >
> > > Since this is a FLIP about improving the user api I'm happy to
> > bikeshed
> > > the names a little more than normal. HashMap makes sense, my other
> > > thought
> > > was InMemory.
> > >
> > > Seth
> > >
> > >
> > >
> > > On Wed, Sep 9, 2020 at 8:04 AM Aljoscha Krettek <
> aljos...@apache.org
> > >
> > > wrote:
> > >
> > >> I like it a lot!
> > >>
> > >> I think it makes sense to clean this up despite the planned new
> > >> fault-tolerance mechanisms.

Re: [ANNOUNCE] New Apache Flink Committer - Yun Tang

[Congxian Qiu]

Congratulations!
Best,
Congxian


Guowei Ma  于2020年9月16日周三 下午12:37写道：

> Congratulations :)
>
> Best,
> Guowei
>
>
> On Wed, Sep 16, 2020 at 11:54 AM Zhijiang
>  wrote:
>
> > Congratulations and welcome, Yun!
> >
> >
> > --
> > From:Jark Wu 
> > Send Time:2020年9月16日(星期三) 11:35
> > To:dev 
> > Cc:tangyun ; Yun Tang 
> > Subject:Re: [ANNOUNCE] New Apache Flink Committer - Yun Tang
> >
> > Congratulations Yun!
> >
> > On Wed, 16 Sep 2020 at 10:40, Rui Li  wrote:
> >
> > > Congratulations Yun!
> > >
> > > On Wed, Sep 16, 2020 at 10:20 AM Paul Lam 
> wrote:
> > >
> > > > Congrats, Yun! Well deserved!
> > > >
> > > > Best,
> > > > Paul Lam
> > > >
> > > > > 2020年9月15日 19:14，Yang Wang  写道：
> > > > >
> > > > > Congratulations, Yun!
> > > > >
> > > > > Best,
> > > > > Yang
> > > > >
> > > > > Leonard Xu  于2020年9月15日周二 下午7:11写道：
> > > > >
> > > > >> Congrats, Yun!
> > > > >>
> > > > >> Best,
> > > > >> Leonard
> > > > >>> 在 2020年9月15日，19:01，Yangze Guo  写道：
> > > > >>>
> > > > >>> Congrats, Yun!
> > > > >>
> > > > >>
> > > >
> > > >
> > >
> > > --
> > > Best regards!
> > > Rui Li
> > >
> >
> >
>

[jira] [Created] (FLINK-19249) Job would wait sometime(~10 min) before failover if some connection broken

[Congxian Qiu(klion26) (Jira)]

Congxian Qiu(klion26) created FLINK-19249:
-

 Summary: Job would wait sometime(~10 min) before failover if some 
connection broken
 Key: FLINK-19249
 URL: https://issues.apache.org/jira/browse/FLINK-19249
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Network
Reporter: Congxian Qiu(klion26)


{quote}encountered this error on 1.7, after going through the master code, I 
think the problem is still there
{quote}
When the network environment is not so good, the connection between the server 
and the client may be disconnected innocently. After the disconnection, the 
server will receive the IOException such as below
{code:java}
java.io.IOException: Connection timed out
 at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
 at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
 at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
 at sun.nio.ch.IOUtil.write(IOUtil.java:51)
 at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:468)
 at 
org.apache.flink.shaded.netty4.io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:403)
 at 
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:934)
 at 
org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.forceFlush(AbstractNioChannel.java:367)
 at 
org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:639)
 at 
org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
 at 
org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
 at 
org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
 at 
org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
 at java.lang.Thread.run(Thread.java:748)
{code}
then release the view reader.

But the job would not fail until the downstream detect the disconnection 
because of {{channelInactive}} later(~10 min). between such time, the job can 
still process data, but the broken channel can't transfer any data or event, so 
snapshot would fail during this time. this will cause the job to replay many 
data after failover.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] New Apache Flink Committer - Niels Basjes

[Congxian Qiu]

Congratulations

Best,
Congxian


Yang Wang  于2020年9月15日周二 上午10:50写道：

> Congratulations!
>
> Best,
> Yang
>
> Xintong Song  于2020年9月15日周二 上午10:41写道：
>
> > Congrats!
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> >
> > On Tue, Sep 15, 2020 at 10:40 AM Zhijiang
> >  wrote:
> >
> > > Congrats, Niels!
> > >
> > > Best,
> > > Zhijiang
> > >
> > >
> > > --
> > > From:Darion Yaphet 
> > > Send Time:2020年9月15日(星期二) 10:02
> > > To:dev 
> > > Subject:Re: [ANNOUNCE] New Apache Flink Committer - Niels Basjes
> > >
> > > Congratulations!
> > >
> > > 刘建刚  于2020年9月15日周二 上午9:53写道：
> > >
> > > > Congratulations!
> > > >
> > > > Best,
> > > > liujiangang
> > > >
> > > > Danny Chan  于2020年9月15日周二 上午9:44写道：
> > > >
> > > > > Congratulations! 💐
> > > > >
> > > > > Best,
> > > > > Danny Chan
> > > > > 在 2020年9月15日 +0800 AM9:31，dev@flink.apache.org，写道：
> > > > > >
> > > > > > Congratulations! 💐
> > > > >
> > > >
> > >
> > >
> > > --
> > >
> > > long is the way and hard  that out of Hell leads up to light
> > >
> > >
> >
>

Re: [ANNOUNCE] New Apache Flink Committer - Arvid Heise

[Congxian Qiu]

Congratulations  Arvid

Best,
Congxian


Benchao Li  于2020年9月15日周二 下午12:52写道：

> Congratulations!
>
> Zhu Zhu  于2020年9月15日周二 上午11:05写道：
>
> > Congratulations, Arvid!
> >
> > Thanks,
> > Zhu
> >
> > jincheng sun  于2020年9月15日周二 上午10:55写道：
> >
> > > Congratulations and welcome, Arvid !
> > >
> > > Best,
> > > Jincheng
> > >
> > >
> > > Dian Fu  于2020年9月15日周二 上午10:53写道：
> > >
> > > > Congratulations!
> > > >
> > > > Regards,
> > > > Dian
> > > >
> > > > > 在 2020年9月15日，上午10:41，Matt Wang  写道：
> > > > >
> > > > > Congrats!
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Best,
> > > > > Matt Wang
> > > > >
> > > > >
> > > > > On 09/15/2020 10:38，Zhijiang
> > > wrote：
> > > > > Hi all,
> > > > >
> > > > > On behalf of the PMC, I’m very happy to announce Arvid Heise as a
> new
> > > > Flink committer.
> > > > >
> > > > > Arvid has been an active community member for more than a year,
> with
> > > 138
> > > > contributions including 116 commits, reviewed many PRs with good
> > quality
> > > > comments.
> > > > > He is mainly working on the runtime scope, involved in critical
> > > features
> > > > like task mailbox model and unaligned checkpoint, etc.
> > > > > Besides that, he was super active to reply questions in the user
> mail
> > > > list (34 emails in March, 51 emails in June, etc), also active in dev
> > > mail
> > > > list and Jira issue discussions.
> > > > >
> > > > > Please join me in congratulating Arvid for becoming a Flink
> > committer!
> > > > >
> > > > > Best,
> > > > > Zhijiang
> > > >
> > > >
> > >
> >
>
>
> --
>
> Best,
> Benchao Li
>

Re: [VOTE] Release 1.11.2, release candidate #1

[Congxian Qiu]

+1 (no-binding)

 checked
- sha verified ok
- gpg verifed ok
- build from source, ok
- check license ok, use the diff generated here[1]
- run some demo locally, ok
[1]

https://github.com/apache/flink/compare/release-1.11.1..release-1.11.2-rc1


Best,
Congxian


David Anderson  于2020年9月14日周一 下午5:33写道：

> +1
>
> Checks:
> - Verified that the fix for FLINK-19109 solves the problem I reported,
> running against the maven artifacts
>
>
>
> On Thu, Sep 10, 2020 at 9:04 AM Zhu Zhu  wrote:
>
> > Hi everyone,
> >
> > Please review and vote on the release candidate #1 for the version
> 1.11.2,
> > as follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> >
> > The complete staging area is available for your review, which includes:
> > * JIRA release notes [1],
> > * the official Apache source release and binary convenience releases to
> be
> > deployed to dist.apache.org [2], which are signed with the key with
> > fingerprint C63E230EFFF519A5BBF2C9AE6767487CD505859C [3],
> > * all artifacts to be deployed to the Maven Central Repository [4],
> > * source code tag "release-1.11.2-rc1" [5],
> > * website pull request listing the new release and adding announcement
> blog
> > post [6].
> >
> > The vote will be open for at least 72 hours. It is adopted by majority
> > approval, with at least 3 PMC affirmative votes.
> >
> > Thanks,
> > Zhu
> >
> > [1]
> >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12348575
> > [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.11.2-rc1/
> > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > [4]
> > https://repository.apache.org/content/repositories/orgapacheflink-1397/
> > [5]
> >
> >
> https://github.com/apache/flink/commit/fe3613574f76201a8d55d572a639a4ce7e18a9db
> > [6] https://github.com/apache/flink-web/pull/377
> >
>

Re: [DISCUSS] Releasing Flink 1.11.2

[Congxian Qiu]

Hi
I'd like to include FLINK-14942 into 1.11.2. FLINK-14942 (this fixes a
bug introduce in 1.11.0), there is a pr for it already.
Best,
Congxian


Zhou, Brian  于2020年9月3日周四 上午11:21写道：

> Hi,
>
> Thanks Becket for addressing the issue. FLINK-18641 is now a blocker for
> Pravega connector integration, hope we can have it included in 1.11.2
> release.
>
> Best Regards,
> Brian
>
> -Original Message-
> From: Becket Qin 
> Sent: Thursday, September 3, 2020 11:18
> To: dev
> Cc: khachatryan.ro...@gmail.com; Till Rohrmann; david; Jingsong Li
> Subject: Re: [DISCUSS] Releasing Flink 1.11.2
>
>
> [EXTERNAL EMAIL]
>
> Hi Zhuzhu,
>
> Thanks for starting the discussion.
>
> I'd like to include FLINK-18641 into 1.11.2 as well. It is a regression
> from previous versions and is currently blocking the development of Pravega
> source on top of FLIP-27.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Wed, Sep 2, 2020 at 11:13 PM Zhu Zhu  wrote:
>
> > Thank you all for the inputs!
> >
> > I agree with Till that we should set a soft deadline first.
> > I'd like to propose next Monday if no new blocker issue pops up.
> > But feel free to raise your concerns if you feel next Monday as a
> > deadline may not work for fixes which should be a blocker for 1.11.2.
> >
> > Here's a summary of the wanted/blocker but still open fixes:
> > - FLINK-19121 Avoid accessing HDFS frequently in HiveBulkWriterFactory
> > - FLINK-19109 Split Reader eats chained periodic watermarks
> > - (not a strict blocker) FLINK-18959 Fail to archiveExecutionGraph
> > because job is not finished when dispatcher close
> > - FLINK-18934 Idle stream does not advance watermark in connected
> > stream
> >
> > Thanks,
> > Zhu
> >
> > Konstantin Knauf  于2020年9月2日周三 下午9:00写道：
> >
> > > I think it would be nice to include a fix for
> > > https://issues.apache.org/jira/browse/FLINK-18934, too, as it
> > > affects a highly requested feature of Flink 1.11 quite severely.
> > >
> > > On Wed, Sep 2, 2020 at 2:51 PM Till Rohrmann 
> > wrote:
> > >
> > > > Thanks a lot for starting this discussion Zhu Zhu and for
> > > > volunteering
> > as
> > > > the release manager. Big +1 for creating the next 1.11 bug fix
> > release. I
> > > > think we already collected quite a bit of fixes which our users
> > > > will benefit from.
> > > >
> > > > For the pending fixes, I would suggest setting a soft deadline
> > > > (maybe
> > > until
> > > > beginning of next week) and then starting to cut the release
> > > > (given
> > that
> > > no
> > > > other blocker issues pop up). Maybe we are able to resolve the
> > > > issues
> > > even
> > > > earlier which would allow us to cut the release also earlier.
> > > >
> > > > From my side I would like to include FLINK-18959 in the release.
> > > > But it
> > > is
> > > > not a strict release blocker.
> > > >
> > > > Cheers,
> > > > Till
> > > >
> > > > On Wed, Sep 2, 2020 at 2:40 PM David Anderson
> > > > 
> > > > wrote:
> > > >
> > > > > I think it's worth considering whether we can get this bugfix
> > included
> > > in
> > > > > 1.11.2:
> > > > >
> > > > > - FLINK-19109 Split Reader eats chained periodic watermarks
> > > > >
> > > > > There is a PR, but it's still a work in progress. Cc'ing Roman,
> > > > > who
> > has
> > > > > been working on this.
> > > > >
> > > > > Regards,
> > > > > David
> > > > >
> > > > >
> > > > > On Wed, Sep 2, 2020 at 2:19 PM Zhu Zhu  wrote:
> > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > It has been about 1 month since we released Flink 1.11.1. It's
> > > > > > not
> > > too
> > > > > far
> > > > > > but
> > > > > > we already have more than 80 resolved improvements/bugs in the
> > > > > release-1.11
> > > > > > branch. Some of them are quite critical. Therefore, I propose
> > > > > > to
> > > create
> > > > > the
> > > > > > next
> > > > > > bugfix release 1.11.2 for Flink 1.11.
> > > > > >
> > > > > > Most noticeable fixes are:
> > > > > > - FLINK-18769 MiniBatch doesn't work with FLIP-95 source
> > > > > > - FLINK-18902 Cannot serve results of asynchronous REST
> > > > > > operations
> > in
> > > > > > per-job mode
> > > > > > - FLINK-18682 Vector orc reader cannot read Hive 2.0.0 table
> > > > > > - FLINK-18608 CustomizedConvertRule#convertCast drops
> > > > > > nullability
> > > > > > - FLINK-18646 Managed memory released check can block RPC
> > > > > > thread
> > > > > > - FLINK-18993 Invoke sanityCheckTotalFlinkMemory method
> > > > > > incorrectly
> > > in
> > > > > > JobManagerFlinkMemoryUtils.java
> > > > > > - FLINK-18663 RestServerEndpoint may prevent server shutdown
> > > > > > - FLINK-18595 Deadlock during job shutdown
> > > > > > - FLINK-18581 Cannot find GC cleaner with java version
> > > > > > previous
> > > > > > jdk8u72(-b01)
> > > > > > - FLINK-17075 Add task status reconciliation between TM and JM
> > > > > >
> > > > > > Furthermore, I think the following blocker issue should be
> > > > > > merged
> > > > before
> > > > > > 1.11.2 release
> > > > > >
> > > > > > - FLINK-19121 A

Re: [ANNOUNCE] New PMC member: Dian Fu

[Congxian Qiu]

Congratulations Dian
Best,
Congxian


Xintong Song  于2020年8月27日周四 下午7:50写道：

> Congratulations Dian~!
>
> Thank you~
>
> Xintong Song
>
>
>
> On Thu, Aug 27, 2020 at 7:42 PM Jark Wu  wrote:
>
> > Congratulations Dian!
> >
> > Best,
> > Jark
> >
> > On Thu, 27 Aug 2020 at 19:37, Leonard Xu  wrote:
> >
> > > Congrats, Dian!  Well deserved.
> > >
> > > Best
> > > Leonard
> > >
> > > > 在 2020年8月27日，19:34，Kurt Young  写道：
> > > >
> > > > Congratulations Dian!
> > > >
> > > > Best,
> > > > Kurt
> > > >
> > > >
> > > > On Thu, Aug 27, 2020 at 7:28 PM Rui Li 
> wrote:
> > > >
> > > >> Congratulations Dian!
> > > >>
> > > >> On Thu, Aug 27, 2020 at 5:39 PM Yuan Mei 
> > > wrote:
> > > >>
> > > >>> Congrats!
> > > >>>
> > > >>> On Thu, Aug 27, 2020 at 5:38 PM Xingbo Huang 
> > > wrote:
> > > >>>
> > >  Congratulations Dian!
> > > 
> > >  Best,
> > >  Xingbo
> > > 
> > >  jincheng sun  于2020年8月27日周四 下午5:24写道：
> > > 
> > > > Hi all,
> > > >
> > > > On behalf of the Flink PMC, I'm happy to announce that Dian Fu is
> > now
> > > > part of the Apache Flink Project Management Committee (PMC).
> > > >
> > > > Dian Fu has been very active on PyFlink component, working on
> > various
> > > > important features, such as the Python UDF and Pandas
> integration,
> > > and
> > > > keeps checking and voting for our releases, and also has
> > successfully
> > > > produced two releases(1.9.3&1.11.1) as RM, currently working as
> RM
> > > to push
> > > > forward the release of Flink 1.12.
> > > >
> > > > Please join me in congratulating Dian Fu for becoming a Flink PMC
> > > > Member!
> > > >
> > > > Best,
> > > > Jincheng(on behalf of the Flink PMC)
> > > >
> > > 
> > > >>
> > > >> --
> > > >> Best regards!
> > > >> Rui Li
> > > >>
> > >
> > >
> >
>

Re: [ANNOUNCE] Apache Flink 1.10.2 released

[Congxian Qiu]

Thanks ZhuZhu for managing this release and everyone else who contributed
to this release!

Best,
Congxian


Xingbo Huang  于2020年8月26日周三 下午1:53写道：

> Thanks Zhu for the great work and everyone who contributed to this release!
>
> Best,
> Xingbo
>
> Guowei Ma  于2020年8月26日周三 下午12:43写道：
>
>> Hi,
>>
>> Thanks a lot for being the release manager Zhu Zhu!
>> Thanks everyone contributed to this!
>>
>> Best,
>> Guowei
>>
>>
>> On Wed, Aug 26, 2020 at 11:18 AM Yun Tang  wrote:
>>
>>> Thanks for Zhu's work to manage this release and everyone who
>>> contributed to this!
>>>
>>> Best,
>>> Yun Tang
>>> 
>>> From: Yangze Guo 
>>> Sent: Tuesday, August 25, 2020 14:47
>>> To: Dian Fu 
>>> Cc: Zhu Zhu ; dev ; user <
>>> u...@flink.apache.org>; user-zh 
>>> Subject: Re: [ANNOUNCE] Apache Flink 1.10.2 released
>>>
>>> Thanks a lot for being the release manager Zhu Zhu!
>>> Congrats to all others who have contributed to the release!
>>>
>>> Best,
>>> Yangze Guo
>>>
>>> On Tue, Aug 25, 2020 at 2:42 PM Dian Fu  wrote:
>>> >
>>> > Thanks ZhuZhu for managing this release and everyone else who
>>> contributed to this release!
>>> >
>>> > Regards,
>>> > Dian
>>> >
>>> > 在 2020年8月25日，下午2:22，Till Rohrmann  写道：
>>> >
>>> > Great news. Thanks a lot for being our release manager Zhu Zhu and to
>>> all others who have contributed to the release!
>>> >
>>> > Cheers,
>>> > Till
>>> >
>>> > On Tue, Aug 25, 2020 at 5:37 AM Zhu Zhu  wrote:
>>> >>
>>> >> The Apache Flink community is very happy to announce the release of
>>> Apache Flink 1.10.2, which is the first bugfix release for the Apache Flink
>>> 1.10 series.
>>> >>
>>> >> Apache Flink® is an open-source stream processing framework for
>>> distributed, high-performing, always-available, and accurate data streaming
>>> applications.
>>> >>
>>> >> The release is available for download at:
>>> >> https://flink.apache.org/downloads.html
>>> >>
>>> >> Please check out the release blog post for an overview of the
>>> improvements for this bugfix release:
>>> >> https://flink.apache.org/news/2020/08/25/release-1.10.2.html
>>> >>
>>> >> The full release notes are available in Jira:
>>> >>
>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12347791
>>> >>
>>> >> We would like to thank all contributors of the Apache Flink community
>>> who made this release possible!
>>> >>
>>> >> Thanks,
>>> >> Zhu
>>> >
>>> >
>>>
>>

Re: [VOTE] Release 1.10.2, release candidate #2

[Congxian Qiu]

+1 (non-binding)

- checksum & signatures, OK
- build from source, OK
- ran some example jobs on clusters, OK
- all pom files point to 1.10.2
- NOTICE/LICENSE files seem OK

Best,
Congxian


Till Rohrmann  于2020年8月18日周二 下午5:24写道：

> +1 (binding)
>
> - verified checksums and signatures
> - verified that dependency changes are reflected in NOTICE/LICENSE files
> - built Flink with Scala 2.12 support from sources
> - Started cluster and ran example jobs, no suspicious log output
>
> Cheers,
> Till
>
> On Mon, Aug 17, 2020 at 4:21 PM Jeff Zhang  wrote:
>
> > +1 （non-binding）
> >
> > Verified it on Zeppelin, it works well.
> >
> >
> >
> > Robert Metzger  于2020年8月17日周一 下午9:41写道：
> >
> > > Thanks a lot for creating another release candidate!
> > >
> > > +1 (binding)
> > >
> > > - checked diff:
> > >
> >
> https://github.com/apache/flink/compare/release-1.10.1...release-1.10.2-rc2
> > >   - kubernetes client was upgraded from 4.5.2 to 4.9.2 + some shading
> > > changes > verified NOTICE file with shade output
> > > - source compiles
> > > - source sha is correct
> > > - checked staging repository: versions set correctly, contents seem in
> > line
> > > with the changes
> > >
> > >
> > >
> > >
> > > On Mon, Aug 17, 2020 at 1:56 PM Zhu Zhu  wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > Please review and vote on the release candidate #2 for the version
> > > 1.10.2,
> > > > as follows:
> > > > [ ] +1, Approve the release
> > > > [ ] -1, Do not approve the release (please provide specific comments)
> > > >
> > > >
> > > > The complete staging area is available for your review, which
> includes:
> > > > * JIRA release notes [1],
> > > > * the official Apache source release and binary convenience releases
> to
> > > be
> > > > deployed to dist.apache.org [2], which are signed with the key with
> > > > fingerprint C63E230EFFF519A5BBF2C9AE6767487CD505859C [3],
> > > > * all artifacts to be deployed to the Maven Central Repository [4],
> > > > * source code tag "release-1.10.2-rc2" [5],
> > > > * website pull request listing the new release and adding
> announcement
> > > blog
> > > > post [6].
> > > >
> > > > The vote will be open for at least 72 hours. It is adopted by
> majority
> > > > approval, with at least 3 PMC affirmative votes.
> > > >
> > > > Thanks,
> > > > Zhu Zhu
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12347791
> > > > [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.10.2-rc2/
> > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > [4]
> > > >
> > https://repository.apache.org/content/repositories/orgapacheflink-1395/
> > > > [5]
> > > >
> > > >
> > >
> >
> https://github.com/apache/flink/commit/68bb8b612932e479ca03c7ae7c0080818d89c8a1
> > > > [6] https://github.com/apache/flink-web/pull/366
> > > >
> > >
> >
> >
> > --
> > Best Regards
> >
> > Jeff Zhang
> >
>

Re: 退订

[Congxian Qiu]

Hi

  Please send an email to dev-unsubscr...@flink.apache.org for
unsubscribing.

  You can read more information[1] here

  PS: I use Chinses because I thought that the email was sent to user-zh
mail list.


[1] https://flink.apache.org/community.html

Best,
Congxian


Congxian Qiu  于2020年8月12日周三 下午1:39写道：

> Hi
> 退订请 发邮件到  user-zh-unsubscr...@flink.apache.org
>
> 详细的可以参考 [1]
>
> [1] https://flink.apache.org/zh/community.html#section-1
>
> Best,
> Congxian
>
>
> 智子 <769257...@qq.com> 于2020年8月12日周三 上午11:15写道：
>
>>

Re: 退订

[Congxian Qiu]

Hi
退订请 发邮件到  user-zh-unsubscr...@flink.apache.org

详细的可以参考 [1]

[1] https://flink.apache.org/zh/community.html#section-1

Best,
Congxian


智子 <769257...@qq.com> 于2020年8月12日周三 上午11:15写道：

>

Re: [DISCUSS] Releasing Flink 1.10.2

[Congxian Qiu]

Hi Zhu Zhu
Thanks for the reply, sorry for the mistake, please ignore my previous
email
Best,
Congxian


Zhu Zhu  于2020年8月9日周日 下午10:28写道：

> Hi Congxian, thanks for the information.
> However, IIUC, FLINK-18675/FLINK-18856 are to solve problems
> of 1.11 and should be included in 1.11.2, while this ML is about 1.10.2.
> Correct me if I'm wrong here.
>
> Thanks,
> Zhu Zhu
>
> Congxian Qiu  于2020年8月9日周日 下午2:45写道：
>
> > Thanks Zhu Zhu for taking care of this. I'd like to
> > include FLINK-18675/FLINK-18856 in 1.11.2, they are the same problem, and
> > there is a pr ready for it now.
> >
> > Best,
> > Congxian
> >
> >
> > Dian Fu  于2020年8月7日周五 下午3:11写道：
> >
> > > Thanks Zhu Zhu for taking care of this.
> > >
> > > Regards,
> > > Dian
> > >
> > > > 在 2020年8月7日，下午2:47，Zhu Zhu  写道：
> > > >
> > > > Thank you all for the confirmation and suggestion.
> > > > And thanks Zhijiang for offering the PMC power to help with the
> > release.
> > > >
> > > > Robert's suggestion sounds good to me. I will keep watching the ML
> and
> > > > JIRA issues until next Monday evening. If there were no new concerns
> > > > nor blocking issues, I will start to create the first release
> > candidate.
> > > >
> > > > Thanks,
> > > > Zhu Zhu
> > > >
> > > > Zhijiang  于2020年8月7日周五 下午2:39写道：
> > > >
> > > >> Thanks for volunteering as the release manager, zhuzhu.
> > > >>
> > > >> +1 for the 1.10.2 release and I am willing to provide any help in
> the
> > > >> procedure if needing PMC permission operations.
> > > >>
> > > >> Best,
> > > >> Zhijiang
> > > >>
> > > >>
> > > >> --
> > > >> From:Robert Metzger 
> > > >> Send Time:2020年8月7日(星期五) 14:16
> > > >> To:dev 
> > > >> Subject:Re: [DISCUSS] Releasing Flink 1.10.2
> > > >>
> > > >> Thanks for taking care of this Zhu Zhu. The list of bugs from your
> > list
> > > >> certainly justifies pushing out a bugfix release.
> > > >> I would propose to wait until Monday for people to speak up if they
> > > want to
> > > >> have a fix included in the release. Otherwise, we could create the
> > > first RC
> > > >> on Monday evening (China time).
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> On Thu, Aug 6, 2020 at 2:53 PM Till Rohrmann 
> > > wrote:
> > > >>
> > > >>> Thanks for kicking this discussion off Zhu Zhu. +1 for the 1.10.2
> > > >> release.
> > > >>> Also thanks for volunteering as the release manager!
> > > >>>
> > > >>> Cheers,
> > > >>> Till
> > > >>>
> > > >>> On Thu, Aug 6, 2020 at 1:26 PM Zhu Zhu  wrote:
> > > >>>
> > > >>>> Hi All,
> > > >>>>
> > > >>>> It has been more than 2 months since we released Flink 1.10.1. We
> > > >> already
> > > >>>> have more than 60 resolved improvements/bugs in the release-1.10
> > > >> branch.
> > > >>>> Therefore, I propose to create the next bugfix release 1.10.2 for
> > > Flink
> > > >>>> 1.10.
> > > >>>>
> > > >>>> Most noticeable fixes are:
> > > >>>> - FLINK-18663 RestServerEndpoint may prevent server shutdown
> > > >>>> - FLINK-18595 Deadlock during job shutdown
> > > >>>> - FLINK-18539 StreamExecutionEnvironment#addSource(SourceFunction,
> > > >>>> TypeInformation) doesn't use the user defined type information
> > > >>>> - FLINK-18048 "--host" option could not take effect for standalone
> > > >>>> application cluster
> > > >>>> - FLINK-18045 Fix Kerberos credentials checking to unblock Flink
> on
> > > >>> secured
> > > >>>> MapR
> > > >>>> - FLINK-18035 Executors#newCachedThreadPool could not work as
> > expected
> > > >>>> - FLINK-18012 Deactivate slot timeout if
> > > >> TaskSlotTable.tryMarkSlotActive
> > > >>> is
> > > >>>> called
> > > >>>> - FLINK-17800 RocksDB optimizeForPointLookup results in missing
> time
> > > >>>> windows
> > > >>>> - FLINK-17558 Partitions are released in TaskExecutor Main Thread
> > > >>>> - FLINK-17466 toRetractStream doesn't work correctly with Pojo
> > > >> conversion
> > > >>>> class
> > > >>>> - FLINK-16451 Fix IndexOutOfBoundsException for DISTINCT AGG with
> > > >>> constants
> > > >>>>
> > > >>>> There is no known blocker issue of 1.10.2 release at the moment.
> > > >>>>
> > > >>>> I would volunteer as the release manager and kick off the release
> > > >>> process.
> > > >>>> What do you think?
> > > >>>>
> > > >>>> Please let me know if there are any concerns or any missing
> blocker
> > > >>> issues
> > > >>>> need to be fixed in 1.10.2.
> > > >>>>
> > > >>>> Thanks,
> > > >>>> Zhu Zhu
> > > >>>>
> > > >>>
> > > >>
> > > >>
> > >
> > >
> >
>

Re: [DISCUSS] Releasing Flink 1.10.2

[Congxian Qiu]

Thanks Zhu Zhu for taking care of this. I'd like to
include FLINK-18675/FLINK-18856 in 1.11.2, they are the same problem, and
there is a pr ready for it now.

Best,
Congxian


Dian Fu  于2020年8月7日周五 下午3:11写道：

> Thanks Zhu Zhu for taking care of this.
>
> Regards,
> Dian
>
> > 在 2020年8月7日，下午2:47，Zhu Zhu  写道：
> >
> > Thank you all for the confirmation and suggestion.
> > And thanks Zhijiang for offering the PMC power to help with the release.
> >
> > Robert's suggestion sounds good to me. I will keep watching the ML and
> > JIRA issues until next Monday evening. If there were no new concerns
> > nor blocking issues, I will start to create the first release candidate.
> >
> > Thanks,
> > Zhu Zhu
> >
> > Zhijiang  于2020年8月7日周五 下午2:39写道：
> >
> >> Thanks for volunteering as the release manager, zhuzhu.
> >>
> >> +1 for the 1.10.2 release and I am willing to provide any help in the
> >> procedure if needing PMC permission operations.
> >>
> >> Best,
> >> Zhijiang
> >>
> >>
> >> --
> >> From:Robert Metzger 
> >> Send Time:2020年8月7日(星期五) 14:16
> >> To:dev 
> >> Subject:Re: [DISCUSS] Releasing Flink 1.10.2
> >>
> >> Thanks for taking care of this Zhu Zhu. The list of bugs from your list
> >> certainly justifies pushing out a bugfix release.
> >> I would propose to wait until Monday for people to speak up if they
> want to
> >> have a fix included in the release. Otherwise, we could create the
> first RC
> >> on Monday evening (China time).
> >>
> >>
> >>
> >>
> >>
> >> On Thu, Aug 6, 2020 at 2:53 PM Till Rohrmann 
> wrote:
> >>
> >>> Thanks for kicking this discussion off Zhu Zhu. +1 for the 1.10.2
> >> release.
> >>> Also thanks for volunteering as the release manager!
> >>>
> >>> Cheers,
> >>> Till
> >>>
> >>> On Thu, Aug 6, 2020 at 1:26 PM Zhu Zhu  wrote:
> >>>
>  Hi All,
> 
>  It has been more than 2 months since we released Flink 1.10.1. We
> >> already
>  have more than 60 resolved improvements/bugs in the release-1.10
> >> branch.
>  Therefore, I propose to create the next bugfix release 1.10.2 for
> Flink
>  1.10.
> 
>  Most noticeable fixes are:
>  - FLINK-18663 RestServerEndpoint may prevent server shutdown
>  - FLINK-18595 Deadlock during job shutdown
>  - FLINK-18539 StreamExecutionEnvironment#addSource(SourceFunction,
>  TypeInformation) doesn't use the user defined type information
>  - FLINK-18048 "--host" option could not take effect for standalone
>  application cluster
>  - FLINK-18045 Fix Kerberos credentials checking to unblock Flink on
> >>> secured
>  MapR
>  - FLINK-18035 Executors#newCachedThreadPool could not work as expected
>  - FLINK-18012 Deactivate slot timeout if
> >> TaskSlotTable.tryMarkSlotActive
> >>> is
>  called
>  - FLINK-17800 RocksDB optimizeForPointLookup results in missing time
>  windows
>  - FLINK-17558 Partitions are released in TaskExecutor Main Thread
>  - FLINK-17466 toRetractStream doesn't work correctly with Pojo
> >> conversion
>  class
>  - FLINK-16451 Fix IndexOutOfBoundsException for DISTINCT AGG with
> >>> constants
> 
>  There is no known blocker issue of 1.10.2 release at the moment.
> 
>  I would volunteer as the release manager and kick off the release
> >>> process.
>  What do you think?
> 
>  Please let me know if there are any concerns or any missing blocker
> >>> issues
>  need to be fixed in 1.10.2.
> 
>  Thanks,
>  Zhu Zhu
> 
> >>>
> >>
> >>
>
>

Re: to be a contributor

[Congxian Qiu]

Hi Dan
   Currently, you do not need to request permission as a contributor[1],
you can directly comment on the issue you would like to contribute to.
   and you can check out the documentation [contribute code][2] for more
information.

[1]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/ANNOUNCE-JIRA-permissions-changed-Only-committers-can-assign-somebody-to-a-ticket-td30695.html
[2] https://flink.apache.org/contributing/contribute-code.html
Best,
Congxian


朱丹  于2020年8月2日周日 上午11:41写道：

> Hi Guys,
>
>
>
> I want to contribute to Apache Flink.
>
> Would you please give me the permission as a contributor?
>
> My JIRA ID is wstcilct.
>
>
>
> Thanks,
>
> Dan
>
>
>
>
>
> --
> Avast 防毒软件已对此电子邮件执行病毒检查。
> https://www.avast.com/antivirus
>

[jira] [Created] (FLINK-18748) Savepoint would be queued unexpected

[Congxian Qiu(klion26) (Jira)]

Congxian Qiu(klion26) created FLINK-18748:
-

 Summary: Savepoint would be queued unexpected
 Key: FLINK-18748
 URL: https://issues.apache.org/jira/browse/FLINK-18748
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Checkpointing
Affects Versions: 1.11.1, 1.11.0
Reporter: Congxian Qiu(klion26)


After FLINK-17342, when triggering a checkpoint/savepoint, we'll check whether 
the request can be triggered in 
{{CheckpointRequestDecider#chooseRequestToExecute}}, the logic is as follow:
{code:java}
Preconditions.checkState(Thread.holdsLock(lock));
// 1. 
if (isTriggering || queuedRequests.isEmpty()) {
   return Optional.empty();
}

// 2 too many ongoing checkpoitn/savepoint
if (pendingCheckpointsSizeSupplier.get() >= maxConcurrentCheckpointAttempts) {
   return Optional.of(queuedRequests.first())
  .filter(CheckpointTriggerRequest::isForce)
  .map(unused -> queuedRequests.pollFirst());
}

// 3 check the timestamp of last complete checkpoint
long nextTriggerDelayMillis = nextTriggerDelayMillis(lastCompletionMs);
if (nextTriggerDelayMillis > 0) {
   return onTooEarly(nextTriggerDelayMillis);
}

return Optional.of(queuedRequests.pollFirst());
{code}
But if currently {{pendingCheckpointsSizeSupplier.get()}} < 
{{maxConcurrentCheckpointAttempts}}, and the request is a savepoint, the 
savepoint will still wait some time in step 3. 

I think we should trigger the savepoint immediately if 
{{pendingCheckpointSizeSupplier.get()}} < {{maxConcurrentCheckpointAttempts}}.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [Discuss] Questions about SortedMapState in Stream API

[Congxian Qiu]

Hi  Sean
   AFAIK, Flink does not support SortedMapState currently, there was an
issue FLINK-6219 about sorted mapstate before. I think you can reach to
this issue if you want sorted mapstate.
  Sorted MapState for RocksDB can be challenging for Java-based comparator,
RocksDB supports the bytes-comparator only.

Best,
Congxian


Sean Z  于2020年7月24日周五 上午3:36写道：

> Hi Flink Community,
>
> I'm new to this community but have been using Flink for a year or so. As a
> user, my team built stuff upon Flink stateful stream processing. We used to
> have a use case that we need a sorted map data structure to store our
> state, something like TreeMap in Java, to query higher/lower keys, do range
> queries, and etc. However, currently, we only have a MapState interface in
> Flink. We found RocksDB is kind of sorted storage in nature. To achieve
> our use case, we have to do some hacky tricks to bypass current limitations
> and use lower RocksDB features as a workaround to implement our own
> SortedMapState. I assume that there should be lots of other users who have
> the same use case, so I have a few questions here.
>
> 1. Do we have a feature like SortedMapState already in place, in
> development or in the future roadmap?
>
> 2. If not, could that be a good feature to have? and are there any other
> concerns?
>
> Just want to start a discussion here from a user perspective. If everything
> goes well, we are also interested in contributing back to the community.
>
>
> Best regards,
> Sean
>

Re: [ANNOUNCE] Apache Flink 1.11.1 released

[Congxian Qiu]

Thanks Dian for the great work and thanks to everyone who makes this
release possible!

Best,
Congxian


Rui Li  于2020年7月23日周四 上午10:48写道：

> Thanks Dian for the great work!
>
> On Thu, Jul 23, 2020 at 10:22 AM Jingsong Li 
> wrote:
>
> > Thanks for being the release manager for the 1.11.1 release, Dian.
> >
> > Best,
> > Jingsong
> >
> > On Thu, Jul 23, 2020 at 10:12 AM Zhijiang 
> > wrote:
> >
> >> Thanks for being the release manager and the efficient work, Dian!
> >>
> >> Best,
> >> Zhijiang
> >>
> >> --
> >> From:Konstantin Knauf 
> >> Send Time:2020年7月22日(星期三) 19:55
> >> To:Till Rohrmann 
> >> Cc:dev ; Yangze Guo ; Dian
> Fu <
> >> dia...@apache.org>; user ; user-zh <
> >> user...@flink.apache.org>
> >> Subject:Re: [ANNOUNCE] Apache Flink 1.11.1 released
> >>
> >> Thank you for managing the quick follow up release. I think this was
> very
> >> important for Table & SQL users.
> >>
> >> On Wed, Jul 22, 2020 at 1:45 PM Till Rohrmann 
> >> wrote:
> >> Thanks for being the release manager for the 1.11.1 release, Dian.
> Thanks
> >> a lot to everyone who contributed to this release.
> >>
> >> Cheers,
> >> Till
> >>
> >> On Wed, Jul 22, 2020 at 11:38 AM Hequn Cheng  wrote:
> >> Thanks Dian for the great work and thanks to everyone who makes this
> >> release possible!
> >>
> >> Best, Hequn
> >>
> >> On Wed, Jul 22, 2020 at 4:40 PM Jark Wu  wrote:
> >>
> >> > Congratulations! Thanks Dian for the great work and to be the release
> >> > manager!
> >> >
> >> > Best,
> >> > Jark
> >> >
> >> > On Wed, 22 Jul 2020 at 15:45, Yangze Guo  wrote:
> >> >
> >> > > Congrats!
> >> > >
> >> > > Thanks Dian Fu for being release manager, and everyone involved!
> >> > >
> >> > > Best,
> >> > > Yangze Guo
> >> > >
> >> > > On Wed, Jul 22, 2020 at 3:14 PM Wei Zhong 
> >> > wrote:
> >> > > >
> >> > > > Congratulations! Thanks Dian for the great work!
> >> > > >
> >> > > > Best,
> >> > > > Wei
> >> > > >
> >> > > > > 在 2020年7月22日，15:09，Leonard Xu  写道：
> >> > > > >
> >> > > > > Congratulations!
> >> > > > >
> >> > > > > Thanks Dian Fu for the great work as release manager, and thanks
> >> > > everyone involved!
> >> > > > >
> >> > > > > Best
> >> > > > > Leonard Xu
> >> > > > >
> >> > > > >> 在 2020年7月22日，14:52，Dian Fu  写道：
> >> > > > >>
> >> > > > >> The Apache Flink community is very happy to announce the
> release
> >> of
> >> > > Apache Flink 1.11.1, which is the first bugfix release for the
> Apache
> >> > Flink
> >> > > 1.11 series.
> >> > > > >>
> >> > > > >> Apache Flink® is an open-source stream processing framework for
> >> > > distributed, high-performing, always-available, and accurate data
> >> > streaming
> >> > > applications.
> >> > > > >>
> >> > > > >> The release is available for download at:
> >> > > > >> https://flink.apache.org/downloads.html
> >> > > > >>
> >> > > > >> Please check out the release blog post for an overview of the
> >> > > improvements for this bugfix release:
> >> > > > >> https://flink.apache.org/news/2020/07/21/release-1.11.1.html
> >> > > > >>
> >> > > > >> The full release notes are available in Jira:
> >> > > > >>
> >> > >
> >> >
> >>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12348323
> >> > > > >>
> >> > > > >> We would like to thank all contributors of the Apache Flink
> >> > community
> >> > > who made this release possible!
> >> > > > >>
> >> > > > >> Regards,
> >> > > > >> Dian
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >>
> >> --
> >>
> >> Konstantin Knauf
> >>
> >> https://twitter.com/snntrable
> >>
> >> https://github.com/knaufk
> >>
> >>
> >>
> >
> > --
> > Best, Jingsong Lee
> >
>
>
> --
> Best regards!
> Rui Li
>

Re: [VOTE] Release 1.11.1, release candidate #1

[Congxian Qiu]

Hi  Dian

Thanks for the information.

Best,
Congxian


Dian Fu  于2020年7月20日周一 上午11:44写道：

> Hi Congxian,
>
> FLINK-18544 was to fix an issue introduced in 1.11.1 and so it should not
> appear in the release note according to the release guide[1].
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/Creating+a+Flink+Release#CreatingaFlinkRelease-ReviewReleaseNotesinJIRA
> <
> https://cwiki.apache.org/confluence/display/FLINK/Creating+a+Flink+Release#CreatingaFlinkRelease-ReviewReleaseNotesinJIRA
> >
>
> Regards,
> Dian
>
> > 在 2020年7月20日，上午11:32，Dian Fu  写道：
> >
> > +1 (non-binding)
> >
> > - checked the checksum and signature
> > - installed PyFlink package on MacOS and run some tests
> >
> > Regards,
> > Dian
> >
> >> 在 2020年7月20日，上午11:11，Congxian Qiu  qcx978132...@gmail.com>> 写道：
> >>
> >> +1 （non-binding)
> >>
> >> I found that the fix version of FLINK-18544 is 1.11.1, and the release
> did not contain it. I think we should fix it in the release note.
> >>
> >> checked
> >> - build from source, ok
> >> - sha512 sum, ok
> >> - gpg key, ok
> >> - License seem ok(checked the change of all pom.xml between 1.11.0 and
> 1.11.1
> https://github.com/apache/flink/compare/release-1.11.0..release-1.11.1-rc1
> <
> https://github.com/apache/flink/compare/release-1.11.0..release-1.11.1-rc1
> >
> >> - Run some demo locally
> >>
> >> Best,
> >> Congxian
> >>
> >>
> >> Rui Li mailto:lirui.fu...@gmail.com>>
> 于2020年7月18日周六 下午7:04写道：
> >> +1 (non-binding)
> >>
> >> - Built from source
> >> - Verified hive connector tests for all hive versions
> >> - Played some simple cases with hive connector and everything seems fine
> >>
> >> On Sat, Jul 18, 2020 at 12:24 AM Rui Li  lirui.fu...@gmail.com>> wrote:
> >>
> >> > OK, I agree FLINK-18588 can wait for the next release.
> >> >
> >> > On Fri, Jul 17, 2020 at 11:56 PM Leonard Xu  <mailto:xbjt...@gmail.com>> wrote:
> >> >
> >> >> +1 (non-binding)
> >> >>
> >> >> - checked/verified signatures and hashes
> >> >> - built from source code with scala 2.11 succeeded
> >> >> - checked that there are no missing artifacts
> >> >> - started a cluster, the Web UI was accessible, submitted a
> wordcount job
> >> >> and ran succeeded, no suspicious log output
> >> >> - test using SQL Client to submit job and the query result is
> expected
> >> >> - go through all issues which fix version property is 1.11.1, all
> issues
> >> >> are closed except FLINK-15794,
> >> >>   and FLINK-15794 has fixed in master and 1.11.1 just wait for being
> >> >> fixed in 1.10.2.
> >> >> - the web PR looks good
> >> >>
> >> >> For FLINK-18588,  I also agree with Timo to put it to 1.11.2 because
> >> >> it's  a `Major` bug rather than `Blocker`.
> >> >>
> >> >> Best,
> >> >> Leonard
> >> >
> >> >
> >> >
> >> > --
> >> > Best regards!
> >> > Rui Li
> >> >
> >>
> >>
> >> --
> >> Best regards!
> >> Rui Li
> >
>
>

Re: [VOTE] Release 1.11.1, release candidate #1

[Congxian Qiu]

+1 （non-binding)

I found that the fix version of FLINK-18544 is 1.11.1, and the release did
not contain it. I think we should fix it in the release note.

checked
- build from source, ok
- sha512 sum, ok
- gpg key, ok
- License seem ok(checked the change of all pom.xml between 1.11.0 and
1.11.1
https://github.com/apache/flink/compare/release-1.11.0..release-1.11.1-rc1
- Run some demo locally

Best,
Congxian


Rui Li  于2020年7月18日周六 下午7:04写道：

> +1 (non-binding)
>
> - Built from source
> - Verified hive connector tests for all hive versions
> - Played some simple cases with hive connector and everything seems fine
>
> On Sat, Jul 18, 2020 at 12:24 AM Rui Li  wrote:
>
> > OK, I agree FLINK-18588 can wait for the next release.
> >
> > On Fri, Jul 17, 2020 at 11:56 PM Leonard Xu  wrote:
> >
> >> +1 (non-binding)
> >>
> >> - checked/verified signatures and hashes
> >> - built from source code with scala 2.11 succeeded
> >> - checked that there are no missing artifacts
> >> - started a cluster, the Web UI was accessible, submitted a wordcount
> job
> >> and ran succeeded, no suspicious log output
> >> - test using SQL Client to submit job and the query result is expected
> >> - go through all issues which fix version property is 1.11.1, all issues
> >> are closed except FLINK-15794,
> >>   and FLINK-15794 has fixed in master and 1.11.1 just wait for being
> >> fixed in 1.10.2.
> >> - the web PR looks good
> >>
> >> For FLINK-18588,  I also agree with Timo to put it to 1.11.2 because
> >> it's  a `Major` bug rather than `Blocker`.
> >>
> >> Best,
> >> Leonard
> >
> >
> >
> > --
> > Best regards!
> > Rui Li
> >
>
>
> --
> Best regards!
> Rui Li
>

Re: [DISCUSS] Releasing Flink 1.11.1 soon?

[Congxian Qiu]

+1 for a quick bug fix release for 1.11

Best,
Congxian


Yu Li  于2020年7月10日周五 上午9:37写道：

> +1, thanks Jark for bringing this up and Dian for volunteering as our
> release manager.
>
> Best Regards,
> Yu
>
>
> On Fri, 10 Jul 2020 at 09:29, Hequn Cheng  wrote:
>
> > +1 for a quick bug fix release and Dian as the release manager.
> >
> > Best,
> > Hequn
> >
> >
> > On Thu, Jul 9, 2020 at 9:22 PM Dian Fu  wrote:
> >
> > > Hi Jark,
> > >
> > > Thanks for offering the help. It would definitely be helpful.
> > >
> > > Regards,
> > > Dian
> > >
> > > > 在 2020年7月9日，下午8:54，Benchao Li  写道：
> > > >
> > > > +1 for a quick bug fix release for 1.11
> > > >
> > > > Aljoscha Krettek  于2020年7月9日周四 下午8:11写道：
> > > >
> > > >> +1
> > > >>
> > > >> I'd also be in favour of releasing a 1.11.1 quickly
> > > >>
> > > >> Aljoscha
> > > >>
> > > >> On 09.07.20 13:57, Jark Wu wrote:
> > > >>> Hi Dian,
> > > >>>
> > > >>> Glad to hear that you want to be the release manager of Flink
> 1.11.1.
> > > >>> I am very willing to help you with the final steps of the release
> > > >> process.
> > > >>>
> > > >>> Best,
> > > >>> Jark
> > > >>>
> > > >>> On Thu, 9 Jul 2020 at 17:57, Jingsong Li 
> > > wrote:
> > > >>>
> > >  FLINK-18461 is really a blocker for the CDC feature.
> > > 
> > >  So +1 for releasing Flink 1.11.1 soon.
> > > 
> > >  Best,
> > >  Jingsong
> > > 
> > >  On Thu, Jul 9, 2020 at 5:34 PM jincheng sun <
> > sunjincheng...@gmail.com
> > > >
> > >  wrote:
> > > 
> > > > Thanks for bring up this discussion Jark.
> > > > +1, looking forward the first bugfix version of Flink 1.11.
> > > >
> > > > Best,
> > > > Jincheng
> > > >
> > > > Dian Fu  于2020年7月9日周四 下午5:28写道：
> > > >
> > > >> Thanks Jark for bringing up this discussion. I also noticed that
> > > there
> > > > are
> > > >> already users trying out the CDC feature and so it makes sense
> to
> > > >> have a
> > > >> quick 1.11.1 release.
> > > >>
> > > >> I would volunteer as the release manager of 1.11.1 if we finally
> > > >> decide
> > > > to
> > > >> have a quick release. Also +1 to create the first RC on next
> > Monday.
> > > >>
> > > >> Thanks,
> > > >> Dian
> > > >>
> > > >>> 在 2020年7月9日，下午3:55，Dawid Wysakowicz 
> 写道：
> > > >>>
> > > >>> I do agree it would be beneficial to have the 1.11.1 rather
> soon.
> > > >>>
> > > >>> Personally additionally to Jark's list I'd like to see:
> > > >>>
> > > >>> FLINK-18419  Can not create a catalog from user jar
> > > >>> (https://issues.apache.org/jira/browse/FLINK-18419)
> > > >>>
> > > >>> incluedd. It has a PR already.
> > > >>>
> > > >>> Best,
> > > >>>
> > > >>> Dawid
> > > >>>
> > > >>> On 09/07/2020 09:51, Leonard Xu wrote:
> > >  Hi, Jark
> > > 
> > >  Thanks fro bringing this discussion.
> > > 
> > >  We canceled the blocker priority of FLINK-18461 when release
> > Flink
> > > >> 1.11.0 from time consideration, IIRC, the majority of us agreed
> to
> > > >> have
> > > > a
> > > >> timely 1.11.1 in the vote mail of Flink 1.11.0.
> > >  And I have noticed some user voice[1][2] that they’re using
> CDC
> > > > feature
> > > >> in mail list and offline communication with community user, I
> > think
> > > >> the
> > > >> following notable issues should be included in 1.11.1:
> > > 
> > >  - FLINK-18461 Changelog source can't be insert into upsert
> sink
> > >  - FLINK-18426 Incompatible deprecated key type for
> registration
> > > > cluster
> > >  - FLINK-18434 Can not select fields with JdbcCatalog
> > >  - FLINK-17726 Scheduler should take care of tasks directly
> > > canceled
> > > > by
> > > >> TaskManager
> > >  - FLINK-17075 Add task status reconciliation between TM and JM
> > > 
> > >  So,  +1 to prepare a quick bug-fix version from my side.
> > > 
> > > 
> > >  Best,
> > >  Leonard Xu
> > >  [1] user:
> > > >>
> > > >
> > > >>
> > >
> >
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flip-105-can-the-debezium-canal-SQL-sink-to-database-directly-td36275.html#a36306
> > > >> <
> > > >>
> > > >
> > > >>
> > >
> >
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flip-105-can-the-debezium-canal-SQL-sink-to-database-directly-td36275.html#a36306
> > > >>>
> > >  [2] user-zh:
> > > >>
> > > >
> > > >>
> > >
> >
> http://apache-flink.147419.n8.nabble.com/flink-1-11-cdc-flink-sql-kafka-topic-tt4567.html#a4590
> > > >> <
> > > >>
> > > >
> > > >>
> > >
> >
> http://apache-flink.147419.n8.nabble.com/flink-1-11-cdc-flink-sql-kafka-topic-tt4567.html#a4590
> > > >>>
> > > 
> > > > 在 2020年7月9日，11:12，Jark Wu  写道：
> > > >
> > > > Besides, it 

Re: [ANNOUNCE] Apache Flink 1.11.0 released

[Congxian Qiu]

Thanks Zhijiang and Piotr for the great work as release manager, and thanks
everyone who makes the release possible!

Best,
Congxian


Benchao Li  于2020年7月8日周三 下午12:39写道：

> Congratulations!  Thanks Zhijiang & Piotr for the great work as release
> managers.
>
> Rui Li  于2020年7月8日周三 上午11:38写道：
>
>> Congratulations! Thanks Zhijiang & Piotr for the hard work.
>>
>> On Tue, Jul 7, 2020 at 10:06 PM Zhijiang 
>> wrote:
>>
>>> The Apache Flink community is very happy to announce the release of
>>> Apache Flink 1.11.0, which is the latest major release.
>>>
>>> Apache Flink® is an open-source stream processing framework for distributed,
>>> high-performing, always-available, and accurate data streaming
>>> applications.
>>>
>>> The release is available for download at:
>>> https://flink.apache.org/downloads.html
>>>
>>> Please check out the release blog post for an overview of the
>>> improvements for this new major release:
>>> https://flink.apache.org/news/2020/07/06/release-1.11.0.html
>>>
>>> The full release notes are available in Jira:
>>>
>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12346364
>>>
>>> We would like to thank all contributors of the Apache Flink community
>>> who made this release possible!
>>>
>>> Cheers,
>>> Piotr & Zhijiang
>>>
>>
>>
>> --
>> Best regards!
>> Rui Li
>>
>
>
> --
>
> Best,
> Benchao Li
>

Re: [ANNOUNCE] New PMC member: Piotr Nowojski

[Congxian Qiu]

Congratulations Piotr!

Best,
Congxian


Zhijiang  于2020年7月7日周二 下午12:25写道：

> Congratulations Piotr!
>
> Best,
> Zhijiang
>
>
> --
> From:Rui Li 
> Send Time:2020年7月7日(星期二) 11:55
> To:dev 
> Cc:pnowojski 
> Subject:Re: [ANNOUNCE] New PMC member: Piotr Nowojski
>
> Congrats!
>
> On Tue, Jul 7, 2020 at 11:25 AM Yangze Guo  wrote:
>
> > Congratulations!
> >
> > Best,
> > Yangze Guo
> >
> > On Tue, Jul 7, 2020 at 11:01 AM Jiayi Liao 
> > wrote:
> > >
> > > Congratulations Piotr!
> > >
> > > Best,
> > > Jiayi Liao
> > >
> > > On Tue, Jul 7, 2020 at 10:54 AM Jark Wu  wrote:
> > >
> > > > Congratulations Piotr!
> > > >
> > > > Best,
> > > > Jark
> > > >
> > > > On Tue, 7 Jul 2020 at 10:50, Yuan Mei 
> wrote:
> > > >
> > > > > Congratulations, Piotr!
> > > > >
> > > > > On Tue, Jul 7, 2020 at 1:07 AM Stephan Ewen 
> > wrote:
> > > > >
> > > > > > Hi all!
> > > > > >
> > > > > > It is my pleasure to announce that Piotr Nowojski joined the
> Flink
> > PMC.
> > > > > >
> > > > > > Many of you may know Piotr from the work he does on the data
> > processing
> > > > > > runtime and the network stack, from the mailing list, or the
> > release
> > > > > > manager work.
> > > > > >
> > > > > > Congrats, Piotr!
> > > > > >
> > > > > > Best,
> > > > > > Stephan
> > > > > >
> > > > >
> > > >
> >
>
>
> --
> Best regards!
> Rui Li
>
>

Re: [VOTE] Release 1.11.0, release candidate #4

[Congxian Qiu]

+1 (non-binding)  % some NOTICE files missing

I found some projects do not contain NOTICE file (I'm not sure these
projects do not need NOTICE file or the NOTICE file is missing. I report
the findings here.).
The pom.xml change is based on the compare between
release-1.10.0..release-1.11-RC4[1], the dependency for below projects has
uploaded to gist[2]
- flink-connector-elasticsearch6
- flink-connector-elasticsearch7
- flink-connector-hbase
- flink-hcatlog
- flink-orc
- flink-parquet
- flink-sequence-file


checked
- sha512 & gpg check, ok
- build from source, ok
- all pom point to 1.11.0
- run some demos on a real cluster, ok
- manual test savepoint relocatable, ok
- some license check, apart from the NOTICE file, looks good to me.

[1] https://
github.com/apache/flink/compare/release-1.10.0..release-1.11.0-rc4
[2] https://gist.github.com/klion26/026a79897334fdeefec381cf7cdd5d93

Best,
Congxian


Steven Wu  于2020年7月5日周日 上午1:41写道：

> +1 (non-binding)
>
> - rolled out to thousands of router jobs in our test env
> - tested with a large-state job. Did simple resilience and
> checkpoint/savepoint tests. General performance metrics look on par.
> - tested with a high-parallelism stateless transformation job. General
> performance metrics look on par.
>
> On Sat, Jul 4, 2020 at 7:39 AM Zhijiang  .invalid>
> wrote:
>
> > Hi Thomas,
> >
> > Thanks for the further update information.
> >
> > I guess we can dismiss the network stack changes, since in your case the
> > downstream and upstream would probably be deployed in the same slot
> > bypassing the network data shuffle.
> > Also I guess release-1.11 will not bring general performance regression
> in
> > runtime engine, as we also did the performance testing for all general
> > cases by [1] in real cluster before and the testing results should fit
> the
> > expectation. But we indeed did not test the specific source and sink
> > connectors yet as I known.
> >
> > Regarding your performance regression with 40%, I wonder it is probably
> > related to specific source/sink changes (e.g. kinesis) or environment
> > issues with corner case.
> > If possible, it would be helpful to further locate whether the regression
> > is caused by kinesis, by replacing the kinesis source & sink and keeping
> > the others same.
> >
> > As you said, it would be efficient to contact with you directly next week
> > to further discuss this issue. And we are willing/eager to provide any
> help
> > to resolve this issue soon.
> >
> > Besides that, I guess this issue should not be the blocker for the
> > release, since it is probably a corner case based on the current
> analysis.
> > If we really conclude anything need to be resolved after the final
> > release, then we can also make the next minor release-1.11.1 come soon.
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-18433
> >
> > Best,
> > Zhijiang
> >
> >
> > --
> > From:Thomas Weise 
> > Send Time:2020年7月4日(星期六) 12:26
> > To:dev ; Zhijiang 
> > Cc:Yingjie Cao 
> > Subject:Re: [VOTE] Release 1.11.0, release candidate #4
> >
> > Hi Zhijiang,
> >
> > It will probably be best if we connect next week and discuss the issue
> > directly since this could be quite difficult to reproduce.
> >
> > Before the testing result on our side comes out for your respective job
> > case, I have some other questions to confirm for further analysis:
> > -  How much percentage regression you found after switching to 1.11?
> >
> > ~40% throughput decline
> >
> > -  Are there any network bottleneck in your cluster? E.g. the network
> > bandwidth is full caused by other jobs? If so, it might have more effects
> > by above [2]
> >
> > The test runs on a k8s cluster that is also used for other production
> jobs.
> > There is no reason be believe network is the bottleneck.
> >
> > -  Did you adjust the default network buffer setting? E.g.
> > "taskmanager.network.memory.floating-buffers-per-gate" or
> > "taskmanager.network.memory.buffers-per-channel"
> >
> > The job is using the defaults, i.e we don't configure the settings. If
> you
> > want me to try specific settings in the hope that it will help to isolate
> > the issue please let me know.
> >
> > -  I guess the topology has three vertexes "KinesisConsumer ->
> Chained
> > FlatMap -> KinesisProducer", and the partition mode for "KinesisConsumer
> ->
> > FlatMap" and "FlatMap->KinesisProducer" are both "forward"? If so, the
> edge
> > connection is one-to-one, not all-to-all, then the above [1][2] should no
> > effects in theory with default network buffer setting.
> >
> > There are only 2 vertices and the edge is "forward".
> >
> > - By slot sharing, I guess these three vertex parallelism task would
> > probably be deployed into the same slot, then the data shuffle is by
> memory
> > queue, not network stack. If so, the above [2] should no effect.
> >
> > Yes, vertices share slots.
> >
> > - I also saw some Jira 

Re: Could someone help me to review PR[12727]?

[Congxian Qiu]

Hi Roc

Thanks for your contribution. I've reviewed it and gave some comments.
please take a look. thanks

Best,
Congxian


Leonard Xu  于2020年6月23日周二 上午10:23写道：

> Hi, Roc
>
> Thanks for the contribution, I’d like to help review this PR.
>
> Best,
> Leonard Xu

Re: S3 Checkpointing taking long time with stateful operations

[Congxian Qiu]

Hi Sandeep

The picture isn't shown.

First, you can try to find out whether there is some operator's e2e time is
big, the e2e time of snapshot for one operator is time${barrier align time} +
time{sync-snapshot} + time{async-snapshot}. exactly-once and at least once
both need to wait for barrier align, but at least once can process record
when barrier aligning.
For at least once mode, you can enable debug log to track the barrier align
procedure.

Best,
Congxian


Kathula, Sandeep  于2020年6月19日周五
上午9:21写道：

> Hi,
>
> We are running a stateful application in Flink with *RocksDB as backend* and
> set *incremental state to true *with checkpoints written to* S3*.
>
>- 10 task managers each with 2 task slots
>- Checkpoint interval 3 minutes
>- Checkpointing mode – At-least once processing
>
>
>
> After running app for 2-3 days, we are seeing end to end checkpoint takes
> almost *2 minutes* with Sync time 2 sec and async time 15 sec max. But
> initially when state is less, it takes 10-15 sec for checkpointing. As
> checkpointing mode is at least once, align duration is 0. We are seeing a
> dip in processing during this time. Couldn’t find out what the actual issue
> is.
>
>
>
> We also tried with remote HDFS for checkpointing but observed similar
> behavior.
>
>
>
> We have couple of questions:
>
>- *When sync time is max 2 sec and async time is 15 sec why is end to
>end checkpointing taking almost 2 minutes?*
>- *How can we reduce the checkpoint time?*
>
> [image: A screenshot of a cell phone Description automatically generated]
>
>
>
> Any help would be appreciated.
>
>
>
>
>
> Thank you
>
> Sandeep Kathula
>
>
>

Re: [ANNOUNCE] Yu Li is now part of the Flink PMC

[Congxian Qiu]

Congratulations Yu !
Best,
Congxian


Thomas Weise  于2020年6月17日周三 下午6:23写道：

> Congratulations!
>
>
> On Wed, Jun 17, 2020, 2:59 AM Fabian Hueske  wrote:
>
> > Congrats Yu!
> >
> > Cheers, Fabian
> >
> > Am Mi., 17. Juni 2020 um 10:20 Uhr schrieb Till Rohrmann <
> > trohrm...@apache.org>:
> >
> > > Congratulations Yu!
> > >
> > > Cheers,
> > > Till
> > >
> > > On Wed, Jun 17, 2020 at 7:53 AM Jingsong Li 
> > > wrote:
> > >
> > > > Congratulations Yu, well deserved!
> > > >
> > > > Best,
> > > > Jingsong
> > > >
> > > > On Wed, Jun 17, 2020 at 1:42 PM Yuan Mei 
> > wrote:
> > > >
> > > >> Congrats, Yu!
> > > >>
> > > >> GXGX & well deserved!!
> > > >>
> > > >> Best Regards,
> > > >>
> > > >> Yuan
> > > >>
> > > >> On Wed, Jun 17, 2020 at 9:15 AM jincheng sun <
> > sunjincheng...@gmail.com>
> > > >> wrote:
> > > >>
> > > >>> Hi all,
> > > >>>
> > > >>> On behalf of the Flink PMC, I'm happy to announce that Yu Li is now
> > > >>> part of the Apache Flink Project Management Committee (PMC).
> > > >>>
> > > >>> Yu Li has been very active on Flink's Statebackend component,
> working
> > > on
> > > >>> various improvements, for example the RocksDB memory management for
> > > 1.10.
> > > >>> and keeps checking and voting for our releases, and also has
> > > successfully
> > > >>> produced two releases(1.10.0&1.10.1) as RM.
> > > >>>
> > > >>> Congratulations & Welcome Yu Li!
> > > >>>
> > > >>> Best,
> > > >>> Jincheng (on behalf of the Flink PMC)
> > > >>>
> > > >>
> > > >
> > > > --
> > > > Best, Jingsong Lee
> > > >
> > >
> >
>

Re: [Reminder] Prefer {% link %} tag in documentation

[Congxian Qiu]

+1 to use {% link %} tag and add a check during CI.
for Chinese doc, will suggest the Chinese translate contributor use the {%
link %} tag when reviewing the translate pr.

Best,
Congxian


Jark Wu  于2020年6月10日周三 上午10:48写道：

> +1 to use  {% link %}  tag and add check in CI.
>
> Tips: if want to link a Chinese page, should write: [CLI]({% link ops/
> cli.zh.md %})
>
> Best,
> Jark
>
> On Wed, 10 Jun 2020 at 10:30, Yangze Guo  wrote:
>
> > Thanks for that reminder, Seth!
> >
> > +1 to add a check during CI if possible.
> >
> > Best,
> > Yangze Guo
> >
> > On Wed, Jun 10, 2020 at 3:04 AM Kostas Kloudas 
> wrote:
> > >
> > > Thanks for the heads up Seth!
> > >
> > > Kostas
> > >
> > > On Tue, Jun 9, 2020 at 7:27 PM Seth Wiesman 
> wrote:
> > > >
> > > > The tag is new to Jekyll 4.0 which we only recently updated to.
> > > >
> > > > There are a lot of existing tags that would need to be updated first
> :)
> > > > I opened a ticket to track that work and then yes that would make
> > sense.
> > > >
> > > > Seth
> > > >
> > > > [1] https://issues.apache.org/jira/browse/FLINK-18193
> > > >
> > > > On Tue, Jun 9, 2020 at 12:05 PM Robert Metzger 
> > wrote:
> > > >
> > > > > Thanks for the reminder. I was also not aware of this tag!
> > > > >
> > > > > How about enforcing the use of this tag through CI?
> > > > > We could for example grep through the added lines of all changes in
> > docs/
> > > > > and fail the build if we see the wrong pattern.
> > > > >
> > > > >
> > > > > On Tue, Jun 9, 2020 at 4:37 PM Seth Wiesman 
> > wrote:
> > > > >
> > > > > > Whoops, responded from the wrong email :)
> > > > > >
> > > > > > Thank you for noticing that, the guide is out of date. I will fix
> > that
> > > > > > immediately!
> > > > > >
> > > > > > On Tue, Jun 9, 2020 at 9:36 AM Seth Wiesman 
> > wrote:
> > > > > >
> > > > > > > Thank you for noticing that, the guide is out of date. I will
> > fix that
> > > > > > > immediately!
> > > > > > >
> > > > > > > On Tue, Jun 9, 2020 at 9:34 AM Dawid Wysakowicz <
> > > > > dwysakow...@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Hi Seth,
> > > > > > >>
> > > > > > >> Thanks I did not know that.
> > > > > > >>
> > > > > > >> I am not entirely sure, but I think our documentation guide is
> > > > > slightly
> > > > > > >> outdated on that manner (
> > > > > > >> https://flink.apache.org/contributing/docs-style.html) Or is
> > there a
> > > > > > >> mistake in your example? Our guide recommends:
> > > > > > >>
> > > > > > >> [CLI]({{ site.baseurl }}{% link ops/cli.md %})
> > > > > > >>
> > > > > > >>
> > > > > > >> Best,
> > > > > > >>
> > > > > > >> Dawid
> > > > > > >> On 09/06/2020 16:20, Seth Wiesman wrote:
> > > > > > >>
> > > > > > >> Hi Everyone!
> > > > > > >>
> > > > > > >> As we are seeing an influx of documentation PRs in
> anticipation
> > of the
> > > > > > 1.11
> > > > > > >> release I would like to remind everyone to use the {% link %}
> > tag when
> > > > > > >> cross-linking pages[1]. This is opposed to creating a link
> > based on
> > > > > > >> site.baseurl.
> > > > > > >>
> > > > > > >> Going forward a link such as:
> > > > > > >>
> > > > > > >> [CLI]({% site.baseurl %}/ops/cli.html)
> > > > > > >>
> > > > > > >> Should be written as:
> > > > > > >>
> > > > > > >> [CLI]({% link ops/cli.md %})
> > > > > > >>
> > > > > > >> This tag will fail the build on broken links which will help
> us
> > > > > prevent
> > > > > > >> 404s on the website.
> > > > > > >>
> > > > > > >> You can see a good example of the link tag in action here[2].
> > > > > > >>
> > > > > > >> Seth
> > > > > > >>
> > > > > > >> [1] https://jekyllrb.com/docs/liquid/tags/
> > > > > > >> [2]
> > > > > >
> > > > >
> >
> https://github.com/apache/flink/blame/b6ea96251d101ca25aa6a6b92170cfa4274b4cc3/docs/index.md#L65-L67
> > > > > > >>
> > > > > > >>
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > > Seth Wiesman | Solutions Architect
> > > > > > >
> > > > > > > +1 314 387 1463
> > > > > > >
> > > > > > > 
> > > > > > >
> > > > > > > Follow us @VervericaData
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > > Join Flink Forward  - The Apache
> > Flink
> > > > > > > Conference
> > > > > > >
> > > > > > > Stream Processing | Event Driven | Real Time
> > > > > > >
> > > > > >
> > > > >
> >
>

Re: [DISCUSS] Add Japanese translation of the flink.apache.org website

[Congxian Qiu]

Share some experience for the current translation(and review) process from
my side. for documentation review, we can pull the patch locally and set up
a local server[1][2], after that, we can reference the rendered
documentation in `localhost:4000`

[1]
https://flink.apache.org/contributing/contribute-documentation.html#update-or-extend-the-documentation
[2]
https://flink.apache.org/contributing/improve-website.html#update-or-extend-the-documentation

Best,
Congxian


Yun Tang  于2020年6月9日周二 下午10:34写道：

> Hi
>
> I think supporting website with another language should be okay. However,
> I'm afraid Flink community lacks of resources
> to maintain technical documentations of another languages currently.
>
> From my experiences of translating documentation to Chinese since the past
> year, I found
> Chinese documentation is easy to be out-of-date and containing broken
> links. Some developers
> might only update English version and forget to update the related Chinese
> version.
>
> Since Jark talked about re-investigating tools for translating, I just
> want to share some painful experiences
> when translating documentations.
> First of all, I think code review should be totally different from doc
> review. Github lacks such kind of power
> to make the markdown review more readable, we often need to read a long
> line without wrapping and give advice for
> just some of the characters.
> Secondly, user who wants to contributes cannot leverage any given powerful
> tool.
>
> I think Crodwin might be a good choice.
>
>   1.  Many popular projects already use this: node.js [1], gitlab [2],
> Minecraft [3].
>   2.  This tool is free for open-source project [4]
>
> [1] https://crowdin.com/project/nodejs-website
> [2] https://crowdin.com/project/gitlab-ee
> [3] https://crowdin.com/project/minecraft
> [4] https://crowdin.com/page/open-source-project-setup-request
>
> Best
> Yun Tang
>
> 
> From: Marta Paes Moreira 
> Sent: Tuesday, June 9, 2020 22:13
> To: dev 
> Subject: Re: [DISCUSS] Add Japanese translation of the flink.apache.org
> website
>
> I had a second look at the PR and maybe it'd be important to clarify the
> scope.
>
> It seems to only target the website and, if that's the case, then
> synchronization might be less of an issue as the core of the website is
> pretty static. I'm not sure how useful it is to have a translated website
> without any other kind of support (e.g. mailing list, documentation) in the
> same language, but the contributor might be able to shed some light on
> this.
>
> Just wanted to amend my original reply as it was pretty blindsided by the
> previous comments. It'd be worth understanding the end goal of the
> contribution first.
>
> Thanks,
>
> Marta
>
> On Tue, Jun 9, 2020 at 2:22 PM Congxian Qiu 
> wrote:
>
> > Hi
> >
> > Thanks for bringing this up, Robert.
> >
> > I agree we may need to investigate better translation/synchronization
> tools
> > again.
> >
> > I'll share some experience when translating the documentation or
> reviewing
> > the translation.
> > 1. Translate documentation using the current procedure needs a lot of
> > resources to keep the translated version fresh, I'm glad that we have so
> > many contributors who want to translate the documentation to Chinese.
> > 2. The documentation may out of sync between the English version and the
> > translated version, this needs the reviewer to know about Flink and the
> > English version doc very well(especially when the documentation has been
> > restructured)
> >
> > Best,
> > Congxian
> >
> >
> > Jark Wu  于2020年6月9日周二 下午6:02写道：
> >
> > > I agree with Dawid and others' opinions.
> > >
> > > We may not have enough resources to maintain more languages.
> > > Maybe it's time to investigate better translation/synchronization tools
> > > again.
> > >
> > > I want to share some background about the current translation process.
> In
> > > the initial proposal of Chinese translation FLIP-35 [1],
> > > we have considered Docusaurus/Crowdin as the localization tool, but it
> > > seems that Crowdin doesn't fit well with Jekyll (Liquid codes).
> > > But it's been a year and a half, maybe it's time to re-investigate them
> > or
> > > other tools.
> > >
> > > Here is the list of how other ASF projects are dealing translation of
> > what
> > > I know:
> > > - Apache Pulsar uses Crowdin:
> > https://github.com/apache/pulsar-translation
> >

Re: [DISCUSS] Add Japanese translation of the flink.apache.org website

[Congxian Qiu]

Hi

Thanks for bringing this up, Robert.

I agree we may need to investigate better translation/synchronization tools
again.

I'll share some experience when translating the documentation or reviewing
the translation.
1. Translate documentation using the current procedure needs a lot of
resources to keep the translated version fresh, I'm glad that we have so
many contributors who want to translate the documentation to Chinese.
2. The documentation may out of sync between the English version and the
translated version, this needs the reviewer to know about Flink and the
English version doc very well(especially when the documentation has been
restructured)

Best,
Congxian


Jark Wu  于2020年6月9日周二 下午6:02写道：

> I agree with Dawid and others' opinions.
>
> We may not have enough resources to maintain more languages.
> Maybe it's time to investigate better translation/synchronization tools
> again.
>
> I want to share some background about the current translation process. In
> the initial proposal of Chinese translation FLIP-35 [1],
> we have considered Docusaurus/Crowdin as the localization tool, but it
> seems that Crowdin doesn't fit well with Jekyll (Liquid codes).
> But it's been a year and a half, maybe it's time to re-investigate them or
> other tools.
>
> Here is the list of how other ASF projects are dealing translation of what
> I know:
> - Apache Pulsar uses Crowdin: https://github.com/apache/pulsar-translation
> - Apache Kylin: the similar way of Flink:
> https://github.com/apache/kylin/tree/document/website
> - Apache RocketMQ: a separate repository, synchronize manually and
> periodically: https://github.com/apache/rocketmq/tree/master/docs/cn
>
> Here is the list of localization tool of what I know:
> - Docusaurus: https://docusaurus.io/
> - Crowdin: https://crowdin.com/
> - GitLocalize: https://gitlocalize.com/
>
> Best,
> Jark
>
> On Tue, 9 Jun 2020 at 16:24, Marta Paes Moreira 
> wrote:
>
> > Thanks for bringing this PR to our attention, Robert!
> >
> > Like Dawid and Xintong, I'm concerned that adding a new language will
> make
> > it unbearable to ensure docs synchronization. Best-effort maintenance of
> > non-english docs may also result in a bad documentation experience for
> > users, and this is something that we should be especially careful about
> in
> > locations where Flink adoption might still be in an early phase, IMO. At
> > the same time, I understand that having a japanese translation would
> > probably be very helpful for Japanese-speaking users and help with Flink
> > adoption in Japan.
> >
> > I see this happening for other up-and-coming locations (e.g.
> > Spanish-speaking countries) in the future, so in order to be able to
> accept
> > these contributions I'm also of the opinion that we first need an
> efficient
> > synchronization tool. Maybe it would be worth investigating how other ASF
> > projects are dealing with this.
> >
> > Marta
> >
> > On Mon, Jun 8, 2020 at 11:46 AM Xintong Song 
> > wrote:
> >
> > > I think Dawid has a good point. With or without another new language,
> > > I'm +1 for trying to have a better tooling/process for synchronizing
> doc
> > > changes.
> > >
> > > Currently, it is not only hard for people who want to modify the
> English
> > > docs, but also for people who try to update the translated copies
> > > accordingly. I'm currently working on updating the Chinese translations
> > for
> > > the memory configuration documents, and I found it very hard to
> identify
> > > the parts that need updates. The English docs are reorganized, contents
> > are
> > > moved across pages, and also small pieces of details are modified. It
> is
> > > not always possible for people who work on the English docs to locate
> the
> > > right place in the translations where the updated contents should be
> > > pasted.
> > >
> > > My two cents on the potential approach. We might label the translation
> > with
> > > the commit id of the original doc where they are in synchronization,
> and
> > > automatically display a warning on the translation if an out-of-sync is
> > > detected.
> > >
> > > Thank you~
> > >
> > > Xintong Song
> > >
> > >
> > >
> > > On Mon, Jun 8, 2020 at 4:30 PM Dawid Wysakowicz <
> dwysakow...@apache.org>
> > > wrote:
> > >
> > > > I don't have a well defined opinion on adding new language. However,
> > one
> > > > thing that I'd like to bring up to the attention on that occasion is
> it
> > > > is already quite cumbersome to update two versions of the docs.
> > > > Especially when we add new sections or change smaller parts of an
> > > > existing document. Right now if I add three sections in an English
> > > > version I have three additional places in the Chinese documents
> where I
> > > > need to paste that. With additional language it doubles, making it 6
> > > > places where I have to manually paste the added parts.
> > > >
> > > > I'd be way more welcoming for adding a new language if we had a
> better
> > > > tooling/process for synchronizing

Re: [ANNOUNCE] New Flink Committer: Benchao Li

[Congxian Qiu]

Congratulations, Benchao!
Best,
Congxian


Marta Paes Moreira  于2020年6月9日周二 下午7:24写道：

> Congrats, Benchao!
>
> On Tue, Jun 9, 2020 at 11:41 AM Till Rohrmann 
> wrote:
>
> > Congratulations!
> >
> > Cheers,
> > Till
> >
> > On Tue, Jun 9, 2020 at 11:19 AM Yang Wang  wrote:
> >
> > > Congratulations!
> > >
> > > Best,
> > > Yang
> > >
> > > Yu Li  于2020年6月9日周二 下午5:11写道：
> > >
> > > > Congratulations, Benchao!
> > > >
> > > > Best Regards,
> > > > Yu
> > > >
> > > >
> > > > On Tue, 9 Jun 2020 at 17:05, Yuan Mei 
> wrote:
> > > >
> > > > > Congrats!
> > > > >
> > > > > On Tue, Jun 9, 2020 at 2:10 PM Jark Wu  wrote:
> > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > On behalf of the PMC, I'm very happy to announce Benchao Li as a
> > new
> > > > > Apache
> > > > > > Flink committer.
> > > > > >
> > > > > > Benchao started contributing to Flink since late 2018. He is very
> > > > active
> > > > > in
> > > > > > Flink SQL component,
> > > > > > and has also participated in many discussions, bug fixes. Over
> the
> > > past
> > > > > few
> > > > > > months, he helped tremendously in answering user questions in the
> > > > mailing
> > > > > > list.
> > > > > >
> > > > > > Please join me in congratulating Benchao for becoming a Flink
> > > > committer!
> > > > > >
> > > > > > Thanks,
> > > > > > Jark
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] (Document) Backwards Compatibility of Savepoints

[Congxian Qiu]

Sorry for jumping in late.

Currently, we only have a forward-compatible guarantee and do not have the
backward-compatible guarantee. And as this may take a large effort to
support the backward-compatible guarantee. so I agree that we should write
this down explicitly.

For the given scenario, I have a little question: Why do we want to restore
from the savepoint taken the new Flink version instead of the previous
savepoint, is that we want to minimize the source rewind?

Best,
Congxian


Steven Wu  于2020年6月3日周三 上午9:08写道：

> Current Flink documentation is actually pretty clear about no guarantees
> for backward compatibility.
>
> https://ci.apache.org/projects/flink/flink-docs-stable/ops/upgrading.html#compatibility-table
>
> On Tue, Jun 2, 2020 at 3:20 AM Yun Tang  wrote:
>
> > Since Flink lacks of such kind of experiments to ensure the backwards
> > compatibility of savepoints before, especially those built-in operators
> > with their own operator state.
> > I am afraid we need huge energy to cover all cases to give the most
> > correct result.
> >
> > I prefer to just point out this in documentation to say explicitly Flink
> > does not guarantee such kind of backwards compatibility.
> >
> > Best
> > Yun Tang
> > 
> > From: Ufuk Celebi 
> > Sent: Wednesday, May 27, 2020 16:42
> > To: dev@flink.apache.org 
> > Subject: Re: [DISCUSS] (Document) Backwards Compatibility of Savepoints
> >
> > I agree with Konstantin and Steven that it makes sense to point this out
> > explicitly.
> >
> > I think that the following would be helpful:
> >
> > 1/ Mention breaking compatibility in release notes
> >
> > 2/ Update the linked table to reflect compatibilities while pointing out
> > what the community commits to maintain going forward (e.g. "happens to
> > work" vs. "guaranteed to work")
> >
> > In general, the table is quite large. Would it make sense to order the
> > releases in reverse order (assuming that the table is more relevant for
> > recent releases)?
> >
> > – Ufuk
> >
> > On Tue, May 26, 2020 at 8:36 PM Steven Wu  wrote:
> >
> > > > A use case for this might be when you want to rollback a framework
> > > upgrade (after some time) due to e.g. a performance
> > > or stability issue.
> > >
> > > Downgrade (that Konstantin called out) is an important and realistic
> > > scenario. It will be great to support backward compatibility for
> > savepoint
> > > or at least document any breaking change.
> > >
> > > On Tue, May 26, 2020 at 4:39 AM Piotr Nowojski 
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > It might have been implicit choice, but so far we were not supporting
> > the
> > > > scenario that you are asking for. It has never been tested and we
> have
> > > > lot’s of state migration code sprinkled among our code base (for
> > example
> > > > upgrading state fields of the operators like [1]), that only supports
> > > > upgrades, not downgrades.
> > > >
> > > > Also we do not have testing infrastructure for checking the
> downgrades.
> > > We
> > > > would need to check if save points taken from master branch, are
> > readable
> > > > by previous releases (not release branch!).
> > > >
> > > > So all in all, I don’t think it can be easily done. It would require
> > some
> > > > effort to start maintaining backward compatibility.
> > > >
> > > > Piotrek
> > > >
> > > > [1]
> > > >
> > >
> >
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011#migrateNextTransactionalIdHindState
> > > >
> > > > > On 26 May 2020, at 13:18, Konstantin Knauf 
> > wrote:
> > > > >
> > > > > Hi everyone,
> > > > >
> > > > > I recently stumbled across the fact that Savepoints created with
> > Flink
> > > > 1.11
> > > > > can not be read by Flink 1.10. A use case for this might be when
> you
> > > want
> > > > > to rollback a framework upgrade (after some time) due to e.g. a
> > > > performance
> > > > > or stability issue.
> > > > >
> > > > > From the documentation [1] it seems as if the Savepoint format is
> > > > generally
> > > > > only forward-compatible although in many cases it is actually also
> > > > > backwards compatible (e.g. Savepoint taken in Flink 1.10, restored
> > with
> > > > > Flink 1.9).
> > > > >
> > > > > Was it a deliberate choice not to document any backwards
> > compatibility?
> > > > If
> > > > > not, should we add the missing entries in the compatibility table?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Konstantin
> > > > >
> > > > > [1]
> > > > >
> > > >
> > >
> >
> https://ci.apache.org/projects/flink/flink-docs-master/ops/upgrading.html#compatibility-table
> > > > >
> > > > > --
> > > > >
> > > > > Konstantin Knauf
> > > > >
> > > > > https://twitter.com/snntrable
> > > > >
> > > > > https://github.com/knaufk
> > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] New Apache Flink Committer - Xintong Song

[Congxian Qiu]

Congratulations! Xintong!
Best,
Congxian


Jark Wu  于2020年6月5日周五 下午3:00写道：

> Congratulations Xintong!
>
> Best,
> Jark
>
> On Fri, 5 Jun 2020 at 14:32, Danny Chan  wrote:
>
> > Congratulations Xintong !
> >
> > Best,
> > Danny Chan
> > 在 2020年6月5日 +0800 PM2:20，dev@flink.apache.org，写道：
> > >
> > > Congratulations Xintong
> >
>

Re: [VOTE] Apache Flink Stateful Functions 2.1.0, release candidate #1

[Congxian Qiu]

@Tzu-Li (Gordon) Tai   Thanks for the info. `mvn clean
install -Prun=e2e-tests` works for me. before verified demo on a clean
source directory.

Best,
Congxian


Tzu-Li (Gordon) Tai  于2020年6月4日周四 下午6:35写道：

> +1 (binding)
>
> Legal
>
> - Verified signatures and hashes of staged Maven artifacts, source
> distribution and Python SDK distribution
> - Checked NOTICE file of statefun-flink-distribution and
> statefun-ridesharing-example-simulator
>
> Functional
>
> - Full build with end-to-end-tests, JDK 8: mvn clean install
> -Prun-e2e-tests
> - Manually verified state TTL for remote functions
> - Manually verified checkpointing with failure recovery
> - Manually verified savepointing + manual restore
> - Generated quickstart project from archetype works
>
>
> On Thu, Jun 4, 2020 at 3:10 PM Hequn Cheng  wrote:
>
> > +1 (binding)
> >
> > - Signatures and hash are correct.
> > - All artifacts to be released to Maven in the staging Nexus repository.
> > - Verify that the source archives do not contain any binaries.
> > - Go through all commits from the last release. No license problem
> spotted.
> > - Check end-to-end tests. All tests have been passed on Travis(both for
> JDK
> > 1.8 and 1.11).
> >
> > Best,
> > Hequn
> >
> > On Thu, Jun 4, 2020 at 12:50 PM Tzu-Li (Gordon) Tai  >
> > wrote:
> >
> > > Hi Hequn,
> > >
> > > Sorry, I mis-tagged the wrong commit.
> > > Just fixed this, the tag [1] [2] should now be pointing to the correct
> > > commit that contains the updated version.
> > >
> > > Gordon
> > >
> > > [1]
> > >
> > >
> >
> https://gitbox.apache.org/repos/asf?p=flink-statefun.git;a=tag;h=c08c9850147d818fc8fed877a01ff87021f3cf21
> > > [2] https://github.com/apache/flink-statefun/tree/release-2.1.0-rc1
> > >
> > > On Thu, Jun 4, 2020 at 12:10 PM Hequn Cheng  wrote:
> > >
> > > > It seems the release tag is not correct? The version in the poms
> should
> > > > be 2.1.0 instead of 2.1-SNAPSHOT.
> > > >
> > > > Best,
> > > > Hequn
> > > >
> > > >
> > > > On Thu, Jun 4, 2020 at 10:33 AM Congxian Qiu  >
> > > > wrote:
> > > >
> > > > > +1  (non-binding)
> > > > >
> > > > > maybe there is something that needs to be updated in
> > > README.md(currently
> > > > > the official docs link points to the master instead of 2.1)
> > > > >
> > > > > and have another question: do we need to add the command used to
> > build
> > > > the
> > > > > base image locally(which was on the README.md in release-2.0.0)?
> > > > >
> > > > > checked
> > > > > - sha & gpg, ok
> > > > > - mvn clean install -Prun-e2e-test on 1.8.0_252, ok
> > > > > - source archives do not contains any binaries
> > > > > - maven clean install -Papache-release, ok (this step need a gpg
> > secret
> > > > > key)
> > > > > - check all pom files, dockerfiles, examples point to the same
> > version,
> > > > ok
> > > > > - check READM.md, nothing unexpected.
> > > > > - but the official docs link points to the master instead of
> 2.1
> > > > > - run greeter&ride-share demo, ok
> > > > >
> > > > > Best,
> > > > > Congxian
> > > > >
> > > > >
> > > > > Tzu-Li (Gordon) Tai  于2020年6月1日周一 下午3:25写道：
> > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > Please review and vote on the *release candidate #1* for the
> > version
> > > > > 2.1.0
> > > > > > of
> > > > > > Apache Flink Stateful Functions,
> > > > > > as follows:
> > > > > > [ ] +1, Approve the release
> > > > > > [ ] -1, Do not approve the release (please provide specific
> > comments)
> > > > > >
> > > > > > ***Testing Guideline***
> > > > > >
> > > > > > You can find here [1] a page in the project wiki on instructions
> > for
> > > > > > testing.
> > > > > > To cast a vote, it is not necessary to perform all listed checks,
> > > > > > but please mention which checks you have performed when voting.
> > > > > >
> > > &g

Re: [VOTE] Apache Flink Stateful Functions 2.1.0, release candidate #1

[Congxian Qiu]

+1  (non-binding)

maybe there is something that needs to be updated in README.md(currently
the official docs link points to the master instead of 2.1)

and have another question: do we need to add the command used to build the
base image locally(which was on the README.md in release-2.0.0)?

checked
- sha & gpg, ok
- mvn clean install -Prun-e2e-test on 1.8.0_252, ok
- source archives do not contains any binaries
- maven clean install -Papache-release, ok (this step need a gpg secret key)
- check all pom files, dockerfiles, examples point to the same version, ok
- check READM.md, nothing unexpected.
- but the official docs link points to the master instead of 2.1
- run greeter&ride-share demo, ok

Best,
Congxian


Tzu-Li (Gordon) Tai  于2020年6月1日周一 下午3:25写道：

> Hi everyone,
>
> Please review and vote on the *release candidate #1* for the version 2.1.0
> of
> Apache Flink Stateful Functions,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> ***Testing Guideline***
>
> You can find here [1] a page in the project wiki on instructions for
> testing.
> To cast a vote, it is not necessary to perform all listed checks,
> but please mention which checks you have performed when voting.
>
> ***Release Overview***
>
> As an overview, the release consists of the following:
> a) Stateful Functions canonical source distribution, to be deployed to the
> release repository at dist.apache.org
> b) Stateful Functions Python SDK distributions to be deployed to PyPI
> c) Maven artifacts to be deployed to the Maven Central Repository
>
> ***Staging Areas to Review***
>
> The staging areas containing the above mentioned artifacts are as follows,
> for your review:
> * All artifacts for a) and b) can be found in the corresponding dev
> repository at dist.apache.org [2]
> * All artifacts for c) can be found at the Apache Nexus Repository [3]
>
> All artifacts are singed with the
> key 1C1E2394D3194E1944613488F320986D35C33D6A [4]
>
> Other links for your review:
> * JIRA release notes [5]
> * source code tag "release-2.1.0-rc1" [6] [7]
>
> ***Vote Duration***
>
> The vote will be open for at least 72 hours *(target end date is Wednesday,
> Jun. 3rd).*
> It is adopted by majority approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Gordon
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/Verifying+a+Flink+Stateful+Functions+Release
> [2] https://dist.apache.org/repos/dist/dev/flink/flink-statefun-2.1.0-rc1/
> [3]
> https://repository.apache.org/content/repositories/orgapacheflink-1373/
> [4] https://dist.apache.org/repos/dist/release/flink/KEYS
> [5]
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12347861
> [6]
>
> https://gitbox.apache.org/repos/asf?p=flink-statefun.git;a=tag;h=a372c69501b8816609f9e90872b1a1e10ab66e8e
> [7] https://github.com/apache/flink-statefun/tree/release-2.1.0-rc1
>

Re: [VOTE] Release flink-shaded 11.0, release candidate #1

[Congxian Qiu]

+1 (non-binding)

checked
- mvn clean verify, ok
- gpg & sha512, ok
- all pom files point to the same version, ok
- checked license, ok

Best,
Congxian


Robert Metzger  于2020年5月27日周三 下午6:05写道：

> +1 (binding)
>
> Checks:
> - diff to flink-shaded 1.10:
>
> https://github.com/apache/flink-shaded/compare/release-10.0...release-11.0-rc1
>
> - mvn clean install passes on the source archive
> - sha of source archive is correct
> - source archive is signed by Chesnay
> - mvn staging repo looks reasonable
> - flink-shaded-zookeeper 3 jar license documentation seems correct
>
>
>
> On Mon, May 25, 2020 at 7:14 PM Chesnay Schepler 
> wrote:
>
> > Hi everyone,
> > Please review and vote on the release candidate #1 for the version 11.0,
> > as follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> >
> > The complete staging area is available for your review, which includes:
> > * JIRA release notes [1],
> > * the official Apache source release to be deployed to dist.apache.org
> > [2], which are signed with the key with fingerprint 11D464BA [3],
> > * all artifacts to be deployed to the Maven Central Repository [4],
> > * source code tag "release-11.0-rc1" [5],
> > * website pull request listing the new release [6].
> >
> > The vote will be open for at least 72 hours. It is adopted by majority
> > approval, with at least 3 PMC affirmative votes.
> >
> > Thanks,
> > Chesnay
> >
> > [1]
> >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12347784
> > [2] https://dist.apache.org/repos/dist/dev/flink/flink-shaded-11.0-rc1/
> > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > [4]
> > https://repository.apache.org/content/repositories/orgapacheflink-1372/
> > [5] https://github.com/apache/flink-shaded/tree/release-11.0-rc1
> > [6] https://github.com/apache/flink-web/pull/340
> >
> >
>

[jira] [Created] (FLINK-17926) Can't build flink-web docker image because of EOL of Ubuntu:18.10

[Congxian Qiu(klion26) (Jira)]

Congxian Qiu(klion26) created FLINK-17926:
-

 Summary: Can't build flink-web docker image because of EOL of 
Ubuntu:18.10
 Key: FLINK-17926
 URL: https://issues.apache.org/jira/browse/FLINK-17926
 Project: Flink
  Issue Type: Bug
  Components: Project Website
Reporter: Congxian Qiu(klion26)


Currently, the Dockerfile[1] in flink-web project is broken because of the EOL 
of Ubuntu 18.10[2], will encounter the error such as bellow when executing 
{{./run.sh}}
{code:java}
Err:3 http://security.ubuntu.com/ubuntu cosmic-security Release
  404  Not Found [IP: 91.189.88.152 80]
Ign:4 http://archive.ubuntu.com/ubuntu cosmic-updates InRelease
Ign:5 http://archive.ubuntu.com/ubuntu cosmic-backports InRelease
Err:6 http://archive.ubuntu.com/ubuntu cosmic Release
  404  Not Found [IP: 91.189.88.142 80]
Err:7 http://archive.ubuntu.com/ubuntu cosmic-updates Release
  404  Not Found [IP: 91.189.88.142 80]
Err:8 http://archive.ubuntu.com/ubuntu cosmic-backports Release
  404  Not Found [IP: 91.189.88.142 80]
Reading package lists...
{code}
The current LTS versions can be found in release website[2].

Apache Flink docker image uses fedora:28[3], so it unaffected.

As fedora does not have LTS release[4], I proposal to use Ubuntu for website 
here, and change the version from {{18.10}} to the closest LTS version {{18.04, 
tried locally, it works successfully.}}

 [1] 
[https://github.com/apache/flink-web/blob/bc66f0f0f463ab62a22e81df7d7efd301b76a6b4/docker/Dockerfile#L17]

[2] [https://wiki.ubuntu.com/Releases]

 
[3]https://github.com/apache/flink/blob/e92b2bf19bdf03ad3bae906dc5fa3781aeddb3ee/docs/docker/Dockerfile#L17

 [4] 
https://fedoraproject.org/wiki/Fedora_Release_Life_Cycle#Maintenance_Schedule



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Semantics of our JIRA fields

[Congxian Qiu]

ct to resolve/close I think that a good practice could be
> > to
> > > > mark
> > > > > automatically a ticket as resolved once that a PR is detected for
> > that
> > > > > ticket, while marking it as closed should be done by the commiter
> who
> > > > merge
> > > > > the PR.
> > > > >
> > > > > Probably this process would slightly increase the work of a
> committer
> > > but
> > > > > this would make things a lot cleaner and will bring the benefit of
> > > > having a
> > > > > reliable and semantically sound JIRA state.
> > > > >
> > > > > Cheers,
> > > > > Flavio
> > > > >
> > > > > Il Dom 24 Mag 2020, 05:05 Israel Ekpo  ha
> > > scritto:
> > > > >
> > > > > > +1 for the proposal
> > > > > >
> > > > > > This will bring some consistency to the process
> > > > > >
> > > > > > Regarding Closed vs Resolved, should we go back and switch all
> the
> > > > > Resolved
> > > > > > issues to Closed so that is is not confusing to new people to the
> > > > project
> > > > > > in the future that may not have seen this discussion?
> > > > > >
> > > > > > I would recommend changing it to Closed just to be consistent
> since
> > > > that
> > > > > is
> > > > > > the majority and the new process will be using Closed going
> forward
> > > > > >
> > > > > > Those are my thoughts for now
> > > > > >
> > > > > > On Sat, May 23, 2020 at 7:48 AM Congxian Qiu <
> > qcx978132...@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > +1 for the proposal. It's good to have a unified description
> and
> > > > write
> > > > > it
> > > > > > > down in the wiki, so that every contributor has the same
> > > > understanding
> > > > > of
> > > > > > > all the fields.
> > > > > > >
> > > > > > > Best,
> > > > > > > Congxian
> > > > > > >
> > > > > > >
> > > > > > > Till Rohrmann  于2020年5月23日周六 上午12:04写道：
> > > > > > >
> > > > > > > > Thanks for drafting this proposal Robert. +1 for the
> proposal.
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Till
> > > > > > > >
> > > > > > > > On Fri, May 22, 2020 at 5:39 PM Leonard Xu <
> xbjt...@gmail.com>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Thanks bringing up this topic @Robert,  +1 to the proposal.
> > > > > > > > >
> > > > > > > > > It clarifies the JIRA fields well and should be a rule to
> > > follow.
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Leonard Xu
> > > > > > > > > > 在 2020年5月22日，20:24，Aljoscha Krettek  >
> > > 写道：
> > > > > > > > > >
> > > > > > > > > > +1 That's also how I think of the semantics of the
> fields.
> > > > > > > > > >
> > > > > > > > > > Aljoscha
> > > > > > > > > >
> > > > > > > > > > On 22.05.20 08:07, Robert Metzger wrote:
> > > > > > > > > >> Hi all,
> > > > > > > > > >> I have the feeling that the semantics of some of our
> JIRA
> > > > fields
> > > > > > > > (mostly
> > > > > > > > > >> "affects versions", "fix versions" and resolve / close)
> > are
> > > > not
> > > > > > > > defined
> > > > > > > > > in
> > > > > > > > > >> the same way by all the core Flink contributors, which
> > leads
> > > > to
> > > > > > > cases
> > > > > > > > > where
> > > > > > > > > >> I spend quite some time o

Re: [DISCUSS] Semantics of our JIRA fields

[Congxian Qiu]

Hi

Currently, when I'm going to create an issue for Project-website. I'm not
very sure what the "Affect Version/s" should be set. The problem is that
the current Dockfile[1] in flink-web is broken because of the EOL of Ubuntu
18.10[2], the project-web does not affect any release of Flink, it does
affect the process to build website, so what's the version should I set to?

[1]
https://github.com/apache/flink-web/blob/bc66f0f0f463ab62a22e81df7d7efd301b76a6b4/docker/Dockerfile#L17
[2] https://wiki.ubuntu.com/Releases

Best,
Congxian


Flavio Pompermaier  于2020年5月24日周日 下午1:23写道：

> In my experience it's quite complicated for a normal reporter to be able to
> fill all the fields correctly (especially for new users).
> Usually you just wanto to report a problem, remember to add a new feature
> or improve code/documentation but you can't really give a priority, assign
> the correct label or decide which releases will benefit of the fix/new
> feature. This is something that only core developers could decide (IMHO).
>
> My experiece says that it's better if normal users could just open tickets
> with some default (or mark ticket as new) and leave them in such a state
> until an experienced user, one that can assign tickets, have the time to
> read it and immediately reject the ticket or accept it and properly assign
> priorities, fix version, etc.
>
> With respect to resolve/close I think that a good practice could be to mark
> automatically a ticket as resolved once that a PR is detected for that
> ticket, while marking it as closed should be done by the commiter who merge
> the PR.
>
> Probably this process would slightly increase the work of a committer but
> this would make things a lot cleaner and will bring the benefit of having a
> reliable and semantically sound JIRA state.
>
> Cheers,
> Flavio
>
> Il Dom 24 Mag 2020, 05:05 Israel Ekpo  ha scritto:
>
> > +1 for the proposal
> >
> > This will bring some consistency to the process
> >
> > Regarding Closed vs Resolved, should we go back and switch all the
> Resolved
> > issues to Closed so that is is not confusing to new people to the project
> > in the future that may not have seen this discussion?
> >
> > I would recommend changing it to Closed just to be consistent since that
> is
> > the majority and the new process will be using Closed going forward
> >
> > Those are my thoughts for now
> >
> > On Sat, May 23, 2020 at 7:48 AM Congxian Qiu 
> > wrote:
> >
> > > +1 for the proposal. It's good to have a unified description and write
> it
> > > down in the wiki, so that every contributor has the same understanding
> of
> > > all the fields.
> > >
> > > Best,
> > > Congxian
> > >
> > >
> > > Till Rohrmann  于2020年5月23日周六 上午12:04写道：
> > >
> > > > Thanks for drafting this proposal Robert. +1 for the proposal.
> > > >
> > > > Cheers,
> > > > Till
> > > >
> > > > On Fri, May 22, 2020 at 5:39 PM Leonard Xu 
> wrote:
> > > >
> > > > > Thanks bringing up this topic @Robert,  +1 to the proposal.
> > > > >
> > > > > It clarifies the JIRA fields well and should be a rule to follow.
> > > > >
> > > > > Best,
> > > > > Leonard Xu
> > > > > > 在 2020年5月22日，20:24，Aljoscha Krettek  写道：
> > > > > >
> > > > > > +1 That's also how I think of the semantics of the fields.
> > > > > >
> > > > > > Aljoscha
> > > > > >
> > > > > > On 22.05.20 08:07, Robert Metzger wrote:
> > > > > >> Hi all,
> > > > > >> I have the feeling that the semantics of some of our JIRA fields
> > > > (mostly
> > > > > >> "affects versions", "fix versions" and resolve / close) are not
> > > > defined
> > > > > in
> > > > > >> the same way by all the core Flink contributors, which leads to
> > > cases
> > > > > where
> > > > > >> I spend quite some time on filling the fields correctly (at
> least
> > > > what I
> > > > > >> consider correctly), and then others changing them again to
> match
> > > > their
> > > > > >> semantics.
> > > > > >> In an effort to increase our efficiency, and since I'm creating
> a
> > > lot
> > > > of
> > > > > >> (test instability-r

Re: [DISCUSS] Semantics of our JIRA fields

[Congxian Qiu]

+1 for the proposal. It's good to have a unified description and write it
down in the wiki, so that every contributor has the same understanding of
all the fields.

Best,
Congxian


Till Rohrmann  于2020年5月23日周六 上午12:04写道：

> Thanks for drafting this proposal Robert. +1 for the proposal.
>
> Cheers,
> Till
>
> On Fri, May 22, 2020 at 5:39 PM Leonard Xu  wrote:
>
> > Thanks bringing up this topic @Robert,  +1 to the proposal.
> >
> > It clarifies the JIRA fields well and should be a rule to follow.
> >
> > Best,
> > Leonard Xu
> > > 在 2020年5月22日，20:24，Aljoscha Krettek  写道：
> > >
> > > +1 That's also how I think of the semantics of the fields.
> > >
> > > Aljoscha
> > >
> > > On 22.05.20 08:07, Robert Metzger wrote:
> > >> Hi all,
> > >> I have the feeling that the semantics of some of our JIRA fields
> (mostly
> > >> "affects versions", "fix versions" and resolve / close) are not
> defined
> > in
> > >> the same way by all the core Flink contributors, which leads to cases
> > where
> > >> I spend quite some time on filling the fields correctly (at least
> what I
> > >> consider correctly), and then others changing them again to match
> their
> > >> semantics.
> > >> In an effort to increase our efficiency, and since I'm creating a lot
> of
> > >> (test instability-related) tickets these days, I would like to discuss
> > the
> > >> semantics, come to a conclusion and document this in our Wiki.
> > >> *Proposal:*
> > >> *Priority:*
> > >> "Blocker": needs to be resolved before a release (matched based on fix
> > >> versions)
> > >> "Critical": strongly considered before a release
> > >> other priorities have no practical meaning in Flink.
> > >> *Component/s:*
> > >> Primary component relevant for this feature / fix.
> > >> For test-related issues, add the component the test belongs to (for
> > example
> > >> "Connectors / Kafka" for Kafka test failures) + "Test".
> > >> The same applies for documentation tickets. For example, if there's
> > >> something wrong with the DataStream API, add it to the "API /
> > DataStream"
> > >> and "Documentation" components.
> > >> *Affects Version/s:*Only for Bug / Task-type tickets: We list all
> > currently
> > >> supported and unreleased Flink versions known to be affected by this.
> > >> Example: If I see a test failure that happens on "master" and
> > >> "release-1.11", I set "affects version" to "1.12.0" and "1.11.0".
> > >> *Fix Version/s:*
> > >> For closed/resolved tickets, this field lists the released Flink
> > versions
> > >> that contain a fix or feature for the first time.
> > >> For open tickets, it indicates that a fix / feature should be
> contained
> > in
> > >> the listed versions. Only blocker issues can block a release, all
> other
> > >> tickets which have "fix version/s" set at the time of a release and
> are
> > >> unresolved will be moved to the next version.
> > >> *Assignee:*
> > >> Person currently working on the ticket. Assigned after conclusion on
> > >> approach by a committer.
> > >> Often, fixes are obvious and committers self-assign w/o discussion.
> > >> *Resolve / Close:*
> > >> You can either Resolve or Close a ticket once it is done (fixed,
> > rejected,
> > >> invalid, ...).
> > >> As a rule, we Close tickets instead of Resolving them when they are
> > done.
> > >> Background: There are semantic differences for Resolve and Close
> > >> (implementor vs reporter considers it done), but I don't see how they
> > >> practically apply to the Flink project. Looking at the numbers, Flink
> > has
> > >> 11066 closed tickets, and 3372 resolved tickets (that's why I propose
> to
> > >> close instead of resolve)
> > >> *Labels:*
> > >> "test-stability" for all test instabilities
> > >> "starter" for tickets suitable for new contributors
> > >> *Release Note:*
> > >> Small notes that will be included into the release notes published
> with
> > the
> > >> release.
> > >> *All other fields are not used not used on a regular basis.*
> > >> Please +1 my proposal if you want it to be published in our Wiki like
> > that
> > >> or let me know if I got something wrong here.
> > >> Best,
> > >> Robert
> > >
> >
> >
>

Re: [DISCUSS] Releasing Stateful Functions 2.1.0 soon?

[Congxian Qiu]

+1 from my side to have smaller and more frequent feature releases for the
project in its early phases.

Best,
Congxian


Marta Paes Moreira  于2020年5月21日周四 下午12:49写道：

> +1 for more frequent releases with a shorter (but feedback-informed)
> feature set.
>
> Thanks, Gordon (and Igal)!
>
> Marta
>
> On Thu, 21 May 2020 at 03:44, Yu Li  wrote:
>
> > +1, it makes a lot of sense for stateful functions to evolve faster.
> >
> > Best Regards,
> > Yu
> >
> >
> > On Wed, 20 May 2020 at 23:36, Zhijiang  > .invalid>
> > wrote:
> >
> > > I also like this idea, considering stateful functions flexible enough
> to
> > > have a faster release cycle. +1 from my side.
> > >
> > > Best,
> > > Zhijiang
> > >
> > >
> > > --
> > > From:Seth Wiesman 
> > > Send Time:2020年5月20日(星期三) 21:45
> > > To:dev 
> > > Subject:Re: [DISCUSS] Releasing Stateful Functions 2.1.0 soon?
> > >
> > > +1 for a fast release cycle
> > >
> > > Seth
> > >
> > > On Wed, May 20, 2020 at 8:43 AM Robert Metzger 
> > > wrote:
> > >
> > > > I like the idea of releasing Statefun more frequently to have faster
> > > > feedback cycles!
> > > >
> > > > No objections for releasing 2.1.0 from my side.
> > > >
> > > > On Wed, May 20, 2020 at 2:22 PM Tzu-Li (Gordon) Tai <
> > tzuli...@apache.org
> > > >
> > > > wrote:
> > > >
> > > > > Hi devs,
> > > > >
> > > > > Since Stateful Functions 2.0 was released early April,
> > > > > we've been getting some good feedback from various channels,
> > > > > including the Flink mailing lists, JIRA issues, as well as Stack
> > > Overflow
> > > > > questions.
> > > > >
> > > > > Some of the discussions have actually translated into new features
> > > > > currently being implemented into the project, such as:
> > > > >
> > > > >- State TTL for the state primitives in Stateful Functions (for
> > both
> > > > >embedded/remote functions)
> > > > >- Transport for remote functions via UNIX domain sockets, which
> > > would
> > > > be
> > > > >useful when remote functions are co-located with Flink StateFun
> > > > workers
> > > > >(i.e. the "sidecar" deployment mode)
> > > > >
> > > > >
> > > > > Besides that, some critical shortcomings have already been
> addressed
> > > > since
> > > > > the last release:
> > > > >
> > > > >- After upgrading to Flink 1.10.1, failure recovery in Stateful
> > > > >Functions now works properly with the new scheduler.
> > > > >- Support for concurrent checkpoints
> > > > >
> > > > >
> > > > > With these ongoing threads, while it's only been just short of 2
> > months
> > > > > since the last release,
> > > > > we (Igal Shilman and I) have been thinking about aiming to already
> > > start
> > > > > the next feature release (2.1.0) soon.
> > > > > This is relatively shorter than the release cycle of what the
> > community
> > > > is
> > > > > used to in Flink (usually 3 months at least),
> > > > > but we think with the StateFun project in its early phases, having
> > > > smaller
> > > > > and more frequent feature releases could potentially help drive
> user
> > > > > adoption.
> > > > >
> > > > > So, what do you think about setting feature freeze for StateFun
> 2.1.0
> > > by
> > > > > next Wednesday (May 27th)?
> > > > > Of course, whether or not to actually have another feature release
> > > > already
> > > > > is still an open discussion - if you prefer a richer feature
> release
> > > with
> > > > > more features included besides the ones listed above, please do
> > > comment!
> > > > >
> > > > > Cheers,
> > > > > Gordon
> > > > >
> > > >
> > >
> > >
> >
>

Re: [VOTE] Guarantee that @PublicEvolving classes are API and binary compatible across bug fix releases (x.y.u and x.y.v)

[Congxian Qiu]

+1 (non-binding)
Best,
Congxian


Yangze Guo  于2020年5月16日周六 上午12:51写道：

> +1
>
> Best,
> Yangze Guo
>
> On Sat, May 16, 2020 at 12:26 AM Yuan Mei  wrote:
> >
> > +1
> >
> > On Sat, May 16, 2020 at 12:21 AM Ufuk Celebi  wrote:
> >
> > > +1
> > >
> > > – Ufuk
> > >
> > >
> > > On Fri, May 15, 2020 at 4:54 PM Zhijiang  > > .invalid>
> > > wrote:
> > >
> > > > Sounds good, +1.
> > > >
> > > > Best,
> > > > Zhijiang
> > > >
> > > >
> > > > --
> > > > From:Thomas Weise 
> > > > Send Time:2020年5月15日(星期五) 21:33
> > > > To:dev 
> > > > Subject:Re: [VOTE] Guarantee that @PublicEvolving classes are API and
> > > > binary compatible across bug fix releases (x.y.u and x.y.v)
> > > >
> > > > +1
> > > >
> > > >
> > > > On Fri, May 15, 2020 at 6:15 AM Till Rohrmann 
> > > > wrote:
> > > >
> > > > > Dear community,
> > > > >
> > > > > with reference to the dev ML thread about guaranteeing API and
> binary
> > > > > compatibility for @PublicEvolving classes across bug fix releases
> [1] I
> > > > > would like to start a vote about it.
> > > > >
> > > > > The proposal is that the Flink community starts to guarantee
> > > > > that @PublicEvolving classes will be API and binary compatible
> across
> > > bug
> > > > > fix releases of the same minor version. This means that a version
> x.y.u
> > > > is
> > > > > API and binary compatible to x.y.v with u <= v wrt all
> @PublicEvolving
> > > > > classes.
> > > > >
> > > > > The voting options are the following:
> > > > >
> > > > > * +1, Provide the above described guarantees
> > > > > * -1, Do not provide the above described guarantees (please provide
> > > > > specific comments)
> > > > >
> > > > > The vote will be open for at least 72 hours. It is adopted by
> majority
> > > > > approval with at least 3 PMC affirmative votes.
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > > >
> > >
> https://lists.apache.org/thread.html/rb0d0f887b291a490ed3773352c90ddf5f11e3d882dc501e3b8cf0ed0%40%3Cdev.flink.apache.org%3E
> > > > >
> > > > > Cheers,
> > > > > Till
> > > > >
> > > >
> > > >
> > >
>

Re: [DISCUSS] Stability guarantees for @PublicEvolving classes

[Congxian Qiu]

Sorry for the late jump in.

+1 to keep the compatibility of @PublicEvolving between minor
releases(x.y.a -> x.y.b), as a user I always think this as a bug-fix
release, break the compatibility between minor releases may give users a
surprise.

As the previous emails said, how and when will a @PublicEvolving
become @Public, and I'm not sure if we can have a technical solution to
keep such a rule. (In my opinion, check such things -- change
@PublicEvolving to @Public -- manually may not so easy)

Best
Congxian


Till Rohrmann  于2020年5月15日周五 下午9:18写道：

> The vote thread can be found here
>
> https://lists.apache.org/thread.html/rc58099fb0e31d0eac951a7bbf7f8bda8b7b65c9ed0c04622f5333745%40%3Cdev.flink.apache.org%3E
> .
>
> Cheers,
> Till
>
> On Fri, May 15, 2020 at 3:03 PM Till Rohrmann 
> wrote:
>
> > I completely agree that there are many other aspect of our guarantees and
> > processes around the @Public and @PublicEvolving classes which need to be
> > discussed and properly defined. For the sake of keeping this discussion
> > thread narrowly scoped, I would suggest to start a separate discussion
> > about the following points (not exhaustive):
> >
> > - What should be annotated with @Public and @PublicEvolving?
> > - Process for transforming @PublicEvolving into @Public; How to ensure
> > that @PublicEvolving will eventually be promoted to @Public?
> > - Process of retiring a @Public/@PublicEvolving API
> >
> > I will start a vote thread about the change I proposed here which is to
> > ensure API and binary compatibility for @PublicEvolving classes between
> > bugfix releases (x.y.z and x.y.u).
> >
> > Cheers,
> > Till
> >
> > On Fri, May 15, 2020 at 6:33 AM Zhu Zhu  wrote:
> >
> >> +1 for "API + binary compatibility for @PublicEvolving classes for all
> bug
> >> fix
> >> releases in a minor release (x.y.z is compatible to x.y.u)"
> >>
> >> This @PublicEnvolving would then be a hard limit to changes.
> >> So it's important to rethink the policy towards using it, as Stephan
> >> proposed.
> >>
> >> I think any Flink interfaces that are visible to users should be
> >> explicitly
> >> marked as @Public or @PublicEnvolving.
> >> Any other interfaces should not be marked as @Public/@PublicEnvolving.
> >> This would be essential for us to check whether we are breaking any user
> >> faced interfaces unexpectedly.
> >> The only exception would be the case that we had to expose a
> method/class
> >> due to implementation limitations, it should be explicitly marked it
> >> as @Internal.
> >>
> >> Thanks,
> >> Zhu Zhu
> >>
> >> Yun Tang  于2020年5月15日周五 上午11:41写道：
> >>
> >> > +1 for this idea, and I also like Xintong's suggestion to make it
> >> > explicitly when the @PublicEvolving API could upgrade to @Public API.
> >> > If we have the rule to upgrade API stable level but not define the
> clear
> >> > timeline, I'm afraid not everyone have the enthusiasm to upgrade this.
> >> >
> >> > The minor suggestion is that I think two major release (which is x.y.0
> >> as
> >> > Chesnay clarified) might be a bit quick. From the release history [1],
> >> > Flink bump major version every 3 ~ 6 months and two major release gap
> >> > could only be at least half a year.
> >> > I think half a year might be a bit too frequent for users to collect
> >> > enough feedbacks, and upgrading API stable level every 3 major
> versions
> >> > should be better.
> >> >
> >> > [1] https://flink.apache.org/downloads.html#flink
> >> >
> >> > Best
> >> > Yun Tang
> >> >
> >> >
> >> > 
> >> > From: Xintong Song 
> >> > Sent: Friday, May 15, 2020 11:04
> >> > To: dev 
> >> > Subject: Re: [DISCUSS] Stability guarantees for @PublicEvolving
> classes
> >> >
> >> > ### Documentation on API compatibility policies
> >> >
> >> > Do we have any formal documentation about the API compatibility
> >> policies?
> >> > The only things I found are:
> >> >
> >> >- In the release announcement (take 1.10.0 as an example) [1]:
> >> >"This version is API-compatible with previous 1.x releases for APIs
> >> >annotated with the @Public annotation."
> >> >- JavaDoc for Public [2] and PublicEvolving [3].
> >> >
> >> > I think we might have a formal documentation, clearly state our
> policies
> >> > for API compatibility.
> >> >
> >> >- What does the annotations mean
> >> >- In what circumstance would the APIs remain compatible / become
> >> >incompatible
> >> >- How do APIs retire (e.g., first deprecated then removed?)
> >> >
> >> > Maybe there is already such kind of documentation that I overlooked?
> If
> >> so,
> >> > we probably want to make it more explicit and easy-to-find.
> >> >
> >> > ### @Public vs. @PublicEvolving for new things
> >> >
> >> > I share Stephan's concern that, with @PublicEvolving used for every
> new
> >> > feature and rarely upgraded to @Public, we are practically making no
> >> > compatibility guarantee between minor versions (x.y.* / x.z.*). On the
> >> > other hand, I think in many 

Re: [DISCUSS] increase "state.backend.fs.memory-threshold" from 1K to 100K

[Congxian Qiu]

Hi,

Overall, I agree with increasing this value. but the default value set to
100K maybe something too large from my side.

I want to share some more information from my side.

The small files problem is indeed a problem many users may encounter in
production env. The states(Keyed state and Operator state) can become small
files in DFS, but increase the value of `state.backend.fs.memory-threshold`
may encounter the JM OOM problem as Yun said previously.
We've tried increase this value in our production env, but some connectors
which UnionState prevent us to do this, the memory consumed by these jobs
can be very large (in our case, thousands of parallelism, set
`state.backend.fs.memory-threshold` to 64K, will consume 10G+ memory for
JM), so in the end, we use the solution proposed in FLINK-11937[1] for both
keyed state and operator state.


[1] https://issues.apache.org/jira/browse/FLINK-11937
Best,
Congxian


Yun Tang  于2020年5月15日周五 下午9:09写道：

> Please correct me if I am wrong, "put the increased value into the default
> configuration" means
> we will update that in default flink-conf.yaml but still leave the default
> value of `state.backend.fs.memory-threshold`as previously?
> It seems I did not get the point why existing setups with existing configs
> will not be affected.
>
> The concern I raised is because one of our large-scale job with 1024
> parallelism source of union state meet the JM OOM problem when we increase
> this value.
> I think if we introduce memory control when serializing TDD asynchronously
> [1], we could be much more confident to increase this configuration as the
> memory footprint
> expands at that time by a lot of serialized TDDs.
>
>
> [1]
> https://github.com/apache/flink/blob/32bd0944d0519093c0a4d5d809c6636eb3a7fc31/flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/Execution.java#L752
>
> Best
> Yun Tang
>
> 
> From: Stephan Ewen 
> Sent: Friday, May 15, 2020 16:53
> To: dev 
> Cc: Till Rohrmann ; Piotr Nowojski <
> pi...@ververica.com>
> Subject: Re: [DISCUSS] increase "state.backend.fs.memory-threshold" from
> 1K to 100K
>
> I see, thanks for all the input.
>
> I agree with Yun Tang that the use of UnionState is problematic and can
> cause issues in conjunction with this.
> However, most of the large-scale users I know that also struggle with
> UnionState have also increased this threshold, because with this low
> threshold, they get an excess number of small files and overwhelm their
> HDFS / S3 / etc.
>
> An intermediate solution could be to put the increased value into the
> default configuration. That way, existing setups with existing configs will
> not be affected, but new users / installations will have a simper time.
>
> Best,
> Stephan
>
>
> On Thu, May 14, 2020 at 9:20 PM Yun Tang  wrote:
>
> > Tend to be not in favor of this proposal as union state is somewhat
> abused
> > in several popular source connectors (e.g. kafka), and increasing this
> > value could lead to JM OOM when sending tdd from JM to TMs with large
> > parallelism.
> >
> > After we collect union state and initialize the map list [1], we already
> > have union state ready to assign. At this time, the memory footprint has
> > not increase too much as the union state which shared across tasks have
> the
> > same reference of ByteStreamStateHandle. However, when we send tdd with
> the
> > taskRestore to TMs, akka will serialize those ByteStreamStateHandle
> within
> > tdd to increases the memory footprint. If the source have 1024
> > parallelisms, and any one of the sub-task would then have 1024*100KB size
> > state handles. The sum of total memory footprint cannot be ignored.
> >
> > If we plan to increase the default value of
> > state.backend.fs.memory-threshold, we should first resolve the above
> case.
> > In other words, this proposal could be a trade-off, which benefit perhaps
> > 99% users, but might bring harmful effects to 1% user with large-scale
> > flink jobs.
> >
> >
> > [1]
> >
> https://github.com/apache/flink/blob/c1ea6fcfd05c72a68739bda8bd16a2d1c15522c0/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/RoundRobinOperatorStateRepartitioner.java#L64-L87
> >
> > Best
> > Yun Tang
> >
> >
> > 
> > From: Yu Li 
> > Sent: Thursday, May 14, 2020 23:51
> > To: Till Rohrmann 
> > Cc: dev ; Piotr Nowojski 
> > Subject: Re: [DISCUSS] increase "state.backend.fs.memory-threshold" from
> > 1K to 100K
> >
> > TL;DR: I have some reservations but tend to be +1 for the proposal,
> > meanwhile suggest we have a more thorough solution in the long run.
> >
> > Please correct me if I'm wrong, but it seems the root cause of the issue
> is
> > too many small files generated.
> >
> > I have some concerns for the case of session cluster [1], as well as
> > possible issues for users at large scale, otherwise I think increasing
> > `state.backend.fs.memory-threshold` to 100K is a good choice, based on
> the
> > assu

Re: [DISCUSS] Exact feature freeze date

[Congxian Qiu]

+1 for Monday morning in Europe.
Best,
Congxian


Danny Chan  于2020年5月16日周六 上午10:40写道：

> +1 for Monday morning in Europe.
>
> Best,
> Danny Chan
> 在 2020年5月15日 +0800 PM10:51，Zhijiang  .invalid>，写道：
> > +1 for Monday morning in Europe.
> >
> > Best,
> > Zhijiang
> >
> >
> > --
> > From:Yun Tang 
> > Send Time:2020年5月15日(星期五) 21:58
> > To:dev 
> > Subject:Re: [DISCUSS] Exact feature freeze date
> >
> > +1 for Monday morning in Europe.
> >
> > Best
> > Yun Tang
> > 
> > From: Jingsong Li 
> > Sent: Friday, May 15, 2020 21:17
> > To: dev 
> > Subject: Re: [DISCUSS] Exact feature freeze date
> >
> > +1 for Monday morning.
> >
> > Best,
> > Jingsong Lee
> >
> > On Fri, May 15, 2020 at 8:45 PM Till Rohrmann 
> wrote:
> >
> > > +1 from my side extend the feature freeze until Monday morning.
> > >
> > > Cheers,
> > > Till
> > >
> > > On Fri, May 15, 2020 at 2:04 PM Robert Metzger 
> > > wrote:
> > >
> > > > I'm okay, but I would suggest to agree on a time of day. What about
> > > Monday
> > > > morning in Europe?
> > > >
> > > > On Fri, May 15, 2020 at 1:43 PM Piotr Nowojski 
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > Couple of contributors asked for extending cutting the release
> branch
> > > > > until Monday, what do you think about such extension?
> > > > >
> > > > > (+1 from my side)
> > > > >
> > > > > Piotrek
> > > > >
> > > > > > On 25 Apr 2020, at 21:24, Yu Li  wrote:
> > > > > >
> > > > > > +1 for extending the feature freeze to May 15th.
> > > > > >
> > > > > > Best Regards,
> > > > > > Yu
> > > > > >
> > > > > >
> > > > > > On Fri, 24 Apr 2020 at 14:43, Yuan Mei 
> > > wrote:
> > > > > >
> > > > > > > +1
> > > > > > >
> > > > > > > On Thu, Apr 23, 2020 at 4:10 PM Stephan Ewen  >
> > > > wrote:
> > > > > > >
> > > > > > > > Hi all!
> > > > > > > >
> > > > > > > > I want to bring up a discussion about when we want to do the
> > > feature
> > > > > > > freeze
> > > > > > > > for 1.11.
> > > > > > > >
> > > > > > > > When kicking off the release cycle, we tentatively set the
> date to
> > > > end
> > > > > of
> > > > > > > > April, which would be in one week.
> > > > > > > >
> > > > > > > > I can say from the features I am involved with (FLIP-27,
> FLIP-115,
> > > > > > > > reviewing some state backend improvements, etc.) that it
> would be
> > > > > helpful
> > > > > > > > to have two additional weeks.
> > > > > > > >
> > > > > > > > When looking at various other feature threads, my feeling is
> that
> > > > there
> > > > > > > are
> > > > > > > > more contributors and committers that could use a few more
> days.
> > > > > > > > The last two months were quite exceptional in and we did
> lose a bit
> > > > of
> > > > > > > > development speed here and there.
> > > > > > > >
> > > > > > > > How do you think about making *May 15th* the feature freeze?
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Stephan
> > > > > > > >
> > > > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> > Best, Jingsong Lee
> >
>

Re: [ANNOUNCE] Apache Flink 1.10.1 released

[Congxian Qiu]

Thanks a lot for the release and your great job, Yu!
Also thanks to everyone who made this release possible!

Best,
Congxian


Yu Li  于2020年5月14日周四 上午1:59写道：

> The Apache Flink community is very happy to announce the release of Apache
> Flink 1.10.1, which is the first bugfix release for the Apache Flink 1.10
> series.
>
> Apache Flink® is an open-source stream processing framework for
> distributed, high-performing, always-available, and accurate data streaming
> applications.
>
> The release is available for download at:
> https://flink.apache.org/downloads.html
>
> Please check out the release blog post for an overview of the improvements
> for this bugfix release:
> https://flink.apache.org/news/2020/05/12/release-1.10.1.html
>
> The full release notes are available in Jira:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12346891
>
> We would like to thank all contributors of the Apache Flink community who
> made this release possible!
>
> Regards,
> Yu
>

Re: [VOTE] Release 1.10.1, release candidate #3

[Congxian Qiu]

+1 (no-binding)

- checksum & gpg ok
- build from source OK
- all pom files points to the same version OK
- LICENSE OK
- run demos OK
Best,
Congxian


Dian Fu  于2020年5月10日周日 下午10:14写道：

> +1 (non-binding)
>
> - checked the dependency changes since 1.10.0
>   1) kafka was bumped from 0.10.2.1 to 0.10.2.2 for
> flink-connector-kafka-0.10 and it has been reflected in the notice file
>   2) amazon-kinesis-producer was bumped from 0.13.1 to 0.14.0 and it has
> been reflected in the notice file
>   3) the dependencies com.carrotsearch:hppc,
> com.github.spullara.mustache.java, org.elasticsearch:elasticsearch-geo and
> org.elasticsearch.plugin:lang-mustache-client was bundled in the jar of
> flink-sql-connector-elasticsearch7 and it has been reflected in the notice
> file
>   4) influxdb-java was bumped from 2.16 to 2.17 and it has been reflected
> in the notice file
> - verified the checksum and signature
> - checked that the PyFlink package could be pip installed
> - have left a few minor comments on the website PR
>
> Regards,
> Dian
>
> On Sat, May 9, 2020 at 12:02 PM Thomas Weise  wrote:
>
> > Thanks for the RC!
> >
> > +1 (binding)
> >
> > - repeated benchmark runs
> >
> >
> > On Fri, May 8, 2020 at 10:52 AM Robert Metzger 
> > wrote:
> >
> > > Thanks a lot for creating another RC!
> > >
> > > +1 (binding)
> > >
> > > - checked diff to last RC:
> > >
> > >
> >
> https://github.com/apache/flink/compare/release-1.10.1-rc2...release-1.10.1-rc3
> > >   - kinesis dependency change is properly documented
> > > - started Flink locally (on Java11 :) )
> > >- seems to be build off the specified commit
> > >- ran example
> > >- checked logs
> > > - staging repo looks ok
> > >
> > >
> > >
> > > On Thu, May 7, 2020 at 2:21 PM Yu Li  wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > Please review and vote on the release candidate #3 for version
> 1.10.1,
> > as
> > > > follows:
> > > > [ ] +1, Approve the release
> > > > [ ] -1, Do not approve the release (please provide specific comments)
> > > >
> > > >
> > > > The complete staging area is available for your review, which
> includes:
> > > > * JIRA release notes [1],
> > > > * the official Apache source release and binary convenience releases
> to
> > > be
> > > > deployed to dist.apache.org [2], which are signed with the key with
> > > > fingerprint D8D3D42E84C753CA5F170BDF93C07902771AB743 [3],
> > > > * all artifacts to be deployed to the Maven Central Repository [4],
> > > > * source code tag "release-1.10.1-rc3" [5],
> > > > * website pull request listing the new release and adding
> announcement
> > > blog
> > > > post [6].
> > > >
> > > > The vote will be open for at least 72 hours. It is adopted by
> majority
> > > > approval, with at least 3 PMC affirmative votes.
> > > >
> > > > Thanks,
> > > > Yu
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12346891
> > > > [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.10.1-rc3/
> > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > [4]
> > > >
> > https://repository.apache.org/content/repositories/orgapacheflink-1364/
> > > > [5]
> > > >
> > > >
> > >
> >
> https://github.com/apache/flink/commit/c5915cf87f96e1c7ebd84ad00f7eabade7e7fe37
> > > > [6] https://github.com/apache/flink-web/pull/330
> > > >
> > >
> >
>

Re: [DISCUSS] Align the behavior of internal return result of MapState#entries, keys, values and iterator.

[Congxian Qiu]

+1 to align the behavior to returning empty iterator. and I think this is
what's the code style[1] said.

Best,
Congxian


Yu Li  于2020年5月11日周一 上午11:42写道：

> +1 to align the behavior to returning empty iterator. There's no `Nullable`
> annotation on the `MapState` interfaces, either.
>
> Best Regards,
> Yu
>
>
> On Mon, 11 May 2020 at 11:21, Pengfei Li  wrote:
>
> > +1 to align the behavior
> >
> > Jark Wu  于2020年5月9日周六 下午9:36写道：
> >
> > > +1 to return emty iterator and align the implementations.
> > >
> > > Best,
> > > Jark
> > >
> > > On Sat, 9 May 2020 at 19:18, SteNicholas  wrote:
> > >
> > > > Hi Tang Yun,
> > > >  I agree with the point you mentioned that align these internal
> > > > behavior
> > > > to return empty iterator instead of null. In my opinion,
> > > > StateMapViewWithKeysNullable handle nullable map keys, and result of
> > > > internal map state should be empty map in the null behavior case.
> > > > Therefore,
> > > > as you mentioned, #iterator() should better return empty iterator.
> > > >
> > > > Thanks,
> > > > Nicholas Jiang
> > > >
> > > >
> > > >
> > > > --
> > > > Sent from:
> > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/
> > > >
> > >
> >
>

[jira] [Created] (FLINK-17571) A better way to show the files used in currently checkpoints

[Congxian Qiu(klion26) (Jira)]

Congxian Qiu(klion26) created FLINK-17571:
-

 Summary: A better way to show the files used in currently 
checkpoints
 Key: FLINK-17571
 URL: https://issues.apache.org/jira/browse/FLINK-17571
 Project: Flink
  Issue Type: New Feature
  Components: Runtime / Checkpointing
Reporter: Congxian Qiu(klion26)


Inspired by the [user 
mail]([http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Shared-Checkpoint-Cleanup-and-S3-Lifecycle-Policy-tt34965.html]).

Currently, there are three types of directory for a checkpoint, the files in 
TASKOWND and EXCLUSIVE directory can be deleted safely, but users can't delete 
the files in the SHARED directory safely(the files may be created a long time 
ago).

I think it's better to give users a better way to know which files are 
currently used(so the others are not used)

maybe a command-line command such as below is ok enough to support such a 
feature.

{{./bin/flink checkpoint list $checkpointDir  # list all the files used in 
checkpoint}}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Send issue and pull request notifications for flink-web and flink-shaded to iss...@flink.apache.org

[Congxian Qiu]

+1 for this

Best,
Congxian


Benchao Li  于2020年5月5日周二 下午5:22写道：

> belated +1
>
> Till Rohrmann  于2020年5月5日周二 下午2:24写道：
>
> > Thanks for all the feedback. I will open the PRs now which change the
> > notification settings of the above-mentioned repositories.
> >
> > Cheers,
> > Till
> >
> > On Tue, May 5, 2020 at 6:26 AM Hequn Cheng  wrote:
> >
> > > +1, thanks a lot for driving this.
> > >
> > > Best, Hequn
> > >
> > > On Tue, May 5, 2020 at 11:56 AM Dian Fu  wrote:
> > >
> > > > +1
> > > >
> > > > Regards,
> > > > Dian
> > > >
> > > > > 在 2020年5月5日，上午9:58，Yangze Guo  写道：
> > > > >
> > > > > +1
> > > > >
> > > > > Best,
> > > > > Yangze Guo
> > > > >
> > > > > On Tue, May 5, 2020 at 6:14 AM Thomas Weise 
> wrote:
> > > > >>
> > > > >> +1
> > > > >>
> > > > >>
> > > > >> On Mon, May 4, 2020 at 10:02 AM Marta Paes Moreira <
> > > ma...@ververica.com
> > > > >
> > > > >> wrote:
> > > > >>
> > > > >>> +1, this is quite annoying and distracting.
> > > > >>>
> > > > >>> Marta
> > > > >>>
> > > > >>> On Mon, May 4, 2020 at 6:27 PM Yu Li  wrote:
> > > > >>>
> > > >  +1
> > > > 
> > > >  Best Regards,
> > > >  Yu
> > > > 
> > > > 
> > > >  On Tue, 5 May 2020 at 00:21, Konstantin Knauf <
> kna...@apache.org>
> > > > wrote:
> > > > 
> > > > > Yes, please.
> > > > >
> > > > > On Mon, May 4, 2020 at 5:50 PM Dawid Wysakowicz <
> > > > >>> dwysakow...@apache.org>
> > > > > wrote:
> > > > >
> > > > >> +1
> > > > >>
> > > > >> Yes, please. I've also observed a lot of noise in the past
> days.
> > > > >>
> > > > >> Best,
> > > > >>
> > > > >> Dawid
> > > > >>
> > > > >> On 04/05/2020 17:48, Tzu-Li (Gordon) Tai wrote:
> > > > >>> +1
> > > > >>>
> > > > >>> All the recent new repos, flink-statefun /
> > flink-statefun-docker
> > > /
> > > > >>> flink-training etc. are also sending notifications to issues@
> .
> > > > >>>
> > > > >>> Gordon
> > > > >>>
> > > > >>>
> > > > >>> On Mon, May 4, 2020, 11:44 PM Till Rohrmann <
> > > trohrm...@apache.org>
> > > > >> wrote:
> > > > >>>
> > > >  Hi everyone,
> > > > 
> > > >  due to some changes on the ASF side, we are now seeing issue
> > and
> > > >  pull
> > > >  request notifications for the flink-web [1] and flink-shaded
> > [2]
> > > >  repo
> > > > > on
> > > >  dev@flink.apache.org. I think this is not ideal since the
> dev
> > > ML
> > > > >>> is
> > > > >> much
> > > >  more noisy now.
> > > > 
> > > >  I would propose to send these notifications to
> > > > > iss...@flink.apache.org
> > > > >> as
> > > >  we are currently doing it for the Flink main repo [3].
> > > > 
> > > >  What do you think?
> > > > 
> > > >  [1] https://github.com/apache/flink-web
> > > >  [2] https://github.com/apache/flink-shaded
> > > >  [3] https://gitbox.apache.org/schemes.cgi?flink
> > > > 
> > > >  Cheers,
> > > >  Till
> > > > 
> > > > >>
> > > > >>
> > > > >
> > > > > --
> > > > >
> > > > > Konstantin Knauf
> > > > >
> > > > > https://twitter.com/snntrable
> > > > >
> > > > > https://github.com/knaufk
> > > > >
> > > > 
> > > > >>>
> > > >
> > > >
> > >
> >
>
>
> --
>
> Benchao Li
> School of Electronics Engineering and Computer Science, Peking University
> Tel:+86-15650713730
> Email: libenc...@gmail.com; libenc...@pku.edu.cn
>

Re: Request of Checking PR [FLINK-17291]

[Congxian Qiu]

Hi, Roc

Thanks for the notice, I'll take a look at it later today.

Best,
Congxian


Roc Marshal  于2020年5月6日周三 上午11:19写道：

> Hi, @klion26 .
>
>
>
>
> I have completed the translation of the page in PR(
> https://github.com/apache/flink/pull/11979) and made corresponding
> improvements according to the suggestions of community members. If you have
> free time, would you please check it for me?
> Thank you for your attention.
>
>
> Best,
> Roc

Re: [VOTE] Release 1.10.1, release candidate #2

[Congxian Qiu]

+1 （no-binding)
- sha and gpg, ok
- all pom files point to same version, ok
- build from souce, ok
- LICENCE, ok
- run demo in standalone cluster, ok

Best,
Congxian


Aljoscha Krettek  于2020年5月5日周二 下午3:50写道：

> Unfortunately, I found this bug which prevents the TaskCancelerWatchdog
> [sic] from working: https://issues.apache.org/jira/browse/FLINK-17514. I
> think it's quite crucial that this failsafe mechanism works correctly.
> We should cancel the release and fix it.
>
> Best,
> Aljoscha
>
> On 05.05.20 05:55, Hequn Cheng wrote:
> > Thanks a lot for managing the release!
> >
> > +1 (binding)
> >
> > - Go through all new commits for 1.10.1 and spot no new license problems.
> > - Built from source archive successfully.
> > - Signatures and hash are correct.
> > - Run SocketWindowWordCount on the local cluster.
> > - Install Python package and run Python WordCount example.
> > - Reviewed website PR
> >
> > Best,
> > Hequn
> >
> > On Sun, May 3, 2020 at 9:10 PM Robert Metzger 
> wrote:
> >
> >> Thanks a lot for addressing the issues from the last release candidate
> and
> >> creating this one!
> >>
> >> +1 (binding)
> >>
> >> - Started Flink on YARN on Google Cloud DataProc by setting
> >> HADOOP_CLASSPATH
> >> - checked staging repo
> >>
> >>
> >>
> >> On Sat, May 2, 2020 at 6:57 PM Thomas Weise  wrote:
> >>
> >>> +1 (binding)
> >>>
> >>> Checked signatures and hashes.
> >>>
> >>> Run internal benchmark applications.
> >>>
> >>> I found a regression that was actually introduced with 1.10.0, hence
> not
> >> a
> >>> blocker for this release:
> >>>
> >>> https://github.com/apache/flink/pull/11975
> >>>
> >>> Thanks,
> >>> Thomas
> >>>
> >>>
> >>> On Fri, May 1, 2020 at 5:37 AM Yu Li  wrote:
> >>>
>  Hi everyone,
> 
>  Please review and vote on the release candidate #2 for version 1.10.1,
> >> as
>  follows:
>  [ ] +1, Approve the release
>  [ ] -1, Do not approve the release (please provide specific comments)
> 
> 
>  The complete staging area is available for your review, which
> includes:
>  * JIRA release notes [1],
>  * the official Apache source release and binary convenience releases
> to
> >>> be
>  deployed to dist.apache.org [2], which are signed with the key with
>  fingerprint D8D3D42E84C753CA5F170BDF93C07902771AB743 [3],
>  * all artifacts to be deployed to the Maven Central Repository [4],
>  * source code tag "release-1.10.1-rc2" [5],
>  * website pull request listing the new release and adding announcement
> >>> blog
>  post [6].
> 
>  The vote will be open for at least 72 hours. It is adopted by majority
>  approval, with at least 3 PMC affirmative votes.
> 
>  Thanks,
>  Yu
> 
>  [1]
> 
> 
> >>>
> >>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12346891
>  [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.10.1-rc2/
>  [3] https://dist.apache.org/repos/dist/release/flink/KEYS
>  [4]
> 
> >> https://repository.apache.org/content/repositories/orgapacheflink-1363/
>  [5]
> 
> 
> >>>
> >>
> https://github.com/apache/flink/commit/f92e8a9d60ef664acd66230da43c6f0a1cd87adc
>  [6] https://github.com/apache/flink-web/pull/330
> 
> >>>
> >>
> >
>
>

Re: Flink 1.9.2 why always checkpoint expired

[Congxian Qiu]

Hi

>From the picture and the previous eamil. you use RocksDBStateBackend, and
all the operators chained together, checkpoint timeout set to 2min.

Do you have keyed state in your job (do you have `keyby` in your job)?

I'll share some experience to find out the reason of checkpoint timeout
problem,
1. does the snapshot thread can get checkpoint lock if you run on version <
1.10
2. does the main thread consumes too much cpu, so that barrier can not be
handled.
3. could you please enable debug log and find out more information.


Best,
Congxian


qq <471237...@qq.com> 于2020年4月28日周二 上午9:20写道：

>
>
> 2020年4月27日 12:40，Jiayi Liao  写道：
>
> Hi,
>
> The picture in your attachment is too vague to see any detail. And beside
> the overview, could you take a look at the details of a specific expired
> checkpoint in history tab? From my experience, the expiration is usually
> because:
>
> 1. The data skew problem, which you can find out from checkpoints' details.
> 2. The processing is too slow (or the job is back-pressured) and the
> checkpoint timeout is set too short.
>
> Best Regards,
> Jiayi Liao
>
> On Mon, Apr 27, 2020 at 12:34 PM qq <471237...@qq.com> wrote:
>
>> Hi all,
>>
>> Why my flink checkpoint always expired, I used RocksDB checkpoint,
>> and I can’t get any useful messages for this. Could you help me ? Thanks
>> very much.
>>
>>
>>
>> <粘贴的图形-1.tiff>
>>
>
>

Re: [VOTE] Release 1.10.1, release candidate #1

[Congxian Qiu]

Thanks a lot for creating a release candidate for 1.10.1!

+1 from my side

checked
- md5/gpg, ok
- source does not contain any binaries, ok
- pom points to the same version 1.10.1, ok
- README file does not contain anything unexpected, ok
- maven clean package -DskipTests, ok
- maven clean verify, encounter a test timeout exception, but I think it
does not block the RC(have created an issue[1] to track it),
- run demos on a stand-alone cluster, ok

[1] https://issues.apache.org/jira/browse/FLINK-17458
Best,
Congxian


Robert Metzger  于2020年4月29日周三 下午2:54写道：

> Thanks a lot for creating a release candidate for 1.10.1!
>
> I'm not sure, but I think found a potential issue in the release while
> checking dependency changes on the ElasticSearch7 connector:
>
> https://github.com/apache/flink/commit/1827e4dddfbac75a533ff2aea2f3e690777a3e5e#diff-bd2211176ab6e7fa83ffeaa89481ff38
>
> In this change, "com.carrotsearch:hppc" has been added to the shaded jar (
>
> https://repository.apache.org/content/repositories/orgapacheflink-1362/org/apache/flink/flink-sql-connector-elasticsearch7_2.11/1.10.1/flink-sql-connector-elasticsearch7_2.11-1.10.1.jar
> ),
> without including proper mention of that dependency in "META-INF/NOTICE".
>
>
> My checking notes:
>
> - checked the diff for dependency changes:
> https://github.com/apache/flink/compare/release-1.10.0...release-1.10.1-rc1
> (w/o
> 
> release commit:
>
> https://github.com/apache/flink/compare/release-1.10.0...0e2b520ec60cc11dce210bc38e574a05fa5a7734
> )
>   - flink-connector-hive sets the derby version for test-scoped
> dependencies:
>
> https://github.com/apache/flink/compare/release-1.10.0...release-1.10.1-rc1#diff-f4dbf40e8457457eb01ae22b53baa3ec
>  - no NOTICE file found, but this module does not forward binaries.
>   - kafka 0.10 minor version upgrade:
>
> https://github.com/apache/flink/compare/release-1.10.0...release-1.10.1-rc1#diff-0287a3f3c37b454c583b6b56de1392e4
>   - NOTICE change found
>- ES7 changes shading:
>
> https://github.com/apache/flink/compare/release-1.10.0...release-1.10.1-rc1#diff-bd2211176ab6e7fa83ffeaa89481ff38
>  - problem found
>   - Influxdb version change
>
> https://github.com/apache/flink/compare/release-1.10.0...release-1.10.1-rc1#diff-0d2cce4875b2804ab89c3343a7de1ca6
>  - NOTICE change found
>
>
>
> On Fri, Apr 24, 2020 at 8:10 PM Yu Li  wrote:
>
> > Hi everyone,
> >
> > Please review and vote on the release candidate #1 for version 1.10.1, as
> > follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> >
> > The complete staging area is available for your review, which includes:
> > * JIRA release notes [1],
> > * the official Apache source release and binary convenience releases to
> be
> > deployed to dist.apache.org [2], which are signed with the key with
> > fingerprint D8D3D42E84C753CA5F170BDF93C07902771AB743 [3],
> > * all artifacts to be deployed to the Maven Central Repository [4],
> > * source code tag "release-1.10.1-rc1" [5],
> > * website pull request listing the new release and adding announcement
> blog
> > post [6].
> >
> > The vote will be open for at least 72 hours. It is adopted by majority
> > approval, with at least 3 PMC affirmative votes.
> >
> > Thanks,
> > Yu
> >
> > [1]
> >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12346891
> > <
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12346891
> > >
> > [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.10.1-rc1/
> > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > [4]
> > https://repository.apache.org/content/repositories/orgapacheflink-1362/
> > [5]
> >
> >
> https://github.com/apache/flink/commit/84b74cc0e21981bf6feceb74b48d7a9d3e215dc5
> > [6] https://github.com/apache/flink-web/pull/330
> >
>

[jira] [Created] (FLINK-17458) TaskExecutorSubmissionTest#testFailingScheduleOrUpdateConsumers

[Congxian Qiu(klion26) (Jira)]

Congxian Qiu(klion26) created FLINK-17458:
-

 Summary: 
TaskExecutorSubmissionTest#testFailingScheduleOrUpdateConsumers
 Key: FLINK-17458
 URL: https://issues.apache.org/jira/browse/FLINK-17458
 Project: Flink
  Issue Type: Test
  Components: Tests
Affects Versions: 1.10.0
Reporter: Congxian Qiu(klion26)


When verifying the RC of release-1.10.1, found that 
`TaskExecutorSubmissionTest#testFailingScheduleOrUpdateConsumers` will fail 
because of Timeout sometime. 

I run this test locally in IDEA, found the following exception(locally in only 
encounter 2 in 1000 times)
{code:java}
java.lang.InterruptedExceptionjava.lang.InterruptedException at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1039)
 at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
 at scala.concurrent.impl.Promise$DefaultPromise.tryAwait(Promise.scala:212) at 
scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:222) at 
scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:227) at 
scala.concurrent.Await$$anonfun$result$1.apply(package.scala:190) at 
scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
 at scala.concurrent.Await$.result(package.scala:190) at 
akka.event.LoggingBus$class.akka$event$LoggingBus$$addLogger(Logging.scala:182) 
at akka.event.LoggingBus$$anonfun$4$$anonfun$apply$4.apply(Logging.scala:117) 
at akka.event.LoggingBus$$anonfun$4$$anonfun$apply$4.apply(Logging.scala:116) 
at scala.util.Success$$anonfun$map$1.apply(Try.scala:237) at 
scala.util.Try$.apply(Try.scala:192) at scala.util.Success.map(Try.scala:237) 
at akka.event.LoggingBus$$anonfun$4.apply(Logging.scala:116) at 
akka.event.LoggingBus$$anonfun$4.apply(Logging.scala:113) at 
scala.collection.TraversableLike$WithFilter$$anonfun$map$2.apply(TraversableLike.scala:683)
 at scala.collection.Iterator$class.foreach(Iterator.scala:891) at 
scala.collection.AbstractIterator.foreach(Iterator.scala:1334) at 
scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at 
scala.collection.AbstractIterable.foreach(Iterable.scala:54) at 
scala.collection.TraversableLike$WithFilter.map(TraversableLike.scala:682) at 
akka.event.LoggingBus$class.startDefaultLoggers(Logging.scala:113) at 
akka.event.EventStream.startDefaultLoggers(EventStream.scala:22) at 
akka.actor.LocalActorRefProvider.init(ActorRefProvider.scala:662) at 
akka.actor.ActorSystemImpl.liftedTree2$1(ActorSystem.scala:874) at 
akka.actor.ActorSystemImpl._start$lzycompute(ActorSystem.scala:870) at 
akka.actor.ActorSystemImpl._start(ActorSystem.scala:870) at 
akka.actor.ActorSystemImpl.start(ActorSystem.scala:891) at 
akka.actor.RobustActorSystem$.internalApply(RobustActorSystem.scala:96) at 
akka.actor.RobustActorSystem$.apply(RobustActorSystem.scala:70) at 
akka.actor.RobustActorSystem$.create(RobustActorSystem.scala:55) at 
org.apache.flink.runtime.akka.AkkaUtils$.createActorSystem(AkkaUtils.scala:125) 
at 
org.apache.flink.runtime.akka.AkkaUtils$.createActorSystem(AkkaUtils.scala:113) 
at 
org.apache.flink.runtime.akka.AkkaUtils$.createLocalActorSystem(AkkaUtils.scala:68)
 at 
org.apache.flink.runtime.akka.AkkaUtils.createLocalActorSystem(AkkaUtils.scala) 
at 
org.apache.flink.runtime.rpc.TestingRpcService.(TestingRpcService.java:74)
 at 
org.apache.flink.runtime.rpc.TestingRpcService.(TestingRpcService.java:67)
 at 
org.apache.flink.runtime.taskexecutor.TaskSubmissionTestEnvironment$Builder.build(TaskSubmissionTestEnvironment.java:349)
 at 
org.apache.flink.runtime.taskexecutor.TaskExecutorSubmissionTest.testFailingScheduleOrUpdateConsumers(TaskExecutorSubmissionTest.java:544)
 at sun.reflect.GeneratedMethodAccessor34.invoke(Unknown Source) at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498) at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
 at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
 at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
 at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
 at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
 at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266) at 
java.lang.Thread.run(Thread.java:748)
org.junit.runners.model.TestTimedOutException: test timed out after 1 
milliseconds
 at sun.misc.Unsafe.park(Native Method) at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at

Re: Flink 1.9.2 why always checkpoint expired

[Congxian Qiu]

Hi
The image is not very clear.
For RocksDBStateBackend, do you enable incremental checkpoint?

Currently, checkpoint on TM side contains some steps:
1 barrier align
2 sync snapshot
3 async snapshot

For expired checkpoint, could you please check the tasks in the first
operator of the DAG to find out why it timed out.
- is there any backpressure? (affect barrier align)
- is the disk util/network util is high? (affect step 2&3)
- is the task thread is too busy? (this can lead to the barrier processed
sometime late)

you can enable the debug log to find out more info.

Best,
Congxian


qq <471237...@qq.com> 于2020年4月27日周一 下午12:34写道：

> Hi all,
>
> Why my flink checkpoint always expired, I used RocksDB checkpoint,
> and I can’t get any useful messages for this. Could you help me ? Thanks
> very much.
>
>
>
>

Re: [ANNOUNCE] New Apache Flink PMC Member - Hequn Chen

[Congxian Qiu]

Congratulations, Hequn!

Best,
Congxian


Yu Li  于2020年4月17日周五 下午9:36写道：

> Congratulations, Hequn!
>
> Best Regards,
> Yu
>
>
> On Fri, 17 Apr 2020 at 21:22, Kurt Young  wrote:
>
> > Congratulations Hequn!
> >
> > Best,
> > Kurt
> >
> >
> > On Fri, Apr 17, 2020 at 8:57 PM Till Rohrmann 
> > wrote:
> >
> > > Congratulations Hequn!
> > >
> > > Cheers,
> > > Till
> > >
> > > On Fri, Apr 17, 2020 at 2:49 PM Shuo Cheng  wrote:
> > >
> > > > Congratulations, Hequn
> > > >
> > > > Best,
> > > > Shuo
> > > >
> > > > On 4/17/20, hufeih...@mails.ucas.ac.cn 
> > > wrote:
> > > > > Congratulations , Hequn
> > > > >
> > > > > Best wish
> > > > >
> > > > >
> > > > > hufeih...@mails.ucas.ac.cn
> > > > > Congratulations, Hequn!
> > > > >
> > > > > Paul Lam  于2020年4月17日周五 下午3:02写道：
> > > > >
> > > > >> Congrats Hequn! Thanks a lot for your contribution to the
> community!
> > > > >>
> > > > >> Best,
> > > > >> Paul Lam
> > > > >>
> > > > >> Dian Fu  于2020年4月17日周五 下午2:58写道：
> > > > >>
> > > > >> > Congratulations, Hequn!
> > > > >> >
> > > > >> > > 在 2020年4月17日，下午2:36，Becket Qin  写道：
> > > > >> > >
> > > > >> > > Hi all,
> > > > >> > >
> > > > >> > > I am glad to announce that Hequn Chen has joined the Flink
> PMC.
> > > > >> > >
> > > > >> > > Hequn has contributed to Flink for years. He has worked on
> > several
> > > > >> > > components including Table / SQL，PyFlink and Flink ML
> Pipeline.
> > > > >> Besides,
> > > > >> > > Hequn is also very active in the community since the
> beginning.
> > > > >> > >
> > > > >> > > Congratulations, Hequn! Looking forward to your future
> > > > contributions.
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > >
> > > > >> > > Jiangjie (Becket) Qin
> > > > >> > > (On behalf of the Apache Flink PMC)
> > > > >> >
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > > > --
> > > > > Best Regards
> > > > >
> > > > > Jeff Zhang
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Releasing Flink 1.10.1

[Congxian Qiu]

Thanks a lot for driving this, Yu! Looking forward to the first RC of 1.10.1
FLINK-16576  has been merged into release-1.10 already.

Best,
Congxian


Dian Fu  于2020年4月16日周四 上午10:35写道：

> Thanks a lot for driving this, Yu! Looking forward for the first RC of
> 1.10.1.
>
> > 在 2020年4月16日，上午10:24，jincheng sun  写道：
> >
> > Looking forward the first RC of Flink 1.10.1 .
> > Good  job Yu!
> >
> > Best,
> > Jincheng
> >
> >
> >
> > Jark Wu  于2020年4月15日周三 下午6:28写道：
> >
> >> +1 to have a 1.10.1 RC soon. It has been a long time since 1.10.0 is
> >> released.
> >>
> >> Best,
> >> Jark
> >>
> >> On Wed, 15 Apr 2020 at 16:10, Till Rohrmann 
> wrote:
> >>
> >>> Great to see that will have the first RC for Flink 1.10.1 soon. Thanks
> a
> >>> lot for driving this effort Yu!
> >>>
> >>> Cheers,
> >>> Till
> >>>
> >>> On Sun, Apr 12, 2020 at 5:03 PM Yu Li  wrote:
> >>>
>  Thanks Weike and all others for the efforts!
> 
>  Here comes the latest status, we are in good shape and plan to produce
> >>> RC1
>  next week.
> 
>  * Blockers (1 left)
>   - [Closed] FLINK-16018 Improve error reporting when submitting batch
> >>> job
>  (instead of AskTimeoutException)
>   - [Closed] FLINK-16142 Memory Leak causes Metaspace OOM error on
> >>> repeated
>  job submission
>   - [Closed] FLINK-16170 SearchTemplateRequest ClassNotFoundException
> >>> when
>  use flink-sql-connector-elasticsearch7
>   - [Closed] FLINK-16262 Class loader problem with
>  FlinkKafkaProducer.Semantic.EXACTLY_ONCE and usrlib directory
>   - [Closed] FLINK-16406 Increase default value for JVM Metaspace to
>  minimise its OutOfMemoryError
>   - [Closed] FLINK-16454 Update the copyright year in NOTICE files
>   - [Closed] FLINK-16705 LocalExecutor tears down MiniCluster before
> >>> client
>  can retrieve JobResult
>   - [Closed] FLINK-16913 ReadableConfigToConfigurationAdapter#getEnum
>  throws UnsupportedOperationException
>   - [Closed] FLINK-16626 Exception encountered when cancelling a job in
>  yarn per-job mode
>   - [Fix for 1.10.1 is Done] FLINK-17093 Python UDF doesn't work when
> >> the
>  input column is of composite type
>   - [PR reviewed] FLINK-16576 State inconsistency on restore with
> >> memory
>  state backends
> 
>  * Critical (1 left)
>   - [Closed] FLINK-16047 Blink planner produces wrong aggregate results
>  with state clean up
>   - [Closed] FLINK-16070 Blink planner can not extract correct unique
> >> key
>  for UpsertStreamTableSink
>   - [Fix for 1.10.1 is Done] FLINK-16225 Metaspace Out Of Memory should
> >>> be
>  handled as Fatal Error in TaskManager
>   - [Closed] FLINK-14316 stuck in "Job leader ... lost leadership"
> >> error
>   - [May Postpone] FLINK-16408 Bind user code class loader to lifetime
> >>> of a
>  slot
> 
>  Please let me know if any concerns/comments. Thanks.
> 
>  Best Regards,
>  Yu
> 
> 
>  On Fri, 3 Apr 2020 at 21:35, DONG, Weike 
> >>> wrote:
> 
> > Hi Yu,
> >
> > Thanks for your updates. I am still working on the fix for
> >> FLINK-16626
>  and
> > it is expected to be completed by this Sunday after thorough testing.
> >
> > Sincerely,
> > Weike
> >
> > On Fri, Apr 3, 2020 at 8:43 PM Yu Li  wrote:
> >
> >> Updates for 1.10.1 watched issues (we are in good progress and
> >> almost
> >> there
> >> to produce the first RC, thanks all for the efforts):
> >>
> >> * Blockers (3 left)
> >>  - [Closed] FLINK-16018 Improve error reporting when submitting
> >> batch
>  job
> >> (instead of AskTimeoutException)
> >>  - [Closed] FLINK-16142 Memory Leak causes Metaspace OOM error on
> >> repeated
> >> job submission
> >>  - [Closed] FLINK-16170 SearchTemplateRequest
> >> ClassNotFoundException
>  when
> >> use flink-sql-connector-elasticsearch7
> >>  - [Closed] FLINK-16262 Class loader problem with
> >> FlinkKafkaProducer.Semantic.EXACTLY_ONCE and usrlib directory
> >>  - [Closed] FLINK-16406 Increase default value for JVM Metaspace to
> >> minimise its OutOfMemoryError
> >>  - [Closed] FLINK-16454 Update the copyright year in NOTICE files
> >>  - [PR reviewed] FLINK-16576 State inconsistency on restore with
> >>> memory
> >> state backends
> >>  - [Under Discussion] FLINK-16626 Exception encountered when
>  cancelling a
> >> job in yarn per-job mode
> >>  - [Closed] FLINK-16705 LocalExecutor tears down MiniCluster before
> >> client
> >> can retrieve JobResult
> >>  - [PR reviewed] FLINK-16913
>  ReadableConfigToConfigurationAdapter#getEnum
> >> throws UnsupportedOperationException
> >>
> >> * Critical (1 left)
> >>  - [Closed] FLINK-14316 stuck in "Job leader ... lost leadership"
> >>> error
> >>  - [Closed] FLINK-16047 Blink planner produces wrong aggregate
> >

Re: [DISCUSS] Releasing Flink 1.9.3

[Congxian Qiu]

+1 to create a new 1.9 bugfix release. and FLINK-16576[1] has merged into
master, filed a pr for release-1.9 already

[1] https://issues.apache.org/jira/browse/FLINK-16576

Best,
Congxian


Yu Li  于2020年4月15日周三 下午9:16写道：

> +1 to create a new 1.9 bug fix release. Thanks Dian for volunteering as our
> RM.
>
> Best Regards,
> Yu
>
>
> On Wed, 15 Apr 2020 at 16:25, Hequn Cheng  wrote:
>
> > +1 for the release and for Dian being the RM.
> > Thanks Jincheng for your continuous efforts on helping the releasing.
> >
> > Best,
> > Hequn
> >
> > On Wed, Apr 15, 2020 at 3:45 PM Till Rohrmann 
> > wrote:
> >
> > > Hi Dian,
> > >
> > > creating a new 1.9 bug fix release is a very good idea. +1 for creating
> > it
> > > soon. Also thanks for volunteering as our release manager.
> > >
> > > Cheers,
> > > Till
> > >
> > > On Fri, Apr 10, 2020 at 7:27 AM Dian Fu  wrote:
> > >
> > > > Hi Jincheng,
> > > >
> > > > Thanks a lot for offering help. It would be very helpful. Thanks
> again!
> > > >
> > > > Regards,
> > > > Dian
> > > >
> > > > > 在 2020年4月10日，下午12:46，jincheng sun  写道：
> > > > >
> > > > > Hi Dian,
> > > > >
> > > > > Thanks for bring up the discussion. I would like to give you a hand
> > at
> > > > the
> > > > > last stage when the RC is finished.  :)
> > > > >
> > > > > Best,
> > > > > Jincheng
> > > > >
> > > > >
> > > > >
> > > > > Dian Fu  于2020年4月10日周五 上午11:08写道：
> > > > >
> > > > >> Hi all,
> > > > >>
> > > > >> It has been more than two months since we released Flink 1.9.2.
> > There
> > > > are
> > > > >> already 36 improvements/bugfixes in the release-1.9 branch.
> > > Therefore, I
> > > > >> propose to create the next bugfix release 1.9.3 for Flink 1.9.
> > > > >>
> > > > >> Most notable fixes are:
> > > > >>
> > > > >> - [FLINK-15085] HistoryServer dashboard config json out of sync
> > > > >> - [FLINK-15575] Azure Filesystem Shades Wrong Package
> > "httpcomponents"
> > > > >> - [FLINK-15638] releasing/create_release_branch.sh does not set
> > > version
> > > > in
> > > > >> flink-python/pyflink/version.py
> > > > >> - [FLINK-16242] BinaryGeneric serialization error cause checkpoint
> > > > failure
> > > > >> - [FLINK-16573] Kinesis consumer does not properly shutdown
> > > > RecordFetcher
> > > > >> threads
> > > > >> - [FLINK-16047] Blink planner produces wrong aggregate results
> with
> > > > state
> > > > >> clean up
> > > > >> - [FLINK-16860] TableException: Failed to push filter into table
> > > source!
> > > > >> when upgrading flink to 1.9.2
> > > > >> - [FLINK-16916] The logic of NullableSerializer#copy is wrong
> > > > >> - [FLINK-16389] Bump Kafka 0.10 to 0.10.2.2
> > > > >> - [FLINK-15812] HistoryServer archiving is done in Dispatcher main
> > > > thread
> > > > >> - [FLINK-17062] Fix the conversion from Java row type to Python
> row
> > > type
> > > > >>
> > > > >> Furthermore, there is one blocker issue which should be merged
> > before
> > > > >> 1.9.3 release:
> > > > >>
> > > > >> - [FLINK-16576] State inconsistency on restore with memory state
> > > > backends
> > > > >> (reviewing)
> > > > >>
> > > > >> I would volunteer as the release manager and kick off the release
> > > > process.
> > > > >> What do you think?
> > > > >>
> > > > >> Please let me know if there are any concerns or any other blocker
> > > issues
> > > > >> need to be fixed in 1.9.3. Thanks.
> > > > >>
> > > > >> Appreciated if there is any PMC could help with the final steps of
> > the
> > > > >> release process.
> > > > >>
> > > > >> Regards,
> > > > >> Dian
> > > >
> > > >
> > >
> >
>

[jira] [Created] (FLINK-17043) Putting more information into accounting when failing a job in FailureManager

[Congxian Qiu(klion26) (Jira)]

Congxian Qiu(klion26) created FLINK-17043:
-

 Summary: Putting more information into accounting when failing a 
job in FailureManager
 Key: FLINK-17043
 URL: https://issues.apache.org/jira/browse/FLINK-17043
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Checkpointing
Reporter: Congxian Qiu(klion26)


Currently, we only fail the job when we received continues 
"CHECKPOINT_DECLINED" message, but ignored the 
"timeout"/"task_failure"/"task_checkpoint_failure"/"finalize_checkpoint_failure"
 and so on.

In my opinion, we should put some checkpoint failure reason above into account 
when failing a job (not only the "CHECKPOINT_DECLINED" reason"


This issue is inspired by a [user mail 
list|[http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Making-job-fail-on-Checkpoint-Expired-tt34051.html]],

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] Apache Flink Stateful Functions 2.0.0 released

[Congxian Qiu]

Thanks a lot for the release and your great job, Gordon!
Also thanks to everyone who made this release possible!

Best,
Congxian


Oytun Tez  于2020年4月8日周三 上午2:55写道：

> I should also add, I couldn't agree more with this sentence in the release
> article: "state access/updates and messaging need to be integrated."
>
> This is something we strictly enforce in our Flink case, where we do not
> refer to anything external for storage, use Flink as our DB.
>
>
>
>  --
>
> [image: MotaWord]
> Oytun Tez
> M O T A W O R D | CTO & Co-Founder
> oy...@motaword.com
>
>   
>
>
> On Tue, Apr 7, 2020 at 12:26 PM Oytun Tez  wrote:
>
>> Great news! Thank you all.
>>
>> On Tue, Apr 7, 2020 at 12:23 PM Marta Paes Moreira 
>> wrote:
>>
>>> Thank you for managing the release, Gordon — you did a tremendous job!
>>> And to everyone else who worked on pushing it through.
>>>
>>> Really excited about the new use cases that StateFun 2.0 unlocks for
>>> Flink users and beyond!
>>>
>>>
>>> Marta
>>>
>>> On Tue, Apr 7, 2020 at 4:47 PM Hequn Cheng  wrote:
>>>
 Thanks a lot for the release and your great job, Gordon!
 Also thanks to everyone who made this release possible!

 Best,
 Hequn

 On Tue, Apr 7, 2020 at 8:58 PM Tzu-Li (Gordon) Tai 
 wrote:

> The Apache Flink community is very happy to announce the release of
> Apache Flink Stateful Functions 2.0.0.
>
> Stateful Functions is an API that simplifies building distributed
> stateful applications.
> It's based on functions with persistent state that can interact
> dynamically with strong consistency guarantees.
>
> Please check out the release blog post for an overview of the release:
> https://flink.apache.org/news/2020/04/07/release-statefun-2.0.0.html
>
> The release is available for download at:
> https://flink.apache.org/downloads.html
>
> Maven artifacts for Stateful Functions can be found at:
> https://search.maven.org/search?q=g:org.apache.flink%20statefun
>
> Python SDK for Stateful Functions published to the PyPI index can be
> found at:
> https://pypi.org/project/apache-flink-statefun/
>
> Official Docker image for building Stateful Functions applications is
> currently being published to Docker Hub.
> Dockerfiles for this release can be found at:
> https://github.com/apache/flink-statefun-docker/tree/master/2.0.0
> Progress for creating the Docker Hub repository can be tracked at:
> https://github.com/docker-library/official-images/pull/7749
>
> The full release notes are available in Jira:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12346878
>
> We would like to thank all contributors of the Apache Flink community
> who made this release possible!
>
> Cheers,
> Gordon
>
 --
>>  --
>>
>> [image: MotaWord]
>> Oytun Tez
>> M O T A W O R D | CTO & Co-Founder
>> oy...@motaword.com
>>
>>   
>>
>

Re: [ANNOUNCE] New Flink committer: Seth Wiesman

[Congxian Qiu]

Congratulations, Seth!

Best,
Congxian


Kurt Young  于2020年4月7日周二 下午2:53写道：

> Congratulations, Seth!
>
> Best,
> Kurt
>
>
> On Tue, Apr 7, 2020 at 2:51 PM Jingsong Li  wrote:
>
> > Congratulations Seth!
> >
> > Best,
> > Jingsong Lee
> >
> > On Tue, Apr 7, 2020 at 2:46 PM Dawid Wysakowicz 
> > wrote:
> >
> > > Congratulations Seth. Happy to have you in the community!
> > >
> > > Best,
> > >
> > > Dawid
> > >
> > > On 07/04/2020 08:43, Dian Fu wrote:
> > > > Congratulations!
> > > >
> > > >> 在 2020年4月7日，下午2:35，Konstantin Knauf  写道：
> > > >>
> > > >> Congratulations, Seth! Well deserved :)
> > > >>
> > > >> On Tue, Apr 7, 2020 at 8:33 AM Tzu-Li (Gordon) Tai <
> > tzuli...@apache.org
> > > >
> > > >> wrote:
> > > >>
> > > >>> Hi everyone!
> > > >>>
> > > >>> On behalf of the PMC, I’m very happy to announce Seth Wiesman as a
> > new
> > > >>> Flink committer.
> > > >>>
> > > >>> Seth started contributing to the project in March 2017. You may
> know
> > > him
> > > >>> from several contributions in the past.
> > > >>> He had helped a lot with Flink documentation, and had contributed
> the
> > > State
> > > >>> Processor API.
> > > >>> Over the past few months, he has also helped tremendously in
> writing
> > > the
> > > >>> majority of the
> > > >>> Stateful Functions documentation.
> > > >>>
> > > >>> Please join me in congratulating Seth for becoming a Flink
> committer!
> > > >>>
> > > >>> Thanks,
> > > >>> Gordon
> > > >>>
> > > >>
> > > >> --
> > > >>
> > > >> Konstantin Knauf | Head of Product
> > > >>
> > > >> +49 160 91394525
> > > >>
> > > >>
> > > >> Follow us @VervericaData Ververica 
> > > >>
> > > >>
> > > >> --
> > > >>
> > > >> Join Flink Forward  - The Apache Flink
> > > >> Conference
> > > >>
> > > >> Stream Processing | Event Driven | Real Time
> > > >>
> > > >> --
> > > >>
> > > >> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
> > > >>
> > > >> --
> > > >> Ververica GmbH
> > > >> Registered at Amtsgericht Charlottenburg: HRB 158244 B
> > > >> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason,
> > Ji
> > > >> (Tony) Cheng
> > >
> > >
> >
> > --
> > Best, Jingsong Lee
> >
>

Re: [VOTE] Apache Flink Stateful Functions Release 2.0.0, release candidate #6

[Congxian Qiu]

+1 （non-binding)

- checked sums and signature: OK
- checked no binaries in source distribution: OK
- checked all POM files/README/Python SDK setup.py point to the same
version 2.0.0 OK
- execute `mvn clean install -Prun-e2e-tests`: OK
- checked quick start: ok
- run greeter example locally: ok
- run Ridesharing example locally: ok

Best,
Congxian


Dian Fu  于2020年4月7日周二 上午10:40写道：

> +1 (non-binding)
>
> - built from source with tests (mvn clean install)
> - verified the checksum and signature
> - checked the bundled licenses and notices
> - verified that the source distribution doesn't container unnecessary
> binaries
> - checked that the version pointed to the same version
> - flink-web PR looks well
> - built and checked the docs, looks well
>
> Regards,
> Dian
>
> > 在 2020年4月6日，下午10:06，Hequn Cheng  写道：
> >
> > Thanks a lot for the new RC!
> >
> > +1 (non-binding)
> >
> > - Signatures and hash are correct.
> > - The source distribution contains no binaries.
> > - The source distribution is building properly with `-Prun-e2e-tests`
> > (JDK8).
> > - All POM files / README / Python SDK setup.py point to the same version.
> > - Verify license and notice.
> >  - Source distribution. Everything looks good and the jquery has been
> > added.
> >  - Jar artifacts. No missing dependencies, no version errors.
> >  - Python source distribution (source and wheel). It contains the license
> > and notice file.
> > - Flink Harness works in IDE.
> >
> > Best,
> > Hequn
> >
> > On Mon, Apr 6, 2020 at 10:05 PM Seth Wiesman 
> wrote:
> >
> >> +1 (non-binding)
> >>
> >> legal / source
> >> - checked sources for binary files
> >> - checked license headers
> >>
> >> functional
> >> - built from source (mvn clean verify -Prun-e2e-tests)
> >> - built python sdk and ran tests
> >> - ran examples
> >> - deployed mixed python / java application on k8s with checkpointing.
> >> Failed TM's and watched it recover.
> >> - deployed application on Flink session cluster
> >> - created a savepoint using the bootstrap api and successfully used it
> to
> >> start an application.
> >>
> >> Seth
> >>
> >> On Mon, Apr 6, 2020 at 5:49 AM Igal Shilman  wrote:
> >>
> >>> +1 (non binding)
> >>>
> >>> legal / source:
> >>> - downloaded and verified the signature
> >>> - verified that pom and versions in the docs match
> >>> - no binary files in the distribution
> >>> - built and run e2e test with Java 8 and Java 11
> >>> - created a project from a maven archetype.
> >>>
> >>> functional:
> >>> - run all the examples
> >>> - deployed to Python greeter example to k8s
> >>> - enabled checkpointing, created an application with two Python
> >> functions,
> >>> that send both local and remote messages, restarted TMs randomly and
> >>> verified
> >>> the sequential output in the output kafka topic (exactly once test)
> >>> -  run the harness tests
> >>> -  run the ridesharing example in paraliisim 10 overnight
> >>> -  created a savepoint with the state bootstrapping tool and
> >>> successfully started a job from that.
> >>>
> >>> Kind regards,
> >>> Igal
> >>>
> >>> On Mon, Apr 6, 2020 at 10:23 AM Robert Metzger 
> >>> wrote:
> >>>
>  Thanks a lot for preparing another RC!
> 
>  +1 (binding)
> 
>  - source archive looks fine (no binaries, copied sources are properly
>  reported)
>  - staging repository looks fine (bundled binaries seem documented,
> >>> versions
>  are correct)
>  - *mvn clean install *(mvn clean verify fails, "install" is required)
> >> w/
>  e2e passes locally from source dir
> 
> 
> 
> 
>  On Mon, Apr 6, 2020 at 9:22 AM Tzu-Li (Gordon) Tai <
> >> tzuli...@apache.org>
>  wrote:
> 
> > FYI -
> > There are these open PRs to add blog posts and update the Flink
> >> website
>  for
> > the Stateful Functions 2.0 release:
> > * https://github.com/apache/flink-web/pull/322
> > * https://github.com/apache/flink-web/pull/321
> >
> > On Mon, Apr 6, 2020 at 2:53 PM Konstantin Knauf <
>  konstan...@ververica.com>
> > wrote:
> >
> >> +1 (non-binding)
> >>
> >> ** Functional **
> >> - Building from source dist with end-to-end tests enabled (mvn
> >> clean
> > verify
> >> -Prun-e2e-tests) passes (JDK 8)
> >> - Flink Harness works in IDE
> >> - Building Python SDK dist from source
> >>
> >> On Mon, Apr 6, 2020 at 5:12 AM Tzu-Li (Gordon) Tai <
>  tzuli...@apache.org>
> >> wrote:
> >>
> >>> +1 (binding)
> >>>
> >>> ** Legal **
> >>> - checksums and GPG files match corresponding release files
> >>> - Source distribution does not contain binaries, contents are
> >> sane
>  (no
> >>> .git* / .travis* / generated html content files)
> >>> - Bundled source LICENSEs and NOTICE looks good. Mentions bundled
> >>> font-awesome, jquery dependency in docs and copied sources from
> > fastutil
> >> (
> >>> http://fastutil.di.unimi.it/)
> >>> - Bundled L

[jira] [Created] (FLINK-17007) Add section "How to handle application parameters" in DataStream documentation

[Congxian Qiu(klion26) (Jira)]

Congxian Qiu(klion26) created FLINK-17007:
-

 Summary: Add section "How to handle application parameters" in 
DataStream documentation
 Key: FLINK-17007
 URL: https://issues.apache.org/jira/browse/FLINK-17007
 Project: Flink
  Issue Type: Improvement
  Components: Documentation
Reporter: Congxian Qiu(klion26)
 Fix For: 1.11.0


This issue wants to add a section “How to handle application parameters” in the 
DataStream page.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] New Committers and PMC member

[Congxian Qiu]

congratulations to all of you

Best,
Congxian


Xingbo Huang  于2020年4月1日周三 下午5:48写道：

> Congratulations to all of you.
>
> Best,
> Xingbo
>
> Rui Li  于2020年4月1日周三 下午5:45写道：
>
> > Congrats to you all!
> >
> > On Wed, Apr 1, 2020 at 5:32 PM Jeff Zhang  wrote:
> >
> > > Congratulations to all of you Well deserved.
> > >
> > > Tzu-Li (Gordon) Tai  于2020年4月1日周三 下午5:28写道：
> > >
> > > > Konstantin, Dawid, and Zhijiang, thanks a lot for your continuous
> > support
> > > > in the community :)
> > > > And congratulations! This is well deserved.
> > > >
> > > > Cheers,
> > > > Gordon
> > > >
> > > > On Wed, Apr 1, 2020 at 5:09 PM Hequn Cheng  wrote:
> > > >
> > > > > Congratulations to all of you!
> > > > >
> > > > > Best, Hequn
> > > > >
> > > > > On Wed, Apr 1, 2020 at 5:07 PM Jiayi Liao  >
> > > > wrote:
> > > > >
> > > > > > Congratulations to you all!
> > > > > >
> > > > > > On Wed, Apr 1, 2020 at 5:05 PM Arvid Heise 
> > > > wrote:
> > > > > >
> > > > > > > Congratulations!
> > > > > > >
> > > > > > > On Wed, Apr 1, 2020 at 11:03 AM Dian Fu  >
> > > > wrote:
> > > > > > >
> > > > > > > > Congratulations to you all.
> > > > > > > >
> > > > > > > >
> > > > > > > > > 在 2020年4月1日，下午5:00，Robert Metzger 
> 写道：
> > > > > > > > >
> > > > > > > > > Welcome & congratulations to all of you!
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Wed, Apr 1, 2020 at 10:58 AM Jingsong Li <
> > > > > jingsongl...@gmail.com>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Congratulations! Konstantin, Dawid, Zhijiang. Well
> deserved.
> > > > > > > > >>
> > > > > > > > >> Best,
> > > > > > > > >> Jingsong Lee
> > > > > > > > >>
> > > > > > > > >> On Wed, Apr 1, 2020 at 4:52 PM Stephan Ewen <
> > se...@apache.org
> > > >
> > > > > > wrote:
> > > > > > > > >>
> > > > > > > > >>> Hi all!
> > > > > > > > >>>
> > > > > > > > >>> Happy to announce that over the last few weeks, several
> > > people
> > > > in
> > > > > > the
> > > > > > > > >>> community joined in new roles:
> > > > > > > > >>>
> > > > > > > > >>>  - Konstantin Knauf joined as a committer. You may know
> > him,
> > > > for
> > > > > > > > >> example,
> > > > > > > > >>> from the weekly community updates.
> > > > > > > > >>>
> > > > > > > > >>>  - Dawid Wysakowicz joined the PMC. Dawid is one of the
> > main
> > > > > > > developers
> > > > > > > > >> on
> > > > > > > > >>> the Table API.
> > > > > > > > >>>
> > > > > > > > >>>  - Zhijiang Wang joined the PMC. Zhijiang is a veteran of
> > > > Flink's
> > > > > > > > >> network
> > > > > > > > >>> / data shuffle system.
> > > > > > > > >>>
> > > > > > > > >>> A warm welcome to your new roles in the Flink project!
> > > > > > > > >>>
> > > > > > > > >>> Best,
> > > > > > > > >>> Stephan
> > > > > > > > >>>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> --
> > > > > > > > >> Best, Jingsong Lee
> > > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > > Best Regards
> > >
> > > Jeff Zhang
> > >
> >
> >
> > --
> > Best regards!
> > Rui Li
> >
>

[jira] [Created] (FLINK-16916) The logic of NullableSerializer#copy is wrong

[Congxian Qiu(klion26) (Jira)]

Congxian Qiu(klion26) created FLINK-16916:
-

 Summary: The logic of NullableSerializer#copy is wrong
 Key: FLINK-16916
 URL: https://issues.apache.org/jira/browse/FLINK-16916
 Project: Flink
  Issue Type: Bug
  Components: API / Type Serialization System
Affects Versions: 1.10.0, 1.9.2, 1.8.3
Reporter: Congxian Qiu(klion26)
 Fix For: 1.11.0


When debugging the problem reported by FLINK-16724, Found that the logic of 
{{NullableSerializer#copy}} is wrong. currently, the logic is such as below:
{code:java}
public void copy(DataInputView source, DataOutputView target) throws 
IOException {
   boolean isNull = source.readBoolean();
   target.writeBoolean(isNull);
   if (isNull) {
  target.write(padding);
   }
   else {
  originalSerializer.copy(source, target);
   }
}

{code}
we forgot to skip {{paddings.length}} bytes when if the {{padding}}'s length is 
not 0.

We can correct the logic such as below 
{code:java}
public void copy(DataInputView source, DataOutputView target) throws 
IOException {
   boolean isNull = deserializeNull(source); // this will skip the padding 
values.
   target.writeBoolean(isNull);
   if (isNull) {
  target.write(padding);
   }
   else {
  originalSerializer.copy(source, target);
   }
}
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Contributor permission application

[Congxian Qiu]

Hi

Welcome to the Flink community

You no longer need contributor permissions. You can simply create a JIRA
ticket and ask to be assigned to it in order to start working. Please also
take a look at the Flink's contribution guidelines [1] for more information.

[1] https://flink.apache.org/contributing/how-to-contribute.html

Best,
Congxian


1101300123  于2020年3月2日周一 下午8:51写道：

> Hi Guys,
>
> I want to contribute to Apache Flink.
> Would you please give me the permission as a contributor?
> My JIRA ID is yutaochina.
>
> 2019-03-20
>
>
> 博彦科技股份有限公司
> --上海泓智信息科技有限公司
> 地址：上海市长宁区天山路8号402室
> 邮编：200336
> 手机：15501079221
> 邮箱：hdxg1101300...@163.com，yuta...@beyondsoft.com
> 发件人："hdxg1101300123"
> 发送时间：2019-02-28 15:27
> 主题：Contributor permission application
> 收件人："dev"
> 抄送：
>
> Hi Guys,
>
> I want to contribute to Apache Flink.
> Would you please give me the permission as a contributor?
> My JIRA ID is yutaochina.
>
> 2019-02-28
>
>
> 博彦科技股份有限公司
> --上海泓智信息科技有限公司
> 地址：上海市长宁区天山路8号402室
> 邮编：200336
> 手机：15501079221
> 邮箱：hdxg1101300...@163.com，yuta...@beyondsoft.com

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

[Congxian Qiu]

Congratulations Jingsong!

Best,
Congxian


jincheng sun  于2020年2月24日周一 下午1:38写道：

> Congratulations Jingsong!
>
> Best,
> Jincheng
>
>
> Zhu Zhu  于2020年2月24日周一 上午11:55写道：
>
>> Congratulations Jingsong!
>>
>> Thanks,
>> Zhu Zhu
>>
>> Fabian Hueske  于2020年2月22日周六 上午1:30写道：
>>
>>> Congrats Jingsong!
>>>
>>> Cheers, Fabian
>>>
>>> Am Fr., 21. Feb. 2020 um 17:49 Uhr schrieb Rong Rong <
>>> walter...@gmail.com>:
>>>
>>> > Congratulations Jingsong!!
>>> >
>>> > Cheers,
>>> > Rong
>>> >
>>> > On Fri, Feb 21, 2020 at 8:45 AM Bowen Li  wrote:
>>> >
>>> > > Congrats, Jingsong!
>>> > >
>>> > > On Fri, Feb 21, 2020 at 7:28 AM Till Rohrmann 
>>> > > wrote:
>>> > >
>>> > >> Congratulations Jingsong!
>>> > >>
>>> > >> Cheers,
>>> > >> Till
>>> > >>
>>> > >> On Fri, Feb 21, 2020 at 4:03 PM Yun Gao 
>>> wrote:
>>> > >>
>>> > >>>   Congratulations Jingsong!
>>> > >>>
>>> > >>>Best,
>>> > >>>Yun
>>> > >>>
>>> > >>> --
>>> > >>> From:Jingsong Li 
>>> > >>> Send Time:2020 Feb. 21 (Fri.) 21:42
>>> > >>> To:Hequn Cheng 
>>> > >>> Cc:Yang Wang ; Zhijiang <
>>> > >>> wangzhijiang...@aliyun.com>; Zhenghua Gao ;
>>> godfrey
>>> > >>> he ; dev ; user <
>>> > >>> u...@flink.apache.org>
>>> > >>> Subject:Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer
>>> > >>>
>>> > >>> Thanks everyone~
>>> > >>>
>>> > >>> It's my pleasure to be part of the community. I hope I can make a
>>> > better
>>> > >>> contribution in future.
>>> > >>>
>>> > >>> Best,
>>> > >>> Jingsong Lee
>>> > >>>
>>> > >>> On Fri, Feb 21, 2020 at 2:48 PM Hequn Cheng 
>>> wrote:
>>> > >>> Congratulations Jingsong! Well deserved.
>>> > >>>
>>> > >>> Best,
>>> > >>> Hequn
>>> > >>>
>>> > >>> On Fri, Feb 21, 2020 at 2:42 PM Yang Wang 
>>> > wrote:
>>> > >>> Congratulations！Jingsong. Well deserved.
>>> > >>>
>>> > >>>
>>> > >>> Best,
>>> > >>> Yang
>>> > >>>
>>> > >>> Zhijiang  于2020年2月21日周五 下午1:18写道：
>>> > >>> Congrats Jingsong! Welcome on board!
>>> > >>>
>>> > >>> Best,
>>> > >>> Zhijiang
>>> > >>>
>>> > >>> --
>>> > >>> From:Zhenghua Gao 
>>> > >>> Send Time:2020 Feb. 21 (Fri.) 12:49
>>> > >>> To:godfrey he 
>>> > >>> Cc:dev ; user 
>>> > >>> Subject:Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer
>>> > >>>
>>> > >>> Congrats Jingsong！
>>> > >>>
>>> > >>>
>>> > >>> *Best Regards,*
>>> > >>> *Zhenghua Gao*
>>> > >>>
>>> > >>>
>>> > >>> On Fri, Feb 21, 2020 at 11:59 AM godfrey he 
>>> > wrote:
>>> > >>> Congrats Jingsong! Well deserved.
>>> > >>>
>>> > >>> Best,
>>> > >>> godfrey
>>> > >>>
>>> > >>> Jeff Zhang  于2020年2月21日周五 上午11:49写道：
>>> > >>> Congratulations！Jingsong. You deserve it
>>> > >>>
>>> > >>> wenlong.lwl  于2020年2月21日周五 上午11:43写道：
>>> > >>> Congrats Jingsong!
>>> > >>>
>>> > >>> On Fri, 21 Feb 2020 at 11:41, Dian Fu 
>>> wrote:
>>> > >>>
>>> > >>> > Congrats Jingsong!
>>> > >>> >
>>> > >>> > > 在 2020年2月21日，上午11:39，Jark Wu  写道：
>>> > >>> > >
>>> > >>> > > Congratulations Jingsong! Well deserved.
>>> > >>> > >
>>> > >>> > > Best,
>>> > >>> > > Jark
>>> > >>> > >
>>> > >>> > > On Fri, 21 Feb 2020 at 11:32, zoudan  wrote:
>>> > >>> > >
>>> > >>> > >> Congratulations! Jingsong
>>> > >>> > >>
>>> > >>> > >>
>>> > >>> > >> Best,
>>> > >>> > >> Dan Zou
>>> > >>> > >>
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> > >>>
>>> > >>> --
>>> > >>> Best Regards
>>> > >>>
>>> > >>> Jeff Zhang
>>> > >>>
>>> > >>>
>>> > >>>
>>> > >>> --
>>> > >>> Best, Jingsong Lee
>>> > >>>
>>> > >>>
>>> > >>>
>>> >
>>>
>>

Re: [VOTE] FLIP-99: Make Max Exception Configurable

[Congxian Qiu]

+1 to this feature.  this is helpful when troubleshooting.

Best,
Congxian


Yadong Xie  于2020年2月20日周四 下午5:39写道：

> Hi all
>
> I want to start the vote for FLIP-99, which proposes to make the max
> exception configurable in web UI.
>
> To help everyone better understand the proposal, we spent some efforts on
> making an online POC
>
> previous web:
>
> http://101.132.122.69:8081/#/job/543e9dc0cb2cca4433116007f0931d1a/exceptions
> POC web:
>
> http://101.132.122.69:8081/web/#/job/543e9dc0cb2cca4433116007f0931d1a/exceptions
>
> The vote will last for at least 72 hours, following the consensus voting
> process.
>
> FLIP wiki:
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-99%3A+Make+Max+Exception+Configurable
>
> Discussion thread:
>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
>
> Thanks,
>
> Yadong
>

Re: [VOTE] FLIP-98: Better Back Pressure Detection

[Congxian Qiu]

+1 for this improvement, It's very helpful to know whether a task is
backpressured or not.

Best,
Congxian


Yadong Xie  于2020年2月20日周四 下午5:34写道：

> Hi all
>
> I want to start the vote for FLIP-98, which proposes to provide better
> backpressure detection experience in web UI.
>
> To help everyone better understand the proposal, we spent some efforts on
> making an online POC
>
> previous web:
> http://101.132.122.69:8081/#/job/add9d3ea1006af15b9f872987eb3befd/overview
> POC web:
>
> http://101.132.122.69:8081/web/#/job/add9d3ea1006af15b9f872987eb3befd/overview
>
> The vote will last for at least 72 hours, following the consensus voting
> process.
>
> FLIP wiki:
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-98%3A+Better+Back+Pressure+Detection
>
> Discussion thread:
>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
>
> Thanks,
>
> Yadong
>

Re: [Discuss] Update the pull request description template.

[Congxian Qiu]

JFYI, there is an issue[1] which I think is related to this thread
[1] https://issues.apache.org/jira/browse/FLINK-15977

Best,
Congxian


Chesnay Schepler  于2020年2月17日周一 下午9:08写道：

> I think it should just be removed since 99% of pull requests ignore it
> anyway.
>
> On 17/02/2020 13:31, Xintong Song wrote:
> > Hi all,
> >
> > It seems our PR description template is a bit outdated, and I would like
> to
> > propose updating it.
> >
> > I was working on a Kubernetes related PR, and realized that our PR
> > description does not mention the new Kubernetes integration questioning
> > about deployment related changes. Currently is is as follows:
> >
> >> Anything that affects deployment or recovery: JobManager (and its
> >> components), Checkpointing, Yarn/Mesos, ZooKeeper:
> >>
> > In addition to outdated contents, there might be other stuff that we want
> > to add to the template. For example, I would suggest add a question about
> > whether there are any memory allocation introduced by the PR, so we
> review
> > them carefully and avoid problems due to un-accounted memory allocations
> > like FLINK-15981. (To be fair, for FLINK-15981 the memory allocation was
> > introduced before we start to account for everything memory usage, but
> > noticing such memory allocations early should help us prevent similar
> > problems in the future).
> >
> > Therefore, I'd also like to collect ideas on how do you think the
> template
> > should be updated in this discussion thread.
> >
> > Looking forward to your feedback~!
> >
> > Thank you~
> >
> > Xintong Song
> >
>
>

Re: [VOTE] Release 1.10.0, release candidate #3

[Congxian Qiu]

+1 (non-binding) for rc3

- build source successfully (inlcude test)
- ran e2e test locally
- test pojo serializer upgrade manually by running flink job.

Best,
Congxian


Zhu Zhu  于2020年2月10日周一 下午12:28写道：

> My bad. The missing commit info is caused by building from the src code zip
> which does not contain the git info.
> So this is not a problem.
>
> +1 (binding) for rc3
> Here's what's were verified :
>  * built successfully from the source code
>  * run a sample streaming and a batch job with parallelism=1000 on yarn
> cluster, with the new scheduler and legacy scheduler, the job runs well
> (tuned some resource configs to enable the jobs to work well)
>  * killed TMs to trigger failures, the jobs can finally recover from the
> failures
>
> Thanks,
> Zhu Zhu
>
> Zhu Zhu  于2020年2月10日周一 上午12:31写道：
>
> > The commit info is shown as  on the web UI and in logs.
> > Not sure if it's a common issue or just happens to my build only.
> >
> > Thanks,
> > Zhu Zhu
> >
> > aihua li  于2020年2月9日周日 下午7:42写道：
> >
> >> Yes, but the results you see in the Performance Code Speed Center [3]
> >> skip FLIP-49.
> >>  The results of the default configurations are overwritten by the latest
> >> results.
> >>
> >> > 2020年2月9日 下午5:29，Yu Li  写道：
> >> >
> >> > Thanks for the efforts Aihua! These could definitely improve our RC
> >> test coverage!
> >> >
> >> > Just to confirm, that the stability tests were executed with the same
> >> test suite for Alibaba production usage, and the e2e performance one was
> >> executed with the test suite proposed in FLIP-83 [1] and FLINK-14917
> [2],
> >> and the result could also be observed from our performance code-speed
> >> center [3], right?
> >> >
> >> > Thanks.
> >> >
> >> > Best Regards,
> >> > Yu
> >> >
> >> > [1]
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-83%3A+Flink+End-to-end+Performance+Testing+Framework
> >> <
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-83%3A+Flink+End-to-end+Performance+Testing+Framework
> >> >
> >> > [2] https://issues.apache.org/jira/browse/FLINK-14917 <
> >> https://issues.apache.org/jira/browse/FLINK-14917>
> >> > [3] https://s.apache.org/nglhm 
> >> >
> >> > On Sun, 9 Feb 2020 at 11:20, aihua li  >> liaihua1...@gmail.com>> wrote:
> >> > +1 (non-binging)
> >> >
> >> > I ran stability tests and end-to-end performance tests in branch
> >> release-1.10.0-rc3,both of them passed.
> >> >
> >> > Stability test: It mainly checks The flink job can revover from
> >> various abnormal situations which concluding disk full,
> >> > network interruption, zk unable to connect, rpc message timeout, etc.
> >> > If job can't be recoverd it means test failed.
> >> > The test passed after running 5 hours.
> >> >
> >> > End-to-end performance test: It containes 32 test scenarios which
> >> designed in FLIP-83.
> >> > Test results: The performance regressions about 3% from 1.9.1 if uses
> >> default parameters;
> >> > The result:
> >> >
> >> >  if skips FLIP-49 (add parameters:taskmanager.memory.managed.fraction:
> >> 0,taskmanager.memory.flink.size: 1568m in flink-conf.yaml),
> >> >  the performance improves about 5% from 1.9.1. The result:
> >> >
> >> >
> >> > I confirm it with @Xintong Song <
> >> https://cwiki.apache.org/confluence/display/~xintongsong> that the
> >> result  makes sense.
> >> >
> >> >> 2020年2月8日 上午5:54，Gary Yao mailto:g...@apache.org>>
> >> 写道：
> >> >>
> >> >> Hi everyone,
> >> >> Please review and vote on the release candidate #3 for the version
> >> 1.10.0,
> >> >> as follows:
> >> >> [ ] +1, Approve the release
> >> >> [ ] -1, Do not approve the release (please provide specific comments)
> >> >>
> >> >>
> >> >> The complete staging area is available for your review, which
> includes:
> >> >> * JIRA release notes [1],
> >> >> * the official Apache source release and binary convenience releases
> >> to be
> >> >> deployed to dist.apache.org  [2], which are
> >> signed with the key with
> >> >> fingerprint BB137807CEFBE7DD2616556710B12A1F89C115E8 [3],
> >> >> * all artifacts to be deployed to the Maven Central Repository [4],
> >> >> * source code tag "release-1.10.0-rc3" [5],
> >> >> * website pull request listing the new release and adding
> announcement
> >> blog
> >> >> post [6][7].
> >> >>
> >> >> The vote will be open for at least 72 hours. It is adopted by
> majority
> >> >> approval, with at least 3 PMC affirmative votes.
> >> >>
> >> >> Thanks,
> >> >> Yu & Gary
> >> >>
> >> >> [1]
> >> >>
> >>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12345845
> >> <
> >>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12345845
> >> >
> >> >> [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.10.0-rc3/ <
> >> https://dist.apache.org/repos/dist/dev/flink/flink-1.10.0-rc3/>
> >> >> [3] https://dist.apache.org/repos/dist/release/flink/KEYS <
> >> https://dist.apache.org/repos/dist/release/flink/KEYS

Re: Question: Modifying Checkpoint Interval at runtime

[Congxian Qiu]

In current Flink, the checkpoint interval can not be modified after job
started, If you want to change the code to implement this feature, maybe
you can try to see the logic in CheckpointCoordinator, and the usage of
function checkMinPauseBetweenCheckpoints[1] and
scheduleTriggerWithDelay()[2]

[1]
https://github.com/apache/flink/blob/19689cb8ddb7cdc57a99d02217e67880c5e938a5/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java#L1463
[2]
https://github.com/apache/flink/blob/19689cb8ddb7cdc57a99d02217e67880c5e938a5/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java#L1485
Best,
Congxian


Morgan Geldenhuys  于2020年2月7日周五 下午7:12写道：

> I am working with the Flink at the moment and am interested in modifying
> the codebase so that the checkpoint interval can be changed at runtime,
> (e.g. if it was set to kickoff the distributed snapshot process every 10
> seconds, and then I wanted to change it to every 15 seconds instead
> without restarting). I was wondering, in your professional opinion, is
> this possible? Of course I am not expecting this to be a simple
> implementation, however, what considerations would I need to take into
> account and which parts of the codebase would this modification involve.
>

Re: [ANNOUNCE] Yu Li became a Flink committer

[Congxian Qiu]

Congratulations Yu, well deserved.
Best,
Congxian


Yun Gao  于2020年1月31日周五 下午4:15写道：

>Congratulations Yu!
>
>   Best,
>   Yun
>
>
> --
> From:felixzheng zheng 
> Send Time:2020 Jan. 31 (Fri.) 10:19
> To:dev 
> Subject:Re: [ANNOUNCE] Yu Li became a Flink committer
>
> Congrats!
>
> Benchao Li  于2020年1月31日周五 上午10:02写道：
>
> > Congratulations!!
> >
> > Biao Liu  于2020年1月29日周三 下午9:25写道：
> >
> > > Congrats!
> > >
> > > On Wed, Jan 29, 2020 at 10:37 aihua li  wrote:
> > >
> > > > Congratulations Yu LI, well deserved.
> > > >
> > > > > 2020年1月23日 下午4:59，Stephan Ewen  写道：
> > > > >
> > > > > Hi all!
> > > > >
> > > > > We are announcing that Yu Li has joined the rank of Flink
> committers.
> > > > >
> > > > > Yu joined already in late December, but the announcement got lost
> > > because
> > > > > of the Christmas and New Years season, so here is a belated proper
> > > > > announcement.
> > > > >
> > > > > Yu is one of the main contributors to the state backend components
> in
> > > the
> > > > > recent year, working on various improvements, for example the
> RocksDB
> > > > > memory management for 1.10.
> > > > > He has also been one of the release managers for the big 1.10
> > release.
> > > > >
> > > > > Congrats for joining us, Yu!
> > > > >
> > > > > Best,
> > > > > Stephan
> > > >
> > > > --
> > >
> > > Thanks,
> > > Biao /'bɪ.aʊ/
> > >
> >
> >
> > --
> >
> > Benchao Li
> > School of Electronics Engineering and Computer Science, Peking University
> > Tel:+86-15650713730
> > Email: libenc...@gmail.com; libenc...@pku.edu.cn
> >
>
>

[jira] [Created] (FLINK-15699) FirstValueAggFunctionWithoutOrderTest failed on Traivs

[Congxian Qiu(klion26) (Jira)]

Congxian Qiu(klion26) created FLINK-15699:
-

 Summary: FirstValueAggFunctionWithoutOrderTest failed on Traivs
 Key: FLINK-15699
 URL: https://issues.apache.org/jira/browse/FLINK-15699
 Project: Flink
  Issue Type: Bug
  Components: Tests
Affects Versions: 1.11.0
Reporter: Congxian Qiu(klion26)


09:48:53.473 [ERROR] COMPILATION ERROR : 
09:48:53.473 [INFO] 
-
09:48:53.473 [ERROR] 
/home/travis/build/flink-ci/flink/flink-table/flink-table-planner-blink/src/test/java/org/apache/flink/table/planner/functions/aggfunctions/FirstValueAggFunctionWithoutOrderTest.java:[78,37]
 incompatible types: inference variable T has incompatible bounds
 equality constraints: java.lang.Byte,T,T,T,T,T,java.lang.Boolean,T
 lower bounds: java.lang.Boolean
09:48:53.473 [INFO] 1 error

[https://travis-ci.com/flink-ci/flink/jobs/277466696]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-15685) org.apache.flink.tests.util.kafka.SQLClientKafkaITCase failed on traivs

[Congxian Qiu(klion26) (Jira)]

Congxian Qiu(klion26) created FLINK-15685:
-

 Summary: org.apache.flink.tests.util.kafka.SQLClientKafkaITCase 
failed on traivs
 Key: FLINK-15685
 URL: https://issues.apache.org/jira/browse/FLINK-15685
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Client, Tests
Reporter: Congxian Qiu(klion26)


 
08:50:02.717 [ERROR] Tests run: 3, Failures: 0, Errors: 3, Skipped: 0, Time 
elapsed: 76.395 s <<< FAILURE! - in 
org.apache.flink.tests.util.kafka.SQLClientKafkaITCase
 
1897208:50:02.722 [ERROR] testKafka[0: kafka-version:0.10 
kafka-sql-version:.*kafka-0.10.jar](org.apache.flink.tests.util.kafka.SQLClientKafkaITCase)
 Time elapsed: 23.806 s <<< ERROR!
 
18973java.io.IOException: 
 
18974Process execution failed due error. Error output:Exception in thread 
"main" org.apache.flink.table.client.SqlClientException: Unexpected exception. 
This is a bug. Please consider filing an issue.
 
18975 at org.apache.flink.table.client.SqlClient.main(SqlClient.java:190)
 
18976Caused by: org.apache.flink.table.client.gateway.SqlExecutionException: 
Could not create execution context.
 
18977 at 
org.apache.flink.table.client.gateway.local.ExecutionContext$Builder.build(ExecutionContext.java:759)
 
18978 at 
org.apache.flink.table.client.gateway.local.LocalExecutor.openSession(LocalExecutor.java:228)
 
18979 at org.apache.flink.table.client.SqlClient.start(SqlClient.java:98)
 
18980 at org.apache.flink.table.client.SqlClient.main(SqlClient.java:178)
 
18981Caused by: java.lang.NoClassDefFoundError: org/apache/avro/io/DatumReader
 
18982 at 
org.apache.flink.formats.avro.AvroRowFormatFactory.createDeserializationSchema(AvroRowFormatFactory.java:64)
 
18983 at 
org.apache.flink.streaming.connectors.kafka.KafkaTableSourceSinkFactoryBase.getDeserializationSchema(KafkaTableSourceSinkFactoryBase.java:285)
 
18984 at 
org.apache.flink.streaming.connectors.kafka.KafkaTableSourceSinkFactoryBase.createStreamTableSource(KafkaTableSourceSinkFactoryBase.java:163)
 
18985 at 
org.apache.flink.table.factories.StreamTableSourceFactory.createTableSource(StreamTableSourceFactory.java:49)
 
18986 at 
org.apache.flink.table.client.gateway.local.ExecutionContext.createTableSource(ExecutionContext.java:371)
 
18987 at 
org.apache.flink.table.client.gateway.local.ExecutionContext.lambda$initializeCatalogs$6(ExecutionContext.java:552)
 
18988 at java.util.LinkedHashMap.forEach(LinkedHashMap.java:684)
 
18989 at 
org.apache.flink.table.client.gateway.local.ExecutionContext.initializeCatalogs(ExecutionContext.java:550)
 
18990 at 
org.apache.flink.table.client.gateway.local.ExecutionContext.initializeTableEnvironment(ExecutionContext.java:487)
 
18991 at 
org.apache.flink.table.client.gateway.local.ExecutionContext.(ExecutionContext.java:159)
 
18992 at 
org.apache.flink.table.client.gateway.local.ExecutionContext.(ExecutionContext.java:118)
 
18993 at 
org.apache.flink.table.client.gateway.local.ExecutionContext$Builder.build(ExecutionContext.java:748)
 
18994 ... 3 more
 
18995Caused by: java.lang.ClassNotFoundException: org.apache.avro.io.DatumReader
 
18996 at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
 
18997 at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
 
18998 at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
 
18999 ... 15 more
 
19000
 
19001 at 
org.apache.flink.tests.util.kafka.SQLClientKafkaITCase.insertIntoAvroTable(SQLClientKafkaITCase.java:178)
 
19002 at 
org.apache.flink.tests.util.kafka.SQLClientKafkaITCase.testKafka(SQLClientKafkaITCase.java:151)
 
19003
 
1900408:50:02.734 [ERROR] testKafka[1: kafka-version:0.11 
kafka-sql-version:.*kafka-0.11.jar](org.apache.flink.tests.util.kafka.SQLClientKafkaITCase)
 Time elapsed: 25.227 s <<< ERROR!
 
19005java.io.IOException: 
 
19006Process execution failed due error. Error output:Exception in thread 
"main" org.apache.flink.table.client.SqlClientException: Unexpected exception. 
This is a bug. Please consider filing an issue.
 
19007 at org.apache.flink.table.client.SqlClient.main(SqlClient.java:190)
 
19008Caused by: org.apache.flink.table.client.gateway.SqlExecutionException: 
Could not create execution context.
 
19009 at 
org.apache.flink.table.client.gateway.local.ExecutionContext$Builder.build(ExecutionContext.java:759)
 
19010 at 
org.apache.flink.table.client.gateway.local.LocalExecutor.openSession(LocalExecutor.java:228)
 
19011 at org.apache.flink.table.client.SqlClient.start(SqlClient.java:98)
 
19012 at org.apache.flink.table.client.SqlClient.main(SqlClient.java:178)
 
19013Caused by: java.lang.NoClassDefFoundError: org/apache/avro/io/DatumReader
 
19014 at 
org.apache.flink.formats.avro.AvroRowFormatFactory.createDeserializationSchema(AvroRowFormatFactory.java:64)
 
19015 at 
org.apache.flink.streaming.connectors.kafka.KafkaTabl