Re: 如何在程序里面判断作业是否是重启了
RuntimeContext 有 getAttemptNumber() 接口，可以看出任务是第几次重跑了。 但是一般来说，我们都是通过外部系统监控 Flink 作业的 numRestarts metric 来判断作业是不是发生了 failover，进行报警。 Thanks, Zhu tison 于2021年2月5日周五 下午12:10写道： > 目前想到的是加一个调度器插件，在重启事件那边 hook 一下。 > > 正常的重启流程貌似没有其他 hook 点了，抄送一下这方面的专家(in cc)看看有没有其他意见。 > > Best, > tison. > > > 熊云昆 于2021年2月5日周五 上午11:30写道： > >> >> super.getRuntimeContext().getAttemptNumber()试试这个方法获取重启次数试试，如果没有重启过是0，反之每重启一次就会加1 >> >> >> | | >> 熊云昆 >> | >> | >> 邮箱：xiongyun...@163.com >> | >> >> 签名由 网易邮箱大师 定制 >> >> 在2021年02月04日 11:42，op 写道： >> 你好，我们下游不是所有需求都会去重，开销有点大。。。 >> >> >> >> >> --原始邮件-- >> 发件人: >> "user-zh" >> < >> zapj...@163.com; >> 发送时间:2021年2月4日(星期四) 中午11:31 >> 收件人:"user-zh"> >> 主题:Re:回复： 如何在程序里面判断作业是否是重启了 >> >> >> >> >> >> >> 下游数据做好幂等操作，就不怕重复操作了。。 >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> 在 2021-02-04 11:26:56，"op" <520075...@qq.com 写道： >> 重启可能会导致数据重发，想加个告警 >> >> >> >> >> --nbsp;原始邮件nbsp;-- >> 发件人: >> "user-zh" >> > 发送时间:nbsp;2021年2月4日(星期四) 中午11:11 >> 收件人:nbsp;"user-zh"> >> 主题:nbsp;Re: 如何在程序里面判断作业是否是重启了 >> >> >> >> 业务上的需求是什么？ >> >> Best, >> tison. >> >> >> op <520075...@qq.comgt; 于2021年2月4日周四 上午11:04写道： >> >> gt; 大家好： >> gt; amp;nbsp; >> gt; >> amp;nbsp;我在程序里通过RestartStrategies设置了重启策略，现在想在算子里面判断是否是触发了Restart，请问有哪些方法能实现呢？ > >
Re: [ANNOUNCE] Apache Flink 1.10.3 released
Thanks Xintong for being the release manager and everyone who helped with the release! Cheers, Zhu Dian Fu 于2021年1月29日周五 下午5:56写道： > Thanks Xintong for driving this release! > > Regards, > Dian > > 在 2021年1月29日，下午5:24，Till Rohrmann 写道： > > Thanks Xintong for being our release manager. Well done! > > Cheers, > Till > > On Fri, Jan 29, 2021 at 9:50 AM Yang Wang wrote: > >> Thanks Xintong for driving this release. >> >> Best, >> Yang >> >> Yu Li 于2021年1月29日周五 下午3:52写道： >> >>> Thanks Xintong for being our release manager and everyone else who made >>> the release possible! >>> >>> Best Regards, >>> Yu >>> >>> >>> On Fri, 29 Jan 2021 at 15:05, Xintong Song wrote: >>> The Apache Flink community is very happy to announce the release of Apache Flink 1.10.3, which is the third bugfix release for the Apache Flink 1.10 series. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming applications. The release is available for download at: https://flink.apache.org/downloads.html Please check out the release blog post for an overview of the improvements for this bugfix release: https://flink.apache.org/news/2021/01/29/release-1.10.3.html The full release notes are available in Jira: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12348668 We would like to thank all contributors of the Apache Flink community who made this release possible! Regards, Xintong Song >>> >
[ANNOUNCE] Apache Flink 1.11.2 released
The Apache Flink community is very happy to announce the release of Apache Flink 1.11.2, which is the second bugfix release for the Apache Flink 1.11 series. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming applications. The release is available for download at: https://flink.apache.org/downloads.html Please check out the release blog post for an overview of the improvements for this bugfix release: https://flink.apache.org/news/2020/09/17/release-1.11.2.html The full release notes are available in Jira: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12348575 We would like to thank all contributors of the Apache Flink community who made this release possible! Thanks, Zhu
Re: Re: [ANNOUNCE] New PMC member: Dian Fu
Congratulations Dian! Thanks, Zhu Zhijiang 于2020年8月27日周四 下午6:04写道： > Congrats, Dian! > > -- > From:Yun Gao > Send Time:2020年8月27日(星期四) 17:44 > To:dev ; Dian Fu ; user < > u...@flink.apache.org>; user-zh > Subject:Re: Re: [ANNOUNCE] New PMC member: Dian Fu > > Congratulations Dian ! > > Best > Yun > > > -- > Sender:Marta Paes Moreira > Date:2020/08/27 17:42:34 > Recipient:Yuan Mei > Cc:Xingbo Huang; jincheng sun >; dev; Dian Fu; user< > u...@flink.apache.org>; user-zh > Theme:Re: [ANNOUNCE] New PMC member: Dian Fu > > Congrats, Dian! > On Thu, Aug 27, 2020 at 11:39 AM Yuan Mei wrote: > > Congrats! > On Thu, Aug 27, 2020 at 5:38 PM Xingbo Huang wrote: > > Congratulations Dian! > > Best, > Xingbo > jincheng sun 于2020年8月27日周四 下午5:24写道： > > Hi all, > > > On behalf of the Flink PMC, I'm happy to announce that Dian Fu is now part of > the Apache Flink Project Management Committee (PMC). > > > Dian Fu has been very active on PyFlink component, working on various > important features, such as the Python UDF and Pandas integration, and keeps > checking and voting for our releases, and also has successfully produced two > releases(1.9.3&1.11.1) as RM, currently working as RM to push forward the > release of Flink 1.12. > > Please join me in congratulating Dian Fu for becoming a Flink PMC Member! > > Best, > Jincheng(on behalf of the Flink PMC) > > >
[ANNOUNCE] Apache Flink 1.10.2 released
The Apache Flink community is very happy to announce the release of Apache Flink 1.10.2, which is the first bugfix release for the Apache Flink 1.10 series. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming applications. The release is available for download at: https://flink.apache.org/downloads.html Please check out the release blog post for an overview of the improvements for this bugfix release: https://flink.apache.org/news/2020/08/25/release-1.10.2.html The full release notes are available in Jira: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12347791 We would like to thank all contributors of the Apache Flink community who made this release possible! Thanks, Zhu
Re: [ANNOUNCE] Apache Flink 1.10.1 released
Thanks Yu for being the release manager. Thanks everyone who made this release possible! Thanks, Zhu Zhu Benchao Li 于2020年5月15日周五 下午7:51写道： > Thanks Yu for the great work, and everyone else who made this possible. > > Dian Fu 于2020年5月15日周五 下午6:55写道： > >> Thanks Yu for managing this release and everyone else who made this >> release possible. Good work! >> >> Regards, >> Dian >> >> 在 2020年5月15日，下午6:26，Till Rohrmann 写道： >> >> Thanks Yu for being our release manager and everyone else who made the >> release possible! >> >> Cheers, >> Till >> >> On Fri, May 15, 2020 at 9:15 AM Congxian Qiu >> wrote: >> >>> Thanks a lot for the release and your great job, Yu! >>> Also thanks to everyone who made this release possible! >>> >>> Best, >>> Congxian >>> >>> >>> Yu Li 于2020年5月14日周四 上午1:59写道： >>> >>>> The Apache Flink community is very happy to announce the release of >>>> Apache Flink 1.10.1, which is the first bugfix release for the Apache Flink >>>> 1.10 series. >>>> >>>> Apache Flink® is an open-source stream processing framework for >>>> distributed, high-performing, always-available, and accurate data streaming >>>> applications. >>>> >>>> The release is available for download at: >>>> https://flink.apache.org/downloads.html >>>> >>>> Please check out the release blog post for an overview of the >>>> improvements for this bugfix release: >>>> https://flink.apache.org/news/2020/05/12/release-1.10.1.html >>>> >>>> The full release notes are available in Jira: >>>> >>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12346891 >>>> >>>> We would like to thank all contributors of the Apache Flink community >>>> who made this release possible! >>>> >>>> Regards, >>>> Yu >>>> >>> >> > > -- > > Benchao Li > School of Electronics Engineering and Computer Science, Peking University > Tel:+86-15650713730 > Email: libenc...@gmail.com; libenc...@pku.edu.cn > >
Re: How to test flink job recover from checkpoint
Hi Eleanore, You can change your application tasks to throw exceptions in a certain frequency. Alternatively, if the application has external dependencies (e.g. source), you can trigger failures manually by manipulating the status of the external service (e.g. shutdown the source service, or break the network connection between the Flink app and the source service). Thanks, Zhu Zhu Eleanore Jin 于2020年3月5日周四 上午8:40写道： > Hi, > > I have a flink application and checkpoint is enabled, I am running locally > using miniCluster. > > I just wonder if there is a way to simulate the failure, and verify that > flink job restarts from checkpoint? > > Thanks a lot! > Eleanore >
Re: [ANNOUNCE] Dian Fu becomes a Flink committer
Congratulations Dian. Thanks, Zhu Zhu hailongwang <18868816...@163.com> 于2020年1月17日周五 上午10:01写道： > > Congratulations Dian ! > > Best, > Hailong Wang > > > > > 在 2020-01-16 21:15:34，"Congxian Qiu" 写道： > > Congratulations Dian Fu > > Best, > Congxian > > > Jark Wu 于2020年1月16日周四 下午7:44写道： > >> Congratulations Dian and welcome on board! >> >> Best, >> Jark >> >> On Thu, 16 Jan 2020 at 19:32, Jingsong Li wrote: >> >> > Congratulations Dian Fu. Well deserved! >> > >> > Best, >> > Jingsong Lee >> > >> > On Thu, Jan 16, 2020 at 6:26 PM jincheng sun >> > wrote: >> > >> >> Congrats Dian Fu and welcome on board! >> >> >> >> Best, >> >> Jincheng >> >> >> >> Shuo Cheng 于2020年1月16日周四 下午6:22写道： >> >> >> >>> Congratulations! Dian Fu >> >>> >> >>> > Xingbo Wei Zhong 于2020年1月16日周四 下午6:13写道： jincheng sun >> >>> 于2020年1月16日周四 下午5:58写道： >> >>> >> >> >> > >> > -- >> > Best, Jingsong Lee >> > >> > > > > >
Re: How long Flink state default TTL，if I don't config the state ttl config?
Yes. State TTL is by default disabled. Thanks, Zhu Zhu LakeShen 于2020年1月6日周一 上午10:09写道： > I saw the flink source code, I find the flink state ttl default is > never expire,is it right? > > LakeShen 于2020年1月6日周一 上午9:58写道： > >> Hi community，I have a question about flink state ttl.If I don't config >> the flink state ttl config, >> How long the flink state retain？Is it forever retain in hdfs? >> Thanks your replay. >> >
