how to setup a ha flink cluster on k8s?

I'm trying to setup a flink cluster on k8s for production use.But the setup here https://ci.apache.org/projects/flink/flink-docs-release-1.9/ops/deployment/kubernetes.htmlthis not ha , when job-manager down and rescheduled the metadata for running job is lost. I tried to use ha setup for

Keyed raw state - example

Hello everyone, I would like to know if anybody has a working example on how to declare a keyed raw state ( in my case a keyedprocessoperator) and how to use it in my UDF (keyedprocessfunction)? Basicaly we have a huge problem with a ValueState w Rocksdb, getting serialized for every element (

Re: How to unsubscribe the Apache projects and jira issues notification

https://apache.org/foundation/mailinglists.html#request-addresses-for-unsubscribing If you want to subscribe to l...@apache.org then you need to send a message to list-subscr...@apache.org To get off a list, send a message to list-unsubscr...@apache.org On Fri, Nov 15, 2019 at 2:40 AM P.

unsubscribe

-- With thanks, Katherin Pudikova

Re: How to unsubscribe the Apache projects and jira issues notification

Hi, Please check the first link on google "unsubscribe user@flink.apache.org” Piotrek > On 15 Nov 2019, at 11:40, P. Ramanjaneya Reddy wrote: > > Hi > > Following blogs want to unsubscribe kindly guide. > > I tried from google..still mails receiving > > Also should unubscribe.. > >

Re: slow checkpoints

Hi Yubraj, So the frequent job failure is the root reason, you need to fix it. Yes, when too many messages are squashed into the message system. If the messages can not be consumed normally, there would exist catchup consuming which will cause your streaming system more pressure than usual.

Broadcast checkpoint serialization fail

Hi all. In Flink 1.8 we have strange exception that causes job failing: 2019-11-14 15:52:52,071 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph- op4 (1/1) (797d4c2b85010dab6be5e1d06ff6493a) switched from RUNNING to FAILED. AsynchronousException{java.lang.Exception: Could not

Re: Initialization of broadcast state before processing main stream

Maxim, great thanks. We'll try buffering. С уважением, Василий Мельник On Thu, 14 Nov 2019 at 19:36, Maxim Parkachov wrote: > Hi Vasily, > > unfortunately, this is known issue with Flink, you could read discussion > under >

How to unsubscribe the Apache projects and jira issues notification

Hi Following blogs want to unsubscribe kindly guide. I tried from google..still mails receiving Also should unubscribe.. j...@apache.org user@flink.apache.org d...@flink.apache.org d...@beam.apache.org u...@beamho.apache.org Thanks

Flink 关于缓冲池被打爆的case求助

业务有个场景：数据量超大，现有的机器资源无法及时处理，需要抽样，但其中有部分数据是必须要处理的，所以数据源分两类： 1）满足维表白名单条件，则直接传到下游； 2）需要采样后传到下游； 过滤后的数据进行业务ETL清洗生成基础数据，基础数据中的白名单数据直接落盘，且基础数据经窗口统计后落盘。 即整个流程如下， 1. sample = source .connect(ruleMap1) //关联维表，若满足白名单则直接发送；反之进行采样后再发送 .process(new BroadcastProcessFunction<>(){ private int j =

通过 yarn-cluster 提交多个任务，Flink 页面只显示最后一个提交的任务

大家好，我想请教一个问题哈。 就是我通过 ./flink run -m yarn-cluster -yn 2 -yjm 1024 -ytm 1024 xxx.jar 的方式提交了两个不同的任务程序，在yarn界面里也能看到两个正在运行的 app ，但是点击 对应的 ApplicationMater 跳转到 Flink-Dashboard 页面之后，在 Running job 页面只看到一个运行的最后提交的程序呢？ Task Slot 也只用了 1 个，这是啥情况呢？ 也木有报错。yarn 页面显示两个任务都是 Running

Re: slow checkpoints

@Congxian , back pressure is due to job failure , some times job can fail and we need to catch up . Thanks Yubraj Singh [image: Mailtrack] Sender notified by Mailtrack

Re: slow checkpoints

Hi Currently, checkpoint may be faile in high back pressure scenario, because the barrier alignment can't be done in expected time, you should fix the back pressure problem first. There is a FLIP[1] that wants to fix this issue. [1]

state TTL 变更问题

有一个程序用到了 state， 设置 TTL 为3天。 运行一段时间后 cancel -s 停止，把过期时间设为 7 天，再从 state 文件恢复运行。 cancel -s 停止时生成的文件里面的所有 key，TTL 都会变成 7 天吗？ 还是依然是 3 天？ 谢谢， 王磊 wangl...@geekplus.com.cn

slow checkpoints

Hi all , I am facing one issue , when i have high back pressure my checkpoints start failing . please let me know how to deal with this kind of situations . Thanks Yubraj Singh . [image: Mailtrack] Sender

回复： yarn-session.sh 启动 报错

| | 李军 | | | | hold_li...@163.com | 签名由网易邮箱大师定制 在2019年11月15日 16:22，tison 写道： 图看不见，你可以外链图床或者 gist 贴 log Best, tison. 李军 于2019年11月15日周五 下午4:07写道： 这个报错实在不知道是什么意思，哪个地方连接不上

Re: yarn-session.sh 启动 报错

图看不见，你可以外链图床或者 gist 贴 log Best, tison. 李军 于2019年11月15日周五 下午4:07写道： > 这个报错实在不知道是什么意思，哪个地方连接不上 > >

yarn-session.sh 启动 报错

这个报错实在不知道是什么意思，哪个地方连接不上