Re: Event is taking a lot of time between the operators

Hi Sanket, I have a similar use case. how are you measuring the time for Async1` function to return the result and external api call On Wed, Sep 29, 2021 at 10:47 AM Sanket Agrawal wrote: > Hi @Piotr Nowojski , > > > > Thank you for replying back. Yes, first async is taking between 1300-1500 >

退订

| | xiaobo77 | | 退订 |

RE: Event is taking a lot of time between the operators

Hi @Piotr Nowojski, Thank you for replying back. Yes, first async is taking between 1300-1500 milliseconds but that is called on a CompletableFuture.supplyAsync and the Async Capacity is set to 1000. Async Code Structure: Inside asyncInvoke we are calling

Re: Flink run different jars

Hi Yangze, Thanks a lot for your reply. References are very helpful! Another quick question. Reference 1 can start a standalone cluster (session Mode). That cluster has a jobManager. I can submit job to run. How about taskManger? Do I need to manually start multiple taskManagers? Is there a

HA setup Flink version flink:1.13.2-scala_2.12-java11

I am following this link for setting up  HA configuration ZooKeeper HA Services | | | | ZooKeeper HA Services ZooKeeper HA Services # Flink’s ZooKeeper HA services use ZooKeeper for high availability services. Flink lever... | | | zookeeper version which I am using is 3.4.10    

回复： flink on native k8s 资源弹性扩容问题

可以看下使用的 k8s 是否有 VPA 的能力 -- Best, Matt Wang 在2021年09月26日 14:23，刘建刚 写道： 这个不支持，你可以通过外部的工具来做到。比如，检测cpu到了一定程度就自动的重启作业来扩容。 赵旭晨 于2021年9月23日周四 下午9:14写道： 目前生产上环境作业参数0.2（cpu），5G 平常增量跑的时候cpu占用率不到5%，上游数据全量初始化时经常会把CPU打满 想问下：flink能否做到弹性扩容？当pod的request cpu打满时自动增加cpu，当高峰期过后处于增量阶段时再收回部分pod资源？

Re: flink sql计算新增用户

Hi！ 听起来 event time temporal join 符合你的需求。详见 https://ci.apache.org/projects/flink/flink-docs-release-1.13/zh/docs/dev/table/sql/queries/joins/#event-time-temporal-join z 于2021年9月28日周二 下午8:27写道： > hi各位，我想计算每日新增用户的数量，用户的登录日志在kafka中，在当日之前登录过的用户即为老用户，目前我的做法是将用户登录信息用flink >

Re: Flink run different jars

Hi, Qihua IIUC, what you want might be a standalone cluster[1] or session cluster[2][3]. [1] https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/overview/ [2]

Re: Unable to connect to Mesos on mesos-appmaster.sh start

Another update: Looking more carefully in my appmaster log, I see the following 2021-09-29 01:15:39.680 [flink-akka.actor.default-dispatcher-3] INFO o.a.f.m.runtime.clusterframework.MesosResourceManagerDriver - Registering as new framework. 2021-09-29 01:15:39.680

Flink Demo with Redpanda

Hey all, I thought this might be of interest to the community. Basically, we took Jark Wu's demo ( https://flink.apache.org/2020/07/28/flink-sql-demo-building-e2e-streaming-application.html) and replaced Kafka with Redpanda: https://vectorized.io/blog/redpanda-flink-docker/ Regards, P

Flink run different jars

Hi, Is that possible to run a flink app without a job? What I am trying to do is I build multiple jars. And switch jar to run different jobs. I am not sure if flink supports this mode. I saw rest API can upload jar, cancel job and run a jar. Right now I can upload a jar to flink. But when I

Re: Unable to connect to Mesos on mesos-appmaster.sh start

Thanks, Matthias! There are lots of apps deployed to the Mesos cluster, the task manager itself is deployed to Mesos via Marathon. In the Mesos log I can see the Job manager agent starting, but no error messages related to it. As you say, TaskManagers don't even have the chance to get confused

Re: Unable to connect to Mesos on mesos-appmaster.sh start

Thanks, Roman! Looking at the log, seems that the TaskManager can resolve $HOSTNAME to its own hostname (07a6b681ee0f), as seen in these lines: 2021-09-27 22:02:41.067 [main] INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Djobmanager.rpc.address=*07a6b681ee0f* 2021-09-27

Re: Questions about keyed streams

Hi! I'm just getting back to this. Questions: 1. Across operators, does the same key group ids get mapped to the same task managers? E.g. if an item is in key group 1 of operator A and that runs on taskmanager-0, will key group 1 of operator B also run on taskmanager-0? 2. Are there any

Re: Unable to connect to Mesos on mesos-appmaster.sh start

Hi Javier, I don't see anything that's configured in the wrong way based on the jobmanager logs you've provided. Have you been able to deploy other applications to this Mesos cluster? Do the Mesos master logs reveal anything? The variable resolution on the TaskManager side is a valid concern

Re: Event is taking a lot of time between the operators

Hi, With Flink 1.8.0 I'm not sure how reliable the backpressure status is in the WebUI when it comes to the Async operators. If I remember correctly until around Flink 1.10 (+/- 2 version) backpressure monitoring was checking for thread dumps stuck in requesting Flink's network memory buffers. If

Re: How to add Flink a Flink connector to stateful functions

I just want to add that the StateFun documentation does cover using custom Flink connectors[1]. [1] https://nightlies.apache.org/flink/flink-statefun-docs-release-3.1/docs/modules/io/flink-connectors/#flink-connectors On Tue, Sep 28, 2021 at 2:52 AM Christian Krudewig (Corporate Development) <

Re: How to enable customize logging library based on SLF4J for Flink deployment in Kubernetes

Could you clarify whether this internal framework uses a custom slfj4/log4j2 version, or is it just using what Flink comes with? Did you only add the configmap and put a single jar into lib, or did you make other changes in Flink? Can you remove just the configmap, start the cluster, and

Event is taking a lot of time between the operators

Hi All, I am new to Flink. While developing a Flink application We observed that our message is taking around 10 seconds between the two Async operators. Below are the details. * Flink Flow: Kinesis Source -> Process -> Async1 -> Async2 -> Process -> Kinesis Sink * Environment:

Re: Unable to connect to Mesos on mesos-appmaster.sh start

Hi, No additional ports need to be open as far as I know. Probably, $HOSTNAME is substituted for something not resolvable on TMs? Please also make sure that the following gets executed before mesos-appmaster.sh: export HADOOP_CLASSPATH=$(hadoop classpath) export

flink sql????????????

hi??kafkaflink sqlmysql??Aid??tsjoin??

Re: rpc invocation exceeds the maximum akka framesize

Hi, There could be many reasons for exceeding akka framesize, for example 1. "inlined" state that is stored in checkpoint .metadata file (rather than "data" files - see [1]) 2. broadcast state as you mentioned (though only the metadata is sent unless the data fits the above limits) 3. too many

Re: 撤回流如何进行窗口分组聚合

这个在 1.14 已经支持了, 详见 FLINK-20487 On 9/28/21, Liu Join wrote: > 我将数据流进行去重后，无法进行窗口聚合操作，一直报错GroupWindowAggregate doesn't support consuming > update and delete changes which is produced by node Deduplicate >

Re:

Hi Violeta, in order to unsubscribe emails from Flink user mail list, send an email to user-unsubscr...@flink.apache.org. For more information, please have a look at https://flink.apache.org/community.html#mailing-lists. Best Ingo On Tue, Sep 28, 2021 at 10:50 AM Violeta Milanović <

[no subject]

unsubscribe

撤回流如何进行窗口分组聚合

我将数据流进行去重后，无法进行窗口聚合操作，一直报错GroupWindowAggregate doesn't support consuming update and delete changes which is produced by node Deduplicate

Re: rpc invocation exceeds the maximum akka framesize

Is the akka framesize a function of broadcast size? From: Deshpande, Omkar Sent: Monday, September 27, 2021 6:31 PM To: user@flink.apache.org Cc: Benenson, Mikhail ; Hwang, Nick ; Canchi, Srivathsan Subject: rpc invocation exceeds the maximum akka framesize

AW: How to add Flink a Flink connector to stateful functions

Hello Igal, Thanks for replying in detail and also so quickly. It’ll take me some time to try it out, thank you! Best, Christian -- Dr. Christian Krudewig Corporate Development – Data Analytics Deutsche Post DHL Headquarters Charles-de-Gaulle-Str. 20 53113 Bonn Germany

Re: How to add Flink a Flink connector to stateful functions

Hello Christian, I'm happy to hear that you are trying out StateFun and like the toolset! Currently StateFun supports "out of the box" only Kafka/Kinesis egresses, simply because so far folks didn't requested anything else. I can create a JIRA issue for that and we'll see how the community