Re: About cluster.evenly-spread-out-slots

2023-07-20 Thread Shammon FY
Hi Kamal, In general, tasks are scheduled in topological order, which means the source tasks will be scheduled first and they will be deployed on different TM if there are enough TMs. However, as Flink does not guarantee this, in some special situations such as failover, multiple source tasks may

RE: About cluster.evenly-spread-out-slots

2023-07-20 Thread Kamal Mittal via user
Hello Shammon, Thanks for quick reply. Is there any way to limit only one task executes on each task manager for an operator like source? If parallelism is set equal to no. of task managers for an operator then it can work and keep cluster.evenly-spread-out-slots=true? Actually there is TCP

Re: About cluster.evenly-spread-out-slots

2023-07-20 Thread Shammon FY
Hi Kamal, Even if `cluster.evenly-spread-out-slots` is set to true, Flink will not guarantee that same operator multiple tasks are never executed/scheduled on the same task manager, it just means Flink will use `LeastUtilizationSlotMatchingStrategy` to find the matching slot for the task. As we

Re: Kafka coordinator not available

2023-07-20 Thread Mason Chen
Hi Lars, You are likely seeing this Kafka client bug: https://issues.apache.org/jira/browse/KAFKA-13840. The latest versions of Flink have updated Kafka clients dependency to include this fix. Best, Mason On Thu, Jul 20, 2023 at 9:21 AM Lars Skjærven wrote: > Hello, > I experienced >

Kafka coordinator not available

2023-07-20 Thread Lars Skjærven
Hello, I experienced CoordinatorNotAvailableException in my flink jobs after our kafka supplier (aiven) did a maintenance update of the cluster. This update is performed by starting up new kafka nodes, copying over data, and switching over internally. The flink jobs runs as expected, with the

flink异步IO超时时,如何释放连接池资源

2023-07-20 Thread 李一飞
代码如下: package com.didichuxing.iov.func import com.didichuxing.iov.util.JedisTool import org.apache.flink.configuration.Configuration import org.apache.flink.streaming.api.scala.async.{ResultFuture, RichAsyncFunction} import org.slf4j.LoggerFactory import redis.clients.jedis.Jedis import

Re: Logs of Kafka payload in Table Api connector of Apache Flink

2023-07-20 Thread Hang Ruan
Hi. elaloya. If you want to log some information about the kafka records, you can add some logs in KafkaRecordEmitter. If you want to know the information about the deserialized value, you should add logs in the avro format. Best, Hang elakiya udhayanan 于2023年7月19日周三 19:44写道: > Hi Team, > > I

Re: ASF jira account

2023-07-20 Thread jinzhuguang
我也遇到类似的问题,我是链接失效了,最后没办法再注册了 > 2023年7月20日 14:54,李天龙 写道: > > 您好! > 我想注册一个flink jira的账号,但由于提出的里有不充分给拒掉了,想再次申请,却提示邮箱已申请过,还未处理: > > > There is already a pending Jira account request associated with this email > address. Please wait for it to be processed > > > 请问怎么解决这个问题,并且成功申请一个账号 > > > >

About cluster.evenly-spread-out-slots

2023-07-20 Thread Kamal Mittal via user
Hello, If property "cluster.evenly-spread-out-slots" is set to TRUE then Flink guarantees that same operator multiple tasks are never executed/scheduled on same task manager? Definitely this will depend upon parallelism value used for an operator and no. of task slots available. Like in below

Re: Elastic Block Store as checkpoint storage

2023-07-20 Thread David Morávek
Using EBS as checkpoint storage doesn't work in a distributed environment if you need to move the state between TMs (e.g., for rescaling and non-local recovery). You'd need something along the lines of RW multi-attach and set up the volumes in a smart way; it won't be easy to set up; I'm not aware

ASF jira account

2023-07-20 Thread 李天龙
您好! 我想注册一个flink jira的账号,但由于提出的里有不充分给拒掉了,想再次申请,却提示邮箱已申请过,还未处理: There is already a pending Jira account request associated with this email address. Please wait for it to be processed 请问怎么解决这个问题,并且成功申请一个账号 -- 发自我的网易邮箱平板适配版

Re: Flink plain avrò deserilization issue

2023-07-20 Thread Girish Narayanan via user
Attaching client code here again as it got blocked previously by the firewall. Thanks, GIrish From: Girish Narayanan Date: Thursday, 20 July 2023 at 12:06 PM To: user@flink.apache.org Subject: Flink plain avrò deserilization issue Hi Team/Users, We are trying to send plain AVRO messages to a

Flink plain avrò deserilization issue

2023-07-20 Thread Girish Narayanan via user
One or more of the following files ( client code.zip ) violates IBM policy and all attachment(s) have been removed from the message. ** Hi Team/Users, We are trying to send plain AVRO messages to a Kafka topic but they are not