Entering a busy loop when adding a new sink to the graph + checkpoint enabled

2024-02-20 Thread Daniel Peled
Hello Guys, Can someone please assist us regarding the following issue ? We have noticed that when we add a *new kafka sink* operator to the graph, *and start from the last save point*, the operator is 100% busy for several minutes and *even 1/2-1 hour* !!! The problematic code seems to be the

Re: Entering a busy loop when adding a new sink to the graph + checkpoint enabled

2024-02-07 Thread Daniel Peled
Good morning, Any updates/progress on this issue ? BR, Danny ‫בתאריך יום א׳, 4 בפבר׳ 2024 ב-13:20 מאת ‪Daniel Peled‬‏ <‪ daniel.peled.w...@gmail.com‬‏>:‬ > Hello, > > We have noticed that when we add a *new kafka sink* operator to the > graph, *and start from the last save point*, the operator

Entering a busy loop when adding a new sink to the graph + checkpoint enabled

2024-02-04 Thread Daniel Peled
Hello, We have noticed that when we add a *new kafka sink* operator to the graph, *and start from the last save point*, the operator is 100% busy for several minutes and *even 1/2-1 hour* !!! The problematic code seems to be the following for-loop in getTransactionalProducer() method:

Flink 1.14.2/3 - KafkaSink vs deprecated FlinkKafkaProducer

2022-01-31 Thread Daniel Peled
Hi everyone, Has anyone encountered any problem with the new KafkaSink that is used in Flink 1.14 ? When running our jobs, the *sinks* of some of our jobs are stuck in initializing for more than an hour. The only thing that helps is deleting the topic *__transaction_state*. After deleting this

Cannot access state from a empty taskmanager - using kubernetes

2021-01-28 Thread Daniel Peled
Hi, We have followed the instructions in the following link ""Enabling Queryable State" with kubernetes: https://ci.apache.org/projects/flink/flink-docs-stable/deployment/resource-providers/standalone/kubernetes.html#enabling-queryable-state *When the replicas of the task-manager pods is 1 we

Re: Chaining 2 flink jobs through a KAFKA topic with checkpoint enabled

2021-01-05 Thread Daniel Peled
ttps://kafka.apache.org/documentation/streams/developer-guide/config-streams.html#processing-guarantee >>> >>> On Mon, Dec 28, 2020 at 12:22 PM Chesnay Schepler >>> wrote: >>> >>>> I don't particularly know the our Kafka connector, but it sounds like a &

Chaining 2 flink jobs through a KAFKA topic with checkpoint enabled

2020-12-27 Thread Daniel Peled
Hello, We have 2 flink jobs that communicate with each other through a KAFKA topic. Both jobs use checkpoints with EXACTLY ONCE semantic. We have seen the following behaviour and we want to make sure and ask if this is the expected behaviour or maybe it is a bug. When the first job produces a