Re: Netty Client Thread - Classloader leak

I suddenly realised that I replied directly to Sudharsan. FYI if anyone wants to check this email too: > On 2 Mar 2022, at 11:36 PM, yu'an huang wrote: > > Hi Sudharsan, > > I think you are right. I just tried your scenario. I set a standalone session > cluster with 2 task managers (with on

Customizing backpressure mechanism for RichParallelSourceFunction

Hello! I have a dataflow pipeline built using Flink's RichParallelSourceFunction as parallel sources. I'm wondering if there are any mechanisms that I could use to implement *ack-based* back-pressure mechanism by sending ACK messages from operators (within the runtime) to source functions to achie

Re: KafkaPartitionSplitReader handleSplitsChanges

Or is the motivation that resolving the committed/latest offsets is an infrequent event (and only for bounded read) so the optimization is not worth it? On Wed, Mar 2, 2022 at 2:16 PM Mason Chen wrote: > Hi all, > > I noticed in the javadocs that SplitReaders should not have a blocking > handleS

Questions regarding connecting local FlinkSQL client to remote JobManager in K8s

Hi Flink users, Wanted to check if any of you tried to run the local FlinkSQL client against JobManager running in the Kubernetes environment. For local FlinkSQL Client and local Flink cluster we set these params: jobmanager.rpc.address: localhost jobmanager.rpc.port: 6123 To make it work, Is t

KafkaPartitionSplitReader handleSplitsChanges

Hi all, I noticed in the javadocs that SplitReaders should not have a blocking handleSplitsChanges implementation: https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/source/reader/splitreader/SplitReader.java#L55 However

Help with pom dependencies for Flink with Table API

Hello, I think I accidentally posted this question on the wrong email list (dev) so I am posting it again here. I am struggling to run my test Flink project with Table API I am trying to run a simple piece of code: final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutio

Re: Flink failure rate restart not work as expect

Hi, Jiaqiao: Since your job enables checkpoint, you can just try to remove the restart strategy config. The default value will be fixed-delay with Integer.MAX_VALUE restart attempts and '1 s' delay, as mentioned in [1]. In this way when a failover occurs, your job will wait for 1 seconds before it

Re: Flink 1.13.6 showing 1.13.5

Hi Razin, Would you mind sharing the Kubernetes manifest you're using to deploy this? Or which docker tag you are using? I spun up a quick example using the docker tag 'flink:1.13.6' and the correct version was shown in the JobManager logs and in the web-ui. Kind regards, Mika On 01.03.2022 13:

Kafka Source in a Geo Replicated Kafka Cluster

Hi, I need to manage geo-redundancy in my Kafka cluster across zones. I am planning to do this with Apache Mirror Maker to maintain an active-passive site. I wanted to understand consumer and producer failover when the primary cluster fails. Is there any way to detect and failover Flink's Kafka s

Left join query not clearing state after migrating from 1.9.0 to 1.14.3

Hello, We recently did a migration of our Flink jobs from version 1.9.0 to 1.14.3. These jobs consume from Kafka and produce to respective sinks. We are using MemoryStateBackend for our checkpointing and GCS as our remote fs. After migration, we found a few jobs that had left join in the SQL query

Re: Flink job recovery after task manager failure

Thanks. At the moment I’m running without a shared file storage, and I’m trying to better understand the recovery process. The behavior I see is as follows: * The job manager identifies the task manager’s failure and tries to restart * The job manager fails to find the checkpoint. Looki