Re: Will job manager restarts lead to repeated savepoint restoration?

Hi Vishal, thanks for the detailed description of the problems. 1. This is currently the intended behaviour of Flink. The reason is that if the system is no longer connected to ZooKeeper then we cannot rule out that there is another process who has taken over the leadership. FLINK-10052 has the g

using DefaultScalaModule

Hello - Flink 1.11 uses a shaded version of Jackson for serialization, which does not support jackson-module-scala. I need to register DefaultScalaModule for some Scala object serialization through Jackson. But when I do a mapper.registerModule(DefaultScalaModule), I get the following compilation

Official Flink 1.12.0 image

Does anyone know when an official 1.12.0 image will be available on Docker Hub? -- Robert Cullen 240-475-4490

Re: Official Flink 1.12.0 image

Hi Robert, there is a discussion about it in FLINK-20632 [1]. PR #9249 [2] still needs to get reviewed. You might want to follow that PR as Xintong suggested in [1]. I hope that helps. Best, Matthias [1] https://issues.apache.org/jira/browse/FLINK-20632 [2] https://github.com/docker-library/offic

DataBricks Spark/Scala Opportunity

> > Hi All, > > Note:- (This is a full time position) > > I copied Geoff in this email chain who is actively looking for Spark/Scala > Developer for a full time position with DataBricks 200k+ > (Salary/Bonus/Stocks/H1b/GC Process). > > If you know anyone or your friends who is good in Spark/Scala e

Flink SQL continuous join checkpointing

Hi everyone! I'm using Flink *1.12.0* with SQL API. My streaming job is basically a join of two dynamic tables (from kafka topics) and insertion the result into PostgreSQL table. I have enabled watermarking based on kafka topic timestamp column for each table in join: CREATE TABLE table1 (

Re: FlinkKafkaProducer Fails with "Topic not present in metadata"

How to enable trace log for Kafka connector? I’m experience periodically same error, but in my case I don’t specify partition Alexey From: Joseph Lorenzini Sent: Thursday, December 10, 2020 7:36:54 AM To: Becket Qin Cc: user@flink.apache.org Subject: Re: Flink

Flink Stateful Function: The program's entry point class not found in the jar file

Hello! I'm trying to deploy a Flink stateful function as a flink jar and I followed the instruction here . However I'm getting the error saying that the program entry point class was not found in the jar even after I added the dependency in my pom. Running

Distribute Parallelism/Tasks within RichOutputFormat?

Hi folks, I've got a single RichOutputFormat which is comprised of two HadoopOutputFormats, let's call them A and B, each writing to different HDFS directories. If a Record matches a certain condition it's written using A, otherwise it's written with B. Currently, the parallelism that is set at

How to implement a WindowableTask(similar to samza) in Apache flink?

I am new to flink and this is my first post in the community. Samza has a concept of windowing where a stream processing job needs to do something in regular intervals, regardless of how many incoming messages

Re: Flink SQL continuous join checkpointing

Hi, Taras > But checkpoint data size for join task is permanently increasing despite the > watermarks on the tables and "Low watermark" mark in UI. > As far as I understand outdated records from both tables must be dropped from > checkpoint after 2 hours, but looks like it holds all job state s

Re: Re: checkpoint delay consume message

Hi Yun, Sorry but we didn't understand your questions. The delay we are experiencing is on the *read* side. The message is written to kafka topic and consumed by flink with a delay that depends on the checkpoints interval When we disabled the checkpoints the messages are immediately consumed We us