Re: A way to meter number of deserialization errors

to understand and > potentially fix the source / parser limitations. > > > > Kind regards, David. > > > > From: Ilya Karpov > Date: Monday, 17 June 2024 at 12:39 > To: user > Subject: [EXTERNAL] A way to meter number of deserialization errors > > Hi all,

Re: A way to meter number of deserialization errors

Does anybody experience the problem of metering deserialization errors? пн, 17 июн. 2024 г. в 14:39, Ilya Karpov : > Hi all, > we are planning to use flink as a connector between kafka and > external systems. We use protobuf as a message format in kafka. If > non-backward compat

A way to meter number of deserialization errors

Hi all, we are planning to use flink as a connector between kafka and external systems. We use protobuf as a message format in kafka. If non-backward compatible changes occur we want to skip those messages ('protobuf.ignore-parse-errors' = 'true') but record an error and raise an alert. I didn't

Testing application pipeline with test Source/Sink implementations

Dear flink community, I'm trying to migrate to flink 1.14 from 1.10, I replaced FlinkKafkaConsumer011->KafkaSource and BucketingSink->FileSink without problems. I want to test whole application pipeline but can't find any Source/Sink classes suitable to substitute KafkaSource/FileSink. Previously

Re: Flink kafka consumers stopped consuming messages

ixed in the meantime. So if you > want to be on the safe side, I'd try to upgrade to more recent versions > (Flink + Kafka consumer). > > Best, > > Arvid > > On Wed, Jun 2, 2021 at 7:01 PM Ilya Karpov <mailto:idkf...@gmail.com>> wrote: > Hi there, > > toda

Flink kafka consumers stopped consuming messages

Hi there, today I've observed strange behaviour of a flink streaming application (flink 1.6.1, per-job cluster deployment, yarn): 3 task managers (2 slots each) are running but only 1 slot is actually consuming messages from kafka (v0.11.0.2), others were idling (currentOutputWatermark was

Re: Flink in k8s operators list

feed-back. It's all free and OSS, so who are we to complain? Though it's > still an important attention point. > > Hope this helps, > > Svend > > > > > > On Fri, 28 May 2021, at 9:09 AM, Ilya Karpov wrote: >> Hi there, >> >> I’m making a li

Flink in k8s operators list

Hi there, I’m making a little research about the easiest way to deploy link job to k8s cluster and manage its lifecycle by k8s operator. The list of solutions is below: - https://github.com/fintechstudios/ververica-platform-k8s-operator -

Importance of calling mapState.clear() when no entries in mapState

Hi, given: - flink 1.6.1 - stateful function with MapState mapState = //init logic; Is there any reason I should call mapState.clear() if I know beforehand that there are no entries in mapState (e.g. mapState.iterator().hasNext() returns false)? Thanks in advance!

Re: kafka corrupt record exception

According to docs (here: https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/kafka.html#the-deserializationschema , last paragraph) that’s an expected behaviour. May

Re: What is Flinks primary API language?

: Replace Scala implemented JobManager.scala and TaskManager.scala to > new JobMaster.java and TaskExecutor.java > FILP32: Make flink-table Scala free. > > But for API level, from my point of view, I have never heard any plan to stop > supporting Scala. > > Best > Yun

What is Flinks primary API language?

Hello, our dev-team is choosing a language for developing Flink jobs. Most likely that we will use flink-streaming api (at least in the very beginning). Because of Spark jobs developing experience we had before the choice for now is scala-api. However recently I’ve found a

Scala API support plans

Hi guys, what are the plans for scala-api support? Asking about it because I’ve recently found that scala-api used to catch up with java one. Thanks!