Forcing consuming one stream completely prior to another starting

I’m joining two streams - one is a “decoration” stream that we have in a compacted Kafka topic, produced using a view on a MySQL table AND using Kafka Connect; the other is the “event data” we want to decorate, coming in over time via Kafka. These streams are keyed the same way - via an “id”

Upgrade to 1.4.0 - Kryo/Avro issue

We're attempting to upgrade our 1.3.2 cluster and jobs to 1.4.0. When submitting jobs to the 1.4.0 Kafka cluster, they fail with a Kryo registration error. My jobs are consuming from Kafka topics with messages in Avro format. The avro schemas are registered with a Confluent avro schema registry.

Kafka Producer timeout causing data loss

Team, One more question to the community regarding hardening Flink Apps. Let me start off by saying we do have known Kafka bottlenecks which we are in the midst of resolving. So during certain times of day, a lot of our Flink Apps are seeing Kafka Producer timeout issues. Most of the logs are

Understanding Restart Strategy

Team, Hopefully, this is a quick one.  We have setup restart strategy as follows in pretty much all of our apps: env.setRestartStrategy(RestartStrategies.fixedDelayRestart(10, Time.of(30, TimeUnit.SECONDS))); This seems pretty straight-forward. App should retry starting 10 times every 30

Task Manager detached under load

Hi All, We have hit some load related issues and was wondering if any one has some suggestions. We are noticing task managers and job managers being detached from each other under load and never really sync up again. As a result, Flink session shows 0 slots available for processing. Even

Re: how to run flink project built with maven

Yep, that's it, so simple... Thanks! On Fri, Jan 19, 2018 at 12:53 PM, Eron Wright wrote: > You must specify the full class name, in this case > `org.example.WordCount`, for the `--class` argument. > > On Fri, Jan 19, 2018 at 9:35 AM, Jesse Lacika wrote: >

Re: how to run flink project built with maven

You must specify the full class name, in this case `org.example.WordCount`, for the `--class` argument. On Fri, Jan 19, 2018 at 9:35 AM, Jesse Lacika wrote: > I feel like this is probably the simplest thing, but I can't seem to > figure it out, and I've searched and searched and

how to run flink project built with maven

I feel like this is probably the simplest thing, but I can't seem to figure it out, and I've searched and searched and can't find the answer, so I thought I'd try the mailing list. I'm using maven to build the sample flink project using the scala API as described here:

Re: What's the meaning of "Registered `TaskManager` at akka://flink/deadLetters " ?

Hi, It seems like you have not opened some of the ports. As I pointed out in the first mail, please go through all of the config options regarding hostnames/ports (not only those that appear in the log files, maybe something is not being logged)