Session ID Error

Hello, In attempting to submit a job via Yarn session on Hadoop cluster (using Flink 1.2.1), I get timed out and receive the following error from the server: Discard message LeaderSessionMessage(----,SubmitJob(JobGraph(jobId: 5d0406fc547af4fc36bf78ed812a3f90),EXEC

Re: Kafka and Flink integration

Hi! For general data exchange between systems, it is often good to have a more standard format. Being able to evolve the schema of types is very helpful if you evolve the data pipeline (which almost always happens eventually). For that reason, Avro and Thrift are very popular for that type of dat

RE: Kafka and Flink integration

Just one more question :). Considering I'm producing into kafka with other application other than flink, which serializer should i use in order to use pojo types when consuming those same messages (now in flink)? -- View this message in context: http://apache-flink-user-mailing-list-archive.233

Re: Access to time in aggregation, or aggregation in ProcessWindowFunction?

Hi, That looks perfect! I realized I could probably use an Evictor together with my WindowProcessFunction to prevent the window from preserving the whole state, but ditching the window looks even better. Thanks a lot! William - Original Message - From: "Nico Kruber" To: Cc:"William Saar

Re: Kafka watermarks

I guess using BoundedOutOfOrdernessTimestampExtractor inside consumer will work. Thanks -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-watermarks-tp13849p13880.html Sent from the Apache Flink User Mailing List archive. mailing list arc

RE: Kafka and Flink integration

Thanks, I'll try to refactor into POJOs. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-and-Flink-integration-tp13792p13879.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: Kafka watermarks

according to the javadoc of FlinkKafkaConsumerBase#assignTimestampsAndWatermarks(): "Specifies an {@link AssignerWithPunctuatedWatermarks} to emit watermarks in a punctuated manner. The watermark extractor will run per Kafka partition, watermarks will be merged across partitions in the same way

RE: Kafka and Flink integration

Yes, POJOs can contain other nested POJO types. You just have to make sure that the nested field is either public, or has a corresponding public getter- and setter- method that follows the Java beans naming conventions. On 21 June 2017 at 12:20:31 AM, nragon (nuno.goncal...@wedotechnologies.com

Re: Can't get my job restarted on job manager failures

Many thanks Nico!! everything is clear now Kind Regards, Mike Pryakhin > On 20 Jun 2017, at 19:24, Nico Kruber wrote: > > yes, this supports my theory: both the ResourceManager and the TaskManager > instances are trying to reconnect to the old JobManager to re-establish a > link > in case

Re: Can't get my job restarted on job manager failures

yes, this supports my theory: both the ResourceManager and the TaskManager instances are trying to reconnect to the old JobManager to re-establish a link in case of temporary failures. * In the best case, the connection is re-established and everything continues. * Since your failure is permanent

RE: Kafka and Flink integration

Can i have pojo has composition of other pojo? My custom object has many dependencies and in order to refactor it I must also change another 5 classes as well. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-and-Flink-integration-tp1379

Re: Access to time in aggregation, or aggregation in ProcessWindowFunction?

Hi William, I'm not quite sure what you are trying to achieve... What constitutes a "new event"? is this based on some key? If so, you may group on that key, create a window and use a custom trigger [1] instead where you can react in onElement() and setup a event time timer for the first one an

Re: Kafka watermarks

So, in order to work with event time I have to options, inside kafka consumer or after kafka consumer. The first I can use: FlinkKafkaConsumer09 consumer. consumer. assignTimestampsAndWatermarks() The other option: FlinkKafkaConsumer09 consumer. DataStream dataStream =env.addSource(consume

Re: Can't get my job restarted on job manager failures

Thanks a lot Niko! It is definitely a permanent failure, I’ve tried multiple times and always got the same behaviour. And as you said, right after the job manager has started the errors stop as well. I also checked the logs of the Task Manager and found huge amount of the following warnings ()

RE: Kafka and Flink integration

Nico, I'll try some different approaches and will be back here, hopefully with some results :) Thanks for this brainstorming :) -Original Message- From: Nico Kruber [mailto:n...@data-artisans.com] Sent: 20 de junho de 2017 16:44 To: Nuno Rafael Goncalves Cc: user@flink.apache.org Subje

RE: Kafka and Flink integration

Hi Nuno, In general, if it is possible, it is recommended that you map your generic classes to Tuples / POJOs [1]. For Tuples / POJOs, Flink will create specialized serializers for them, whereas for generic classes (i.e. types which cannot be treated as POJOs) Flink simply fallbacks to using Kryo

Re: Kafka and Flink integration

I can only repeat what Gordon wrote on Friday: "It’s usually always recommended to register your classes with Kryo [using registerKryoType()], to avoid the somewhat inefficient classname writing. Also, depending on the case, to decrease serialization overhead, nothing really beats specific custo

Re: Can't get my job restarted on job manager failures

My best guess here is that the resource manager is still trying to connect to the JobManager which failed - after all, how should it know whether this is a temporary network failure or a permanent failure?! If, after your new JobManager starts, the errors stop as well, I'd say, you don't have t

RE: Kafka and Flink integration

I believe there are some performance impact while de/serializing, which is "normal". What I'm trying to understand is if there are any tips to improve this process. For instance, tuples vs general class types. Do you know if it's worth it to map a custom object into tuple just for de/serializati

Re: Kafka watermarks

Can you clarify a bit more on what you want to achieve? Also, what is "BOOTE"? Nico On Tuesday, 20 June 2017 13:45:06 CEST nragon wrote: > When consuming from kafka should we use BOOTE inside consumer or after? > Thanks > > > > -- > View this message in context: > http://apache-flink-user-mai

Re: Can't get my job restarted on job manager failures

Hi Niko, Thanks for your reply! Having zookeeper-related properties configured everything works smoothly! I was confused because the doc references high availability configuration for yarn session mode, anyway thanks a lot! Now I noticed another problem, when I kill the job manager, it then gets

Re: Kafka and Flink integration

No, this is only necessary if you want to register a custom serializer itself [1]. Also, in case you are wondering about registerKryoType() - this is only needed as a performance optimisation. What exactly is your problem? What are you trying to solve? (I can't read JFR files here, and from what

Re: Flink CEP not emitting timed out events properly

I know that there wouldn't be a scenario where the first event type(coming from topic t1) would be coming with a timestamp higher than the current watermark. Although I am still investigating whether the other events from other topics (specifically t3 and t4) are arriving after the watermark update

Re: Flink CEP not emitting timed out events properly

Are you sure that after incrementing the wm by 1sec, there is no element that will come with a timestamp smaller than this? Or, that after 10sec of inactivity, no element will come with such a timestamp? Kostas > On Jun 20, 2017, at 4:18 PM, Biplob Biswas wrote: > > currentMaxTimestamp = cur

Re: Flink CEP not emitting timed out events properly

Hi Kostas, Yes, I have a flag in my timestampextractor. As you can see from the code below, I am checking whether currentTime - systemTimeSinceLastModification > 10 sec. as new events come then the watermark wouldn't be incremented. But as soon as I have a difference of more than 10 seconds,

Re: Can't get my job restarted on job manager failures

Hi Mike, have you configured zookeeper [1] ? afaik, it is required for a high- availability (YARN) session and is used to store JobManager state. Without it, a recovery would not know what to recover from. Nico [1] https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/ jobmanager_hi

Re: Flink CEP not emitting timed out events properly

You are correct that elements are waiting until a watermark with a higher timestamp than theirs (or the patterns timeout) arrives. Now for the Watermark emitter, 1) how do you measure the 10sec in processing time and ii) by how much do you advance the watermark. If you advance it by a lot, th

Re: Flink CEP not emitting timed out events properly

Hi Kostas, I have out-of-orderness of around 5 seconds from what I have observed but that too from events coming from a different topic. The initial topic doesn't have out-of-order events still I have added a generous time bound of 20 seconds. Still, I will try for a higher number just in order to

Re: Flink CEP not emitting timed out events properly

Hi Biplob, You are correct that only a higher watermark leads to discarded events. Are you sure that your custom watermark emitter does not emit a high watermark? E.g. your partition has elements that are far out-of-order. In addition, are you sure that your elements are not simply buffered and

Re: Queries regarding FlinkCEP

For example if there is no events in one of the partitions the partition will always generate Watermark(Long.MIN_VALUE) which will result in Watermark not being advanced. There is open JIRA to improve such situations: FLINK-5479 . Z pozdrowieniami!

Re: Flink CEP not emitting timed out events properly

But if that's the case, I don't understand why some of my events are just lost If the watermark which is used is the smallest ... They either I expect a match or I expect a timed out event. The only way I can imagine my events getting lost is higher watermark than the incoming event and thus

Re: Queries regarding FlinkCEP

Hi dawid, Yes I am reading from multiple topics and yes a few topics have multiple partitions, not all of them. But I didn't understand the concept of stalled partition. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Queries-regarding-Fli

Re: Flink CEP not emitting timed out events properly

It is possible that two Watermarks will be passed through the Stream, but at an operator that has more than one input streams(in your case from different topics and partitions) the smallest Watermark is considered valid and passed downstream. Z pozdrowieniami! / Cheers! Dawid Wysakowicz *Data/So

Re: Queries regarding FlinkCEP

Hi Biplop, Your recent post on reading from different topics made me realise it may be a problem with "stalled" partitions. Did your topic have more than one partition? If it did, it may be the problem that Watermark is generated independently per partition and then the smallest one is taken as a

Re: Flink CEP not emitting timed out events properly

Hi dawid, First of all congratulations on being a Flink committer, saw your tweet in the morning. Now regarding that link, that talks about multiple partitions for a single topic, here I am talking about multiple topics each having different number of partitions. I tried adding tinestampextract

Kafka watermarks

When consuming from kafka should we use BOOTE inside consumer or after? Thanks -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-watermarks-tp13849.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble

Re: Flink CEP not emitting timed out events properly

Hi Biplop, If an operator has two inputs, the resulting Watermark is the smallest one from the two upstreams. More on that you can check here .

Can't get my job restarted on job manager failures

Hello, I'm currently trying to check whether my job is restarted in case of Job Manager failure. The job is submitted as a single job on YARN with the following options set in the flink-conf.yaml: restart-strategy: fixed-delay restart-strategy.fixed-delay.attempts: 3 restart-strategy.fixed-de

Re: Flink CEP not emitting timed out events properly

Hi Kostas, Implementing my custom timestamp assigned made em realise a problem which we have in our architecture you may say. Any inputs would be really appreciated. So, for now, we are reading from 4 different kafka topics, and we have a flow similar to something like this: Event 1(on topic t

Re: Custom Gauge Metric is not getting updated on Job Dashboard

how long does the job run? The metrics are only updated every 10 seconds; and are not updated when the job finishes. On 20.06.2017 12:12, sohimankotia wrote: I was testing flink Gauge metric in my flat map function . I was just sending a value (end time -start time ) of a DB Op in Flatmap . But

Re: Kafka and Flink integration

Attaching jfr flight_recording_10228245112.jfr -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-and-Flink-integration-

Access to time in aggregation, or aggregation in ProcessWindowFunction?

Hi, I am looking to implement a window that sends out updates for each new event it receives and also when an expiration timer fires and purges the window (the expiration time can be determined from a timestamp in the first event). I can't figure out a way to do this that does not require preserv

Custom Gauge Metric is not getting updated on Job Dashboard

I was testing flink Gauge metric in my flat map function . I was just sending a value (end time -start time ) of a DB Op in Flatmap . But on dashboard is only showing first value instead of updated value . I am using following code in my flat map . getRuntimeContext() .getMetricGroup() .g

Re: Kafka and Flink integration

Do I need to use registerTypeWithKryoSerializer() in my execution environment? My serialization into kafka is done with the following snippet try (ByteArrayOutputStream byteArrayOutStream = new ByteArrayOutputStream(); Output output = new Output(byteArrayOutStream)) { Kryo kryo = new Kryo();