RE: Session windows - Evaluation of addition of element + window expires/gets discarded after some set time of inactivity

Hi Bart, Thank you so much for sharing the approach. Looks like this solved my problem. Here's what I have as an implementation for my use-case: package org.apache.flink.quickstart import org.apache.flink.api.common.state.{ ReducingState, ReducingStateDescriptor, ValueState, ValueStateDescriptor

Re: multiple processing of streams

Hi Robert, it is certainly possible to feed the same DataStream into two (or more) operators. Both operators should then process the complete input stream. What you describe is an unintended behavior. Can you explain how you figure out that both window operators only receive half of the events?

FlinkKafkaConsumerBase - Received confirmation for unknown checkpoint

Hello, Am getting the following warning upon executing a checkpoint /2016-10-21 16:31:54,229 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 5 @ 1477063914229 2016-10-21 16:31:54,233 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator -

Re: ValueState in RichCoFlatMap, possible 1.2-SNAPSHOT regression

Hi, the problem is this line > object FlatMapper extends RichCoFlatMapFunction[Long, String, (Long, String)] > { which should use „class" instead of „object". Otherwise, one singleton instance of the FlatMapper is used by Flink across multiple operator instances, which leads to the whole bunc

RE: Session windows - Evaluation of addition of element + window expires/gets discarded after some set time of inactivity

Here is a session trigger that I wrote (not quite the same rules around what a session is, but should hopefully be a good start to work from). I'd love to get any feedback on how it could be improved. - bart import org.apache.flink.api.common.state.{ReducingState, ReducingStateDescriptor, Valu

Unit testing a Kafka stream based application?

Hi, In addition to having unit tests for the individual components (map, flatmap, reduce, etc) of my application I would like to write unit tests for the entire flow of my Flink application. My application reads from Kafka, does various processing and writes out put to both kafka and files. This

Re: NoClassDefFoundError on cluster with httpclient 4.5.2

Hi Till, The httpclient jar is included in the job jar. Looking at a similar issue FLINK-4587 , It turns out the problem is with maven shade plugin, since I'm building Flink from sources with maven 3.3.x. I was able to solve the problem by rebuildi

Re: Flink error: Too few memory segments provided

thank you so much, it worked immediately. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-error-Too-few-memory-segments-provided-tp9657p9669.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com

Re: Flink error: Too few memory segments provided

Hi, On 21 October 2016 at 11:17, otherwise777 wrote: > I tried increasing the taskmanager.network.numberOfBuffers to 4k and > later to > 8k, i'm not sure if my configuration file is even read, it's stored inside > my IDE as follows: http://prntscr.com/cx0vrx > i buil

Re: Flink error: Too few memory segments provided

I tried increasing the taskmanager.network.numberOfBuffers to 4k and later to 8k, i'm not sure if my configuration file is even read, it's stored inside my IDE as follows: http://prntscr.com/cx0vrx i build the flink program from the IDE and run it. I created several a

Re: Session windows - Evaluation of addition of element + window expires/gets discarded after some set time of inactivity

Hi Anchit, I think you need a customized EventTimeTrigger which returns "TriggerResult.FIRE" both on new element and watermark. Thanks, Manu

Session windows - Evaluation of addition of element + window expires/gets discarded after some set time of inactivity

Hi All, I have a use case where in I'm supposed to work with Session Windows to maintain some values for some sessionIDs/keys. The use case is as follows: I need to maintain a session window for the incoming data and discard the window after some set gap/period of inactivity but what I want is t