Re: Multiple consumers and custom triggers

Got it. Thanks! > On Dec 15, 2016, at 02:58, Jamie Grier wrote: > > Ahh, sorry, for #2: A single Flink job can have as many sources as you like. > They can be combined in multiple ways, via things like joins, or connect(), > etc. They can also be completely

Updating a Tumbling Window every second?

Hello, I have a rather simple problem with a difficult explanation... I have 3 streams, one of objects of class A (stream A), one of class B (stream B) and one of class C (stream C). The elements of A are generated at a rate of about 3 times every second. Elements of type B encapsulates some key

Re: Multiple consumers and custom triggers

Ahh, sorry, for #2: A single Flink job can have as many sources as you like. They can be combined in multiple ways, via things like joins, or connect(), etc. They can also be completely independent — in other words the data flow graph can be completely disjoint. You never to need to call execute()

Re: In 1.2-SNAPSHOT, EventTimeSessionWindows are not firing untill the whole stream is processed

Hi Aljoscha, Thanks a lot for the explanation. Using readFile with PROCESS_CONTINUOUSLY solves it. Two more questions though: 1. Is it possible to gracefully stop the job once it has read the input once? 2. Does the watermark extraction period depend on the watch interval, or should any watch

Flink 1.1.3 RollingSink - understanding output blocks/parallelism

Hi everyone, although this question might sound trivial, I’ve been curious about the following. Given a Flink topology with parallelism level set to 6 for example and outputting the data stream to HDFS using an instance RollingSink, how is the output file structured? By structured, I refer to

Re: Multiple consumers and custom triggers

For #1 there are a couple of ways to do this. The easiest is probably stream1.connect(stream2).map(...) where the MapFunction maps the two input types to a common type that you can then process uniformly. For #3 There must always be a WindowAssigner specified. There are some convenient ways to

Multiple consumers and custom triggers

Hello people, I've written down some quick questions for which I couldn't find much or anything in the documentation. I hope you can answer some of them! *# Multiple consumers* *1.* Is it possible to .union() streams of different classes? It is useful to create a consumer that counts elements

Re: Standalone cluster layout

Hi Avihai, 1. As much as possible (I would leave the operating system at least 1 GB of memory). It depends also on the workload you have. For streaming workload with very small state, you can use Flink with 1-2 GB of heap space and still get very good performance. 2. Yes, I would recommend to run

Re: PartitionedState and watermark of Window coGroup()

Hi, elements are coGrouped on the specified key. So only elements with the same key in both streams end up in the same group. Yes, the watermark uses the minimum of both streams. On Tue, Dec 13, 2016 at 7:02 PM, Sendoh wrote: > Hi Flink users, > > I'm a bit

Re: checkpoint notifier not found?

Is this more appropriate for dev list? Anyway here is my first: https://github.com/apache/flink/pull/3006 > On Dec 14, 2016, at 2:38 AM, Robert Metzger wrote: > > Hi Abhishek, > you can not push to the Flink repository directly.

Re: Checkpointing

Hi, for Flink 1.2 the ListCheckpointed interface is intended to replace Checkpointed. From a user perspective they work very similar, but the new interface allows you to break down the state into smaller, redistributable units (the list items) to support job rescaling. A proper documentation

Re: use case for sliding windows

I think so, you can use sliding windows and basically do a WordCount-like job that counts the occurences in each window. Then, you would have a filter afterwards that filters out those elements where the count is lower than a given threshold. Cheers, Aljoscha On Mon, 12 Dec 2016 at 22:30

Re: WindowFunction-extension, WindowedStream apply signature mismatch

Hi, are you using the WindowFunction in org.apache.flink.streaming.api.scala.function? It's a bit tricky because there is another WindowFunction in another package. Cheers, Aljoscha On Tue, 13 Dec 2016 at 13:48 MIkkel Islay wrote: > (The following is a cross-post of a

Re: checkpoint notifier not found?

Hi Abhishek, you can not push to the Flink repository directly. Only Flink committers are allowed to do that. But you can fork the Flink repository on github to your own GitHub account and then push the changes to your Github. Then, you can create a pull request to offer those changes to the main

[ANNOUNCE] New Flink community mailing list

Hi everyone, We have created a new Flink mailing lists, commun...@flink.apache.org where we can post everything related to the broader Flink community including job offers, upcoming meetups and conferences, exciting reads, and everything else that is deemed worthy for the greater Flink community.

Re: In 1.2-SNAPSHOT, EventTimeSessionWindows are not firing untill the whole stream is processed

Hi Yassine, for a bit more detailed explanation: We internally changed how the timer system works, this timer system is also used to periodically extract watermarks. Due to this change, in your case we don't extract watermarks anymore. Internally, your call resolves to something like this: