Re: Generate watermarks per key in a KeyedStream

We are contending with the same issue, as it happens.  We have dozens, and potentially down the line, may need to deal with thousands of different "time systems" as you put it, and may not be know at compile time or job start time.  In a practical sense, how

Re: Generate watermarks per key in a KeyedStream

Thanks for your reply, Xingcan. On Wed, Nov 8, 2017 at 10:42 PM, Xingcan Cui wrote: > Hi Shailesh, > > actually, the watermarks are generated per partition, but all of them will > be forcibly aligned to the minimum one during processing. That is decided > by the semantics of

Re: Generate watermarks per key in a KeyedStream

Hi Shailesh, actually, the watermarks are generated per partition, but all of them will be forcibly aligned to the minimum one during processing. That is decided by the semantics of watermark and KeyedStream, i.e., the watermarks belong to a whole stream and a stream is made up of different

Generate watermarks per key in a KeyedStream

Hi, I'm working on implementing a use case wherein different physical devices are sending events, and due to network/power issues, there can be a delay in receiving events at Flink source. One of the operators within the flink job is the Pattern operator, and there are certain patterns which are

Re: Watermarks per key

. So is that what's technically preventing assigning timestamps per key from working? I'm curious to hear Aljoscha's thoughts on watermark management across keys. Thanks! -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Watermarks-per-key

Re: Watermarks per key

Hi Jordan, it is not possible to generate watermarks per key. This feature has been requested a couple of times but I think there are no plans to implement that. As far as I understand, the management of watermarks would be quite expensive (maintaining several watermarks, purging watermarks

Watermarks per key

Hi, I’m designing a streaming job whose elements need to be windowed by event time across a large set of keys. All elements are read from the same source. Event time progresses independently across keys. Is it possible to assign timestamps, and thus generate independent watermarks, per keyed

Re: Maintaining watermarks per key, instead of per operator instance

> AggregateReadings()); > > DataStream aggregatesPerHour = readings > .timeWindow(Time.hours(1)) > .allowedLateness(Time.hours(2)) > .apply(new ReadingAggregate(), new AggregateReadings(), new > AggregateReadings()); > > > > Feedback is

Re: Maintaining watermarks per key, instead of per operator instance

DataStream aggregatesPerHour = readings >> .timeWindow(Time.hours(1)) >> .allowedLateness(Time.hours(2)) >> .apply(new ReadingAggregate(), new AggregateReadings(), new >> AggregateReadings()); >> >> >> Feedback is very welcome. >> >

Re: Maintaining watermarks per key, instead of per operator instance

t; I am pretty new to flink, could you elaborate on a possible solution? We > can guarantee good ordering by sensor_id, thus watermarking by key would be > the only reasonable way for us ( > *sensorData.keyBy('id').timeWindow(1.minute).sum('value')*), could I do > my own watermarking after*se

Re: Maintaining watermarks per key, instead of per operator instance

han > > > > >> On 11 Nov 2016, at 00:29, Fabian Hueske-2 [via Apache Flink User Mailing >> List archive.] <[hidden email] >> <http://user/SendEmail.jtp?type=node=10179=0>> wrote: >> > >> Hi Stephan, >> >> I just wrote an

Re: Maintaining watermarks per key, instead of per operator instance

[via Apache Flink User Mailing > List archive.] <[hidden email] > <http:///user/SendEmail.jtp?type=node=10179=0>> wrote: > > Hi Stephan, > > I just wrote an answer to your SO question. > > Best, Fabian > > > 2016-11-10 11:01 GMT+01:00 Stephan Epping &l

Re: Maintaining watermarks per key, instead of per operator instance

lt;[hidden email] > >: > Hello, > > I found this question in the Nabble archive > (http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Maintaining-watermarks-per-key-instead-of-per-operator-instance-tp7288.html > > <http://apache-flink-user-mailing

Re: Maintaining watermarks per key, instead of per operator instance

t; about Snapshots? Will they work with the mentioned design? >> >> kind regards, >> Stephan > >>> On 11 Nov 2016, at 00:39, Fabian Hueske <>> href="x-msg://10/user/SendEmail.jtp?type=nodenode=10094i=1 >>> " >>> target=&qu

Re: Maintaining watermarks per key, instead of per operator instance

; On 11 Nov 2016, at 00:39, Fabian Hueske < href="x-msg://10/user/SendEmail.jtp?type=nodenode=10094i=1" > target="_top" rel="nofollow" link="external" class="">[hidden email]> wrote: > > Hi Stephan, > > I just wrote an answer

Re: Maintaining watermarks per key, instead of per operator instance

6-11-10 11:01 GMT+01:00 Stephan Epping <[hidden email] >> >: >> Hello, >> >> I found this question in the Nabble archive >> (http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Maintaining-watermarks-per-key-instead-of-per-operator-instance-tp7

Re: Maintaining watermarks per key, instead of per operator instance

1:00 Stephan Epping <stephan.epp...@zweitag.de>: > >> Hello, >> >> I found this question in the Nabble archive ( >> http://apache-flink-user-mailing-list-archive.2336050.n4. >> nabble.com/Maintaining-watermarks-per-key-instead-of-per- >> operator-insta

Re: Maintaining watermarks per key, instead of per operator instance

n. > > Best, Fabian > > 2016-11-10 11:01 GMT+01:00 Stephan Epping <stephan.epp...@zweitag.de > <mailto:stephan.epp...@zweitag.de>>: > Hello, > > I found this question in the Nabble archive > (http://apache-flink-user-mailing-list-archive.2336050.n4.nabbl

Re: Maintaining watermarks per key, instead of per operator instance

ning- > watermarks-per-key-instead-of-per-operator-instance-tp7288.html) but was > unable/dont know how to reply. > > Here is my question regarding the mentioned thread: > > Hello, > > I have similar requirements (see StackOverflor http:// > stackoverflow.com/quest

Re: Maintaining watermarks per key, instead of per operator instance

Hi Aljoscha! Is it possible somehow to use the RichXFunction in CEP? The task is pretty similar, but I have to ignore once the next triggered event for the same key. On Wed, Jun 1, 2016 at 2:54 PM, Aljoscha Krettek wrote: > Hi, > yeah, in that case per-key watermarks would

Re: Maintaining watermarks per key, instead of per operator instance

Hi again Aljoscha, understood. Thanks for the link. I really like the straightforward approach concerning storing state. It makes things very easy. The improvements are very interesting, particularly the composite triggers. That would significantly improve flexibility. Kind regards Leon 1.

Re: Maintaining watermarks per key, instead of per operator instance

Hi, yeah, in that case per-key watermarks would be useful for you. I won't be possible to add such a feature, though, due to the (possibly) dynamic nature of the key space and how watermark tracking works. You should be able to implement it with relatively low overhead using a RichFlatMapFunction

Re: Maintaining watermarks per key, instead of per operator instance

Hi Aljoscha, thanks for the speedy reply. I am processing measurements delivered by smart meters. I use windows to gather measurements and calculate values such as average consumption. The key is simply the meter ID. The challenge is that meters may undergo network partitioning, under which

Maintaining watermarks per key, instead of per operator instance

My use case primarily concerns applying transformations per key, with the keys remaining fixed throughout the topology. I am using event time for my windows. The problem i am currently facing is that watermarks in windows propagate per operator instance, meaning the operator event time