[Window] Per key window

Hello, How should i create a tumbling window referring same key. each key have it own window instance. Session window referring key, but must have a gap. Regards, Shu li Zheng

Re: AKA and quarantine

>> If you enable taskmanager.exit-on-fatal-akka-error, then it will stop TMs which got quarantined. This will automatically restart TMs in case that you are running Flink on Yarn. Thus, I would recommend enabling this if possible We do not use yarn. This would end up restarting the jobs on the rem

Re: AKA and quarantine

Hi Vishal, Akka usually quarantines remote ActorSystems in case of a system message delivery failure or if the death watch was triggered. This can, for example, happen if your machine is under heavy load or has a high GC pressure and does not find enough time to respond to the heartbeats. - If yo

Re: Scheduled/timed source/sink

AFAIK Flink's source/sink interface do not expose something akin to the ProcessFunction's TimerService, so I would suggest to use an ScheduledExecutorService or similar. Technically you can use process functions as sources, the easiest way being to create a dummy source which broadcasts a singl

Re: ElasticsearchSink in Flink 1.4.0 with Elasticsearch 5.2+

As of right now there is no specific date, see also https://flink.apache.org/news/2017/11/22/release-1.4-and-1.5-timeline.html. On 29.01.2018 13:41, Christophe Jolif wrote: Thanks a lot. Is there any timeline for 1.5 by the way? -- Christophe On Mon, Jan 29, 2018 at 11:36 AM, Tzu-Li (Gordon)

Re: AKA and quarantine

Thank you. On Mon, Jan 29, 2018 at 3:17 AM, Fabian Hueske wrote: > Hi Vishal, > > sorry for the late response. > Till (in CC) might be able to answer your Akka / coordination related > questions. > > Best, Fabian > > 2018-01-24 1:22 GMT+01:00 Vishal Santoshi : > >> Any suggestions ? I know thes

Re: GetExecutionPlan fails with IllegalArgumentException in Comparator

@Julian Could you provide an example job to reproduce this? (Can also be shared privately) On 23.01.2018 18:05, Fabian Hueske wrote: Great! Thank you :-) 2018-01-23 17:58 GMT+01:00 Bauss, Julian >: Hi Fabian, sure! See: https://issues.apache.org

Trigger Time vs. Latest Acknowledgement

I'm triggering nightly savepoints at 23:59:00 with crontab on the flink cluster. For example last night's savepoint has this information: Trigger Time: 23:59:14 Latest Acknowledgement: 00:00:59 What are the min/max boundaries for the data contained by the savepoint? Can I deduce from this either

Scheduled/timed source/sink

What is the best way to add timer to a source/sink? (Apologies if I’m asking the question that’s already being answered elsewhere), also is it possible to use process function as source or sink? Thanks Ishwara

Re: ElasticsearchSink in Flink 1.4.0 with Elasticsearch 5.2+

Thanks a lot. Is there any timeline for 1.5 by the way? -- Christophe On Mon, Jan 29, 2018 at 11:36 AM, Tzu-Li (Gordon) Tai wrote: > Hi Christophe, > > Thanks a lot for the contribution! I’ll add reviewing the PR to my backlog. > I would like / will try to take a look at the PR by the end of th

Re: How to make savepoints more robust in the face of refactorings ?

>One thing that I’m not completely certain with yet, is where in your demonstrated code a anonymous-classed serializer is generated for some type. >From what I see, there shouldn’t be any anonymous-class serializers for the code. Is the code you provided a “simplified” version of the actual code in

Re: How to make savepoints more robust in the face of refactorings ?

Hi, In the Scala API, type serializers may be anonymous classes generated by Scala macros, and would therefore contain a reference to the wrapping class (i.e., your `Operators` class). Since Flink currently serializes serializers into the savepoint to be used for deserialization on restore, and

Re: ElasticsearchSink in Flink 1.4.0 with Elasticsearch 5.2+

Hi Christophe, Thanks a lot for the contribution! I’ll add reviewing the PR to my backlog. I would like / will try to take a look at the PR by the end of this week, after some 1.4.1 blockers which I’m still busy with. Cheers, Gordon On 29 January 2018 at 9:25:27 AM, Fabian Hueske (fhue...@gmail

Advice or best practices on adding metadata to stream events

Hi everyone, I am fairly new to the world of stream processing and I was wondering about best practices when needing to add metadata to a stream in Flink (or stream processing in general). Searching for examples/discussions of this topic did not yield the results I was hoping for, so I figured

Re: Flink CEP exception during RocksDB update

Hi again Varun, I am investigating the problem you mentioned and I found a bug in the SharedBuffer, but I am not sure if it is the only bug that affects you. Could you please try this branch https://github.com/kl0u/flink/tree/cep-inv and let me know

Re: ElasticsearchSink in Flink 1.4.0 with Elasticsearch 5.2+

Hi Christophe, great! Thanks for your contribution. I'm quite busy right now, but I agree that we should have support for ES 5.3 and Es 6.x for the next minor release 1.5. Best, Fabian 2018-01-26 23:09 GMT+01:00 Christophe Jolif : > Ok, I got it "done". I have a PR for ES5.3 (FLINK-7386) just

Re: AKA and quarantine

Hi Vishal, sorry for the late response. Till (in CC) might be able to answer your Akka / coordination related questions. Best, Fabian 2018-01-24 1:22 GMT+01:00 Vishal Santoshi : > Any suggestions ? I know these are very general issue but these are edge > conditions that we want the community t