[jira] [Created] (FLINK-4267) Adding async I/O flat map

2016-07-26 Thread Till Rohrmann (JIRA)
Till Rohrmann created FLINK-4267: Summary: Adding async I/O flat map Key: FLINK-4267 URL: https://issues.apache.org/jira/browse/FLINK-4267 Project: Flink Issue Type: New Feature Com

Re: [DISCUSS][FLIP-4] Enhance Window Evictor in Flink

2016-07-26 Thread Vishnu Viswanath
Hi Aljoscha, Regarding your concern - to not expose the StreamRecord in the Evictor, were you able to find any alternative? I tried to make the methods take Iterable input similar to the WindowFunction, but that didn't work since we have to clear the state and add the elements back to the state

Re: Flink 1.1.0 Preview RC0

2016-07-26 Thread Ufuk Celebi
Everything has been merged and I've now created the release-1.1 branch from commit 12bf7c1. I will create the RC1 asap and start the vote soon. Thanks to everyone who was involved in fixing and reporting issues! :-) On Tue, Jul 26, 2016 at 4:32 PM, Ufuk Celebi wrote: > Very happy to hear this. I

Re: Flink 1.1.0 Preview RC0

2016-07-26 Thread Ufuk Celebi
Very happy to hear this. I will create the RC and release branch later today after both PRs have been merged. :-) On Tue, Jul 26, 2016 at 4:21 PM, Aljoscha Krettek wrote: > I'm running tests locally now because of the Travis problems. Will merge as > soon as they pass for both the window slowness

Re: Flink 1.1.0 Preview RC0

2016-07-26 Thread Aljoscha Krettek
I'm running tests locally now because of the Travis problems. Will merge as soon as they pass for both the window slowness fix and the multiple metrics reporters change. On Tue, 26 Jul 2016 at 14:47 Maximilian Michels wrote: > Yes, I'm done with the investigation for > https://github.com/apache/

Re: Flink 1.1.0 Preview RC0

2016-07-26 Thread Maximilian Michels
Yes, I'm done with the investigation for https://github.com/apache/flink/pull/2257. Merging before 6 pm CET. On Tue, Jul 26, 2016 at 10:52 AM, Ufuk Celebi wrote: > The following PRs are still open. I will create the RC as soon as they > are merged. In any case I will fork off the release-1.1 bran

Re: [DISCUSS] Allowed Lateness in Flink

2016-07-26 Thread Kostas Kloudas
And also I think that the shouldFire has to take as an additional argument the time. This will help differentiate between ON_TIME and EARLY, LATE firings. > On Jul 26, 2016, at 11:02 AM, Kostas Kloudas > wrote: > > Hello, > > This is a nice proposal that I think covers most of the cases. > The

Re: Question about Apache Flink Use Case

2016-07-26 Thread Kostas Kloudas
Hi Suma Cherukuri, I also replied to your question in the dev list, but I repeat the answer here just in case you missed in. From what I understand you have many small files and you want to aggregate them into bigger ones containing the logs of the last 24h. As Max said RollingSinks will allow

Re: Question about Apache Flink Use Case

2016-07-26 Thread Kostas Kloudas
Hi Suma Cherukuri, From what I understand you have many small files and you want to aggregate them into bigger ones containing the logs of the last 24h. As Max said RollingSinks will allow you to have exactly-once semantics when writing your aggregated results to your FS. As far as reading your

Re: Question about Apache Flink Use Case

2016-07-26 Thread Maximilian Michels
Hi Suma Cherukuri, Apache Flink can certainly serve your use case very well. Here's why: 1) Apache Flink has a connectors for Kafka and ElasticSearch. It supports reading and writing to the S3 file system. 2) Apache Flink includes a RollingSink which splits up data into files with a configurable

Re: [DISCUSS] Allowed Lateness in Flink

2016-07-26 Thread Kostas Kloudas
Hello, This is a nice proposal that I think covers most of the cases. The only thing that is missing would be a method: void onFire(Window window, TriggerContext ctx) that will be responsible for doing whatever is necessary if the windowOperator decides to fire. You can imagine resetting the c

Re: Flink 1.1.0 Preview RC0

2016-07-26 Thread Ufuk Celebi
The following PRs are still open. I will create the RC as soon as they are merged. In any case I will fork off the release-1.1 branch by the end of today (CET 6:00 PM), so that we are not blocking regular Flink development. - https://github.com/apache/flink/pull/2257: Allow re-registration of TMs