Re: [DISCUSS] Enhance Window Evictor in Flink

My thought process when asking if we can use state backend in window function was : can we add the elements to be evicted into some state and allow the evictAfter to read it from some context and remove it from the window? On Fri, Jul 8, 2016 at 7:30 PM, Vishnu Viswanath <

Re: [DISCUSS] Enhance Window Evictor in Flink

Hi Aljoscha, Thanks for the explanation, and sorry for late reply was busy with work. I did think about this scenario, in fact in my previous mail I thought of posting this question, then I understood that this problem will be there which ever method we choose(Trigger looking for pattern or

[jira] [Created] (FLINK-4185) Reflecting rename from Tachyon to Alluxio

Jiri Simsa created FLINK-4185: - Summary: Reflecting rename from Tachyon to Alluxio Key: FLINK-4185 URL: https://issues.apache.org/jira/browse/FLINK-4185 Project: Flink Issue Type: Improvement

Re: [DISCUSS] FLIP 1 - Flink Improvement Proposal

I was planning to ask people to do that once the process is in place, yes. Wouldn't it be "forward porting", though? :-) On Fri, 8 Jul 2016 at 16:38 Robert Metzger wrote: > What about "backporting" some of the currently ongoing FLIPs to the new > process? > In particular

Re: Contribute to Flink

Hi Kevin, welcome to the Flink community! Did you have a look into the web-page: https://flink.apache.org/how-to-contribute.html If you have follow-up question, just go for it -Matthias On 07/08/2016 09:12 AM, Kevin wrote: > Hi, > > I am relatively new to the development process of Apache

Re: [DISCUSS] FLIP 1 - Flink Improvement Proposal

What about "backporting" some of the currently ongoing FLIPs to the new process? In particular those which are not yet fully implemented (Security, Mesos, Key Groups, ...) ? On Fri, Jul 8, 2016 at 4:21 PM, Aljoscha Krettek wrote: > Of course, this would make the list look

[jira] [Created] (FLINK-4184) Ganglia and GraphiteReporter report metric names with invalid characters

Till Rohrmann created FLINK-4184: Summary: Ganglia and GraphiteReporter report metric names with invalid characters Key: FLINK-4184 URL: https://issues.apache.org/jira/browse/FLINK-4184 Project:

[jira] [Created] (FLINK-4183) Move checking for StreamTableEnvironment into validation layer

Timo Walther created FLINK-4183: --- Summary: Move checking for StreamTableEnvironment into validation layer Key: FLINK-4183 URL: https://issues.apache.org/jira/browse/FLINK-4183 Project: Flink

Re: [DISCUSS] Enhance Window Evictor in Flink

Hi Vishnu, how long would these patterns be? The Trigger would not have to sort the elements for every new element but just insert the new element into an internal data structure. Only when it sees that the watermark is past a certain point would it check whether the pattern matches and actually

Re: Evaluating Apache Flink

Hey Kevin, Your questions regarding the future of this space are so difficult to answer that investors are betting millions of dollars on many possible outcomes. You should consult them. :) When you are choosing a system your best bet is current capabilities, the roadmap and the community of the

Re: [DISCUSS] FLIP 1 - Flink Improvement Proposal

Of course, this would make the list look like this: All of the following are public interfaces that people build around: - DataStream and DataSet API, including classes related to that, such as StreamExecutionEnvironment - Classes marked with the @Public annotation - On-disk binary formats,

[jira] [Created] (FLINK-4181) Add a basic streaming Table API example

Timo Walther created FLINK-4181: --- Summary: Add a basic streaming Table API example Key: FLINK-4181 URL: https://issues.apache.org/jira/browse/FLINK-4181 Project: Flink Issue Type: New Feature

[jira] [Created] (FLINK-4180) Create a batch SQL example

Timo Walther created FLINK-4180: --- Summary: Create a batch SQL example Key: FLINK-4180 URL: https://issues.apache.org/jira/browse/FLINK-4180 Project: Flink Issue Type: New Feature

Contribute to Flink

Hi, I am relatively new to the development process of Apache Flink. Where can I start to help you developing Flink? Kind regards, Kevin

[jira] [Created] (FLINK-4179) Update TPCHQuery3Table example

Timo Walther created FLINK-4179: --- Summary: Update TPCHQuery3Table example Key: FLINK-4179 URL: https://issues.apache.org/jira/browse/FLINK-4179 Project: Flink Issue Type: Improvement

Re: [DISCUSS] FLIP 1 - Flink Improvement Proposal

What about the scripts? On Fri, Jul 8, 2016 at 2:47 PM, Aljoscha Krettek wrote: > Thanks a lot, Matthias! I created an actual "FLIP-Template" based on the > template you copied over. > > I propose to change the list of public interfaces to this: > > All of the following are

Re: Evaluating Apache Flink

Hi Marton, Thank you for your elaborate answer. I will comment in your e-mail below: On 08.07.2016 15:13, Márton Balassi wrote: Hi Kevin, Thanks for being willing to contribute such an effort. I think it is a completely valid discussion to ask in your organization and please feel free to ask

Re: Evaluating Apache Flink

Hi Kevin, Thanks for being willing to contribute such an effort. I think it is a completely valid discussion to ask in your organization and please feel free to ask us questions during your evaluation. Putting statements on the Flink website highlighting the differences would be very tricky

[DISCUSS] Move Documentation from Wiki to Documentation

Hi, I recently was checking out the wiki because I want to add a page for FLIPs (Flink Improvement Proposals, see that other mail thread.). I noticed that we have some documentation there that IMHO should reside in the proper Flink documentation. Specifically, I'm talking about these: -

Re: [DISCUSS] FLIP 1 - Flink Improvement Proposal

Thanks a lot, Matthias! I created an actual "FLIP-Template" based on the template you copied over. I propose to change the list of public interfaces to this: All of the following are public interfaces that people build around: - DataStream and DataSet API, including classes related to that,

Evaluating Apache Flink

Hi, I am currently working working for an organization which is using Apache Spark as main data processing framework. Now the organization is wondering whether Apache Flink is better at processing their data than Apache Spark. Therefore, I am evaluating Apache Flink and I am comparing it to

Re: [Discuss] Read from S3 using AvroInputFormat

I was not aware that there is separate mailing list for discussions. >From next time I will use that mailing list for my queries. Thanks Till for that information. Regards, Vinay Patil On Fri, Jul 8, 2016 at 5:38 PM, Till Rohrmann wrote: > Hi Vinay, > > have you tried

Re: [DISCUSS] Enhance Window Evictor in Flink

Hi Aljoscha, That is a good idea, trying to tie it back to the use case, e.g., suppose trigger is looking for a pattern, a-b-a and when it sees such a pattern, it will trigger the window and it knows that now the Evictor is going to evict the element b, and trigger updates its state as a-a (even

Re: [Discuss] Read from S3 using AvroInputFormat

Hi Vinay, have you tried only specifying the directory path you want to read [1]? I think in the future these kind of questions should be posted to the user mailing list u...@flink.apache.org. The dev mailing list should be used for discussions about Flink's development. [1]

[Discuss] Read from S3 using AvroInputFormat

Hi, I am able to read avro file stored in S3 using AvroInputFormat by providing the full path. Is there a way I can read all the folders inside the bucket using existing Flink API. I know we can do this using the S3 REST Service, but I am more interested to do it in Flink (may be using Flink

Re: [DISCUSS] FLIP 1 - Flink Improvement Proposal

Done. I replaced KIP -> FLIP and Kafka -> Flink. Still there are some things we need to rework. Please have a look and propose changes. https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals Same for FLIP Template

Re: Contributing

Thank you for your answer! I will take a look at it :-). Regards, Kevin On 08.07.2016 13:18, Matthias J. Sax wrote: Done. I replaced KIP -> FLIP and Kafka -> Flink. Still there are some things we need to rework. Please have a look and propose changes.

Re: Contributing

Done. I replaced KIP -> FLIP and Kafka -> Flink. Still there are some things we need to rework. Please have a look and propose changes. https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals Same for FLIP Template

[jira] [Created] (FLINK-4178) CoStreamCheckpointingITCase deadlocks

Robert Metzger created FLINK-4178: - Summary: CoStreamCheckpointingITCase deadlocks Key: FLINK-4178 URL: https://issues.apache.org/jira/browse/FLINK-4178 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-4177) CassandraConnectorTest.testCassandraCommitter causing unstable builds

Robert Metzger created FLINK-4177: - Summary: CassandraConnectorTest.testCassandraCommitter causing unstable builds Key: FLINK-4177 URL: https://issues.apache.org/jira/browse/FLINK-4177 Project: Flink

Re: [DISCUSS] Enhance Window Evictor in Flink

Hi, come to think of it, the right place to put such checks is actually the Trigger. It would have to be a custom trigger that observes time but also keeps some internal state machine to decide when it has observed the right pattern in the window. Then the window function would just have to do the

Re: Contributing

Hi Kevin, welcome to the Flink community! Did you have a look into the web-page: https://flink.apache.org/how-to-contribute.html If you have follow-up question, just go for it :) -Matthias On 07/08/2016 10:02 AM, Kevin Jacobs wrote: > Hi, > > I am relatively new to the development process of

[jira] [Created] (FLINK-4176) Travis build fails at flink-connector-kinesis for JDK: openjdk 7

Tzu-Li (Gordon) Tai created FLINK-4176: -- Summary: Travis build fails at flink-connector-kinesis for JDK: openjdk 7 Key: FLINK-4176 URL: https://issues.apache.org/jira/browse/FLINK-4176 Project:

Contributing

Hi, I am relatively new to the development process of Apache Flink. Where can I start to help you developing Flink? Kind regards, Kevin

Re: [DISCUSS] Allowed Lateness in Flink

@Chen I added a section at the end of the document regarding access to the elements that are dropped as late. Right now, the section just mentions that we have to do this but there is no real proposal yet for how to do it. Only a rough sketch so that we don't forget about it. On Fri, 8 Jul 2016

Re: [DISCUSS] FLIP 1 - Flink Improvement Proposal

Hi, I got this reply from one of the Kafka committers: Thanks for sharing your intention to use a process similar to our KIP process. You are more than welcome to copy the structures and docs that we have for the KIP process. :) Ismael So it seems we're good to go. @Matthias, since you are a