Re: Junit Issue while testing Kafka Source

Yeah understood. Thank you for helping guys. Regards, Vinay Patil *+91-800-728-4749* On Thu, May 26, 2016 at 5:40 PM, Stephan Ewen wrote: > The SuccessException does not really have a dependency. > > It is just a special Exception class that you throw in your code when you > want to stop. > Th

Iteration Intermediate Output

Hi y'all, I think this is an oft-requested feature [0] and there are many graph algorithms for which intermediate output is the desired result. I'd like to take Stephan up on his offer [1] for pointers. I have yet to get in deep, but I see that iteration tasks are treated specially as IterationIn

[jira] [Created] (FLINK-3980) Remove ExecutionConfig.PARALLELISM_UNKNOWN

Greg Hogan created FLINK-3980: - Summary: Remove ExecutionConfig.PARALLELISM_UNKNOWN Key: FLINK-3980 URL: https://issues.apache.org/jira/browse/FLINK-3980 Project: Flink Issue Type: Improvement

Re: buffering in operators, implementing statistics

Hi Stavros! I think what Aljoscha wants to say is that the community is a bit hard pressed reviewing new and complex things right now. There are a lot of threads going on already. If you want to work on this, why not make your own GitHub project "Approximate algorithms on Apache Flink" or so? Gr

[jira] [Created] (FLINK-3979) [documentation]add missed import classes in run_example_quickstart

Jia Zhai created FLINK-3979: --- Summary: [documentation]add missed import classes in run_example_quickstart Key: FLINK-3979 URL: https://issues.apache.org/jira/browse/FLINK-3979 Project: Flink Issue

Re: Error related to JMX and metrics

Ok, there is an OutOfMemory error apparently unrelated to the one reported here that made the job fail. 2016-05-25 18:36 GMT+02:00 Simone Robutti : > There were many errors after these (null pointers among them) so I assumed > they were a consequence. If you say this shouldn't be the primary caus

[jira] [Created] (FLINK-3978) Add contains methods to RuntimeContext

Greg Hogan created FLINK-3978: - Summary: Add contains methods to RuntimeContext Key: FLINK-3978 URL: https://issues.apache.org/jira/browse/FLINK-3978 Project: Flink Issue Type: Improvement

Re: Motivation for Delta Iterations

Thanks Till and Slim. Trevor Grant Data Scientist https://github.com/rawkintrevo http://stackexchange.com/users/3002022/rawkintrevo http://trevorgrant.org *"Fortunate is he, who is able to know the causes of things." -Virgil* On Thu, May 26, 2016 at 3:10 AM, Slim Baltagi wrote: > Hi Trevor >

Re: Junit Issue while testing Kafka Source

The SuccessException does not really have a dependency. It is just a special Exception class that you throw in your code when you want to stop. The code that calls "env.execute()" catches the exception and checks whether the failure cause is that special exceptions. Flink propagates the exceptions

Re: Junit Issue while testing Kafka Source

Hi Stephan, Yes using DeserializationSchema solution will definitely work. I am not able to get the dependency for SuccessException. Any help on this Regards, Vinay Patil *+91-800-728-4749* On Thu, May 26, 2016 at 3:32 PM, Stephan Ewen wrote: > Hi! > > On Flink 1.0, there is the "flink-test-u

Re: Junit Issue while testing Kafka Source

Hi! On Flink 1.0, there is the "flink-test-utils_2.10" dependency that has a some useful things. The "SuccessException" seems a quite common thing - I have seen that in other infinite program tests as well (Google Dataflow / Beam) Another way you can architect tests is to have an element in the

Re: Junit Issue while testing Kafka Source

Hi Aljoscha, Thank you for answering. Throwing SuccessException is a good idea , however when I am adding following dependency, no classes are getting added to the jar: org.apache.flink flink-tests_2.10 1.0.3 Is there any other dependency that I have to add ? I have also added t

Re: Motivation for Delta Iterations

Hi Trevor A summary of this 12 pages academic paper: 'Spinning Fast Iterative Data Flows’ by Ewen and al published in 2012 http://vldb.org/pvldb/vol5/p1268_stephanewen_vldb2012.pdf is done by Adrian Colyer and published on June 18, 20

Re: Fold error when window evictor is set

Hi, I'm afraid this is a bug. I open a Jira Issue for it: https://issues.apache.org/jira/browse/FLINK-3977 Cheers, Aljoscha On Thu, 26 May 2016 at 09:13 Erik wrote: > Hi, > > I get a "No initial value was serialized for the fold window function. > Probably the setOutputmethod was not called" wh

[jira] [Created] (FLINK-3977) Subclasses of InternalWindowFunction must support OutputTypeConfigurable

Aljoscha Krettek created FLINK-3977: --- Summary: Subclasses of InternalWindowFunction must support OutputTypeConfigurable Key: FLINK-3977 URL: https://issues.apache.org/jira/browse/FLINK-3977 Project:

Re: Motivation for Delta Iterations

Hi Trevor, delta iterations are described in this paper [1]. [1] http://stratosphere.eu/assets/papers/spinningFastIterativeDataFlows_12.pdf Cheers, Till On Thu, May 26, 2016 at 1:57 AM, Trevor Grant wrote: > I was wondering where the idea for delta iterations came from. > > Many of the featur

Re: Junit Issue while testing Kafka Source

Hi, what we are doing in most internal tests is to verify in a sink whether the data is correct and then throw a SuccessException. This brings down the job and we check whether we catch a SuccessException to verify that the test was successful. Look, for example, at the ValidatingSink in EventTimeW

Fold error when window evictor is set

Hi, I get a "No initial value was serialized for the fold window function. Probably the setOutputmethod was not called" when I use the fold function on a WindowedStream where an evictor has been set. If no evictor is set, it works fine. There was a similair issue