Re: Figuring out when a job has successfully restored state

2017-03-29 Thread Till Rohrmann
Hi Gyula, there exists a related issue [1]. Fixing this issue will move the state restoration in the state DEPLOYING. This means that when you see a task being in state RUNNING, then it will have restored all of its eager state. [1] https://issues.apache.org/jira/browse/FLINK-4714 Cheers, Till

Re: deploying flink in AWS - some teething issues

2017-03-29 Thread Chakravarthy varaga
Hi, Any updates here? I'm sure many would have faced similar issues like these, any help here is highly appreciated. Best Regards CVP On Tue, Mar 28, 2017 at 5:47 PM, Chakravarthy varaga < chakravarth...@gmail.com> wrote: > Hi Team, > >If the flink cluster is conainerized and managed th

Re: Figuring out when a job has successfully restored state

2017-03-29 Thread Gyula Fóra
Thanks Till, This is exactly what I was looking for :) Gyula Till Rohrmann ezt írta (időpont: 2017. márc. 29., Sze, 10:23): > Hi Gyula, > > there exists a related issue [1]. Fixing this issue will move the state > restoration in the state DEPLOYING. This means that when you see a task > being i

20 times higher throughput with Window function vs fold function, intended?

2017-03-29 Thread Kamil Dziublinski
Hi guys, I’m using flink on production in Mapp. We recently swapped from storm. Before I have put this live I was doing performance tests and I found something that “feels” a bit off. I have a simple streaming job reading from kafka, doing window for 3 seconds and then storing into hbase. Initial

Gelly - which partitioning

2017-03-29 Thread Kaepke, Marc
Hi guys, I can’t found on web which graph partitioning are supported by Gelly. During my search I found this link. But the ticket is still open. https://cwiki.apache.org/confluence/display/FLINK/Flink+Gelly Thanks for help! Best Marc

Re: ClassNotFoundException upon Savepoint Disposal

2017-03-29 Thread Konstantin Gregor
Hi Ufuk, hi Stefan, thanks a lot for your replies. Ufuk, we are using the HDFS state backend. Stefan, I installed 1.1.5 on our machines and built our software with the Flink 1.1.5 dependency, but the problem remains. Below are the logs for savepoint creation [1] and savepoint disposal [2] as we

Re: Fail to call jobs/:jobid/cancel-with-savepoint/ with local Flink mini cluseter started by IDE

2017-03-29 Thread Chesnay Schepler
Hello, The cancel-with-savepoint command is not available in 0.10.1 . I'm pretty sure it was added in 1.2 , so you'll have to upgrade the runtime-web dependency. Note that all Flink dependencies should always have the same version. Regards, Chesnay On 28.03.2017 18:12, Sendoh wrote: Hi Flin

Re: 20 times higher throughput with Window function vs fold function, intended?

2017-03-29 Thread Timo Walther
Hi Kamil, the performance implications might be the result of which state the underlying functions are using internally. WindowFunctions use ListState or ReducingState, fold() uses FoldingState. It also depends on the size of your state and the state backend you are using. I recommend the fol

Re: Gelly - which partitioning

2017-03-29 Thread Timo Walther
Hi Marc, maybe Greg (in CC) can help answering your question? Regards, Timo Am 29/03/17 um 11:50 schrieb Kaepke, Marc: Hi guys, I can’t found on web which graph partitioning are supported by Gelly. During my search I found this link. But the ticket is still open. https://cwiki.apache.org/co

Re: deploying flink in AWS - some teething issues

2017-03-29 Thread Patrick Lucas
For 1., I think the standard approach would be to specify from without what the heap size should be. If you want an *x* MB heap, you could set your container memory limit to 1.3 * *x* or so (to account for overhead) and set taskmanager.heap.mb: *x* in your config. The other way around—e.g. from in

Re: Gelly - which partitioning

2017-03-29 Thread Greg Hogan
Hi Marc, I’ll defer to Vasia’s comment below from FLINK-1536 as she has much more knowledge and experience with graph partitioning. This is certainly an area of interest so please let us know if you would like to contribute! The referenced list of papers is at: http://www.citeulike.org/user/v

Re: ClassNotFoundException upon Savepoint Disposal

2017-03-29 Thread Konstantin Gregor
Hi Ufuk, yes the class is part of the jar. Actually, don't worry about this issue too much, we were just wondering if we broke something. Since that does not seem to be the case, we will deal with this until we have 1.2. So no need to write a tool, but thanks for the offer :) We want to upgrade t

Re: ClassNotFoundException upon Savepoint Disposal

2017-03-29 Thread Ufuk Celebi
Thanks for providing the logs. It looks like the JARs are uploaded (modulo any bugs ;-))... could you double check that the class is actually part of the JAR and has not been moved around via jar -tf | grep EventDataRecord If everything looks good in the JAR, I could write a short tool that pri

GroupReduce is interrupted on reading large CSV files

2017-03-29 Thread Sapei, Ferry Syafei
Hallo everyone, I have a Flink batch job, which reads four CSV files. The rows in the files= will be read and grouped together. When the four CSV Files are small enough, the job can finish successfully. = However when the input files are large, the job could not successfully exec= uted an

Re: How to rewind Kafka cursors into a Flink job ?

2017-03-29 Thread Tzu-Li (Gordon) Tai
Hi Dominique, What your plan A is suggesting is that a downstream operator can provide signals to upstream operators and alter their behaviour. In general, this isn’t possible, as in a distributed streaming environment it’s hard to guarantee what records exactly will be altered by the behaviour.

Re: Apache Flink Hackathon

2017-03-29 Thread Tzu-Li (Gordon) Tai
Sounds like a cool event! Thanks for sharing this! On March 27, 2017 at 11:40:24 PM, Lior Amar (lior.a...@parallelmachines.com) wrote: Hi all, My name is Lior and I am working at Parallel Machines (a startup company located in the Silicon Valley). We are hosting a Flink Hackathon on April 10

Re: Flink 1.2 time window operation

2017-03-29 Thread Tzu-Li (Gordon) Tai
Hi Dominik, Was the job running with processing time or event time? If event time, how are you producing the watermarks? Normally to understand how windows are firing in Flink, these two factors would be the place to look at. I can try to further explain this once you provide info with these. Al

Re: Flink 1.2 time window operation

2017-03-29 Thread Dominik Safaric
Hi Gordon, The job was run using processing time. The Kafka broker version I’ve used was 0.10.1.1. Dominik > On 30 Mar 2017, at 08:35, Tzu-Li (Gordon) Tai wrote: > > Hi Dominik, > > Was the job running with processing time or event time? If event time, how > are you producing the watermark