Re: Flink 0.9.1 Kafka 0.8.1

2015-09-10 Thread Robert Metzger
Good news: I was able to reproduce the issue and there is already a fix: https://github.com/apache/flink/pull/1117 I'm wondering how the offset became out of range. Was this caused by Flink or did you change the offset in Zookeeper with an external tool? I'm just asking to make sure that Flink is

Re: Flink 0.9.1 Kafka 0.8.1

2015-09-10 Thread Ufuk Celebi
Thanks for reporting the issue. I think this warrants a 0.9.2 release after the fix is in. – Ufuk > On 10 Sep 2015, at 16:52, Gwenhael Pasquiers > wrote: > > Thanks, > > In the mean time we’ll go back to 0.9.0 J > > From: Robert Metzger [mailto:rmetz...@apache.org] > Sent: jeudi 10 septe

RE: Flink 0.9.1 Kafka 0.8.1

2015-09-10 Thread Gwenhael Pasquiers
Thanks, In the mean time we’ll go back to 0.9.0 ☺ From: Robert Metzger [mailto:rmetz...@apache.org] Sent: jeudi 10 septembre 2015 16:49 To: user@flink.apache.org Subject: Re: Flink 0.9.1 Kafka 0.8.1 Hi Gwen, sorry that you ran into this issue. The implementation of the Kafka Consumer has been

Re: Flink 0.9.1 Kafka 0.8.1

2015-09-10 Thread Robert Metzger
Hi Gwen, sorry that you ran into this issue. The implementation of the Kafka Consumer has been changed completely in 0.9.1 because there were some corner-case issues with the exactly-once guarantees in 0.9.0. I'll look into the issue immediately. On Thu, Sep 10, 2015 at 4:26 PM, Gwenhael Pasqui

Flink 0.9.1 Kafka 0.8.1

2015-09-10 Thread Gwenhael Pasquiers
Hi everyone, We're trying to use consume a 0.8.1 Kafka on Flink 0.9.1 and we've run into the following issue : My offset became OutOfRange however now when I start my job, it loops on the OutOfRangeException, no matter what the value of auto.offset.reset is... (earliest, latest, largest, small

Re: The main method caused an error. HDFS flink

2015-09-10 Thread Robert Metzger
@Florian: There is now a pull request which is probably fixing the issue: https://github.com/apache/flink/pull/1114 So if you want to can build a custom Flink version based on the pull request. On Tue, Sep 8, 2015 at 9:26 PM, Stephan Ewen wrote: > We'll try to fix the issue for the upcoming mile

Re: Flink HA mode

2015-09-10 Thread Ufuk Celebi
Hey Emmanuel, as Fabian said it's currently under development in 0.10-SNAPSHOT. It doesn't recover jobs yet (wait a few more days for this), but the infrastructure for leader election, starting ZooKeeper, etc. is in place. Comments are very welcome if you find time to play around with it. If you