Re: streaming using DeserializationSchema

2016-02-11 Thread Gyula Fóra
Hey, A very simple thing you could do is to set up a simple kafka producer in a java program that will feed the data into a topic. This also has the additional benefit that you are actually testing against kafka. Cheers, Gyula Martin Neumann ezt írta (időpont: 2016. febr. 12., P, 0:20): > Hej,

Re: Compile issues with Flink 1.0-SNAPSHOT and Scala 2.11

2016-02-11 Thread Stephan Ewen
Hi! The CI system has just finished uploading an new snapshot. In that one, the scalatest dependency is now correctly at 2.11 again. https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-test-utils_2.11/1.0-SNAPSHOT/flink-test-utils_2.11-1.0-20160211.232156-288.pom

Re: Compile issues with Flink 1.0-SNAPSHOT and Scala 2.11

2016-02-11 Thread Stephan Ewen
Hi! I examined the Apache Snapshot Repository, and I could see that in the latest snapshot a "scalatest_2.10" version was introduced. I could not figure out how, yet. I could not find a "flink-core_2.10" or "flink-annotations_2.10" dependency, yet. Previous snapshot: https://repository.apache.or

streaming using DeserializationSchema

2016-02-11 Thread Martin Neumann
Hej, I have a stream program reading data from Kafka where the data is in avro. I have my own DeserializationSchema to deal with it. For testing reasons I want to read a dump from hdfs instead, is there a way to use the same DeserializationSchema to read from an avro file stored on hdfs? cheers

Re: Compile issues with Flink 1.0-SNAPSHOT and Scala 2.11

2016-02-11 Thread Cory Monty
Ufuk, Thanks for the link. I've double-checked everything in our dependencies list and it's all correct. Stephan, We don't explicitly depend on "flink-java", so there should be no suffix. It's curious, to me, that scalatest is showing in the stack trace. I also tried clearing ~/.sbt/staging and

Re: Compile issues with Flink 1.0-SNAPSHOT and Scala 2.11

2016-02-11 Thread Stephan Ewen
Hi Cory! Hmmm, curios... I just double check the code, there are no more references to a Scala-versioned "flink-core" and "flink-annotations" project in the code base. The projects you use with Scala version suffix look good, actually. Just to be safe, can you check that the "flink-java" dependen

Re: Compile issues with Flink 1.0-SNAPSHOT and Scala 2.11

2016-02-11 Thread Ufuk Celebi
> On 11 Feb 2016, at 22:24, Cory Monty wrote: > > Hmm. We don't explicitly include "flink-annotations" and we do not append the > Scala suffix for "flink-core": > > `"org.apache.flink" % "flink-core" % "1.0-SNAPSHOT"` > > Here are the packages we currently include with a Scala suffix: > > fl

Re: Compile issues with Flink 1.0-SNAPSHOT and Scala 2.11

2016-02-11 Thread Cory Monty
Hmm. We don't explicitly include "flink-annotations" and we do not append the Scala suffix for "flink-core": `"org.apache.flink" % "flink-core" % "1.0-SNAPSHOT"` Here are the packages we currently include with a Scala suffix: flink-scala flink-clients flink-streaming-scala flink-connector-kafka-

Re: Compile issues with Flink 1.0-SNAPSHOT and Scala 2.11

2016-02-11 Thread Stephan Ewen
Hi Cory! "flink-core" and "flink-annotations" should not have Scala suffixes, because they do not depend on Scala. So far, we mark the Scala independent projects without suffixes. Is that very confusing, or does that interfere with build tools? Greetings, Stephan On Thu, Feb 11, 2016 at 9:50 P

Compile issues with Flink 1.0-SNAPSHOT and Scala 2.11

2016-02-11 Thread Cory Monty
As of this afternoon, SBT is running into issues compiling with the following error: [error] Modules were resolved with conflicting cross-version suffixes in [error]org.scalatest:scalatest _2.10, _2.11 [error]org.apache.flink:flink-core _2.11, [error]org.apache.flink:flink-annotations

Re: Flink 1.0-SNAPSHOT scala 2.11 in S3 has scala 2.10

2016-02-11 Thread David Kim
Thanks Stephan for following up! On Wed, Feb 10, 2016 at 10:58 AM, Stephan Ewen wrote: > We discovered yesterday that the snapshot builds were not updated in a > while (because the build server experienced timeouts). > Hence the SNAPSHOT build may have quite stale. > > It is updating frequently

ApacheCon NA 2016 - Important Dates!!!

2016-02-11 Thread Melissa Warnkin
Hello everyone! I hope this email finds you well.  I hope everyone is as excited about ApacheCon as I am! I'd like to remind you all of a couple of important dates, as well as ask for your assistance in spreading the word! Please use your social media platform(s) to get the word out! The more v

RE: Distribution of sinks among the nodes

2016-02-11 Thread Gwenhael Pasquiers
Thanks, One more thing to expect from the next version ! -Original Message- From: Aljoscha Krettek [mailto:aljos...@apache.org] Sent: lundi 8 février 2016 13:18 To: user@flink.apache.org Subject: Re: Distribution of sinks among the nodes Hi, I just merged the new feature, so once this ma

Re: job manager timeout

2016-02-11 Thread Robert Metzger
Hi Radu, did you check the JobManager logs as well? Maybe there you can see why the JobManager is failing. The timeout is configurable through the "akka.client.timeout" variable. The default value is "60 s". On Wed, Feb 10, 2016 at 7:35 PM, Radu Tudoran wrote: > Hi, > > > > I am running a prog

Re: IOException when trying flink-twitter example

2016-02-11 Thread Maximilian Michels
Hi Ram, This is an Exception thrown in the Twitter client. Unfortunately the exception is caught and not printed to the screen. I would suggest to set a breakpoint in the ClientBase to find out about the reason for the failed connection. Cheers, Max On Thu, Feb 11, 2016 at 9:30 AM, ram kumar w

Re: Simple Flink - Kafka Test

2016-02-11 Thread Robert Metzger
Quick clarification on Stephan's comment: In Flink 0.10, no suffix means scala 2.10, for Scala 2.11 you have to add the _2.11 suffix to ALL dependencies (including flink-java_2.11, flink-core_2.11 and so on). In Flink 1.0, all artifacts depending on scala have a version suffix. For example flink-cl

Re: Compilation error while instancing FlinkKafkaConsumer082

2016-02-11 Thread Robert Metzger
Hi, which build system are you using? Can you maybe post the configuration file of that build system ? (pom.xml / sbt file). I suspect that some of the dependencies are wrong. Maybe not all have the right scala version suffix or there is a version mix. On Wed, Feb 10, 2016 at 5:17 PM, Simone Rob

IOException when trying flink-twitter example

2016-02-11 Thread ram kumar
Hi, Got following exception in taskmanager logs when running TwitterFilterSourceExample, 08:25:19,207 WARN com.twitter.hbc.httpclient.ClientBase - twitterSourceClient failed to establish connection properly 08:25:19,207 INFO com.twitter.hbc.httpclient.ClientBase