Dibyendu - I am using the Kafka consumer built into Spark streaming.
Pulled the jar from here:
http://search.maven.org/remotecontent?filepath=org/apache/spark/spark-streaming-kafka_2.10/1.0.0/spark-streaming-kafka_2.10-1.0.0.jar
Thanks for the sbt-assembly link, Soumitra.
On Wed, Sep 17, 2014 at
Hmm, no response to this thread!
Adding to it, please share experiences of building an enterprise grade product
based on Spark Streaming.
I am exploring Spark Streaming for enterprise software and am cautiously
optimistic about it. I see huge potential to improve debuggability of Spark.
-
I don't have anything in production yet but I now at least have a
stable (running for more than 24 hours) streaming app. Earlier, the
app would crash for all sorts of reasons. Caveats/setup:
- Spark 1.0.0 (I have no input flow control unlike Spark 1.1)
- Yarn for RM
- Input and Output to Kafka
-
cannot process 1G file with this configuration. I tried various
things but could not succeed yet.
- Original Message -
From: Tim Smith secs...@gmail.com
Cc: spark users user@spark.apache.org
Sent: Wednesday, September 17, 2014 1:11:12 PM
Subject: Re: Stable spark streaming app
I don't
Nice write-up... very helpful!
-Original Message-
From: Tim Smith [mailto:secs...@gmail.com]
Sent: Wednesday, September 17, 2014 1:11 PM
Cc: spark users
Subject: Re: Stable spark streaming app
I don't have anything in production yet but I now at least have a stable
(running for more
Thanks :)
On Wed, Sep 17, 2014 at 2:10 PM, Paul Wais pw...@yelp.com wrote:
Thanks Tim, this is super helpful!
Question about jars and spark-submit: why do you provide
myawesomeapp.jar as the program jar but then include other jars via
the --jars argument? Have you tried building one uber
