Dibyendu - I am using the Kafka consumer built into Spark streaming.
Pulled the jar from here:
http://search.maven.org/remotecontent?filepath=org/apache/spark/spark-streaming-kafka_2.10/1.0.0/spark-streaming-kafka_2.10-1.0.0.jar
Thanks for the sbt-assembly link, Soumitra.
On Wed, Sep 17, 2014 at
ot;
Sent: Wednesday, September 17, 2014 4:10:12 PM
Subject: Re: Stable spark streaming app
Thanks :)
On Wed, Sep 17, 2014 at 2:10 PM, Paul Wais wrote:
> Thanks Tim, this is super helpful!
>
> Question about jars and spark-submit: why do you provide
> myawesomeapp.jar as the program jar
Thanks :)
On Wed, Sep 17, 2014 at 2:10 PM, Paul Wais wrote:
> Thanks Tim, this is super helpful!
>
> Question about jars and spark-submit: why do you provide
> myawesomeapp.jar as the program jar but then include other jars via
> the --jars argument? Have you tried building one uber jar with al
Nice write-up... very helpful!
-Original Message-
From: Tim Smith [mailto:secs...@gmail.com]
Sent: Wednesday, September 17, 2014 1:11 PM
Cc: spark users
Subject: Re: Stable spark streaming app
I don't have anything in production yet but I now at least have a stable
(running for
spark/log4j.properties' for logging
. Currently I cannot process > 1G file with this configuration. I tried various
things but could not succeed yet.
- Original Message -
From: "Tim Smith"
Cc: "spark users"
Sent: Wednesday, September 17, 2014 1:11:12 PM
Subject: Re:
Thanks Tim, this is super helpful!
Question about jars and spark-submit: why do you provide
myawesomeapp.jar as the program jar but then include other jars via
the --jars argument? Have you tried building one uber jar with all
dependencies and just sending that to Spark as your app jar?
Also, h
I don't have anything in production yet but I now at least have a
stable (running for more than 24 hours) streaming app. Earlier, the
app would crash for all sorts of reasons. Caveats/setup:
- Spark 1.0.0 (I have no input flow control unlike Spark 1.1)
- Yarn for RM
- Input and Output to Kafka
- CD
Hmm, no response to this thread!
Adding to it, please share experiences of building an enterprise grade product
based on Spark Streaming.
I am exploring Spark Streaming for enterprise software and am cautiously
optimistic about it. I see huge potential to improve debuggability of Spark.
-