Yes, what Nick said is the recommended way. In most usecases, a spark streaming program in production is not usually run from the shell. Hence, we chose not to make the external stuff (twitter, kafka, etc.) available to spark shell to avoid dependency conflicts brought it by them with spark's dependencies. That said, you could tweak things at your own risk as Praveen suggested. Specifically for twitter, its quite fine I think, as the dependencies of the twitter4j library is pretty thin, and do not conflict with spark's dependencies (At least the current version).
TD On Tue, Jul 15, 2014 at 1:03 AM, Nick Pentreath <nick.pentre...@gmail.com> wrote: > You could try the following: create a minimal project using sbt or Maven, > add spark-streaming-twitter as a dependency, run sbt assembly (or mvn > package) on that to create a fat jar (with Spark as provided dependency), > and add that to the shell classpath when starting up. > > > On Tue, Jul 15, 2014 at 9:06 AM, Praveen Seluka <psel...@qubole.com> > wrote: > >> If you want to make Twitter* classes available in your shell, I believe >> you could do the following >> 1. Change the parent pom module ordering - Move external/twitter before >> assembly >> 2. In assembly/pom.xm, add external/twitter dependency - this will >> package twitter* into the assembly jar >> >> Now when spark-shell is launched, assembly jar is in classpath - hence >> twitter* too. I think this will work (remember trying this sometime back) >> >> >> On Tue, Jul 15, 2014 at 11:59 AM, Nicholas Chammas < >> nicholas.cham...@gmail.com> wrote: >> >>> Hmm, I'd like to clarify something from your comments, Tathagata. >>> >>> Going forward, is Twitter Streaming functionality not supported from the >>> shell? What should users do if they'd like to process live Tweets from the >>> shell? >>> >>> Nick >>> >>> >>> On Mon, Jul 14, 2014 at 11:50 PM, Nicholas Chammas < >>> nicholas.cham...@gmail.com> wrote: >>> >>>> At some point, you were able to access TwitterUtils from spark shell >>>>> using Spark 1.0.0+ ? >>>> >>>> >>>> Yep. >>>> >>>> >>>>> If yes, then what change in Spark caused it to not work any more? >>>> >>>> >>>> It still works for me. I was just commenting on your remark that it >>>> doesn't work through the shell, which I now understand to apply to versions >>>> of Spark before 1.0.0. >>>> >>>> Nick >>>> >>> >>> >> >
