A couple of issues: 1) the jar doesn't show up on the classpath even though SparkSubmit had it in the --jars options. I tested this by running > :cp in spark-shell 2) After adding it the classpath using (:cp /Users/rhoover/Work/spark-etl/target/scala-2.10/spark-etl_2.10-1.0.jar), it still fails. When I do that in the scala repl, it works.
BTW, I'm using the latest code from the master branch (8421034e793c0960373a0a1d694ce334ad36e747) On Mon, Apr 28, 2014 at 3:40 PM, Roger Hoover <roger.hoo...@gmail.com>wrote: > Matei, thank you. That seemed to work but I'm not able to import a class > from my jar. > > Using the verbose options, I can see that my jar should be included > > Parsed arguments: > ... > jars > /Users/rhoover/Work/spark-etl/target/scala-2.10/spark-etl_2.10-1.0.jar > > And I see the class I want to load in the jar: > > jar -tf > /Users/rhoover/Work/spark-etl/target/scala-2.10/spark-etl_2.10-1.0.jar | > grep IP2IncomeJob > etl/IP2IncomeJob$$anonfun$1.class > etl/IP2IncomeJob$$anonfun$4.class > etl/IP2IncomeJob$.class > etl/IP2IncomeJob$$anonfun$splitOverlappingRange$1.class > etl/IP2IncomeJob.class > etl/IP2IncomeJob$$anonfun$3.class > etl/IP2IncomeJob$$anonfun$2.class > > But the import fails > > scala> import etl.IP2IncomeJob > <console>:10: error: not found: value etl > import etl.IP2IncomeJob > > Any ideas? > > > > On Sun, Apr 27, 2014 at 3:46 PM, Matei Zaharia <matei.zaha...@gmail.com>wrote: > >> Hi Roger, >> >> You should be able to use the --jars argument of spark-shell to add JARs >> onto the classpath and then work with those classes in the shell. (A recent >> patch, https://github.com/apache/spark/pull/542, made spark-shell use >> the same command-line arguments as spark-submit). But this is a great >> question, we should test it out and see whether anything else would make >> development easier. >> >> SBT also has an interactive shell where you can run classes in your >> project, but unfortunately Spark can’t deal with closures typed directly in >> that the right way. However you write your Spark logic in a method and just >> call that method from the SBT shell, that should work. >> >> Matei >> >> On Apr 27, 2014, at 3:14 PM, Roger Hoover <roger.hoo...@gmail.com> wrote: >> >> > Hi, >> > >> > From the meetup talk about the 1.0 release, I saw that spark-submit >> will be the preferred way to launch apps going forward. >> > >> > How do you recommend launching such jobs in a development cycle? For >> example, how can I load an app that's expecting to a given to spark-submit >> into spark-shell? >> > >> > Also, can anyone recommend other tricks for rapid development? I'm new >> to Scala, sbt, etc. I think sbt can watch for changes in source files and >> compile them automatically. >> > >> > I want to be able to make code changes and quickly get into a >> spark-shell to play around with them. >> > >> > I appreciate any advice. Thanks, >> > >> > Roger >> >> >
