Re: Working with Avro Generic Records in the interactive scala shell

2014-05-27 Thread Jeremy Lewi
variable SPARK_JAVA_OPTS might be better than shutting down and restarting the spark context in the console. J On Sat, May 24, 2014 at 11:16 AM, Jeremy Lewi jer...@lewi.us wrote: Hi Josh, Thanks for the help. The class should be on the path on all nodes. Here's what I did: 1) I built a jar from

Re: Working with Avro Generic Records in the interactive scala shell

2014-05-27 Thread Jeremy Lewi
/in/mattmassie/ Massiehttp://www.twitter.com/matt_massie UC, Berkeley AMPLab https://twitter.com/amplab On Tue, May 27, 2014 at 7:18 AM, Jeremy Lewi jer...@lewi.us wrote: I was able to work around this by switching to the SpecificDatum interface and following this example: https

Re: Working with Avro Generic Records in the interactive scala shell

2014-05-24 Thread Jeremy Lewi
(with its dependencies) and including the path to that jar on the command line in an environment variable (e.g. SPARK_CLASSPATH=path ./spark-shell)? --j On Saturday, May 24, 2014, Jeremy Lewi jer...@lewi.us wrote: Hi Spark Users, I'm trying to read and process an Avro dataset using the interactive

Working with Avro Generic Records in the interactive scala shell

2014-05-23 Thread Jeremy Lewi
Hi Spark Users, I'm trying to read and process an Avro dataset using the interactive spark scala shell. When my pipeline executes I get the ClassNotFoundException pasted at the end of this email. I'm trying to use the Generic Avro API (not the Specific API). Here's a gist of the commands I'm