I am happy to report that after set spark.dirver.userClassPathFirst, I can use protobuf 3 with spark-shell. Looks like the classloading issue in the driver, not executor.
Marcelo, thank you very much for the tip! Lan > On Sep 15, 2015, at 1:40 PM, Marcelo Vanzin <van...@cloudera.com> wrote: > > Hi, > > Just "spark.executor.userClassPathFirst" is not enough. You should > also set "spark.driver.userClassPathFirst". Also not that I don't > think this was really tested with the shell, but that should work with > regular apps started using spark-submit. > > If that doesn't work, I'd recommend shading, as others already have. > > On Tue, Sep 15, 2015 at 9:19 AM, Lan Jiang <ljia...@gmail.com> wrote: >> I used the --conf spark.files.userClassPathFirst=true in the spark-shell >> option, it still gave me the eror: java.lang.NoSuchFieldError: unknownFields >> if I use protobuf 3. >> >> The output says spark.files.userClassPathFirst is deprecated and suggest >> using spark.executor.userClassPathFirst. I tried that and it did not work >> either. > > -- > Marcelo --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
