I have used protobuf 3 successfully with Spark on CDH 5.4, even though Hadoop itself comes with protobuf 2.5. I think the steps apply to HDP too. You need to do the following
1. Set the below parameter spark.executor.userClassPathFirst=true spark.driver.userClassPathFirst=true 2. Include protobuf 3 jar file either through —jars during the spark-submit or package it into a uber jar file with your own classes. Lan > On Nov 4, 2015, at 4:07 PM, Cassa L <lcas...@gmail.com> wrote: > > Hi, > Does spark support protobuff 3.0? I used protobuff 2.5 with spark-1.4 built > for HDP 2.3. Given that protobuff has compatibility issues , want to know if > spark supports protbuff 3.0 > > LCassa --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
