Thanks Michael, you are correct. I also opened https://issues.apache.org/jira/browse/SPARK-4702 -- if someone can comment on why this might be happening that would be great. This would be a blocker to me using 1.2 and it used to work so I'm a bit puzzled. I was hoping that it's again a result of the default profile switch but it didn't seem to be the case
(ps. please advise if this is more user-list appropriate. I'm posting to dev as it's an RC) On Tue, Dec 2, 2014 at 8:37 PM, Michael Armbrust <mich...@databricks.com> wrote: > In Hive 13 (which is the default for Spark 1.2), parquet is included and > thus we no longer include the Hive parquet bundle. You can now use the > included > ParquetSerDe: org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe > > If you want to compile Spark 1.2 with Hive 12 instead you can pass > -Phive-0.12.0 and parquet.hive.serde.ParquetHiveSerDe will be included as > before. > > Michael > > On Tue, Dec 2, 2014 at 9:31 AM, Yana Kadiyska <yana.kadiy...@gmail.com> > wrote: > >> Apologies if people get this more than once -- I sent mail to dev@spark >> last night and don't see it in the archives. Trying the incubator list >> now...wanted to make sure it doesn't get lost in case it's a bug... >> >> ---------- Forwarded message ---------- >> From: Yana Kadiyska <yana.kadiy...@gmail.com> >> Date: Mon, Dec 1, 2014 at 8:10 PM >> Subject: [Thrift,1.2 RC] what happened to >> parquet.hive.serde.ParquetHiveSerDe >> To: dev@spark.apache.org >> >> >> Hi all, apologies if this is not a question for the dev list -- figured >> User list might not be appropriate since I'm having trouble with the RC >> tag. >> >> I just tried deploying the RC and running ThriftServer. I see the >> following >> error: >> >> 14/12/01 21:31:42 ERROR UserGroupInformation: PriviledgedActionException >> as:anonymous (auth:SIMPLE) >> cause:org.apache.hive.service.cli.HiveSQLException: >> java.lang.RuntimeException: >> MetaException(message:java.lang.ClassNotFoundException Class >> parquet.hive.serde.ParquetHiveSerDe not found) >> 14/12/01 21:31:42 WARN ThriftCLIService: Error executing statement: >> org.apache.hive.service.cli.HiveSQLException: java.lang.RuntimeException: >> MetaException(message:java.lang.ClassNotFoundException Class >> parquet.hive.serde.ParquetHiveSerDe not found) >> at >> >> org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.run(Shim13.scala:192) >> at >> >> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:231) >> at >> >> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:212) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >> at >> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:606) >> at >> >> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:79) >> at >> >> org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:37) >> at >> >> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:64) >> at java.security.AccessController.doPrivileged(Native Method) >> at javax.security.auth.Subject.doAs(Subject.java:415) >> >> >> >> I looked at a working installation that I have(build master a few weeks >> ago) and this class used to be included in spark-assembly: >> >> ls *.jar|xargs grep parquet.hive.serde.ParquetHiveSerDe >> Binary file spark-assembly-1.2.0-SNAPSHOT-hadoop2.0.0-mr1-cdh4.2.0.jar >> matches >> >> but with the RC build it's not there? >> >> I tried both the prebuilt CDH drop and later manually built the tag with >> the following command: >> >> ./make-distribution.sh --tgz -Phive -Dhadoop.version=2.0.0-mr1-cdh4.2.0 >> -Phive-thriftserver >> $JAVA_HOME/bin/jar -tvf spark-assembly-1.2.0-hadoop2.0.0-mr1-cdh4.2.0.jar >> |grep parquet.hive.serde.ParquetHiveSerDe >> >> comes back empty... >> > >
