What Spark and Hadoop versions are you on? I have it working in my Spark
app with the parquet-hive-bundle-1.5.0.jar bundled into my app fat-jar.
I¹m running Spark 1.0.2 and CDH5.
bin/spark-shell --master local[*] --driver-class-path
~/parquet-hive-bundle-1.5.0.jar
To see if that works?
On 8/26/14, 6:03 PM, "lyc" <yanchen....@huawei.com> wrote:
>Hi Silvio,
>
>I re-downloaded hive-0.12-bin and reset the related path in spark-env.sh.
>However, I still got some error. Do you happen to know any step I did
>wrong?
>Thank you!
>
>My detailed step is as follows:
>#enter spark-shell (successful)
>/bin/spark-shell --master spark://S4:7077 --jars
>/home/hduser/parquet-hive-bundle-1.5.0.jar
>#import related hiveContext (successful)
>...
># create parquet table:
>hql("CREATE TABLE parquet_test (id int, str string, mp MAP<STRING,STRING>,
>lst ARRAY<STRING>, strct STRUCT<A:STRING,B:STRING>) PARTITIONED BY (part
>string) ROW FORMAT SERDE 'parquet.hive.serde.ParquetHiveSerDe' STORED AS
>INPUTFORMAT 'parquet.hive.DeprecatedParquetInputFormat' OUTPUTFORMAT
>'parquet.hive.DeprecatedParquetOutputFormat'")
>
>get error:
>
>14/08/26 21:59:20 ERROR exec.DDLTask: java.lang.NoClassDefFoundError:
>Could
>not initialize class
>org.apache.hadoop.hive.ql.io.parquet.serde.primitive.ParquetPrimitiveInspe
>ctorFactory
> at
>org.apache.hadoop.hive.ql.io.parquet.serde.ArrayWritableObjectInspector.ge
>tObjectInspector(ArrayWritableObjectInspector.java:77)
> at
>org.apache.hadoop.hive.ql.io.parquet.serde.ArrayWritableObjectInspector.<i
>nit>(ArrayWritableObjectInspector.java:59)
> at
>org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe.initialize(Par
>quetHiveSerDe.java:113)
> at
>org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreU
>tils.java:218)
> at
>org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Tabl
>e.java:272)
> at
>org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:265)
> at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:597)
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:576)
> at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:3661)
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:252)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151)
> at
>org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65
>)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1414)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1192)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1020)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888)
> at org.apache.spark.sql.hive.HiveContext.runHive(HiveContext.scala:186)
> at
>org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:160)
> at
>org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd$lzycompute(Hive
>Context.scala:250)
> at
>org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd(HiveContext.sca
>la:247)
> at org.apache.spark.sql.hive.HiveContext.hiveql(HiveContext.scala:85)
> at org.apache.spark.sql.hive.HiveContext.hql(HiveContext.scala:90)
> at $line34.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:18)
> at $line34.$read$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:23)
> at $line34.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:25)
> at $line34.$read$$iwC$$iwC$$iwC.<init>(<console>:27)
> at $line34.$read$$iwC$$iwC.<init>(<console>:29)
> at $line34.$read$$iwC.<init>(<console>:31)
> at $line34.$read.<init>(<console>:33)
> at $line34.$read$.<init>(<console>:37)
> at $line34.$read$.<clinit>(<console>)
> at $line34.$eval$.<init>(<console>:7)
> at $line34.$eval$.<clinit>(<console>)
> at $line34.$eval.$print(<console>)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
>57)
> at
>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
>pl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
>org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:788)
> at
>org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1056)
> at
> org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:614)
> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:645)
> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:609)
> at
>org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:796)
> at
>org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:84
>1)
> at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:753)
> at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:601)
> at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:608)
> at org.apache.spark.repl.SparkILoop.loop(SparkILoop.scala:611)
> at
>org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply$mcZ$sp(SparkILoo
>p.scala:936)
> at
>org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala
>:884)
> at
>org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala
>:884)
> at
>scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoade
>r.scala:135)
> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:884)
> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:982)
> at org.apache.spark.repl.Main$.main(Main.scala:31)
> at org.apache.spark.repl.Main.main(Main.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
>57)
> at
>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
>pl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:292)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
>14/08/26 21:59:20 INFO ql.Driver: </PERFLOG method=task.DDL.Stage-0
>start=1409090360562 end=1409090360564 duration=2>
>14/08/26 21:59:20 ERROR ql.Driver: FAILED: Execution Error, return code 1
>from org.apache.hadoop.hive.ql.exec.DDLTask. Could not initialize class
>org.apache.hadoop.hive.ql.io.parquet.serde.primitive.ParquetPrimitiveInspe
>ctorFactory
>14/08/26 21:59:20 INFO ql.Driver: </PERFLOG method=Driver.execute
>start=1409090360561 end=1409090360564 duration=3>
>14/08/26 21:59:20 INFO ql.Driver: <PERFLOG method=releaseLocks>
>14/08/26 21:59:20 INFO ql.Driver: </PERFLOG method=releaseLocks
>start=1409090360564 end=1409090360564 duration=0>
>14/08/26 21:59:20 INFO ql.Driver: <PERFLOG method=releaseLocks>
>14/08/26 21:59:20 INFO ql.Driver: </PERFLOG method=releaseLocks
>start=1409090360564 end=1409090360564 duration=0>
>14/08/26 21:59:20 ERROR hive.HiveContext:
>======================
>HIVE FAILURE OUTPUT
>======================
>OK
>OK
>OK
>OK
>OK
>OK
>OK
>OK
>OK
>OK
>OK
>OK
>OK
>OK
>OK
>OK
>OK
>FAILED: Execution Error, return code 1 from
>org.apache.hadoop.hive.ql.exec.DDLTask. tried to access method
>org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaStringObjectIn
>spector.<init>()V
>from class
>org.apache.hadoop.hive.serde2.objectinspector.primitive.ParquetStringInspe
>ctor
>FAILED: Execution Error, return code 1 from
>org.apache.hadoop.hive.ql.exec.DDLTask. Could not initialize class
>org.apache.hadoop.hive.ql.io.parquet.serde.primitive.ParquetPrimitiveInspe
>ctorFactory
>
>======================
>END HIVE FAILURE OUTPUT
>======================
>
>org.apache.spark.sql.execution.QueryExecutionException: FAILED: Execution
>Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Could
>not
>initialize class
>org.apache.hadoop.hive.ql.io.parquet.serde.primitive.ParquetPrimitiveInspe
>ctorFactory
> at org.apache.spark.sql.hive.HiveContext.runHive(HiveContext.scala:190)
> at
>org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:160)
> at
>org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd$lzycompute(Hive
>Context.scala:250)
> at
>org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd(HiveContext.sca
>la:247)
> at org.apache.spark.sql.hive.HiveContext.hiveql(HiveContext.scala:85)
> at org.apache.spark.sql.hive.HiveContext.hql(HiveContext.scala:90)
> at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:18)
> at $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:23)
> at $iwC$$iwC$$iwC$$iwC.<init>(<console>:25)
> at $iwC$$iwC$$iwC.<init>(<console>:27)
> at $iwC$$iwC.<init>(<console>:29)
> at $iwC.<init>(<console>:31)
> at <init>(<console>:33)
> at .<init>(<console>:37)
> at .<clinit>(<console>)
> at .<init>(<console>:7)
> at .<clinit>(<console>)
> at $print(<console>)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
>57)
> at
>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
>pl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
>org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:788)
> at
>org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1056)
> at
> org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:614)
> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:645)
> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:609)
> at
>org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:796)
> at
>org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:84
>1)
> at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:753)
> at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:601)
> at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:608)
> at org.apache.spark.repl.SparkILoop.loop(SparkILoop.scala:611)
> at
>org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply$mcZ$sp(SparkILoo
>p.scala:936)
> at
>org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala
>:884)
> at
>org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala
>:884)
> at
>scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoade
>r.scala:135)
> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:884)
> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:982)
> at org.apache.spark.repl.Main$.main(Main.scala:31)
> at org.apache.spark.repl.Main.main(Main.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
>57)
> at
>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
>pl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:292)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
>
>
>
>
>--
>View this message in context:
>http://apache-spark-user-list.1001560.n3.nabble.com/Does-HiveContext-suppo
>rt-Parquet-tp12209p12858.html
>Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>For additional commands, e-mail: user-h...@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
