We plan to upgrade our spark cluster to 1.4, and I just have a test in
local mode which reference here:
http://hortonworks.com/blog/bringing-orc-support-into-apache-spark/
but an exception caused when running the example, the stack trace as below:
*Exception in thread "main" java.lang.NoSuchFieldError: defaultVal*
at
org.apache.spark.sql.hive.HiveContext$$anonfun$newTemporaryConfiguration$1.apply(HiveContext.scala:536)
at
org.apache.spark.sql.hive.HiveContext$$anonfun$newTemporaryConfiguration$1.apply(HiveContext.scala:534)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:105)
at
org.apache.spark.sql.hive.HiveContext$.newTemporaryConfiguration(HiveContext.scala:534)
at
org.apache.spark.sql.hive.HiveContext.executionHive$lzycompute(HiveContext.scala:165)
at
org.apache.spark.sql.hive.HiveContext.executionHive(HiveContext.scala:161)
at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:168)
*at com.newegg.ec.bigdata.ORCSpark$.main(ORCSpark.scala:24)*
at com.newegg.ec.bigdata.ORCSpark.main(ORCSpark.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
the code of the 24th line was: val sqlContext = new
org.apache.spark.sql.hive.HiveContext(sc)
I use the Spark core with 1.4.1 and Hive with 1.1.0-cdh5.4.0
On Sat, Aug 22, 2015 at 11:18 PM, Ted Yu <yuzhih...@gmail.com> wrote:
> In Spark 1.4, there was considerable refactoring around interaction with
> Hive, such as SPARK-7491.
>
> It would not be straight forward to port ORC support to 1.3
>
> FYI
>
> On Fri, Aug 21, 2015 at 10:21 PM, dong.yajun <dongt...@gmail.com> wrote:
>
>> hi Ted,
>>
>> thanks for your reply, are there any other way to do this with spark 1.3?
>> such as write the orcfile manually in foreachPartition method?
>>
>> On Sat, Aug 22, 2015 at 12:19 PM, Ted Yu <yuzhih...@gmail.com> wrote:
>>
>>> ORC support was added in Spark 1.4
>>> See SPARK-2883
>>>
>>> On Fri, Aug 21, 2015 at 7:36 PM, dong.yajun <dongt...@gmail.com> wrote:
>>>
>>>> Hi list,
>>>>
>>>> Is there a way to save the RDD result as Orcfile in spark1.3? due to
>>>> some reasons we can't upgrade our spark version to 1.4 now.
>>>>
>>>> --
>>>> *Ric Dong*
>>>>
>>>>
>>>
>>
>>
>> --
>> *Ric Dong*
>>
>>
>
--
*Ric Dong*
