Re: ORC files created by Spark job can't be accessed using hive table

Thanks Michael so the following code written using Spark 1.5.1 should be able to recognise by Hive table right dataFrame.write().mode(SaveMode.Append).partitionBy(" entity","date").format("orc").save("baseTable"); Hive console: Create external table bla bla stored as ORC Location

Re: ORC files created by Spark job can't be accessed using hive table

See this thread: http://search-hadoop.com/m/q3RTtwwjNxXvPEe1 A brief search in Spark JIRAs didn't find anything opened on this subject. On Tue, Oct 6, 2015 at 8:51 AM, unk1102 wrote: > Hi I have a spark job which creates ORC files in partitions using the > following code

Re: ORC files created by Spark job can't be accessed using hive table

I believe this is fixed in Spark 1.5.1 as long as the table is only using types that hive understands and is not partitioned. The problem with partitioned tables it that hive does not support dynamic discovery unless you manually run the repair command. On Tue, Oct 6, 2015 at 9:33 AM, Umesh

Re: ORC files created by Spark job can't be accessed using hive table

Hi Ted thanks I know I solved that by using dataframe for both reading and writing. I am running into different problem now if spark can read hive orc files why can't hive read orc files created by Spark? On Oct 6, 2015 9:28 PM, "Ted Yu" wrote: > See this thread: >

ORC files created by Spark job can't be accessed using hive table

Hi I have a spark job which creates ORC files in partitions using the following code dataFrame.write().mode(SaveMode.Append).partitionBy("entity","date").format("orc").save("baseTable"); Above code creates successfully orc files which is readable in Spark dataframe But when I try to load orc