Re: How to write data into Hive partitioned Parquet table?

2014-10-16 Thread Michael Armbrust
Support for dynamic partitioning is available in master and will be part of Spark 1.2 On Thu, Oct 16, 2014 at 1:08 AM, Banias H banias4sp...@gmail.com wrote: I got tipped by an expert that the error of Unsupported language features in query that I had was due to the fact that SparkSQL does not

Re: How to write data into Hive partitioned Parquet table?

2014-10-15 Thread Banias H
I got tipped by an expert that the error of Unsupported language features in query that I had was due to the fact that SparkSQL does not support dynamic partitions, and I can do saveAsParquetFile() for each partition. My inefficient implementation is to: //1. run the query without DISTRIBUTE BY

How to write data into Hive partitioned Parquet table?

2014-10-14 Thread Banias H
Hi, I am still new to Spark. Sorry if similar questions are asked here before. I am trying to read a Hive table; then run a query and save the result into a Hive partitioned Parquet table. For example, I was able to run the following in Hive: INSERT INTO TABLE target_table PARTITION