Re: Spark SQL insert overwrite table not showing all the partition.

[Zhan Zhang]

INSERT OVERWRITE will overwrite any existing data in the table or partition

  *   unless IF NOT EXISTS is provided for a partition (as of Hive 
0.9.0<https://issues.apache.org/jira/browse/HIVE-2612>).


Thanks.

Zhan Zhang

On Apr 21, 2016, at 3:20 PM, Bijay Kumar Pathak 
<bkpat...@mtu.edu<mailto:bkpat...@mtu.edu>> wrote:

Hi,

I have a job which writes to the Hive table with dynamic partition. Inside the 
job,  I am writing into the table two-time but I am only seeing the partition 
with last write although I can see in the Spark UI it is processing data fro 
both the partition.

Below is the query I am using to write to the table.

hive_c.sql("""INSERT OVERWRITE TABLE base_table PARTITION (date='{1}', date_2)
              SELECT * from temp_table
                """.format(date_val)
                 )


Thanks,
Bijay