Spark SQL insert overwrite table not showing all the partition.

[Bijay Kumar Pathak]

Hi,

I have a job which writes to the Hive table with dynamic partition. Inside
the job,  I am writing into the table two-time but I am only seeing the
partition with last write although I can see in the Spark UI it is
processing data fro both the partition.

Below is the query I am using to write to the table.

hive_c.sql("""INSERT OVERWRITE TABLE base_table PARTITION (date='{1}', date_2)
              SELECT * from temp_table
                """.format(date_val)
                 )



Thanks,
Bijay