Re: SPARK SQL: returns null for a column, while HIVE query returns data for the same column

2018-05-11 Thread ARAVIND ARUMUGHAM Sethurathnam
- this column was added in later partitions and not present in earlier ones. - - i assume partition pruning should just load from that particular partition i am specifying when using spark sql ? - (spark version 2.2) On Fri, May 11, 2018 at 2:24 PM, ARAVIND ARUMUGHAM

SPARK SQL: returns null for a column, while HIVE query returns data for the same column

2018-05-11 Thread ARAVIND ARUMUGHAM Sethurathnam
I have a hive table created on top of s3 DATA in parquet format and partitioned by one column named eventdate. 1) When using HIVE QUERY, it returns data for a column named "headertime" which is in the schema of BOTH the table and the file. select headertime from dbName.test_bug where