Just found out that I need following option while reading:
.option("basePath", "hdfs://localhost:9000/ptest/")
https://stackoverflow.com/questions/43192940/why-is-partition-key-column-missing-from-dataframe
On Tue, Jul 17, 2018 at 3:48 PM, Nirav Patel <npa...@xactlycorp.com> wrote:
> I created a hive table with parquet storage using sparkSql. Now in hive
> cli when I do describe and Select I can see partition columns in both as
> regular columns as well as partition column. However if I try to do same in
> sparkSql (Dataframe) I don't see partition columns.
>
> I need to do projection on partition column in spark. How do I do that now?
>
--
<http://www.xactlycorp.com/email-click/>
<https://www.instagram.com/xactlycorp/>
<https://www.linkedin.com/company/xactly-corporation>
<https://twitter.com/Xactly> <https://www.facebook.com/XactlyCorp>
<http://www.youtube.com/xactlycorporation>
