Hi all,
We have a hive table stored in S3 and registered in a hive metastore. This table is partitionned with a key "day". So we access this table through the spark dataframe API as : sqlContext.read() .table("tablename) .where(col("day").between("2016-08-01","2016-08-02")) When the job is launched, we can see that spark have tasks "table" which have a small duration (seconds) but takes minutes. In the logs we see that every paths for every partitions are listed, regardless the partition key values, during minutes. 16/08/03 13:17:16 INFO HadoopFsRelation: Listing s3a://buckets3/day=2016-07-24 16/08/03 13:17:16 INFO HadoopFsRelation: Listing s3a://buckets3/day=2016-07-25 .... Is it a normal behaviour? Do we could specify something in the read().table, maybe some options? I tried to find such options but i cannot find anything. Thanks, Mehdi