please try spark.sql.hive.verifyPartitionPath true
------------------ Original ------------------
From: "Steve Loughran";<ste...@hortonworks.com>;
Date: Sat, May 20, 2017 09:19 PM
To: "Bajpai, Amit X. -ND"<amit.x.bajpai....@disney.com>;
Cc: "user@spark.apache.org"<user@spark.apache.org>;
Subject: Re: SparkSQL not able to read a empty table location
On 20 May 2017, at 01:44, Bajpai, Amit X. -ND <amit.x.bajpai....@disney.com>
wrote:
Hi,
I have a hive external table with the S3 location having no files (but the S3
location directory does exists). When I am trying to use Spark SQL to count the
number of records in the table it is throwing error saying ??File
s3n://data/xyz does not exist. null/0??.
select * from tablex limit 10
Can someone let me know how we can fix this issue.
Thanks
There isn't really a "directory" in S3, just a set of objects whose paths
begin with a string. Try creating an empty file with an _ prefix in the
directory; it should be ignored by Spark SQL but will cause the "directory" to
come into being
