I don’t think Hive picks up partitions automatically in this scenario. Maybe a ticket could be added to add partitions based on some additional syntax, as this seems to be an occasionally used scenario. I’ve seen msck used as a hack to “restore” partitions into metastore (it will find the directories and create the partitions if all goes well), note that new partitions also won’t be picked up. Make sure to try it first on test directory so you could see if it works for you.
From: Hafiz Mujadid <hafizmujadi...@gmail.com<mailto:hafizmujadi...@gmail.com>> Reply-To: "user@hive.apache.org<mailto:user@hive.apache.org>" <user@hive.apache.org<mailto:user@hive.apache.org>> Date: Thursday, October 22, 2015 at 03:57 To: "user@hive.apache.org<mailto:user@hive.apache.org>" <user@hive.apache.org<mailto:user@hive.apache.org>> Subject: Hive s3 external table with sub directories I have following s3 directory structure. Data/ Year=2015/ Month=01/ Day=01/ files Day=02/ files Month=02/ Day=01/ files Day=02/ files . . . Year=2014/ Month=01/ Day=01/ files Day=02/ files Month=02/ Day=01/ files Day=02/ files So i am creating hive external table as follow CREATE external TABLE trips ( trip_id STRING,probe_id STRING,provider_id STRING, is_moving TINYINT,is_completed BOOLEAN,start_time STRING, start_lat DOUBLE,start_lon DOUBLE,start_lat_adj DOUBLE) PARTITIONED BY (year INT,month INT,day INT) STORED AS TEXTFILE LOCATION 's3n://accesskey:secretkey@bucket/data/'; When i run query on this table no data is returned without any exception. If i place same files in one directory only and without partitioning, then it runs fine. I also tried bey setting set mapred.input.dir.recursive=true; set hive.mapred.supports.subdirectories=true; Any idea where i am wrong?
