Gentle Ping. Please help me on below issue. Has any one faced same issue
On 4/27/2018 1:28 AM, Mahender Sarangam wrote:
Hi,
Can any one faced issue while fetching data from external table. We are copying
data from upstream system into our storage S3. As part of copy, directories
along with Zero bytes files are been copied. Source File Format is in JSON
format. Below is Folder Hierarchy Structure
DATE --> <Folder>
<DAY=201803250> ---> Folder
1.json.gz --> File
2.json.gz
<day=201803250> ---> Empty Zero Bytes Files.
Please find below screenshot
[cid:[email protected]]
We are trying to create external table with JSON Serde.
ADD JAR
wasb://[email protected]/json/json-serde-1.3.9.jar<mailto:wasb://[email protected]/json/json-serde-1.3.9.jar>;
SET hive.mapred.supports.subdirectories=TRUE;
SET mapred.input.dir.recursive=TRUE;
SET hive.merge.mapfiles = true;
SET hive.merge.mapredfiles = true;
SET hive.merge.tezfiles = true;
DROP TABLE IF EXISTS Ext_STG1;
CREATE EXTERNAL TABLE Ext_STG1(Col1 String, Col2 String, Col3 String) ROW
FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe' WITH SERDEPROPERTIES
("case.insensitive" = "true", "ignore.malformed.json" = "true")
STORED AS TEXTFILE LOCATION
'wasb://[email protected]/date/day=201803250/<mailto:wasb://[email protected]/date/day=201803250/>'
TBLPROPERTIES ('serialization.null.format' = '');
select * from Ext_STG1 limit 100;
Above Query shows Empty Results.
When I delete Zero bytes files, then i could see data from select external
table. Is this expected behaviour. Is there any setting for ignoring Zero bytes
files in hive external table
-Mahens
