Kolluru Som Shekhar Sharma created HIVE-9012:
------------------------------------------------
Summary: Not able to move and populate the data fully on to the
table when the scratch directory is on S3
Key: HIVE-9012
URL: https://issues.apache.org/jira/browse/HIVE-9012
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.13.1
Environment: Amazon AMI and S3 as storage service
Reporter: Kolluru Som Shekhar Sharma
Priority: Blocker
Fix For: 0.13.1
I have set the hive.exec.scratchDir to point to a directory on S3 and external
table is on S3 level.
I ran a simple query which extracts the key value pairs from JSON string
without any WHERE clause, and the about of data is ~500GB. The query ran fine,
but when it is trying to move the data from the scratch directory it doesn't
complete. So i need to kill the process and manually need to move the data.
The data size in the scratch directory was nearly ~550GB
I tried the same scenario with less data and putting where clause, it completed
successfully and data also gets populated in the table. I checked the size in
the table and in the scratch directory. The data in the table was showing 2MB
and the data in the scratch directory is 48.6GB
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
