Rajesh Balamohan created HCATALOG-465:
-----------------------------------------
Summary: Dynamic Partition in HCatalog 0.4 throws
FileAlreadyExists exception
Key: HCATALOG-465
URL: https://issues.apache.org/jira/browse/HCATALOG-465
Project: HCatalog
Issue Type: Bug
Components: pig
Affects Versions: 0.4
Environment: Hadoop 0.20.20x, HCatalog 0.4, Pig 0.9.2
Reporter: Rajesh Balamohan
Here is a simple use case which can reproduce this error. I have also attached
the stacktrace
1. In HCat, create 2 tables (table_1 and table_2). Populate some data in
table_1.
2. Load from table_1 to table_2 with dynamic partition
hcat -e "create table table_1( HIT_TIME_GMT string,SERVICE
string,ACCEPT_LANGUAGE string, DATE_TIME string)
partitioned by (load_date string,repo_name string) row format delimited fields
terminated by '\t' stored as textfile";
hcat -e "create table table_2( HIT_TIME_GMT string,SERVICE
string,ACCEPT_LANGUAGE string, DATE_TIME string)
partitioned by (load_date string,repo_name string) row format delimited fields
terminated by '\t' stored as textfile";
Have some data populated to this with load_date='20120101' and
repo_name='testRepo'
a = load 'table_1' using org.apache.hcatalog.pig.HCatLoader();
b = filter a by (load_date == '20120101' and repo_name == 'testRepo');
store b into 'table_2' using org.apache.hcatalog.pig.HCatStorer();
org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory
hdfs://cluster:54310/user/hive8/warehouse/db/table_1/_DYN0.4448079902737385/load_date=20120515/repo_name=testRepo
already exists
at
org.apache.hadoop.mapred.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:117)
at
org.apache.hcatalog.mapreduce.FileRecordWriterContainer.write(FileRecordWriterContainer.java:201)
at
org.apache.hcatalog.mapreduce.FileRecordWriterContainer.write(FileRecordWriterContainer.java:52)
at
org.apache.hcatalog.pig.HCatBaseStorer.putNext(HCatBaseStorer.java:235)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:139)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:98)
at
org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:531)
at
org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map.collect(PigMapOnly.java:48)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:248)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
at org.apache.hadoop.mapred.Child.main(Child.java:264)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
