[ https://issues.apache.org/jira/browse/HIVE-27951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
László Bodor updated HIVE-27951: -------------------------------- Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) > hcatalog dynamic partitioning fails with partition already exist error when > exist parent partitions path > -------------------------------------------------------------------------------------------------------- > > Key: HIVE-27951 > URL: https://issues.apache.org/jira/browse/HIVE-27951 > Project: Hive > Issue Type: Bug > Components: HCatalog > Affects Versions: 4.0.0-beta-1 > Reporter: Yi Zhang > Assignee: Yi Zhang > Priority: Critical > Labels: pull-request-available > Fix For: 4.0.0 > > > if a table have multiple partitions (part1=x1, part2=y1), when insert into a > new partition(part1=x1, part2=y2) hcatalog FileOutputCommitterContainer > throws path already exists error > > reproduce: > create table source(id int, part1 string, part2 string); > create table target(id int) partitioned by (part1 string, part2 string) > insert into table source values (1, "x1", "y1"), (2, "x1", "y2"); > > pig -useHcatalog > A = load 'source' using org.apache.hive.hcatalog.pig.HCatLoader(); > B = filter A by (part2 == 'y1'); > // following succeeds > store B into 'target' USING org.apache.hive.hcatalog.pig.HCatStorer(); > //following fails with duplicate publishing error > C = filter A by (part2 == 'y2'); > store C into 'target' USING org.apache.hive.hcatalog.pig.HCatStorer(); > > > ``` > Partition already present with given partition key values : Data already > exists in /user/hive/warehouse/target_data/part1=x1, duplicate publish not > possible. > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.commitJob(PigOutputCommitter.java:243) > at > org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:286) > > Caused by: org.apache.hive.hcatalog.common.HCatException : 2002 : Partition > already present with given partition key values : Data already exists in > /user/hive/warehouse/target_data/part1=x1, duplicate publish not possible. > at > org.apache.hive.hcatalog.mapreduce.FileOutputCommitterContainer.moveTaskOutputs(FileOutputCommitterContainer.java:564) > at > org.apache.hive.hcatalog.mapreduce.FileOutputCommitterContainer.registerPartitions(FileOutputCommitterContainer.java:949) > at > org.apache.hive.hcatalog.mapreduce.FileOutputCommitterContainer.commitJob(FileOutputCommitterContainer.java:273) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.commitJob(PigOutputCommitter.java:241) > ``` -- This message was sent by Atlassian Jira (v8.20.10#820010)