David Chen created HIVE-7855:
--------------------------------
Summary: Invalid partition values specified when writing to static
partitioned tables via HCatalog
Key: HIVE-7855
URL: https://issues.apache.org/jira/browse/HIVE-7855
Project: Hive
Issue Type: Bug
Components: HCatalog
Reporter: David Chen
Assignee: David Chen
While working on my fix for HIVE-4329, I found that on trunk, writing to static
partitioned tables via HCatalog causes the following exception to be thrown:
{code}
org.apache.hive.hcatalog.common.HCatException : 2010 : Invalid partition values
specified : Unable to configure dynamic partitioning for storage handler,
mismatch between number of partition values obtained[0] and number of partition
values required[2]
at
org.apache.hive.hcatalog.mapreduce.HCatBaseOutputFormat.configureOutputStorageHandler(HCatBaseOutputFormat.java:143)
at
org.apache.hive.hcatalog.mapreduce.DynamicPartitionFileRecordWriterContainer.configureDynamicStorageHandler(DynamicPartitionFileRecordWriterContainer.java:209)
at
org.apache.hive.hcatalog.mapreduce.DynamicPartitionFileRecordWriterContainer.getLocalFileWriter(DynamicPartitionFileRecordWriterContainer.java:132)
at
org.apache.hive.hcatalog.mapreduce.FileRecordWriterContainer.write(FileRecordWriterContainer.java:117)
at
org.apache.hive.hcatalog.mapreduce.FileRecordWriterContainer.write(FileRecordWriterContainer.java:54)
at
org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:635)
at
org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
at
org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
at
org.apache.hive.hcatalog.mapreduce.HCatMapReduceTest$MapCreate.map(HCatMapReduceTest.java:249)
at
org.apache.hive.hcatalog.mapreduce.HCatMapReduceTest$MapCreate.map(HCatMapReduceTest.java:240)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
While the tests for static partitioned tables (TestHCatPartitioned,
TestHCatExternalPartitioned, TestHCatMutablePartitioned) do not fail, this
exception is thrown (and then caught and ignored) during the write path.
This exception is thrown because the partition values list is empty while the
partition keys list is not. It seems to me that partition columns for static
partitioned tables are currently not handled correctly in HCatalog and the
implementation only works incidentally.
However, this bug actually manifests in failures with my fix for HIVE-4329. In
FileRecordWriterContainer.write, the partition columns are removed from the
record being written. However, since the partition values list is empty,
jobinfo.getPosOfPartCols ends up returning an empty list, resulting in the
SerDe throwing an exception when it finds that the number of columns in the
record does not match those of the schema.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
