[ https://issues.apache.org/jira/browse/HIVE-12606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Z. S. updated HIVE-12606: ------------------------- Description: When reading via HCatalog an ORC table that has null values in fields it fails with the following exception: {code} 15/12/07 19:47:42 INFO mapred.Task: Using ResourceCalculatorProcessTree : null 15/12/07 19:47:42 INFO mapred.MapTask: Processing split: org.apache.hive.hcatalog.mapreduce.HCatSplit@4c8c30bc 15/12/07 19:47:42 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) 15/12/07 19:47:42 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100 15/12/07 19:47:42 INFO mapred.MapTask: soft limit at 83886080 15/12/07 19:47:42 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600 15/12/07 19:47:42 INFO mapred.MapTask: kvstart = 26214396; length = 6553600 15/12/07 19:47:42 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 15/12/07 19:47:42 INFO orc.ReaderImpl: Reading ORC rows from hdfs://[REDACTED]/000000_0 with {include: null, offset: 0, length: 1628} 15/12/07 19:47:42 INFO mapred.MapTask: Ignoring exception during close for org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader@5096bee4 java.lang.NullPointerException at org.apache.hive.hcatalog.mapreduce.HCatRecordReader.close(HCatRecordReader.java:223) at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.close(MapTask.java:520) at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:1999) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 15/12/07 19:47:42 INFO mapred.MapTask: Starting flush of map output 15/12/07 19:47:42 INFO mapred.LocalJobRunner: map task executor complete. 15/12/07 19:47:42 INFO mapreduce.FileOutputCommitterContainer: Job failed. Try cleaning up temporary directory [hdfs://bd/user/hive/warehouse/test.db/billing_aolon_revenue_output_stream/_DYN0.44164173619220104]. 15/12/07 19:47:42 INFO mapreduce.FileOutputCommitterContainer: Cancelling delegation token for the job. 15/12/07 19:47:42 WARN conf.Configuration: file:/tmp/hadoop-/mapred/local/localRunner/job_local413328602_0001/job_local413328602_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring. 15/12/07 19:47:42 WARN conf.Configuration: file:/tmp/hadoop-/mapred/local/localRunner/job_local413328602_0001/job_local413328602_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring. 15/12/07 19:47:42 INFO hive.metastore: Trying to connect to metastore with URI thrift://bd:9083 15/12/07 19:47:42 INFO hive.metastore: Connected to metastore. 15/12/07 19:47:42 WARN mapred.LocalJobRunner: job_local413328602_0001 java.lang.Exception: java.lang.NullPointerException at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522) Caused by: java.lang.NullPointerException at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$StringDictionaryTreeReader.startStripe(RecordReaderImpl.java:1545) at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$StringTreeReader.startStripe(RecordReaderImpl.java:1337) at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$StructTreeReader.startStripe(RecordReaderImpl.java:1825) at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.readStripe(RecordReaderImpl.java:2537) at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.advanceStripe(RecordReaderImpl.java:2950) at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.advanceToNextRow(RecordReaderImpl.java:2992) at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.<init>(RecordReaderImpl.java:284) at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:480) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.createReaderFromFile(OrcInputFormat.java:214) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$OrcRecordReader.<init>(OrcInputFormat.java:146) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:1010) at org.apache.hive.hcatalog.mapreduce.HCatRecordReader.createBaseRecordReader(HCatRecordReader.java:116) at org.apache.hive.hcatalog.mapreduce.HCatRecordReader.initialize(HCatRecordReader.java:91) at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:545) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:783) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 15/12/07 19:47:43 INFO mapreduce.Job: Job job_local413328602_0001 running in uber mode : false 15/12/07 19:47:43 INFO mapreduce.Job: map 0% reduce 0% 15/12/07 19:47:43 INFO mapreduce.Job: Job job_local413328602_0001 failed with state FAILED due to: NA 15/12/07 19:47:43 INFO mapreduce.Job: Counters: 0 {/code} I could not test it with later versions because those introduce the HIVE-10213 bug with dynamic partitions which is not solved until 1.12 and I can't use that since we're running on cloudera. Sorry. was: When reading via HCatalog an ORC table that has null values in fields it fails with the following exception: 15/12/07 19:47:42 INFO mapred.Task: Using ResourceCalculatorProcessTree : null 15/12/07 19:47:42 INFO mapred.MapTask: Processing split: org.apache.hive.hcatalog.mapreduce.HCatSplit@4c8c30bc 15/12/07 19:47:42 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) 15/12/07 19:47:42 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100 15/12/07 19:47:42 INFO mapred.MapTask: soft limit at 83886080 15/12/07 19:47:42 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600 15/12/07 19:47:42 INFO mapred.MapTask: kvstart = 26214396; length = 6553600 15/12/07 19:47:42 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 15/12/07 19:47:42 INFO orc.ReaderImpl: Reading ORC rows from hdfs://[REDACTED]/000000_0 with {include: null, offset: 0, length: 1628} 15/12/07 19:47:42 INFO mapred.MapTask: Ignoring exception during close for org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader@5096bee4 java.lang.NullPointerException at org.apache.hive.hcatalog.mapreduce.HCatRecordReader.close(HCatRecordReader.java:223) at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.close(MapTask.java:520) at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:1999) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 15/12/07 19:47:42 INFO mapred.MapTask: Starting flush of map output 15/12/07 19:47:42 INFO mapred.LocalJobRunner: map task executor complete. 15/12/07 19:47:42 INFO mapreduce.FileOutputCommitterContainer: Job failed. Try cleaning up temporary directory [hdfs://bd/user/hive/warehouse/test.db/billing_aolon_revenue_output_stream/_DYN0.44164173619220104]. 15/12/07 19:47:42 INFO mapreduce.FileOutputCommitterContainer: Cancelling delegation token for the job. 15/12/07 19:47:42 WARN conf.Configuration: file:/tmp/hadoop-/mapred/local/localRunner/job_local413328602_0001/job_local413328602_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring. 15/12/07 19:47:42 WARN conf.Configuration: file:/tmp/hadoop-/mapred/local/localRunner/job_local413328602_0001/job_local413328602_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring. 15/12/07 19:47:42 INFO hive.metastore: Trying to connect to metastore with URI thrift://bd:9083 15/12/07 19:47:42 INFO hive.metastore: Connected to metastore. 15/12/07 19:47:42 WARN mapred.LocalJobRunner: job_local413328602_0001 java.lang.Exception: java.lang.NullPointerException at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522) Caused by: java.lang.NullPointerException at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$StringDictionaryTreeReader.startStripe(RecordReaderImpl.java:1545) at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$StringTreeReader.startStripe(RecordReaderImpl.java:1337) at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$StructTreeReader.startStripe(RecordReaderImpl.java:1825) at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.readStripe(RecordReaderImpl.java:2537) at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.advanceStripe(RecordReaderImpl.java:2950) at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.advanceToNextRow(RecordReaderImpl.java:2992) at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.<init>(RecordReaderImpl.java:284) at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:480) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.createReaderFromFile(OrcInputFormat.java:214) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$OrcRecordReader.<init>(OrcInputFormat.java:146) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:1010) at org.apache.hive.hcatalog.mapreduce.HCatRecordReader.createBaseRecordReader(HCatRecordReader.java:116) at org.apache.hive.hcatalog.mapreduce.HCatRecordReader.initialize(HCatRecordReader.java:91) at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:545) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:783) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 15/12/07 19:47:43 INFO mapreduce.Job: Job job_local413328602_0001 running in uber mode : false 15/12/07 19:47:43 INFO mapreduce.Job: map 0% reduce 0% 15/12/07 19:47:43 INFO mapreduce.Job: Job job_local413328602_0001 failed with state FAILED due to: NA 15/12/07 19:47:43 INFO mapreduce.Job: Counters: 0 I could not test it with later versions because those introduce the HIVE-10213 bug with dynamic partitions which is not solved until 1.12 and I can't use that since we're running on cloudera. Sorry. > HCatalog ORC Null values in fields results in NullPointer exception > ------------------------------------------------------------------- > > Key: HIVE-12606 > URL: https://issues.apache.org/jira/browse/HIVE-12606 > Project: Hive > Issue Type: Bug > Components: HCatalog > Affects Versions: 0.13.1 > Environment: Linux > Reporter: Z. S. > > When reading via HCatalog an ORC table that has null values in fields it > fails with the following exception: > {code} > 15/12/07 19:47:42 INFO mapred.Task: Using ResourceCalculatorProcessTree : > null > 15/12/07 19:47:42 INFO mapred.MapTask: Processing split: > org.apache.hive.hcatalog.mapreduce.HCatSplit@4c8c30bc > 15/12/07 19:47:42 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) > 15/12/07 19:47:42 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100 > 15/12/07 19:47:42 INFO mapred.MapTask: soft limit at 83886080 > 15/12/07 19:47:42 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600 > 15/12/07 19:47:42 INFO mapred.MapTask: kvstart = 26214396; length = 6553600 > 15/12/07 19:47:42 INFO mapred.MapTask: Map output collector class = > org.apache.hadoop.mapred.MapTask$MapOutputBuffer > 15/12/07 19:47:42 INFO orc.ReaderImpl: Reading ORC rows from > hdfs://[REDACTED]/000000_0 with {include: null, offset: 0, length: 1628} > 15/12/07 19:47:42 INFO mapred.MapTask: Ignoring exception during close for > org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader@5096bee4 > java.lang.NullPointerException > at > org.apache.hive.hcatalog.mapreduce.HCatRecordReader.close(HCatRecordReader.java:223) > at > org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.close(MapTask.java:520) > at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:1999) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at > org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > 15/12/07 19:47:42 INFO mapred.MapTask: Starting flush of map output > 15/12/07 19:47:42 INFO mapred.LocalJobRunner: map task executor complete. > 15/12/07 19:47:42 INFO mapreduce.FileOutputCommitterContainer: Job failed. > Try cleaning up temporary directory > [hdfs://bd/user/hive/warehouse/test.db/billing_aolon_revenue_output_stream/_DYN0.44164173619220104]. > 15/12/07 19:47:42 INFO mapreduce.FileOutputCommitterContainer: Cancelling > delegation token for the job. > 15/12/07 19:47:42 WARN conf.Configuration: > file:/tmp/hadoop-/mapred/local/localRunner/job_local413328602_0001/job_local413328602_0001.xml:an > attempt to override final parameter: > mapreduce.job.end-notification.max.retry.interval; Ignoring. > 15/12/07 19:47:42 WARN conf.Configuration: > file:/tmp/hadoop-/mapred/local/localRunner/job_local413328602_0001/job_local413328602_0001.xml:an > attempt to override final parameter: > mapreduce.job.end-notification.max.attempts; Ignoring. > 15/12/07 19:47:42 INFO hive.metastore: Trying to connect to metastore with > URI thrift://bd:9083 > 15/12/07 19:47:42 INFO hive.metastore: Connected to metastore. > 15/12/07 19:47:42 WARN mapred.LocalJobRunner: job_local413328602_0001 > java.lang.Exception: java.lang.NullPointerException > at > org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462) > at > org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522) > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$StringDictionaryTreeReader.startStripe(RecordReaderImpl.java:1545) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$StringTreeReader.startStripe(RecordReaderImpl.java:1337) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$StructTreeReader.startStripe(RecordReaderImpl.java:1825) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.readStripe(RecordReaderImpl.java:2537) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.advanceStripe(RecordReaderImpl.java:2950) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.advanceToNextRow(RecordReaderImpl.java:2992) > at > org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.<init>(RecordReaderImpl.java:284) > at > org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:480) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.createReaderFromFile(OrcInputFormat.java:214) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$OrcRecordReader.<init>(OrcInputFormat.java:146) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:1010) > at > org.apache.hive.hcatalog.mapreduce.HCatRecordReader.createBaseRecordReader(HCatRecordReader.java:116) > at > org.apache.hive.hcatalog.mapreduce.HCatRecordReader.initialize(HCatRecordReader.java:91) > at > org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:545) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:783) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at > org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > 15/12/07 19:47:43 INFO mapreduce.Job: Job job_local413328602_0001 running in > uber mode : false > 15/12/07 19:47:43 INFO mapreduce.Job: map 0% reduce 0% > 15/12/07 19:47:43 INFO mapreduce.Job: Job job_local413328602_0001 failed with > state FAILED due to: NA > 15/12/07 19:47:43 INFO mapreduce.Job: Counters: 0 > {/code} > I could not test it with later versions because those introduce the > HIVE-10213 bug with dynamic partitions which is not solved until 1.12 and I > can't use that since we're running on cloudera. Sorry. -- This message was sent by Atlassian JIRA (v6.3.4#6332)