Hello !
I have an HDP 2.2.4.2.(it has normally hive 0.14.0).
I have a ORC table with multiple fields :
###
CREATE TABLE `table1`(
`field1` varchar(10),
`field2` varchar(10),
`field3` date,
`field4` varchar(8),
`field5` varchar(64)
PARTITIONED BY (
`year` string,
`month` string,
`day` string)
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
LOCATION
'hdfs://<NAMENODE>/apps/hive/warehouse/database1.db/table1'
TBLPROPERTIES (
'orc.compress'='SNAPPY'
)
###
It is partitioned on year, month and day.
When I perform this HQL query on this table, it works :
SELECT field1
FROM table1
When i perform this HDL query on this table, it does not work :
SELECT field1
FROM table1
GROUP BY field1
In this case, I got the following error message :
###
2016-08-09 10:43:35,090 WARN [main] org.apache.hadoop.mapred.YarnChild:
Exception running child : java.io.IOException: java.io.IOException:
java.io.IOException: Error reading file:
hdfs://<NAMENODE>/apps/hive/warehouse/database1.db/table1/year=2016/month=08/day=08/000000_0
at
org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
at
org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
at
org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.doNextWithExceptionHandler(HadoopShimsSecure.java:273)
at
org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.next(HadoopShimsSecure.java:183)
at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:199)
at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:185)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:52)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.io.IOException: java.io.IOException: Error reading file:
hdfs://<NAMENODE>/apps/hive/warehouse/database1.db/table1/year=2016/month=08/day=08/000000_0
at
org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
at
org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
at
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:352)
at
org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:101)
at
org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:41)
at
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:115)
at
org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.doNextWithExceptionHandler(HadoopShimsSecure.java:271)
... 11 more
Caused by: java.io.IOException: Error reading file:
hdfs://<NAMENODE>/apps/hive/warehouse/database1.db/table1/year=2016/month=08/day=08/000000_0
at
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.next(RecordReaderImpl.java:3185)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$OrcRecordReader.next(OrcInputFormat.java:164)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$OrcRecordReader.next(OrcInputFormat.java:138)
at
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:347)
... 15 more
Caused by: java.io.EOFException: Read past end of RLE integer from
compressed stream Stream for column 1 kind DATA position: 290916 length:
290916 range: 0 offset: 290916 limit: 290916 range 0 = 0 to 290916
uncompressed: 106643 to 106643
at
org.apache.hadoop.hive.ql.io.orc.RunLengthIntegerReaderV2.readValues(RunLengthIntegerReaderV2.java:56)
at
org.apache.hadoop.hive.ql.io.orc.RunLengthIntegerReaderV2.next(RunLengthIntegerReaderV2.java:302)
at
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$StringDictionaryTreeReader.next(RecordReaderImpl.java:1664)
at
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$StringTreeReader.next(RecordReaderImpl.java:1402)
at
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$VarcharTreeReader.next(RecordReaderImpl.java:1839)
at
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$StructTreeReader.next(RecordReaderImpl.java:1936)
at
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.next(RecordReaderImpl.java:3174)
... 18 more
###
Question1 - Does someone know why I got this error when I do the GROUP BY ?
Question2 - Does the GROUP BY use something in the ORC file 00000_0 ? If
yes, do you know what ?
Question3 - I found the following JIRAs on Internet :
https://issues.apache.org/jira/browse/HIVE-6711, but it is normally solved
in Hive 0.14.0, so I don't think it can be my problem ?
Question4 - I found also this JIRA :
https://issues.apache.org/jira/browse/HIVE-10916, but this file is only 110
MB sized.
I'm open to all suggestion ^_^.
BR.
Tale
