[
https://issues.apache.org/jira/browse/HIVE-8474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14176496#comment-14176496
]
Matt McCline commented on HIVE-8474:
------------------------------------
Some background on vectorization.
There are are "shadow" VectorizationContext data structures that track which
columns of vectorized row batches for used by each vectorized operators.
In row-by-row mode an operator can easily form a new row Object Array to
correspond to the outputObjInspector.
However, in Vectorization we mask or project away columns in a
VectorizedRowBatch (e.g. VectorFilterOperator) so the same batch can travel
down the operators without being copied. Or, in the case of computing new
columns, VectorSelectOperator will compute new scratch columns.
So, the VectorizationContext starts as all the table columns for Map or the
keys and values for Reduce and then as we go down the operators new
VectorizationContext objects are cloned and their column and scratch column
maps are modified.
So, some operators do not use inputObjInspectors or outputObjInspector.
Others, do use them when the vector operator unpacks batches into rows to call
an row mode operator.
> Vectorized reads of transactional tables fail when not all columns are
> selected
> -------------------------------------------------------------------------------
>
> Key: HIVE-8474
> URL: https://issues.apache.org/jira/browse/HIVE-8474
> Project: Hive
> Issue Type: Bug
> Components: Transactions, Vectorization
> Affects Versions: 0.14.0
> Reporter: Alan Gates
> Assignee: Alan Gates
> Priority: Critical
> Fix For: 0.14.0
>
> Attachments: HIVE-8474.patch
>
>
> {code}
> create table concur_orc_tab(name varchar(50), age int, gpa decimal(3, 2))
> clustered by (age) into 2 buckets stored as orc TBLPROPERTIES
> ('transactional'='true');
> select name, age from concur_orc_tab order by name;
> {code}
> results in
> {code}
> Diagnostic Messages for this Task:
> Error: java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:352)
> at
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:79)
> at
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:33)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:115)
> at
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:199)
> at
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:185)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:52)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorizedBatchUtil.setNullColIsNullValue(VectorizedBatchUtil.java:63)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorizedBatchUtil.addRowToBatchFrom(VectorizedBatchUtil.java:443)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorizedBatchUtil.addRowToBatch(VectorizedBatchUtil.java:214)
> at
> org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowReader.next(VectorizedOrcAcidRowReader.java:95)
> at
> org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowReader.next(VectorizedOrcAcidRowReader.java:43)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:347)
> ... 13 more
> {code}
> The issue is that the object inspector passed to VectorizedOrcAcidRowReader
> has all of the columns in the file rather than only the projected columns.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
