jqysn commented on issue #3913:
URL: https://github.com/apache/iceberg/issues/3913#issuecomment-1016104942
@pvary Thanks for your help. It works with above settings.
However on tez side, the latest release version is 0.10.0, which should have
come compatible issue on guava if using hive 2.3.6. And I tried to find out
what happened when running on tez with partition col:
`Caused by: java.lang.RuntimeException: **Record: Record(d1)**, StructField:
org.apache.iceberg.mr.hive.serde.objectinspector.IcebergRecordObjectInspector$IcebergRecordStructField@27fa5cb1
at
org.apache.iceberg.mr.hive.serde.objectinspector.IcebergRecordObjectInspector.getStructFieldData(IcebergRecordObjectInspector.java:75)
at
org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator._evaluate(ExprNodeColumnEvaluator.java:95)
at
org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:80)
at
org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator$DeferredExprObject.get(ExprNodeGenericFuncEvaluator.java:88)
at
org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPEqual.evaluate(GenericUDFOPEqual.java:103)
at
org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:187)
at
org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:80)
at
org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:68)
at
org.apache.hadoop.hive.ql.exec.FilterOperator.process(FilterOperator.java:112)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
at
org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:148)
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:550)
... 18 more`
I throw a runtime exception to view the content of GenericRecord, and it
looks like there is a miss match between the record and schema: the struct
filed position(evaluated as 2) doesn't match with the Record schema (position
should be 0). That's probably the reason why it throws
ArrayIndexOutOfBoundsException.
To make iceberg support tez-0.9.2 (most recent version in use), we need to
use _getFieldName_ from record schema, rather than using the _position()_ of
the structfield.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
