I reproduced you issue here, I guess it is a bug.
As a workaround, we can first cast the column as timestamp.
SQL>showddl hive.hive.t_p;
/* Hive DDL */
CREATE TABLE DEFAULT.T_P
(
A decimal(15,2)
, B timestamp
)
stored as parquet
;
/* Trafodion DDL */
--- SQL operation complete.
SQL>select * from hive.hive.t_p where b >= timestamp '1998-12-01 00:00:00';
*** ERROR[8442] Unable to access EXT scanFetch interface. Call to Java
exception in fetchNextRow() returned error java.lang.IllegalArgumentException:
FilterPredicate column: b's declared type (java.lang.Long) does not match the
schema found in file metadata. Column b is of type: INT96
Valid types for this column are: [class parquet.io.api.Binary]
parquet.filter2.predicate.ValidTypeMap.assertTypeValid(ValidTypeMap.java:138)
parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumn(SchemaCompatibilityValidator.java:176)
parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumnFilterPredicate(SchemaCompatibilityValidator.java:151)
parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:115)
parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:58)
parquet.filter2.predicate.Operators$GtEq.accept(Operators.java:248)
parquet.filter2.predicate.SchemaCompatibilityValidator.validate(SchemaCompatibilityValidator.java:63)
parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:59)
parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:40)
parquet.filter2.compat.FilterCompat$FilterPredicateCompat.accept(FilterCompat.java:126)
parquet.filter2.compat.RowGroupFilter.filterRowGroups(RowGroupFilter.java:46)
parquet.hadoop.ParquetReader.initReader(ParquetReader.java:148)
parquet.hadoop.ParquetReader.read(ParquetReader.java:128)
org.trafodion.sql.TrafParquetFileReader.fetchNextBlock(TrafParquetFileReader.java:591)(18).
Error detail 0. [2017-11-08 18:13:39]
SQL>select * from hive.hive.t_p where cast(b as timestamp) >= timestamp
'1998-12-01 00:00:00';
A B
----------------- --------------------------
1.20 2016-01-01 04:30:30.000000
--- 1 row(s) selected.
Best regards,
Yuan
-----Original Message-----
From: Liu, Yuan (Yuan) [mailto:[email protected]]
Sent: Wednesday, November 08, 2017 6:01 PM
To: [email protected]
Subject: RE: timestamp type in Parquet format
Are you sure you use Trafodion 2.3? Trafodion 2.2 have not been released yet.
Best regards,
Yuan
-----Original Message-----
From: Zhu, Wen-Jun [mailto:[email protected]]
Sent: Wednesday, November 08, 2017 5:47 PM
To: [email protected]
Subject: timestamp type in Parquet format
Hi all,
I am doing TPC-H benchmark test in Trafodion, with version of Trafodion
2.3, and that of Hive 1.1 .
The column l_shipdate in table lineitem is of type date, which is not
supported in Hive until version 1.2..
So I change the type to timestamp.
Then I got the following error message in trafci:
SQL>showddl hive.hive.parquet_lineitem;
/* Hive DDL */
CREATE TABLE DEFAULT.PARQUET_LINEITEM
(
L_ORDERKEY int
, L_PARTKEY int
, L_SUPPKEY int
, L_LINENUMBER int
, L_QUANTITY decimal(15,2)
, L_EXTENDEDPRICE decimal(15,2)
, L_DISCOUNT decimal(15,2)
, L_TAX decimal(15,2)
, L_RETURNFLAG char(1)
, L_LINESTATUS char(1)
, L_SHIPDATE timestamp
, L_COMMITDATE timestamp
, L_RECEIPTDATE timestamp
, L_SHIPINSTRUCT char(25)
, L_SHIPMODE char(10)
, L_COMMENT varchar(44)
)
stored as parquet
;
SQL> select
l_shipdate
from
hive.hive.parquet_lineitem
where
l_shipdate <= timestamp '1998-12-01 00:00:00'
limit 10;
*** ERROR[8442] Unable to access EXT scanFetch interface. Call to Java
exception in fetchNextRow() returned error java.lang.IllegalArgumentException:
FilterPredicate column: l_shipdate's declared type (java.lang.Long) does not
match the schema found in file metadata. Column l_shipdate is of type:
FullTypeDescriptor(PrimitiveType: INT96, OriginalType: null) Valid types for
this column are: null
parquet.filter2.predicate.ValidTypeMap.assertTypeValid(ValidTypeMap.java:132)
parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumn(SchemaCompatibilityValidator.java:185)
parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumnFilterPredicate(SchemaCompatibilityValidator.java:160)
parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:112)
parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:59)
parquet.filter2.predicate.Operators$LtEq.accept(Operators.java:221)
parquet.filter2.predicate.SchemaCompatibilityValidator.validate(SchemaCompatibilityValidator.java:64)
parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:59)
parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:40)
parquet.filter2.compat.FilterCompat$FilterPredicateCompat.accept(FilterCompat.java:126)
parquet.filter2.compat.RowGroupFilter.filterRowGroups(RowGroupFilter.java:46)
parquet.hadoop.ParquetReader.initReader(ParquetReader.java:152)
parquet.hadoop.ParquetReader.read(ParquetReader.java:132)
org.trafodion.sql.TrafParquetFileReader.fetchNextBlock(TrafParquetFileReader.java:591)(18).
Error detail 0. [2017-11-08 17:35:08]
Can anyone help me to find out what happened? If needed, I can provide the data.
Thank you in advance!
