It works. Thanks.
-----邮件原件----- 发件人: Liu, Yuan (Yuan) [mailto:[email protected]] 发送时间: 2017年11月8日 18:19 收件人: [email protected] 主题: RE: timestamp type in Parquet format I reproduced you issue here, I guess it is a bug. As a workaround, we can first cast the column as timestamp. SQL>showddl hive.hive.t_p; /* Hive DDL */ CREATE TABLE DEFAULT.T_P ( A decimal(15,2) , B timestamp ) stored as parquet ; /* Trafodion DDL */ --- SQL operation complete. SQL>select * from hive.hive.t_p where b >= timestamp '1998-12-01 SQL>00:00:00'; *** ERROR[8442] Unable to access EXT scanFetch interface. Call to Java exception in fetchNextRow() returned error java.lang.IllegalArgumentException: FilterPredicate column: b's declared type (java.lang.Long) does not match the schema found in file metadata. Column b is of type: INT96 Valid types for this column are: [class parquet.io.api.Binary] parquet.filter2.predicate.ValidTypeMap.assertTypeValid(ValidTypeMap.java:138) parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumn(SchemaCompatibilityValidator.java:176) parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumnFilterPredicate(SchemaCompatibilityValidator.java:151) parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:115) parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:58) parquet.filter2.predicate.Operators$GtEq.accept(Operators.java:248) parquet.filter2.predicate.SchemaCompatibilityValidator.validate(SchemaCompatibilityValidator.java:63) parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:59) parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:40) parquet.filter2.compat.FilterCompat$FilterPredicateCompat.accept(FilterCompat.java:126) parquet.filter2.compat.RowGroupFilter.filterRowGroups(RowGroupFilter.java:46) parquet.hadoop.ParquetReader.initReader(ParquetReader.java:148) parquet.hadoop.ParquetReader.read(ParquetReader.java:128) org.trafodion.sql.TrafParquetFileReader.fetchNextBlock(TrafParquetFileReader.java:591)(18). Error detail 0. [2017-11-08 18:13:39] SQL>select * from hive.hive.t_p where cast(b as timestamp) >= timestamp SQL>'1998-12-01 00:00:00'; A B ----------------- -------------------------- 1.20 2016-01-01 04:30:30.000000 --- 1 row(s) selected. Best regards, Yuan -----Original Message----- From: Liu, Yuan (Yuan) [mailto:[email protected]] Sent: Wednesday, November 08, 2017 6:01 PM To: [email protected] Subject: RE: timestamp type in Parquet format Are you sure you use Trafodion 2.3? Trafodion 2.2 have not been released yet. Best regards, Yuan -----Original Message----- From: Zhu, Wen-Jun [mailto:[email protected]] Sent: Wednesday, November 08, 2017 5:47 PM To: [email protected] Subject: timestamp type in Parquet format Hi all, I am doing TPC-H benchmark test in Trafodion, with version of Trafodion 2.3, and that of Hive 1.1 . The column l_shipdate in table lineitem is of type date, which is not supported in Hive until version 1.2.. So I change the type to timestamp. Then I got the following error message in trafci: SQL>showddl hive.hive.parquet_lineitem; /* Hive DDL */ CREATE TABLE DEFAULT.PARQUET_LINEITEM ( L_ORDERKEY int , L_PARTKEY int , L_SUPPKEY int , L_LINENUMBER int , L_QUANTITY decimal(15,2) , L_EXTENDEDPRICE decimal(15,2) , L_DISCOUNT decimal(15,2) , L_TAX decimal(15,2) , L_RETURNFLAG char(1) , L_LINESTATUS char(1) , L_SHIPDATE timestamp , L_COMMITDATE timestamp , L_RECEIPTDATE timestamp , L_SHIPINSTRUCT char(25) , L_SHIPMODE char(10) , L_COMMENT varchar(44) ) stored as parquet ; SQL> select l_shipdate from hive.hive.parquet_lineitem where l_shipdate <= timestamp '1998-12-01 00:00:00' limit 10; *** ERROR[8442] Unable to access EXT scanFetch interface. Call to Java exception in fetchNextRow() returned error java.lang.IllegalArgumentException: FilterPredicate column: l_shipdate's declared type (java.lang.Long) does not match the schema found in file metadata. Column l_shipdate is of type: FullTypeDescriptor(PrimitiveType: INT96, OriginalType: null) Valid types for this column are: null parquet.filter2.predicate.ValidTypeMap.assertTypeValid(ValidTypeMap.java:132) parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumn(SchemaCompatibilityValidator.java:185) parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumnFilterPredicate(SchemaCompatibilityValidator.java:160) parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:112) parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:59) parquet.filter2.predicate.Operators$LtEq.accept(Operators.java:221) parquet.filter2.predicate.SchemaCompatibilityValidator.validate(SchemaCompatibilityValidator.java:64) parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:59) parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:40) parquet.filter2.compat.FilterCompat$FilterPredicateCompat.accept(FilterCompat.java:126) parquet.filter2.compat.RowGroupFilter.filterRowGroups(RowGroupFilter.java:46) parquet.hadoop.ParquetReader.initReader(ParquetReader.java:152) parquet.hadoop.ParquetReader.read(ParquetReader.java:132) org.trafodion.sql.TrafParquetFileReader.fetchNextBlock(TrafParquetFileReader.java:591)(18). Error detail 0. [2017-11-08 17:35:08] Can anyone help me to find out what happened? If needed, I can provide the data. Thank you in advance!
