[
https://issues.apache.org/jira/browse/DRILL-6670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16574488#comment-16574488
]
ASF GitHub Bot commented on DRILL-6670:
---------------------------------------
okalinin opened a new pull request #1428: DRILL-6670: align Parquet
TIMESTAMP_MICROS logical type handling with earlier versions
URL: https://github.com/apache/drill/pull/1428
## DRILL-6670: align Parquet TIMESTAMP_MICROS logical type handling with
earlier versions
### Background
Parquet dependencies were upgraded with DRILL-6353 which changed behavior in
how Parquet handles `TIMESTAMP_MICROS` logical type. Previously, calling
`SchemaElement.getConverted_type()` was returning null and logical type was
ignored. With updated Parquet version above call returns `TIMESTAMP_MICROS`
which triggers query exception.
Issue impacts both ParquetRecordReader and DrillParquetReader.
This change aims at restoring original functionality - handling
`TIMESTAMP_MICROS` as `INT64` with no logical type in both Parquet readers. It
doesn't seem to make sense to do more since `TIMESTAMP_MICROS` is deprecated
logical type as per Parquet [current
documentation](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md).
### Change description
- Added `TIMESTAMP_MICROS` logical type in both Parquet readers in order to
handle it as regular `INT64`
- Modified `ParquetSimpleTestFileGenerator` to include `TIMESTAMP_MICROS`
columns in Parquet test files and updated existing test data files
- Modified respective tests with minor regrouping
- Fixed `TestDrillParquetReader` to make sure correct reader is actually
used to handle test queries
- Improved debug lines to avoid using ambiguous 'new/old Parquet reader'
terminology
### Level of testing
- Sample file and query provided by issue reporter
- Unit tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Error in parquet record reader - previously readable file fails to be read in
> 1.14
> ----------------------------------------------------------------------------------
>
> Key: DRILL-6670
> URL: https://issues.apache.org/jira/browse/DRILL-6670
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Parquet
> Affects Versions: 1.14.0
> Reporter: Dave Challis
> Assignee: Oleksandr Kalinin
> Priority: Major
> Attachments: example.parquet
>
>
> Parquet file which was generated by PyArrow was readable in Apache Drill 1.12
> and 1.13, but fails to be read with 1.14.
> Running the query "SELECT * FROM dfs.`foo.parquet`" results in the following
> error message from the Drill web query UI:
> {code}
> Query Failed: An Error Occurred
> org.apache.drill.common.exceptions.UserRemoteException: INTERNAL_ERROR ERROR:
> Error in parquet record reader. Message: Failure in setting up reader Parquet
> Metadata: ParquetMetaData{FileMetaData{schema: message schema { optional
> binary name (UTF8); optional binary creation_parameters (UTF8); optional
> int64 creation_date (TIMESTAMP_MICROS); optional int32 data_version; optional
> int32 schema_version; } , metadata: {pandas={"index_columns": [],
> "column_indexes": [], "columns": [{"name": "name", "field_name": "name",
> "pandas_type": "unicode", "numpy_type": "object", "metadata": null}, {"name":
> "creation_parameters", "field_name": "creation_parameters", "pandas_type":
> "unicode", "numpy_type": "object", "metadata": null}, {"name":
> "creation_date", "field_name": "creation_date", "pandas_type": "datetime",
> "numpy_type": "datetime64[ns]", "metadata": null}, {"name": "data_version",
> "field_name": "data_version", "pandas_type": "int32", "numpy_type": "int32",
> "metadata": null}, {"name": "schema_version", "field_name": "schema_version",
> "pandas_type": "int32", "numpy_type": "int32", "metadata": null}],
> "pandas_version": "0.22.0"}}}, blocks: [BlockMetaData{1, 27142
> [ColumnMetaData{SNAPPY [name] optional binary name (UTF8) [PLAIN, RLE], 4},
> ColumnMetaData{SNAPPY [creation_parameters] optional binary
> creation_parameters (UTF8) [PLAIN, RLE], 252}, ColumnMetaData{SNAPPY
> [creation_date] optional int64 creation_date (TIMESTAMP_MICROS) [PLAIN, RLE],
> 46334}, ColumnMetaData{SNAPPY [data_version] optional int32 data_version
> [PLAIN, RLE], 46478}, ColumnMetaData{SNAPPY [schema_version] optional int32
> schema_version [PLAIN, RLE], 46593}]}]} Fragment 0:0 [Error Id:
> bdb2e4d5-5982-4cc6-b95e-244782f827d2 on f9d0456cddd2:31010]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
