[
https://issues.apache.org/jira/browse/DRILL-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15686374#comment-15686374
]
ASF GitHub Bot commented on DRILL-4980:
---------------------------------------
Github user vdiravka commented on a diff in the pull request:
https://github.com/apache/drill/pull/644#discussion_r89088416
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetReaderUtility.java
---
@@ -189,10 +195,16 @@ public static DateCorruptionStatus
detectCorruptDates(ParquetMetadata footer,
String createdBy = footer.getFileMetaData().getCreatedBy();
String drillVersion =
footer.getFileMetaData().getKeyValueMetaData().get(ParquetRecordWriter.DRILL_VERSION_PROPERTY);
- String isDateCorrect =
footer.getFileMetaData().getKeyValueMetaData().get(ParquetRecordWriter.IS_DATE_CORRECT_PROPERTY);
+ String stringWriterVersion =
footer.getFileMetaData().getKeyValueMetaData().get(ParquetRecordWriter.WRITER_VERSION_PROPERTY);
+ // This flag can be present in parquet files which were generated with
1.9.0-SNAPSHOT drill version.
+ // If this flag is present it means that the version of the drill
parquet writer is 2
+ final String isDateCorrectFlag = "is.date.correct";
+ String isDateCorrect =
footer.getFileMetaData().getKeyValueMetaData().get(isDateCorrectFlag);
if (drillVersion != null) {
- return Boolean.valueOf(isDateCorrect) ?
DateCorruptionStatus.META_SHOWS_NO_CORRUPTION
- : DateCorruptionStatus.META_SHOWS_CORRUPTION;
+ int writerVersion = (stringWriterVersion != null) ?
Integer.parseInt(stringWriterVersion)
--- End diff --
Agree. It makes sense.
I even found one redundant check in the return statement.
Changes in a new commit.
> Upgrading of the approach of parquet date correctness status detection
> ----------------------------------------------------------------------
>
> Key: DRILL-4980
> URL: https://issues.apache.org/jira/browse/DRILL-4980
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - Parquet
> Affects Versions: 1.8.0
> Reporter: Vitalii Diravka
> Assignee: Parth Chandra
> Fix For: Future
>
>
> This jira is an addition for the
> [DRILL-4203|https://issues.apache.org/jira/browse/DRILL-4203].
> The date correctness label for the new generated parquet files should be
> upgraded.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
