JingsongLi commented on code in PR #10601:
URL: https://github.com/apache/paimon/pull/10601#discussion_r4294819221
##########
paimon-format/src/main/java/org/apache/paimon/format/parquet/ParquetSchemaConverter.java:
##########
@@ -214,7 +240,11 @@ public static Type convertToParquetType(String name,
DataType type, int fieldId,
case TIMESTAMP_WITH_LOCAL_TIME_ZONE:
LocalZonedTimestampType localZonedTimestampType =
(LocalZonedTimestampType) type;
return createTimestampWithLogicalType(
- name, localZonedTimestampType.getPrecision(),
repetition, true)
+ name,
+ localZonedTimestampType.getPrecision(),
+ repetition,
+ true,
+ localZonedTimestampMillisAsMicros)
Review Comment:
[P2] Preserve timestamp predicate pruning for the new MICROS encoding
This changes mirrored LTZ(3) files to MICROS, but
`ParquetFilters.timestampColumn()` still requires MILLIS for logical precision
<= 3. It rejects these columns and `convert()` silently drops the predicate. I
verified both an equality and a 10-second range on 120,000 monotonically
increasing LTZ(3) events with 64 KiB row groups: both files contain 15 groups,
but MILLIS reads only group 7 and emits 7,686 rows, while the new MICROS file
gets NoOpFilter, reads all 15 groups and emits 120,000 rows (actual matches
remain 1/10). This loses within-file time pruning on a common selective-query
path. Please make both timestamp column validation and literal conversion
follow the physical file unit, and add a multi-row-group test that asserts
actual exclusion; the new single-group predicate test only proves rows are
retained.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
