0dunay0 opened a new issue, #10600: URL: https://github.com/apache/paimon/issues/10600
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. (Related: #8230 and #8238 are open, stale and conflicting. They write MICROS for every `TIMESTAMP(<=3)` column unconditionally. #8801 and #8797 are the closed pushdown bug and its fix.) ### Paimon version master (c9a36a81d7). Also reproduced on 1.4.x. ### Compute Engine Flink and Spark write the table. Athena (Iceberg reader) fails to read it. ### Minimal reproduce step 1. Create an append table with `'metadata.iceberg.storage' = 'table-location'` (or `hadoop-catalog`) and a `TIMESTAMP_LTZ(3)` column, for example `ts TIMESTAMP_LTZ(3)`. 2. Write rows with Flink or Spark. Paimon commits Iceberg metadata, which declares `ts` as `timestamptz`. 3. Query the table through the Iceberg metadata from Athena, Trino, or any reader that enforces the Parquet type of the file. ### What doesn't meet your expectations? The query fails with an error like this (Athena): ``` ICEBERG_BAD_DATA: Field ts's type INT64 in parquet file s3://.../data-xxx.parquet is incompatible with type timestamp(6) with time zone defined in table schema ``` Paimon declares the column as Iceberg `timestamptz`, but `ParquetSchemaConverter` writes `TIMESTAMP_LTZ(<=3)` as `INT64 (TIMESTAMP(MILLIS, isAdjustedToUTC=true))`. The Iceberg spec requires `INT64 TIMESTAMP_MICROS` for `timestamptz`. Paimon itself reads the files fine. Expected: when Iceberg compatibility is enabled, the Parquet data files use the unit that Iceberg requires. Files already written with MILLIS must remain readable, so a table can hold both. ### Anything else? #8230 proposes MICROS for all `TIMESTAMP(<=3)` columns unconditionally. That changes the on-disk format for everybody, including tables that never use Iceberg compatibility. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
