Jefffrey commented on PR #10569:
URL: https://github.com/apache/arrow-rs/pull/10569#issuecomment-5324261079
i'm kinda of the mind to just enable it by default, instead of honoring the
previous default behaviour. for reference, was able to use codex to generate
some duckdb code which shows that they can generate such files so i'm fairly
sure its only a parquet-java limitation:
```sql
DuckDB v1.5.5 (Variegata)
Enter ".help" for usage hints.
memory D CREATE TABLE values_to_write AS
SELECT
CASE
WHEN i % 2 = 0
THEN UUID '00000000-0000-0000-0000-000000000001'
ELSE UUID '00000000-0000-0000-0000-000000000002'
END AS value
FROM range(200) AS rows(i);
memory D
memory D COPY values_to_write
TO 'duckdb-v1-flba-dictionary.parquet'
(
FORMAT PARQUET,
PARQUET_VERSION V1,
COMPRESSION UNCOMPRESSED
);
memory D SELECT value, count(*)
FROM read_parquet('duckdb-v1-flba-dictionary.parquet')
GROUP BY value
ORDER BY value;
┌──────────────────────────────────────┬──────────────┐
│ value │ count_star() │
│ uuid │ int64 │
├──────────────────────────────────────┼──────────────┤
│ 00000000-0000-0000-0000-000000000001 │ 100 │
│ 00000000-0000-0000-0000-000000000002 │ 100 │
└──────────────────────────────────────┴──────────────┘
memory D SELECT
path_in_schema,
type,
encodings,
dictionary_page_offset
FROM parquet_metadata('duckdb-v1-flba-dictionary.parquet');
┌────────────────┬──────────────────────┬──────────────────┬────────────────────────┐
│ path_in_schema │ type │ encodings │
dictionary_page_offset │
│ varchar │ varchar │ varchar │ int64
│
├────────────────┼──────────────────────┼──────────────────┼────────────────────────┤
│ value │ FIXED_LEN_BYTE_ARRAY │ PLAIN_DICTIONARY │
4 │
└────────────────┴──────────────────────┴──────────────────┴────────────────────────┘
memory D SELECT format_version
FROM parquet_file_metadata('duckdb-v1-flba-dictionary.parquet');
┌────────────────┐
│ format_version │
│ int64 │
├────────────────┤
│ 1 │
└────────────────┘
memory D
```
the issue mentions
> I found one outlier (parquet-cpp), but that is besides the point. The only
conclusion to draw from this is that readers can read those files; I created a
test table with pyiceberg and read it back with iceberg-rust and data is
intact, no errors.
maybe if we double check against parquet-java, to see if it can read such
files, we can just simplify this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
