Yibo Dong created FLINK-41554:
---------------------------------
Summary: Parquet reader fails on reordered nested ROW fields with
different physical types
Key: FLINK-41554
URL: https://issues.apache.org/jira/browse/FLINK-41554
Project: Flink
Issue Type: Bug
Environment: Apache Flink 2.3.0 (stock), batch Table API, local
filesystem connector with CSV format. No optimizer rules disabled.
Reporter: Yibo Dong
h2. Description
Flink's filesystem Parquet sink writes nested ROW fields successfully, but the
Parquet reader fails when the physical field order differs from the declared
read schema and the reordered fields have different physical types.
Both files contain the same named nested fields and values. The file written
with ROW<x INT, y STRING> can be read successfully, whereas the file written
with ROW<y STRING, x INT> fails when read using ROW<x INT, y STRING>.
h2. How to Reproduce
Run the following SQL in a fresh batch TableEnvironment, using initially
nonexistent filesystem paths:
{code:sql}
CREATE VIEW input_rows AS
SELECT * FROM (VALUES
(1, 11, 'a'),
(2, 12, 'b')
) AS t(id, x, y);
CREATE TABLE write_ordered (
id INT,
r ROW<x INT, y STRING>
) WITH (
'connector' = 'filesystem',
'path' = 'file:///tmp/flink-parquet-reorder/ordered',
'format' = 'parquet'
);
CREATE TABLE write_reordered (
id INT,
r ROW<y STRING, x INT>
) WITH (
'connector' = 'filesystem',
'path' = 'file:///tmp/flink-parquet-reorder/reordered',
'format' = 'parquet'
);
INSERT INTO write_ordered
SELECT id, CAST(ROW(x, y) AS ROW<x INT, y STRING>)
FROM input_rows;
INSERT INTO write_reordered
SELECT id, CAST(ROW(y, x) AS ROW<y STRING, x INT>)
FROM input_rows;
CREATE TABLE read_ordered (
id INT,
r ROW<x INT, y STRING>
) WITH (
'connector' = 'filesystem',
'path' = 'file:///tmp/flink-parquet-reorder/ordered',
'format' = 'parquet'
);
CREATE TABLE read_reordered (
id INT,
r ROW<x INT, y STRING>
) WITH (
'connector' = 'filesystem',
'path' = 'file:///tmp/flink-parquet-reorder/reordered',
'format' = 'parquet'
);
SELECT id, x, y FROM input_rows ORDER BY id;
SELECT id, r.x, r.y FROM read_ordered ORDER BY id;
SELECT id, r.x, r.y FROM read_reordered ORDER BY id;
{code}
h2. Expected Result
All three queries should return the same values when accessing nested fields by
name:
{code:text}
+I[1, 11, a]
+I[2, 12, b]
{code}
h2. Actual Result
The input view and the ordered Parquet file return the expected rows.
The reordered Parquet file is written successfully, but reading it fails
without returning any rows:
{code:text}
IllegalArgumentException: Unexpected type: BINARY
{code}
The exception originates from
{{ParquetSplitReaderUtil.createWritableColumnVector}}. The outer query reports
{{Failed to fetch next result}}.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)