Hemanth Boyina created SPARK-60940:
--------------------------------------
Summary: Vectorized Parquet/ORC readers return wrong values for
map column DEFAULTs
Key: SPARK-60940
URL: https://issues.apache.org/jira/browse/SPARK-60940
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Hemanth Boyina
When a column with a MAP DEFAULT is added to a table with ALTER TABLE ADD
COLUMN, files written before the column existed do not contain it, so the
vectorized Parquet and ORC readers fill in the default for every row with
WritableColumnVector.appendObjects.
repro steps:
CREATE TABLE t (id INT) USING PARQUET;
INSERT INTO t VALUES (1), (2), (3), (4), (5);
ALTER TABLE t ADD COLUMN m MAP<STRING, INT> DEFAULT map('a', 1, 'b', 2);
SELECT id, size(m), map_entries(m) FROM t ORDER BY id;
Every row should be \{a=1, b=2}, but some rows come back as empty maps.
The same happens with ORC.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
