[
https://issues.apache.org/jira/browse/SPARK-60123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yibo Dong updated SPARK-60123:
------------------------------
Description:
h2. What happened
Spark can create and write an ORC table with a {{STRING COLLATE UTF8_LCASE}}
column, but reading the stored values fails with an exception. The same column
type works with Parquet.
h2. How to reproduce
Run the following SQL using {{{}spark-sql{}}}:
{code:sql}
CREATE TABLE t_parquet_collated_string_control (
k STRING COLLATE UTF8_LCASE
) USING PARQUET;
INSERT INTO t_parquet_collated_string_control VALUES ('a'), ('A'), ('b');
SELECT k FROM t_parquet_collated_string_control WHERE k = 'a';
CREATE TABLE t_orc_collated_string (
k STRING COLLATE UTF8_LCASE
) USING ORC;
INSERT INTO t_orc_collated_string VALUES ('a'), ('A'), ('b');
SELECT k FROM t_orc_collated_string;
{code}
Save the SQL as {{repro.sql}} and run:
{code:bash}
spark-sql --master 'local[2]'
--conf spark.ui.enabled=false
--conf spark.sql.catalogImplementation=in-memory
--conf "spark.sql.warehouse.dir=$(mktemp -d)"
-f repro.sql
{code}
h2. Expected result
The Parquet query should return {{a}} and {{{}A{}}}, since {{UTF8_LCASE}}
provides case-insensitive comparison.
The ORC query should return all three inserted values: {{{}a{}}}, {{{}A{}}},
and {{{}b{}}}.
h2. Actual result
The Parquet query returns {{a}} and {{A}} as expected. The ORC query fails with:
{code}
org.apache.spark.SparkException:
[FAILED_READ_FILE.NO_HINT] Encountered error while reading file
file:///.../part-....orc. SQLSTATE: KD001
Caused by: java.lang.IllegalArgumentException:
Can't parse category at 'struct<`k`:string collate UTF^8_LCASE>'
at org.apache.orc.impl.ParserUtils.parseCategory(ParserUtils.java:70)
at org.apache.orc.TypeDescription.fromString(TypeDescription.java:230)
at
org.apache.spark.sql.execution.datasources.orc.OrcColumnarBatchReader.initialize(OrcColumnarBatchReader.java:142)
{code}
h2. Additional information
The failure also occurs with an equality filter, so it is not specific to
filter pushdown.
The likely cause is that {{OrcUtils.getOrcSchemaString}} uses Spark's
{{catalogString}} for a collated {{{}StringType{}}}, producing a schema string
that ORC's {{TypeDescription}} parser cannot handle.
The existing [CollationSuite
test|https://github.com/apache/spark/blob/38422aefaf599ffb4cfa579d98898d95b1f02081/sql/core/src/test/scala/org/apache/spark/sql/collation/CollationSuite.scala#L491-L510]
covers table creation and insertion for file-based data sources, including
ORC, but does not read back the stored column values.
was:
h2. What happened
Spark can create and write an ORC table with a {{STRING COLLATE UTF8_LCASE}}
column, but reading the stored values fails with an exception. The same column
type works with Parquet.
h2. How to reproduce
Run the following SQL using {{spark-sql}}:
{code:sql}
CREATE TABLE t_parquet_collated_string_control (
k STRING COLLATE UTF8_LCASE
) USING PARQUET;
INSERT INTO t_parquet_collated_string_control VALUES ('a'), ('A'), ('b');
SELECT k FROM t_parquet_collated_string_control WHERE k = 'a';
CREATE TABLE t_orc_collated_string (
k STRING COLLATE UTF8_LCASE
) USING ORC;
INSERT INTO t_orc_collated_string VALUES ('a'), ('A'), ('b');
SELECT k FROM t_orc_collated_string;
{code}
Save the SQL as {{repro.sql}} and run:
{code:bash}
spark-sql --master 'local[2]'
--conf spark.ui.enabled=false
--conf spark.sql.catalogImplementation=in-memory
--conf "spark.sql.warehouse.dir=$(mktemp -d)"
-f repro.sql
{code}
h2. Expected result
The Parquet query should return {{a}} and {{A}}, since {{UTF8_LCASE}} provides
case-insensitive comparison.
The ORC query should return all three inserted values: {{a}}, {{A}}, and {{b}}.
h2. Actual result
The Parquet query returns {{a}} and {{A}} as expected. The ORC query fails with:
{code:text}
org.apache.spark.SparkException:
[FAILED_READ_FILE.NO_HINT] Encountered error while reading file
file:///.../part-....orc. SQLSTATE: KD001
Caused by: java.lang.IllegalArgumentException:
Can't parse category at 'struct<`k`:string collate UTF^8_LCASE>'
at org.apache.orc.impl.ParserUtils.parseCategory(ParserUtils.java:70)
at org.apache.orc.TypeDescription.fromString(TypeDescription.java:230)
at
org.apache.spark.sql.execution.datasources.orc.OrcColumnarBatchReader.initialize(OrcColumnarBatchReader.java:142)
{code}
h2. Additional information
The failure also occurs with an equality filter, so it is not specific to
filter pushdown.
The likely cause is that {{OrcUtils.getOrcSchemaString}} uses Spark's
{{catalogString}} for a collated {{StringType}}, producing a schema string that
ORC's {{TypeDescription}} parser cannot handle.
The existing [CollationSuite
test|https://github.com/apache/spark/blob/38422aefaf599ffb4cfa579d98898d95b1f02081/sql/core/src/test/scala/org/apache/spark/sql/collation/CollationSuite.scala#L491-L510]
covers table creation and insertion for file-based data sources, including
ORC, but does not read back the stored column values.
> Reading ORC tables with collated STRING columns fails
> -----------------------------------------------------
>
> Key: SPARK-60123
> URL: https://issues.apache.org/jira/browse/SPARK-60123
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Environment: Spark: 5.0.0-SNAPSHOT
> Commit: 38422aefaf599ffb4cfa579d98898d95b1f02081
> Java: OpenJDK 17.0.17
> OS: macOS 15.7.4
> Reporter: Yibo Dong
> Priority: Major
> Labels: sql
>
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
