Cheng Su created SPARK-34862:
--------------------------------
Summary: Support nested column in Spark ORC vectorized readers
Key: SPARK-34862
URL: https://issues.apache.org/jira/browse/SPARK-34862
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 3.2.0
Reporter: Cheng Su
The task is to support nested column type in Spark ORC vectorized reader.
Currently both ORC vectorized reader does not support nested column type
(struct, array and map). We implemented nested column vectorized reader for
FB-ORC in our internal fork of Spark. We are seeing performance improvement
compared to non-vectorized reader when reading nested columns. In addition,
this can also help improve the non-nested column performance when reading
non-nested and nested columns together in one query.
ORC:
[https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFileFormat.scala#L138]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
