Muhammad Samir Khan created SPARK-32731:
-------------------------------------------
Summary: Added tests for arrays/maps of nested structs to
ReadSchemaSuite to test structs reuse
Key: SPARK-32731
URL: https://issues.apache.org/jira/browse/SPARK-32731
Project: Spark
Issue Type: Test
Components: SQL, Tests
Affects Versions: 3.0.0
Reporter: Muhammad Samir Khan
Splitting tests originally posted in
[PR|[https://github.com/apache/spark/pull/29352]] for SPARK-32531. The added
tests cover cases for maps and arrays of nested structs for different file
formats. Eg, [https://github.com/apache/spark/pull/29353] and
[https://github.com/apache/spark/pull/29354] add object reuse when reading ORC
and Avro files. However, for dynamic data structures like arrays and maps, we
do not know just by looking at the schema what the size of the data structure
will be so it has to be allocated when reading the data points. The added tests
provide coverage so that objects are not accidentally reused when encountering
maps and arrays.
AFAIK this is not covered by existing tests.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
