Zamil Majdy created SPARK-43264:
-----------------------------------

             Summary: Avoid allocation of unwritten ColumnVector in 
VectorizedReader
                 Key: SPARK-43264
                 URL: https://issues.apache.org/jira/browse/SPARK-43264
             Project: Spark
          Issue Type: Improvement
          Components: Spark Core, SQL
    Affects Versions: 3.4.1, 3.5.0
            Reporter: Zamil Majdy


Spark Vectorized Reader allocates the array for every fields for each value 
count even the array is ended up empty. This causes a high memory consumption 
when reading a table with large struct+array or many columns with sparse value. 
One way to fix this is by lazily allocating the column vector and only 
allocates the array only when it is needed (array is written).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to