Liang-Chi Hsieh created SPARK-27832:
---------------------------------------
Summary: Don't decompress and create column batch when the task is
completed
Key: SPARK-27832
URL: https://issues.apache.org/jira/browse/SPARK-27832
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 3.0.0
Reporter: Liang-Chi Hsieh
Cached relation decompresses and creates column batch when accessing cache.
It's possible that a thread doesn't stop immediately reading cached relation
after the task is completed. Due to race condition, cached relation might still
decompresses and creates new and unnecessary batch. At the moment, the returned
batch is also immediately closed. At the reader side, it can cause null
exception, and we probably need to hide such exception.
We don't need to create the batch if the task is completed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
