James Turton created DRILL-8366:
-----------------------------------

             Summary: Late release of compressor memory in the Parquet writer
                 Key: DRILL-8366
                 URL: https://issues.apache.org/jira/browse/DRILL-8366
             Project: Apache Drill
          Issue Type: Bug
          Components: Storage - Parquet
    Affects Versions: 1.20.2
            Reporter: James Turton
            Assignee: James Turton
             Fix For: 1.20.3


The Parquet writer waits until the end of the entire write before releasing its 
compression codec factory. The factory in turn releases compressors which 
release direct memory buffers used during compression. This deferred release 
leads a build up of direct memory use and can cause large write jobs to fail. 
The Parquet writer can instead release the abovementioned each time that a 
file/row group is flushed.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Reply via email to