[ https://issues.apache.org/jira/browse/DRILL-4823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sudheesh Katkam updated DRILL-4823: ----------------------------------- Assignee: Arina Ielchiieva (was: Jinfeng Ni) > Fix OOM while trying to prune partitions with reasonable data size > ------------------------------------------------------------------ > > Key: DRILL-4823 > URL: https://issues.apache.org/jira/browse/DRILL-4823 > Project: Apache Drill > Issue Type: Bug > Components: Functions - Drill, Query Planning & Optimization > Affects Versions: 1.6.0 > Reporter: Arina Ielchiieva > Assignee: Arina Ielchiieva > Fix For: 1.9.0 > > > _Example query:_ > {code:sql} > select '/'||dir0||'/'||dir1||'/'||dir2||'/'||dir3 , count(*) > FROM dfs.`/path/to/parquet/files` > WHERE ('/'||dir0||'/'||dir1||'/'||dir2||'/'||dir3) IN ( > '/2015/11/30', > '//2015/09/01', > '/2015/09/02', > '/2015/09/03', > '/2015/09/04', > '/2015/09/09', > '/2016/03/30' > ) > group by '/'||dir0||'/'||dir1||'/'||dir2||'/'||dir3 > order by 1; > {code} > _Error:_ > OOM while trying to prune partitions: > {noformat} > org.apache.drill.exec.exception.OutOfMemoryException: Unable to allocate > buffer of size 256 due to memory limit. Current allocation: 5242880 > at > org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:216) > ~[drill-memory-base-1.6.0.jar:1.6.0] > at > org.apache.drill.exec.ops.BufferManagerImpl.getManagedBuffer(BufferManagerImpl.java:60) > ~[drill-java-exec-1.6.0.jar:1.6.0] > at > org.apache.drill.exec.ops.BufferManagerImpl.getManagedBuffer(BufferManagerImpl.java:56) > ~[drill-java-exec-1.6.0.jar:1.6.0] > at > org.apache.drill.exec.ops.QueryContext.getManagedBuffer(QueryContext.java:241) > ~[drill-java-exec-1.6.0.jar:1.6.0] > at > org.apache.drill.exec.expr.fn.interpreter.InterpreterEvaluator$EvalVisitor.getManagedBufferIfAvailable(InterpreterEvaluator.java:158) > ~[drill-java-exec-1.6.0.jar:1.6.0] > {noformat} > _Cause:_ > Interpreter always asks for a new buffer to hold varchar/varbinary or decimal > constant values. That's why the memory size required would be proportion to # > of constant expressions multiplied by # of input rows (partition). This is > different from evaluation from run-time generated where constant expression > will be evaluated once and use only one buffer per value. > _Fix:_ > To use one buffer for each unique constant value in query. -- This message was sent by Atlassian JIRA (v6.3.4#6332)