[
https://issues.apache.org/jira/browse/DRILL-5544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16031465#comment-16031465
]
ASF GitHub Bot commented on DRILL-5544:
---------------------------------------
Github user paul-rogers commented on the issue:
https://github.com/apache/drill/pull/846
Another critical issue here. Direct memory is not a limitless resource,
unfortunately. Allocating memory larger than 16 MB causes memory fragmentation
as the allocation must come from the system, but all free memory is cached in
Netty in 16 MB chunks. So, if the buffer size here is larger than 16 MB, the
operation may fail with an OOM.
If this operation must buffer in memory, and must use larger than 16 MB,
then you need something like what the original code provided. That buffered
writer can be backed by a chain of 16 MB direct memory blocks.
> Out of heap running CTAS against text delimited
> -----------------------------------------------
>
> Key: DRILL-5544
> URL: https://issues.apache.org/jira/browse/DRILL-5544
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Parquet
> Affects Versions: 1.10.0
> Environment: - 2 or 4 nodes cluster
> - 4G or 8G of Java heap and more than 8G of direct memory
> - planner.width.max_per_node = 40
> - store.parquet.compression = none
> To generate lineitem.tbl file unzip dbgen.tgz archive and run:
> {code}dbgen -TL -s 500{code}
> Reporter: Vitalii Diravka
> Assignee: Vitalii Diravka
> Fix For: 1.11.0
>
> Attachments: dbgen.tgz
>
>
> This query causes the drillbit to hang:
> {code}
> create table xyz as
> select
> cast(columns[0] as bigint) l_orderkey,
> cast(columns[1] as integer) l_poartkey,
> cast(columns[2] as integer) l_suppkey,
> cast(columns[3] as integer) l_linenumber,
> cast(columns[4] as double) l_quantity,
> cast(columns[5] as double) l_extendedprice,
> cast(columns[6] as double) l_discount,
> cast(columns[7] as double) l_tax,
> cast(columns[8] as char(1)) l_returnflag,
> cast(columns[9] as char(1)) l_linestatus,
> cast(columns[10] as date) l_shipdate,
> cast(columns[11] as date) l_commitdate,
> cast(columns[12] as date) l_receiptdate,
> cast(columns[13] as char(25)) l_shipinstruct,
> cast(columns[14] as char(10)) l_shipmode,
> cast(columns[15] as varchar(44)) l_comment
> from
> `lineitem.tbl`;
> {code}
> OOM "Java heap space" from the drillbit.log:
> {code:title=drillbit.log|borderStyle=solid}
> ...
> 2017-02-07 22:38:11,031 [2765b496-0b5b-a3df-c252-a8bb9cd2e52f:frag:1:53]
> DEBUG o.a.d.e.s.p.ParquetDirectByteBufferAllocator -
> ParquetDirectByteBufferAllocator: Allocated 209715 bytes. Allocated
> ByteBuffer id: 1563631814
> 2017-02-07 22:38:16,478 [2765b496-0b5b-a3df-c252-a8bb9cd2e52f:frag:1:1] ERROR
> o.a.d.exec.server.BootStrapContext -
> org.apache.drill.exec.work.WorkManager$WorkerBee$1.run() leaked an exception.
> java.lang.OutOfMemoryError: Java heap space
> 2017-02-07 22:38:17,391 [2765b496-0b5b-a3df-c252-a8bb9cd2e52f:frag:1:13]
> ERROR o.a.drill.common.CatastrophicFailure - Catastrophic Failure Occurred,
> exiting. Information message: Unable to handle out of memory condition in
> FragmentExecutor.
> ...
> {code}
> To reproduce the issue please see environment details.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
