[jira] [Commented] (DRILL-8489) Sender memory leak when rpc encode exception
[
https://issues.apache.org/jira/browse/DRILL-8489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17842679#comment-17842679
]
ASF GitHub Bot commented on DRILL-8489:
---
cgivre merged PR #2901:
URL: https://github.com/apache/drill/pull/2901
> Sender memory leak when rpc encode exception
>
>
> Key: DRILL-8489
> URL: https://issues.apache.org/jira/browse/DRILL-8489
> Project: Apache Drill
> Issue Type: Bug
> Components: Server
>Affects Versions: 1.21.1
>Reporter: shihuafeng
>Priority: Major
> Fix For: 1.22.0
>
>
> When encode throw Exception, if encode msg instanceof ReferenceCounted, netty
> can release msg, but drill convert msg to OutboundRpcMessage, so netty can
> not release msg. this causes sender memory leaks
> exception info
> {code:java}
> 2024-04-16 16:25:57,998 [DataClient-7] ERROR
> o.a.d.exec.rpc.RpcExceptionHandler - Exception in RPC communication.
> Connection: /10.32.112.138:47924 <--> /10.32.112.138:31012 (data client).
> Closing connection.
> io.netty.handler.codec.EncoderException:
> org.apache.drill.exec.exception.OutOfMemoryException: Unable to allocate
> buffer of size 4096 due to memory limit (9223372036854775807). Current
> allocation: 0
> at
> io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:107)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:881)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:940)
> at
> io.netty.channel.AbstractChannelHandlerContext$WriteTask.run(AbstractChannelHandlerContext.java:1247)
> at
> io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
> at
> io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
> at
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.drill.exec.exception.OutOfMemoryException: Unable to
> allocate buffer of size 4096 due to memory limit (9223372036854775807).
> Current allocation: 0
> at
> org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:245)
> at
> org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:220)
> at
> org.apache.drill.exec.memory.DrillByteBufAllocator.buffer(DrillByteBufAllocator.java:55)
> at
> org.apache.drill.exec.memory.DrillByteBufAllocator.buffer(DrillByteBufAllocator.java:50)
> at org.apache.drill.exec.rpc.RpcEncoder.encode(safeRelease.java:87)
> at org.apache.drill.exec.rpc.RpcEncoder.encode(RpcEncoder.java:38)
> at
> io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:90){code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8489) Sender memory leak when rpc encode exception
[
https://issues.apache.org/jira/browse/DRILL-8489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17837949#comment-17837949
]
ASF GitHub Bot commented on DRILL-8489:
---
shfshihuafeng opened a new pull request, #2901:
URL: https://github.com/apache/drill/pull/2901
# [DRILL-8489](https://issues.apache.org/jira/browse/DRILL-8489): Sender
memory leak when rpc encode exception
## Description
When encode throw Exception, if encode msg instanceof ReferenceCounted,
netty can release msg, but drill convert msg to OutboundRpcMessage, so netty
can not release msg.
## Documentation
(Please describe user-visible changes similar to what should appear in the
Drill documentation.)
## Testing
1. export DRILL_MAX_DIRECT_MEMORY=${DRILL_MAX_DIRECT_MEMORY:-"2G"}
2. tpch 1s
3. tpch sql 8
```
select
o_year,
sum(case when nation = 'CHINA' then volume else 0 end) / sum(volume) as
mkt_share
from (
select
extract(year from o_orderdate) as o_year,
l_extendedprice * 1.0 as volume,
n2.n_name as nation
from hive.tpch1s.part, hive.tpch1s.supplier, hive.tpch1s.lineitem,
hive.tpch1s.orders, hive.tpch1s.customer, hive.tpch1s.nation n1,
hive.tpch1s.nation n2, hive.tpch1s.region
where
p_partkey = l_partkey
and s_suppkey = l_suppkey
and l_orderkey = o_orderkey
and o_custkey = c_custkey
and c_nationkey = n1.n_nationkey
and n1.n_regionkey = r_regionkey
and r_name = 'ASIA'
and s_nationkey = n2.n_nationkey
and o_orderdate between date '1995-01-01'
and date '1996-12-31'
and p_type = 'LARGE BRUSHED BRASS') as all_nations
group by o_year
order by o_year;
```
5. This scenario is relatively easy to Reproduce by running the following
script
```
drill_home=/data/shf/apache-drill-1.22.0-SNAPSHOT/bin
fileName=/data/shf/1s/shf.txt
random_sql(){
#for i in `seq 1 3`
while true
do
num=$((RANDOM%22+1))
if [ -f $fileName ]; then
echo "$fileName" " is exit"
exit 0
else
$drill_home/sqlline -u
\"jdbc:drill:zk=jupiter-2:2181/drill_shf/jupiterbits_shf1\" -f tpch_sql8.sql >>
sql8.log 2>&1
fi
done
}
main(){
unset HADOOP_CLASSPATH
#TPCH power test
for i in `seq 1 25`
do
random_sql &
done
}
```
> Sender memory leak when rpc encode exception
>
>
> Key: DRILL-8489
> URL: https://issues.apache.org/jira/browse/DRILL-8489
> Project: Apache Drill
> Issue Type: Bug
> Components: Server
>Affects Versions: 1.21.1
>Reporter: shihuafeng
>Priority: Major
> Fix For: 1.22.0
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
