[
https://issues.apache.org/jira/browse/DRILL-8490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
shihuafeng updated DRILL-8490:
------------------------------
Description:
*1.DES*
** when ChannelClosedException, .ReconnectingConnection#CloseHandler
release sendingAccountor reference counter before netty release buffer, so
operator was closed before memory is released by netty .
*2 .exception info*
2024-04-13 08:45:39,909 [DataClient-3] WARN
o.apache.drill.exec.rpc.RequestIdMap - Failure while attempting to fail rpc
response.
java.lang.IllegalArgumentException: Self-suppression not permitted
at java.lang.Throwable.addSuppressed(Throwable.java:1072)
at
org.apache.drill.common.DeferredException.addException(DeferredException.java:88)
at
org.apache.drill.common.DeferredException.addThrowable(DeferredException.java:97)
at
org.apache.drill.exec.work.fragment.FragmentExecutor.fail(FragmentExecutor.java:502)
at
org.apache.drill.exec.work.fragment.FragmentExecutor.access$400(FragmentExecutor.java:131)
at
org.apache.drill.exec.work.fragment.FragmentExecutor$ExecutorStateImpl.fail(FragmentExecutor.java:518)
at
org.apache.drill.exec.ops.FragmentContextImpl.fail(FragmentContextImpl.java:298)
at
org.apache.drill.exec.ops.FragmentContextImpl$1.accept(FragmentContextImpl.java:152)
at
org.apache.drill.exec.ops.FragmentContextImpl$1.accept(FragmentContextImpl.java:149)
at
org.apache.drill.exec.ops.DataTunnelStatusHandler.failed(DataTunnelStatusHandler.java:45)
at
org.apache.drill.exec.rpc.data.DataTunnel$ThrottlingOutcomeListener.failed(DataTunnel.java:125)
at
org.apache.drill.exec.rpc.RequestIdMap$RpcListener.setException(RequestIdMap.java:145)
at
org.apache.drill.exec.rpc.RequestIdMap$SetExceptionProcedure.apply(RequestIdMap.java:78)
at
org.apache.drill.exec.rpc.RequestIdMap$SetExceptionProcedure.apply(RequestIdMap.java:68)
at
com.carrotsearch.hppc.IntObjectHashMap.forEach(IntObjectHashMap.java:692)
at
org.apache.drill.exec.rpc.RequestIdMap.channelClosed(RequestIdMap.java:64)
at
org.apache.drill.exec.rpc.AbstractRemoteConnection.channelClosed(AbstractRemoteConnection.java:192)
at
org.apache.drill.exec.rpc.AbstractClientConnection.channelClosed(AbstractClientConnection.java:97)
at
org.apache.drill.exec.rpc.RpcBus$ChannelClosedHandler.operationComplete(RpcBus.java:158)
at
org.apache.drill.exec.rpc.RpcBus$ChannelClosedHandler.operationComplete(RpcBus.java:135)
at
org.apache.drill.exec.rpc.ReconnectingConnection$CloseHandler.operationComplete(ReconnectingConnection.java:205)
at
org.apache.drill.exec.rpc.ReconnectingConnection$CloseHandler.operationComplete(ReconnectingConnection.java:192)
at
io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578)
at
io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:552)
at
io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491)
at
io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616)
at
io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:605)
at
io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
at
io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84)
at
io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:1164)
at
io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:755)
at
io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:731)
at
io.netty.channel.AbstractChannel$AbstractUnsafe.handleWriteError(AbstractChannel.java:950)
at
io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:933)
at
io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.forceFlush(AbstractNioChannel.java:361)
at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:716)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
at
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.drill.exec.rpc.ChannelClosedException: Channel closed
/10.32.112.138:51108 <--> /10.32.112.138:31012.
at
org.apache.drill.exec.rpc.RpcBus$ChannelClosedHandler.operationComplete(RpcBus.java:156)
{noformat}
*no* further _formatting_ is done here{noformat}
Summary: Sender operator fake memory leak result to sql when
ChannelClosedException (was: Sender operator fake memory leak result to sql
failed and parent allocator exception)
> Sender operator fake memory leak result to sql when ChannelClosedException
> ---------------------------------------------------------------------------
>
> Key: DRILL-8490
> URL: https://issues.apache.org/jira/browse/DRILL-8490
> Project: Apache Drill
> Issue Type: Bug
> Components: Server
> Affects Versions: 1.21.1
> Reporter: shihuafeng
> Priority: Major
> Fix For: 1.22.0
>
>
> *1.DES*
> ** when ChannelClosedException, .ReconnectingConnection#CloseHandler
> release sendingAccountor reference counter before netty release buffer, so
> operator was closed before memory is released by netty .
>
> *2 .exception info*
>
> 2024-04-13 08:45:39,909 [DataClient-3] WARN
> o.apache.drill.exec.rpc.RequestIdMap - Failure while attempting to fail rpc
> response.
> java.lang.IllegalArgumentException: Self-suppression not permitted
> at java.lang.Throwable.addSuppressed(Throwable.java:1072)
> at
> org.apache.drill.common.DeferredException.addException(DeferredException.java:88)
> at
> org.apache.drill.common.DeferredException.addThrowable(DeferredException.java:97)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.fail(FragmentExecutor.java:502)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.access$400(FragmentExecutor.java:131)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor$ExecutorStateImpl.fail(FragmentExecutor.java:518)
> at
> org.apache.drill.exec.ops.FragmentContextImpl.fail(FragmentContextImpl.java:298)
> at
> org.apache.drill.exec.ops.FragmentContextImpl$1.accept(FragmentContextImpl.java:152)
> at
> org.apache.drill.exec.ops.FragmentContextImpl$1.accept(FragmentContextImpl.java:149)
> at
> org.apache.drill.exec.ops.DataTunnelStatusHandler.failed(DataTunnelStatusHandler.java:45)
> at
> org.apache.drill.exec.rpc.data.DataTunnel$ThrottlingOutcomeListener.failed(DataTunnel.java:125)
> at
> org.apache.drill.exec.rpc.RequestIdMap$RpcListener.setException(RequestIdMap.java:145)
> at
> org.apache.drill.exec.rpc.RequestIdMap$SetExceptionProcedure.apply(RequestIdMap.java:78)
> at
> org.apache.drill.exec.rpc.RequestIdMap$SetExceptionProcedure.apply(RequestIdMap.java:68)
> at
> com.carrotsearch.hppc.IntObjectHashMap.forEach(IntObjectHashMap.java:692)
> at
> org.apache.drill.exec.rpc.RequestIdMap.channelClosed(RequestIdMap.java:64)
> at
> org.apache.drill.exec.rpc.AbstractRemoteConnection.channelClosed(AbstractRemoteConnection.java:192)
> at
> org.apache.drill.exec.rpc.AbstractClientConnection.channelClosed(AbstractClientConnection.java:97)
> at
> org.apache.drill.exec.rpc.RpcBus$ChannelClosedHandler.operationComplete(RpcBus.java:158)
> at
> org.apache.drill.exec.rpc.RpcBus$ChannelClosedHandler.operationComplete(RpcBus.java:135)
> at
> org.apache.drill.exec.rpc.ReconnectingConnection$CloseHandler.operationComplete(ReconnectingConnection.java:205)
> at
> org.apache.drill.exec.rpc.ReconnectingConnection$CloseHandler.operationComplete(ReconnectingConnection.java:192)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:552)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491)
> at
> io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616)
> at
> io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:605)
> at
> io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
> at
> io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84)
> at
> io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:1164)
> at
> io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:755)
>
> at
> io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:731)
> at
> io.netty.channel.AbstractChannel$AbstractUnsafe.handleWriteError(AbstractChannel.java:950)
> at
> io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:933)
> at
> io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.forceFlush(AbstractNioChannel.java:361)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:716)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
> at
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.drill.exec.rpc.ChannelClosedException: Channel closed
> /10.32.112.138:51108 <--> /10.32.112.138:31012.
> at
> org.apache.drill.exec.rpc.RpcBus$ChannelClosedHandler.operationComplete(RpcBus.java:156)
> {noformat}
> *no* further _formatting_ is done here{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
