[
https://issues.apache.org/jira/browse/IGNITE-26969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18039188#comment-18039188
]
Pavel Tupitsyn commented on IGNITE-26969:
-----------------------------------------
Tried:
* With and without JDBC
* Multithreaded scenarios
* Unstable cluster scenarios
(org.apache.ignite.client.RetryPolicyTest#testUnstableCluster)
No leaks found, memory usage is stable after millions of iterations. OOM might
happen if something else consumed all the memory, so the stack trace does not
indicate any issues.
[~akhitrin] I will close this for now. Please collect a heap dump if you
encounter this again and open a new ticket.
> IgniteClient: OutOfMemoryError when working with JDBC under C3P0 pool
> ----------------------------------------------------------------------
>
> Key: IGNITE-26969
> URL: https://issues.apache.org/jira/browse/IGNITE-26969
> Project: Ignite
> Issue Type: Bug
> Components: jdbc ai3
> Affects Versions: 3.2
> Reporter: Andrey Khitrin
> Assignee: Pavel Tupitsyn
> Priority: Critical
> Labels: ignite-3
> Fix For: 3.2
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> h2. How to reproduce
> Use a C3P0 connection pool to work with AI3 cluster.
> {code:java}
> ComboPooledDataSource c3p0Pool = new ComboPooledDataSource();
> c3p0Pool.setJdbcUrl("jdbc:ignite:thin://" + thinClientEndpoint);
> try (Connection connection = cluster().dataSource().getConnection()) {
> ...
> {code}
> After some amount of time, the following error may occur:
> {code}
> java.sql.SQLException: Connections could not be acquired from the underlying
> database!
> at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:118)
> at
> com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:692)
> at
> com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:140)
> at
> org.gridgain.ai3tests.tests.failover.ClusterFailoverTestBase.createTables(ClusterFailoverTestBase.java:193)
> at
> org.gridgain.ai3tests.tests.failover.ClusterFailoverTestBase.getFilledTables(ClusterFailoverTestBase.java:182)
> at
> org.gridgain.ai3tests.tests.failover.ClusterFailover1NodeTest.singleKillAndRestartNodeWhenDataIsLoaded(ClusterFailover1NodeTest.java:60)
> at
> java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
> at java.base/java.lang.reflect.Method.invoke(Method.java:580)
> at
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:787)
> at
> org.junit.platform.commons.support.ReflectionSupport.invokeMethod(ReflectionSupport.java:479)
> at
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
> at
> org.gridgain.ai3tests.tests.extensions.cluster.IgniteClusterByIgniteRunnerExtension.interceptTestTemplateMethod(IgniteClusterByIgniteRunnerExtension.java:146)
> at
> org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:112)
> at
> org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:94)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
> at
> io.qameta.allure.junit5.AllureJunit5.interceptTestTemplateMethod(AllureJunit5.java:59)
> at
> org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:112)
> at
> org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:94)
> at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
> at
> org.junit.jupiter.api.AssertTimeoutPreemptively.lambda$submitTask$3(AssertTimeoutPreemptively.java:95)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
> at java.base/java.lang.Thread.run(Thread.java:1583)
> Caused by: com.mchange.v2.resourcepool.CannotAcquireResourceException: A
> ResourcePool could not acquire a resource from its primary factory or source.
> at
> com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1507)
> at
> com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:644)
> at
> com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:554)
> at
> com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutAndMarkConnectionInUse(C3P0PooledConnectionPool.java:758)
> at
> com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:685)
> ... 23 common frames omitted
> Caused by: java.sql.SQLException: Failed to connect to server
> at
> org.apache.ignite.internal.jdbc.JdbcConnection.<init>(JdbcConnection.java:141)
> at
> org.apache.ignite.jdbc.IgniteJdbcDriver.connect(IgniteJdbcDriver.java:177)
> at
> com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:161)
> at
> com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:161)
> at
> com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:147)
> at
> com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:202)
> at
> com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1176)
> at
> com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1163)
> at
> com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:44)
> at
> com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1908)
> at
> com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696)
> Caused by: org.apache.ignite.lang.IgniteException: Cannot reserve 131072
> bytes of direct buffer memory (allocated: 536847170, limit: 536870912)
> at
> org.apache.ignite.internal.util.ViewUtils.ensurePublicException(ViewUtils.java:72)
> at org.apache.ignite.internal.util.ViewUtils.sync(ViewUtils.java:46)
> at
> org.apache.ignite.internal.jdbc.JdbcConnection.buildClient(JdbcConnection.java:189)
> at
> org.apache.ignite.internal.jdbc.JdbcConnection.<init>(JdbcConnection.java:139)
> ... 10 common frames omitted
> Caused by: java.lang.OutOfMemoryError: Cannot reserve 131072 bytes of direct
> buffer memory (allocated: 536847170, limit: 536870912)
> at java.base/java.nio.Bits.reserveMemory(Bits.java:178)
> at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:111)
> at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:360)
> at io.netty.util.internal.CleanerJava9.allocate(CleanerJava9.java:86)
> at
> io.netty.util.internal.PlatformDependent.allocateDirect(PlatformDependent.java:600)
> at
> io.netty.buffer.UnpooledDirectByteBuf.allocateDirectBuffer(UnpooledDirectByteBuf.java:138)
> at
> io.netty.buffer.UnpooledDirectByteBuf.<init>(UnpooledDirectByteBuf.java:81)
> at
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:48)
> at
> io.netty.buffer.UnsafeByteBufUtil.newUnsafeDirectByteBuf(UnsafeByteBufUtil.java:698)
> at
> io.netty.buffer.AdaptiveByteBufAllocator$DirectChunkAllocator.allocate(AdaptiveByteBufAllocator.java:115)
> at
> io.netty.buffer.AdaptivePoolingAllocator$SizeClassChunkController.newChunkAllocation(AdaptivePoolingAllocator.java:580)
> at
> io.netty.buffer.AdaptivePoolingAllocator$Magazine.allocate(AdaptivePoolingAllocator.java:983)
> at
> io.netty.buffer.AdaptivePoolingAllocator$Magazine.tryAllocate(AdaptivePoolingAllocator.java:854)
> at
> io.netty.buffer.AdaptivePoolingAllocator$MagazineGroup.allocate(AdaptivePoolingAllocator.java:396)
> at
> io.netty.buffer.AdaptivePoolingAllocator.allocate(AdaptivePoolingAllocator.java:261)
> at
> io.netty.buffer.AdaptivePoolingAllocator.allocate(AdaptivePoolingAllocator.java:248)
> at
> io.netty.buffer.AdaptiveByteBufAllocator.newDirectBuffer(AdaptiveByteBufAllocator.java:67)
> at
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:168)
> at
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:154)
> at
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:88)
> at
> org.apache.ignite.internal.client.io.netty.NettyClientConnection.getBuffer(NettyClientConnection.java:86)
> at
> org.apache.ignite.internal.client.TcpClientChannel.handshakeReqAsync(TcpClientChannel.java:710)
> at
> org.apache.ignite.internal.client.TcpClientChannel.handshakeAsync(TcpClientChannel.java:678)
> at
> org.apache.ignite.internal.client.TcpClientChannel.lambda$initAsync$0(TcpClientChannel.java:202)
> at
> java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
> at
> java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179)
> at
> org.apache.ignite.internal.client.io.netty.NettyClientConnectionMultiplexer.lambda$openAsync$1(NettyClientConnectionMultiplexer.java:177)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:604)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:597)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:573)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:506)
> at
> io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:650)
> at
> io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:639)
> at
> io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:119)
> at
> io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84)
> at
> io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:350)
> at
> io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:385)
> at
> io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.handle(AbstractNioChannel.java:432)
> at
> io.netty.channel.nio.NioIoHandler$DefaultNioRegistration.handle(NioIoHandler.java:388)
> at
> io.netty.channel.nio.NioIoHandler.processSelectedKey(NioIoHandler.java:596)
> at
> io.netty.channel.nio.NioIoHandler.processSelectedKeysOptimized(NioIoHandler.java:571)
> at
> io.netty.channel.nio.NioIoHandler.processSelectedKeys(NioIoHandler.java:512)
> at io.netty.channel.nio.NioIoHandler.run(NioIoHandler.java:484)
> at
> io.netty.channel.SingleThreadIoEventLoop.runIo(SingleThreadIoEventLoop.java:225)
> at
> io.netty.channel.SingleThreadIoEventLoop.run(SingleThreadIoEventLoop.java:196)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:1193)
> at
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:1583)
> {code}
> h2. Context
> It's highly likely that this issue may appear after IGNITE-26719. There are
> two arguments for this:
> 1. It wasn't observed before 2025-10-31, and after this date it has started
> to appear quite often. IGNITE-26719 was
> [merged|https://github.com/apache/ignite-3/commit/dd091a9ab1a70cae3f9ab0d38cc5b90c7293e2bc]
> at 2025-10-29.
> 2. A provided stack trace includes TcpClientChannel.handshakeAsync method
> which was changed by the provided issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
