[ https://issues.apache.org/jira/browse/ARTEMIS-2623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17835898#comment-17835898 ]
Justin Bertram edited comment on ARTEMIS-2623 at 4/10/24 9:49 PM: ------------------------------------------------------------------ I'm not really sure what the broker can do about this when [the JVM can't (or won't) do anything about this|https://bugs.openjdk.org/browse/JDK-8155004]. This looks like a potential work-around: https://github.com/airlift/jvmkill. was (Author: jbertram): I'm not really sure what the broker can do about this when the JVM can't (or won't) do anything about this. This looks like a potential work-around: https://github.com/airlift/jvmkill. > Artemis continues to run in broken state after OOME (thread) > ------------------------------------------------------------ > > Key: ARTEMIS-2623 > URL: https://issues.apache.org/jira/browse/ARTEMIS-2623 > Project: ActiveMQ Artemis > Issue Type: Bug > Affects Versions: 2.11.0 > Environment: openjdk 11.0.3 2019-04-16 LTS > OpenJDK Runtime Environment 18.9 (build 11.0.3+7-LTS) > OpenJDK 64-Bit Server VM 18.9 (build 11.0.3+7-LTS, mixed mode, sharing) > Reporter: Keith Wall > Priority: Major > > If Artemis runs out of threads with the JVM signalling this with an OOME > error, the process continues in a broken state logging exceptions. It would > be better if the process exited so the running environment (Kubernetes, job > control etc) can restart it. > Note that stack trace shows a {{ClassCastException}} where an attempt to made > to assign the OOME (Error) to an Exception. > I read from [JDK-8155004|https://bugs.openjdk.java.net/browse/JDK-8155004] > that the JVM flag {{XxxxOnOutOfMemoryError}} is not applied to this sort of > OOME. > {noformat} > [104223.528s][warning][os,thread] Failed to start thread - pthread_create > failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached. > 2020-02-19T14:41:04.437Z WARN [ProtonServerSenderContext] > java.lang.ClassCastException: class java.lang.OutOfMemoryError cannot be cast > to class java.lang.Exception (java.lang.OutOfMemoryError and > java.lang.Exception are in module java.base of loader 'bootstrap') > java.lang.ClassCastException: class java.lang.OutOfMemoryError cannot be cast > to class java.lang.Exception (java.lang.OutOfMemoryError and > java.lang.Exception are in module java.base of loader 'bootstrap') > at > org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.individualAcknowledge(ServerConsumerImpl.java:998) > at > org.apache.activemq.artemis.protocol.amqp.broker.AMQPSessionCallback.ack(AMQPSessionCallback.java:399) > at > org.apache.activemq.artemis.protocol.amqp.proton.ProtonServerSenderContext.onMessage(ProtonServerSenderContext.java:621) > at > org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.onDelivery(AMQPConnectionContext.java:583) > at > org.apache.activemq.artemis.protocol.amqp.proton.handler.Events.dispatch(Events.java:92) > at > org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.dispatch(ProtonHandler.java:485) > at > org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.flush(ProtonHandler.java:285) > at > org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.inputBuffer(ProtonHandler.java:242) > at > org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.inputBuffer(AMQPConnectionContext.java:180) > at > org.apache.activemq.artemis.protocol.amqp.broker.ActiveMQProtonRemotingConnection.bufferReceived(ActiveMQProtonRemotingConnection.java:145) > at > org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:654) > at > org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:73) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) > at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1470) > at > io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1219) > at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1266) > at > io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498) > at > io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437) > at > io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) > at > io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) > at > io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) > at > io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796) > at > io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427) > at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328) > at > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) > at > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > at > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > 2020-02-19T14:41:04.437Z WARN [ProtonHandler] AMQ119007: error acknowledging > message > org.apache.activemq.artemis.protocol.amqp.broker.AMQPMessage@51dd2f4c, class > java.lang.OutOfMemoryError cannot be cast to class java.lang.Exception > (java.lang.OutOfMemoryError and java.lang.Exception are in module java.base > of loader 'bootstrap') > org.apache.activemq.artemis.protocol.amqp.exceptions.ActiveMQAMQPIllegalStateException: > AMQ119007: error acknowledging message > org.apache.activemq.artemis.protocol.amqp.broker.AMQPMessage@51dd2f4c, class > java.lang.OutOfMemoryError cannot be cast to class java.lang.Exception > (java.lang.OutOfMemoryError and java.lang.Exception are in module java.base > of loader 'bootstrap') > at > org.apache.activemq.artemis.protocol.amqp.proton.ProtonServerSenderContext.onMessage(ProtonServerSenderContext.java:624) > at > org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.onDelivery(AMQPConnectionContext.java:583) > at > org.apache.activemq.artemis.protocol.amqp.proton.handler.Events.dispatch(Events.java:92) > at > org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.dispatch(ProtonHandler.java:485) > at > org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.flush(ProtonHandler.java:285) > at > org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.inputBuffer(ProtonHandler.java:242) > at > org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.inputBuffer(AMQPConnectionContext.java:180) > at > org.apache.activemq.artemis.protocol.amqp.broker.ActiveMQProtonRemotingConnection.bufferReceived(ActiveMQProtonRemotingConnection.java:145) > at > org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:654) > at > org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:73) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) > at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1470) > at > io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1219) > at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1266) > at > io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498) > at > io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437) > at > io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) > at > io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) > at > io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) > at > io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796) > at > io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427) > at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328) > at > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) > at > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > at > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > [104223.571s][warning][os,thread] Failed to start thread - pthread_create > failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached. > {noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010)