Essentially just from this log output I assume the server your broker is
running out of ram to use.
This can be either
A) genuine memory leak in artemis
B) you simply don't have enough ram for the load/throughout.
Some questions:
Is the load constant?
Do you have server ram usage metrics available?
You should ensure there is more ram available to the broker instance than just
heap allocation, for network buffers etc.
Cheers
Mike
Sent from my iPhone
> On 2 Jun 2017, at 09:44, Helge Waastad <he...@waastad.org> wrote:
>
> Hi,
> I'm running artemis 2.2.0 as a docker container.
>
> I'm collecting MQTT messages an these are consumed by a JMS consumer
> (artemis-jms-client)
>
> It's running fine for a while, but suddenly this appear (docker *-
> json.log):
>
> {"log":"19:16:12,338 WARN
> [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnection]
> Trying to allocate 712 bytes, System is throwing OutOfMemoryError on
> NettyConnection org.apache.activemq.art
> emis.core.remoting.impl.netty.NettyServerConnection@6f035b0a[local=
> /10.42.154.105:61616, remote=/10.42.21.198:40844], there are currently
> pendingWrites: [NETTY] -\u003e 0[EVENT LOOP] -\u003e 0 causes: fail
> ed to allocate 16777216 byte(s) of direct memory (used: 1057466368,
> max:
> 1073741824): io.netty.util.internal.OutOfDirectMemoryError: failed to
> allocate 16777216 byte(s) of direct memory (used: 1057466368, m
> ax:
> 1073741824)\r\n","stream":"stdout","time":"2017-06-
> 01T19:16:12.342853929Z"}
> {"log":"19:16:12,342 WARN [org.apache.activemq.artemis.core.server]
> AMQ222151: removing consumer which did not handle a message,
> consumer=ServerConsumerImpl [id=0, filter=null,
> binding=LocalQueueBinding [a
> ddress=CentreonTopic, queue=QueueImpl[name=CentreonTopic,
> postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::serverUUID=772ad6f8-4630-11e7-93cd-
> 02a837635b7b],
> temp=false]@3e389a2d, filter=null, name=Cent
> reonTopic,
> clusterName=CentreonTopic772ad6f8-4630-11e7-93cd-02a837635b7b]],
> message=Reference[715739]:NON-
> RELIABLE:CoreMessage[messageID=715739,durable=false,userID=null,priorit
> y=0,
> timestamp=0,expiration=0
> , durable=false,
> address=CentreonTopic,properties=TypedProperties[mqtt.message.retain=fa
> lse,mqtt.qos.level=0]]@1623021181:
> io.netty.util.internal.OutOfDirectMemoryError: failed to allocate
> 16777216 byte(s)
> of direct memory (used: 1057466368, max:
> 1073741824)\r\n","stream":"stdout","time":"2017-06-
> 01T19:16:12.347107296Z"}
> {"log":"19:31:54,236 WARN
> [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnection]
> Trying to allocate 548 bytes, System is throwing OutOfMemoryError on
> NettyConnection org.apache.activemq.art
> emis.core.remoting.impl.netty.NettyServerConnection@7b18e1a6[local=
> /10.42.154.105:61616, remote=/10.42.162.183:48376], there are
> currently
> pendingWrites: [NETTY] -\u003e 0[EVENT LOOP] -\u003e 0 causes: fai
> led to allocate 16777216 byte(s) of direct memory (used: 1057466368,
> max: 1073741824): io.netty.util.internal.OutOfDirectMemoryError:
> failed
> to allocate 16777216 byte(s) of direct memory (used: 1057466368,
> max:
> 1073741824)\r\n","stream":"stdout","time":"2017-06-
> 01T19:31:54.238904544Z"}
> {"log":"19:31:54,238 WARN [org.apache.activemq.artemis.core.server]
> AMQ222151: removing consumer which did not handle a message,
> consumer=ServerConsumerImpl [id=0, filter=null,
> binding=LocalQueueBinding [a
> ddress=CentreonTopic, queue=QueueImpl[name=CentreonTopic,
> postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::serverUUID=772ad6f8-4630-11e7-93cd-
> 02a837635b7b],
> temp=false]@3e389a2d, filter=null, name=Cent
> reonTopic,
> clusterName=CentreonTopic772ad6f8-4630-11e7-93cd-02a837635b7b]],
> message=Reference[722892]:NON-
> RELIABLE:CoreMessage[messageID=722892,durable=false,userID=null,priorit
> y=0,
> timestamp=0,expiration=0
> , durable=false,
> address=CentreonTopic,properties=TypedProperties[mqtt.message.retain=fa
> lse,mqtt.qos.level=0]]@1252621657:
> io.netty.util.internal.OutOfDirectMemoryError: failed to allocate
> 16777216 byte(s)
> of direct memory (used: 1057466368, max:
> 1073741824)\r\n","stream":"stdout","time":"2017-06-
> 01T19:31:54.239955162Z"}
>
>
>
> Then after a couple of hours:
>
> {"log":"23:22:24,013 WARN [io.netty.channel.DefaultChannelPipeline]
> An
> exceptionCaught() event was fired, and it reached at the tail of the
> pipeline. It usually means the last handler in the pipeline did n
> ot handle the exception.:
> io.netty.util.internal.OutOfDirectMemoryError:
> failed to allocate 16777216 byte(s) of direct memory (used:
> 1057466368,
> max: 1073741824)\r\n","stream":"stdout","time":"2017-06-01T23
> :22:24.015087347Z"}
> {"log":"23:22:24,014 WARN [io.netty.channel.DefaultChannelPipeline]
> An
> exceptionCaught() event was fired, and it reached at the tail of the
> pipeline. It usually means the last handler in the pipeline did n
> ot handle the exception.:
> io.netty.util.internal.OutOfDirectMemoryError:
> failed to allocate 16777216 byte(s) of direct memory (used:
> 1057466368,
> max: 1073741824)\r\n","stream":"stdout","time":"2017-06-01T23
> :22:24.015759902Z"}
> {"log":"23:22:24,015 WARN [io.netty.channel.DefaultChannelPipeline]
> An
> exceptionCaught() event was fired, and it reached at the tail of the
> pipeline. It usually means the last handler in the pipeline did n
> ot handle the exception.:
> io.netty.util.internal.OutOfDirectMemoryError:
> failed to allocate 16777216 byte(s) of direct memory (used:
> 1057466368,
> max: 1073741824)\r\n","stream":"stdout","time":"2017-06-01T23
> :22:24.016623101Z"}
>
>
> And this message is looping and in 5 mins it's filled my 12GB drive.
>
> Any clues what to do? I'll do some more debugging.
>
> /hw
