We have two brokers configured in a network of brokers.
<networkConnector
networkTTL="-1"
conduitSubscriptions="false"
decreaseNetworkConsumerPriority="true"
uri="static:(ssl://${activemq.networkPartner}:61616)"
userName="${activemq.username}"
password="${activemq.password}"
/>
</networkConnectors>
Both stores the messages on local disk using KahaDB.
<persistenceAdapter>
<kahaDB directory="${activemq.data}/kahadb"/>
</persistenceAdapter>
And they both run ActiveMQ-5.13.4 in java 8u45.
Clients are allowed to connect through openwires and amqps:
<transportConnectors>
<transportConnector name="nio+ssl"
uri="nio+ssl://0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="amqp+ssl"
uri="amqp+nio+ssl://0.0.0.0:5671?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
</transportConnectors>
We have about 10 queues and 1 topic configured. The major part of these
messages are sent by Nagios that frequently checks that everything is
fine by sending and receiving messages on all accepted protocols.
Since we setup this configuration, ActiveMQ has leaked file descriptors
and we haven't been able to figure out the cause.
According to `lsof` there is a lot of pipes and anon_inodes open:
[...]
java 33212 root 6462w FIFO 0,8 0t0 76388774 pipe
java 33212 root 6463r FIFO 0,8 0t0 76326567 pipe
java 33212 root 6464w FIFO 0,8 0t0 76326567 pipe
java 33212 root 6465u 0000 0,9 0 6827 anon_inode
java 33212 root 6466r FIFO 0,8 0t0 76344481 pipe
java 33212 root 6467w FIFO 0,8 0t0 76344481 pipe
java 33212 root 6468u 0000 0,9 0 6827 anon_inode
java 33212 root 6469u 0000 0,9 0 6827 anon_inode
java 33212 root 6470r FIFO 0,8 0t0 76372935 pipe
java 33212 root 6471w FIFO 0,8 0t0 76372935 pipe
java 33212 root 6472u 0000 0,9 0 6827 anon_inode
java 33212 root 6473r FIFO 0,8 0t0 76514685 pipe
java 33212 root 6474r FIFO 0,8 0t0 76435445 pipe
java 33212 root 6475r FIFO 0,8 0t0 76406235 pipe
java 33212 root 6476w FIFO 0,8 0t0 76406235 pipe
java 33212 root 6477u 0000 0,9 0 6827 anon_inode
java 33212 root 6478w FIFO 0,8 0t0 76435445 pipe
[...]
Graph of open files growing over time:
https://imgur.com/a/mM7MC
Other ActiveMQ "too many open files" related issues seems to be about
sockets that hangs on "CLOSE_WAIT" but that doesn't seems to be the
problem here.
Any idea how to fix this?
--
jocar
