Hi guys,
We are doing a pilot project adding JBM to existing EJB application.
Configuration is as following:
JBoss 4.2.3, JBM 1.4.2.GA-SP1, JBossRemoting 2.5.1
EJB3 application is running on a cluster of 7 servers.
All servers have the same application farmed across them.
We have a clustered queue and using clustered connection factory.
Initially everything seemed to work fine
But what we noticed is that during peak server load times (nightly) some
messages are getting "stuck" in the queues with
MessageCount=DeliveringCount=non-zero. This numbers stays non-zero until the
node is restarted. The symptoms are very similar to the open Jira issue
https://jira.jboss.org/jira/browse/JBMESSAGING-1456
This issue links to the JBoss Remoting issue
https://jira.jboss.org/jira/browse/JBREM-1112 which seems to be fixed in 2.5.1
release.
We've upgraded to this release but the issue seems to stay there.
While reading the Jira issues above and looking at the sympthoms it seems that
the issue may be caused by network timeouts. So one thing we've done is we
bumped the following values up in remoting-bisocket-service.xml file:
| <attribute name="clientLeasePeriod" isParam="true">15000</attribute>
| <attribute name="validatorPingPeriod" isParam="true">15000</attribute>
| <attribute name="validatorPingTimeout" isParam="true">10000</attribute>
| <attribute name="timeout" isParam="true">120000</attribute>
|
This (or maybe upgrade to the latest JBoss Remoting) seemed to help somewhat.
If previous night saw us loosing all messages in the queue, last night had just
10% of them stuck.
And actually it's a bit different tonight. We only see 4 messages stuck with
MessageCount=DeliveringCount=non-zero but on one of the nodes we see a bunch of
messages just sitting there with MessageCount=non-zero, DeliveringCount=0 and
ConsumerCount=0. These messages are just piling up. While checking other nodes,
four of them have same 0 ConsumerCount, two nodes have ConsumerCount=2 and one
node has ConsumerCount=6 (the first one).
With all nodes having same code deployed with MDB listening on the queue why
would the ConsumerCount be zero? And more importantly: why would the messages
go to the node that doesn't have the consumers?
With this piece I think we might be just missing something in our
configuration. I'll post our config below and hopefully you guys can point to
what we're doing wrong.
As for the messages "stuck" I'll try bumping timeouts even higher but this
doesn't seem to be a proper solution. I would definitely appreciate a good
advice here.
Our config:
Clustered queue defined in destinations-service.xml like this:
| <mbean code="org.jboss.jms.server.destination.QueueService"
|
name="jboss.messaging.destination:service=Queue,name=/queue/recommendationSentQueue"
| xmbean-dd="xmdesc/Queue-xmbean.xml">
| <depends
optional-attribute-name="ServerPeer">jboss.messaging:service=ServerPeer</depends>
| <depends>jboss.messaging:service=PostOffice</depends>
| <attribute name="Clustered">true</attribute>
| <attribute name="RedeliveryDelay">1000</attribute>
| </mbean>
|
MDB is annotated like this:
| @MessageDriven(activationConfig =
| {
| @ActivationConfigProperty(propertyName="destinationType",
propertyValue="javax.jms.Queue"),
| @ActivationConfigProperty(propertyName="destination",
propertyValue="/queue/recommendationSentQueue"),
| @ActivationConfigProperty(propertyName="DLQMaxResent",
propertyValue="10")
| })
|
JmsXA connection factory that we're using in the code is pointing to
ClusteredXAConnectionFactory in hajndi-jms-ds.xml :
| <mbean code="org.jboss.jms.jndi.JMSProviderLoader"
| name="jboss.messaging:service=JMSProviderLoader,name=HAJNDIJMSProvider">
| <attribute name="ProviderName">DefaultJMSProvider</attribute>
| <attribute name="ProviderAdapterClass">
| org.jboss.jms.jndi.JNDIProviderAdapter
| </attribute>
| <!-- The combined connection factory -->
| <attribute name="FactoryRef">ClusteredXAConnectionFactory</attribute>
| <!-- The queue connection factory -->
| <attribute
name="QueueFactoryRef">ClusteredXAConnectionFactory</attribute>
| <!-- The topic factory -->
| <attribute
name="TopicFactoryRef">ClusteredXAConnectionFactory</attribute>
| <!-- Access JMS via HAJNDI -->
| <attribute name="Properties">
| java.naming.factory.initial=org.jnp.interfaces.NamingContextFactory
| java.naming.factory.url.pkgs=org.jboss.naming:org.jnp.interfaces
| java.naming.provider.url=${jboss.bind.address:localhost}:1100
| jnp.disableDiscovery=false
| jnp.partitionName=${jboss.partition.name:DefaultPartition}
| jnp.discoveryGroup=${jboss.partition.udpGroup:230.0.0.4}
| jnp.discoveryPort=1102
| jnp.discoveryTTL=16
| jnp.discoveryTimeout=5000
| jnp.maxRetries=1
| </attribute>
| </mbean>
|
| <!-- The server session pool for Message Driven Beans -->
| <mbean code="org.jboss.jms.asf.ServerSessionPoolLoader"
| name="jboss.messaging:service=ServerSessionPoolMBean,name=StdJMSPool">
| <depends
optional-attribute-name="XidFactory">jboss:service=XidFactory</depends>
| <attribute name="PoolName">StdJMSPool</attribute>
| <attribute name="PoolFactoryClass">
| org.jboss.jms.asf.StdServerSessionPoolFactory
| </attribute>
| </mbean>
|
| <!-- JMS XA Resource adapter, use this to get transacted JMS in beans -->
| <tx-connection-factory>
| <jndi-name>JmsXA</jndi-name>
| <xa-transaction/>
| <rar-name>jms-ra.rar</rar-name>
|
<connection-definition>org.jboss.resource.adapter.jms.JmsConnectionFactory</connection-definition>
| <config-property name="SessionDefaultType"
type="java.lang.String">javax.jms.Topic</config-property>
| <config-property name="JmsProviderAdapterJNDI"
type="java.lang.String">java:/DefaultJMSProvider</config-property>
| <max-pool-size>20</max-pool-size>
|
<security-domain-and-application>JmsXARealm</security-domain-and-application>
| </tx-connection-factory>
|
Thank you guys!
Looking forward to hear any ideas or advice.
Victor
View the original post :
http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4230528#4230528
Reply to the post :
http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4230528
_______________________________________________
jboss-user mailing list
jboss-user@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/jboss-user
