[
https://issues.apache.org/jira/browse/AMQ-6865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16418074#comment-16418074
]
Rohan Mars commented on AMQ-6865:
---------------------------------
I've been able to easily reproduce this error on 5.15.3. Use a packet loss
strategy between amq and db and the problem will appear instantly.
{code:java}
tc qdisc add dev enp0s9 root netem loss 100%
{code}
1) You need to configure the mysql jdbc connection string to timeout before the
lease check interval does. The default jdbc mysql connection doesn't timeout,
which equivalent timeout settings will be required for any alternative database
connection string.
{code:java}
jdbc:mysql://dbhost/activemq?relaxAutoCommit=true&useSSL=false&useJDBCCompliantTimezoneShift=true&connectTimeout=3000&socketTimeout=3000"{code}
2) The use of the LeaseLockerIOExceptionHandler is not enabled by default when
you use a lease-database-locker, major documentation gap. You need to declare
the bean and inject it. Additionally, you need to set back StopStartConnectors
to false, so a full restart of the broker occurs (which if it can't reconnect
will actually shut down the process). I wasn't convinced that keeping it as
"true" property shuts down the broker and/or cleans up it's internal state to
be a slave mode trying to reconnect, which only occurs in the logs when you
restore the packet loss after the original salve has become master.
{code:java}
<bean id="ioExceptionHandler"
class="org.apache.activemq.util.LeaseLockerIOExceptionHandler">
<property name="stopStartConnectors"><value>false</value></property>
</bean>
...
<broker xmlns="http://activemq.apache.org/schema/core"; brokerName="amq1"
dataDirectory="${activemq.data}" ioExceptionHandler="#ioExceptionHandler">
....
{code}
Ideally, ActiveMQ would support starting up in a "retrying warm mode" when the
persistence adapter is using JDBC by going in a loop trying to startup, i.e.
connecting to the database. Shutting down the whole process requires some
additional operational overhead. But it appears that would not be a straight
forward change based on the current code.
> JDBC Master/Slave: Master did not leave lock when the master loose database
> connection.
> ---------------------------------------------------------------------------------------
>
> Key: AMQ-6865
> URL: https://issues.apache.org/jira/browse/AMQ-6865
> Project: ActiveMQ
> Issue Type: Bug
> Reporter: Pranjal Bathia
> Priority: Major
>
> We have JBoss AMQ instance of jboss-a-mq-6.3.0.redhat-187, and we use JDBC
> for master/slave configuration.
> As per our observation, during networking disruption for connecting to the
> database, a broker was unable to recover cleanly.
> In hawtio, both master/slave was showing as master, and ActiveMQ tab with amq
> connection was available on both broker and which resulted in a situation
> where one app was sending messages to broker01 (which was acting as producer)
> and all consumers were connected to broker02.
> We have below setting :
> <persistenceAdapter>
> <jdbcPersistenceAdapter dataSource="#mysql-ds"
> lockKeepAlivePeriod="5000">
> <locker>
> <lease-database-locker lockAcquireSleepInterval="10000"/>
> </locker>
> </jdbcPersistenceAdapter>
> </persistenceAdapter>
> We had seen this issue in production, we did not get a chance to look at an
> activemq_lock table.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
