Hey Dan/Adrian,
Re: https://issues.jboss.org/browse/ISPN-2541
I'm looking at this intermittent failure, and it seems to be caused by the fact
that the test does not wait for the cluster to be formed when the new node is
started, which can lead a replication timeout failure from the new joining node.
The test can easily be fixed by waiting for cluster to form, and then do the
call.
But, I think there's an underlying issue, which I'm not sure if it has been
fixed already:
Node A sends a put to Node-B and newly joined Node-C:
test.log:11206:2012-11-23 09:11:29,925 148390 TRACE
[org.infinispan.remoting.transport.jgroups.JGroupsTransport]
(testng-MarshallExternalPojosTest:org.infinispan.marshall.MarshallExternalPojosTest)
dests=null,
command=SingleRpcCommand{cacheName='org.infinispan.marshall.MarshallExternalPojosTest',
command=PutKeyValueCommand{key=k0-testReplicateJBossExternalizePojoToNewJoiningNode,
value=org.infinispan.marshall.PojoWithJBossExternalize@413f7ce9, flags=null,
putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1}}, mode=SYNCHRONOUS,
timeout=15000
Node-C receives the put request but it's in the middle of state transfer, so it
forwards it to node B:
2012-11-23 09:11:29,960 148425 TRACE
[org.infinispan.statetransfer.StateTransferManagerImpl]
(OOB-2,ISPN,MarshallExternalPojosTest-NodeC-10724:org.infinispan.marshall.MarshallExternalPojosTest)
Forwarding command
PutKeyValueCommand{key=k0-testReplicateJBossExternalizePojoToNewJoiningNode,
value=org.infinispan.marshall.PojoWithJBossExternalize@413f7ce9, flags=null,
putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1} to new targets
[MarshallExternalPojosTest-NodeB-22814]
2012-11-23 09:11:29,960 148425 TRACE
[org.infinispan.remoting.rpc.RpcManagerImpl]
(OOB-2,ISPN,MarshallExternalPojosTest-NodeC-10724:org.infinispan.marshall.MarshallExternalPojosTest)
MarshallExternalPojosTest-NodeC-10724 broadcasting call
PutKeyValueCommand{key=k0-testReplicateJBossExternalizePojoToNewJoiningNode,
value=org.infinispan.marshall.PojoWithJBossExternalize@413f7ce9, flags=null,
putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1} to recipient list
[MarshallExternalPojosTest-NodeB-22814]
At the same time, Node-B is forwarding the request to Node-C:
2012-11-23 09:11:29,965 148430 TRACE
[org.infinispan.statetransfer.StateTransferManagerImpl]
(OOB-2,ISPN,MarshallExternalPojosTest-NodeB-22814:org.infinispan.marshall.MarshallExternalPojosTest)
Forwarding command
PutKeyValueCommand{key=k0-testReplicateJBossExternalizePojoToNewJoiningNode,
value=org.infinispan.marshall.PojoWithJBossExternalize@413f7ce9, flags=null,
putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1} to new targets
[MarshallExternalPojosTest-NodeC-10724]
In the end, Node-A does not get a reply from Node-C:
2012-11-23 09:11:44,960 163425 TRACE
[org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher]
(testng-MarshallExternalPojosTest:org.infinispan.marshall.MarshallExternalPojosTest)
Responses: [sender=MarshallExternalPojosTest-NodeC-10724, received=false,
suspected=false]
Does this ring a bell?
Cheers,
--
Galder Zamarreño
gal...@redhat.com
twitter.com/galderz
Project Lead, Escalante
http://escalante.io
Engineer, Infinispan
http://infinispan.org
_______________________________________________
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
