[jboss-user] [JBossCache] - Re: Data gravitation synchronization issue
Glad to hear you got it working (somewhat). Correct that such an access pattern is not recommended. This is why we strongly stress the need for session affinity with BR. We're working on some ideas to remove the need for session affinity (wiki - http://wiki.jboss.org/wiki/Wiki.jsp?page=JBossCachePartitioning[/url] and discussion thread - [url]http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4061375) Cheers, Manik View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4098606#4098606 Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4098606 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBossCache] - Re: Data gravitation synchronization issue
I have been investigating the issue further and it seems like we actually access the node right after creating it. This causes concurrent access from both the server that currently owns the data and the server that is assigned to the node and this is what is triggering the lock-situation described above. I understand that accessing a node this way is not desired (permitted?) when running buddy replication and I will change the usage pattern to avoid this. However, I don't see why the locks should hang and cause a timeout (if configured). Arguably the scenario should work functionally, but will of course be suboptimal. I would argue this mostly since the lock-issue could be symptomatic of a more generic lock problem at hand. Anyway, it seems like we got buddy rep up and running now, which is great! View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4098452#4098452 Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4098452 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBossCache] - Re: Data gravitation synchronization issue
Also, have you got data gravitation remove on find and search subtrees set to true? View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4097944#4097944 Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4097944 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBossCache] - Re: Data gravitation synchronization issue
Just to understand the issue better, when you say from starting up, what has happened so far? Also, how many caches have you got in the cluster and how many buddy backups per cache? I'm presuming the error occurs when you try and access the data on a node that is *not* the original data owner nor the buddy? View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4097940#4097940 Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4097940 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBossCache] - Re: Data gravitation synchronization issue
Using 2.1 beta does not solve the problem. I still get timeout exceptions from unsuccesful lock acquisition when enabling buddy replication. JBoss Cache version: JBossCache 'Alegrias' 2.1.0.BETA1[ $Id: Version.java 4592 2007-10-10 16:44:36Z [EMAIL PROTECTED] $] | | JGroups version: 2.6.0 beta-1 | | Incoming Thread,TableSpace,172.16.0.5:7500 ERROR jboss.cache.interceptors.TxInterceptor - method invocation failed | org.jboss.cache.lock.TimeoutException: failure acquiring lock: fqn=/5, caller=GlobalTransaction:<172.16.0.7:7500>:1, lock=write owner=GlobalTransaction:<172.16.0.6:7500>:2 (activeReaders=0, activeWriter=Thread[Incoming Thread,TableSpace, | 172.16.0.5:7500,5,Incoming], waitingReaders=0, waitingWriters=0, waitingUpgrader=0) | at org.jboss.cache.lock.IdentityLock.acquire(IdentityLock.java:528) | at org.jboss.cache.interceptors.PessimisticLockInterceptor$LockManager.acquire(PessimisticLockInterceptor.java:598) | at org.jboss.cache.interceptors.PessimisticLockInterceptor.acquireNodeLock(PessimisticLockInterceptor.java:412) | at org.jboss.cache.interceptors.PessimisticLockInterceptor.lock(PessimisticLockInterceptor.java:348) | at org.jboss.cache.interceptors.PessimisticLockInterceptor.invoke(PessimisticLockInterceptor.java:185) | at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:76) | at org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:34) | at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:76) | at org.jboss.cache.interceptors.NotificationInterceptor.invoke(NotificationInterceptor.java:32) | at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:76) | at org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:299) | at org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:131) | at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:76) | at org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:123) | at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:76) | at org.jboss.cache.interceptors.InvocationContextInterceptor.invoke(InvocationContextInterceptor.java:62) | at org.jboss.cache.CacheImpl.invokeMethod(CacheImpl.java:3954) | at org.jboss.cache.CacheImpl._dataGravitationCleanup(CacheImpl.java:3026) | | Caused by: org.jboss.cache.lock.TimeoutException: write lock for /5 could not be acquired after 5000 ms. Locks: Read lock owners: [] | Write lock owner: GlobalTransaction:<172.16.0.6:7500>:2 | (caller=GlobalTransaction:<172.16.0.7:7500>:1, lock info: write owner=GlobalTransaction:<172.16.0.6:7500>:2 (activeReaders=0, activeWriter=Thread[Incoming Thread,TableSpace,172.16.0.5:7500,5,Incoming], waitingReaders=0, waitingWriters=0 | , waitingUpgrader=0)) | at org.jboss.cache.lock.IdentityLock.acquireWriteLock0(IdentityLock.java:244) | at org.jboss.cache.lock.IdentityLock.acquireWriteLock(IdentityLock.java:167) | at org.jboss.cache.lock.IdentityLock.acquire(IdentityLock.java:497) | View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4097891#4097891 Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4097891 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBossCache] - Re: Data gravitation synchronization issue
I have also seen this behaviour in our application. Buddy replication does not seem to work under concurrent load and/or usage. Just from starting up (no load applied) I get this in the logs: | 2007-10-23 14:25:52,664 Incoming Thread,TableSpace,172.16.0.5:7500 ERROR jboss.cache.interceptors.TxInterceptor - method invocation failed | org.jboss.cache.lock.TimeoutException: failure acquiring lock: fqn=/5, caller=GlobalTransaction:<172.16.0.7:7500>:1, lock=write owner=GlobalTransaction:<172.16.0.6:7500>:2 (activeReaders=0, activeWriter=Thread[Incoming Thread,TableSpace, | 172.16.0.5:7500,5,Incoming], waitingReaders=0, waitingWriters=0, waitingUpgrader=0) | at org.jboss.cache.lock.IdentityLock.acquire(IdentityLock.java:528) | at org.jboss.cache.interceptors.PessimisticLockInterceptor$LockManager.acquire(PessimisticLockInterceptor.java:579) | at org.jboss.cache.interceptors.PessimisticLockInterceptor.acquireNodeLock(PessimisticLockInterceptor.java:393) | at org.jboss.cache.interceptors.PessimisticLockInterceptor.lock(PessimisticLockInterceptor.java:329) | at org.jboss.cache.interceptors.PessimisticLockInterceptor.invoke(PessimisticLockInterceptor.java:187) | at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:76) | at org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:34) | at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:76) | at org.jboss.cache.interceptors.NotificationInterceptor.invoke(NotificationInterceptor.java:32) | at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:76) | at org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:298) | at org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:131) | at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:76) | at org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:123) | at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:76) | at org.jboss.cache.interceptors.InvocationContextInterceptor.invoke(InvocationContextInterceptor.java:62) | at org.jboss.cache.CacheImpl.invokeMethod(CacheImpl.java:3939) | at org.jboss.cache.CacheImpl._dataGravitationCleanup(CacheImpl.java:3055) | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) | at java.lang.reflect.Method.invoke(Method.java:585) | at org.jgroups.blocks.MethodCall.invoke(MethodCall.java:330) | at org.jboss.cache.interceptors.CallInterceptor.invoke(CallInterceptor.java:49) | at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:76) | at org.jboss.cache.interceptors.DataGravitatorInterceptor.invoke(DataGravitatorInterceptor.java:167) | at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:76) | at org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:37) | at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:76) | at org.jboss.cache.interceptors.PessimisticLockInterceptor.invoke(PessimisticLockInterceptor.java:203) | at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:76) | at org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:34) | at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:76) | at org.jboss.cache.interceptors.NotificationInterceptor.invoke(NotificationInterceptor.java:32) | at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:76) | at org.jboss.cache.interceptors.TxInterceptor.replayModifications(TxInterceptor.java:511) | at org.jboss.cache.interceptors.TxInterceptor.handlePessimisticPrepare(TxInterceptor.java:394) | at org.jboss.cache.interceptors.TxInterceptor.handleRemotePrepare(TxInterceptor.java:254) | at org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:100) | at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:76) | at org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:123) | at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:76) | at org.jboss.cache.interceptors.InvocationContextInterceptor.invoke(InvocationContextInterceptor.java:62) | at org.jboss.cache.CacheImpl.invokeMethod(CacheImpl.java:3939) | at org.jboss.cache.CacheImpl._replicate(CacheImpl.java:2853) | at sun.reflect.GeneratedMethodAcc
[jboss-user] [JBossCache] - Re: Data gravitation synchronization issue
Have you tried with 2.1.0.BETA1? This needs JGroups 2.6.BETA1 as well (packaged with JBoss Cache) View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4096917#4096917 Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4096917 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBossCache] - Re: Data gravitation synchronization issue
We're using version 2.0.0 GA (with JGroups 2.5). There's no eviction policy configured. View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4096574#4096574 Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4096574 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBossCache] - Re: Data gravitation synchronization issue
what version of JBoss Cache do you use? There have been some issues with data gravitation finding state that may already have been evicted (JBCACHE-1192) which is fixed in 2.1.0.BETA1. Does your setup involve any eviction configuration? View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4096537#4096537 Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4096537 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBossCache] - Re: Data gravitation synchronization issue
The UnitTest models our real application. It goes like this: It is an application processing events for units (areas) where the area is an object stored in the cache and session affinity is enforced by a message bus. Sometimes during the lifetime of the application new areas will be added by a coordinator (which is restricted to one of the nodes, ie. new areas are only ever added at one node). The message bus acts as a load balancer and will distribute events for areas, using a sticky algorithm so that session affinity is honored. So: | * The coordinator adds an area object to the cache. | * The message bus is notified (by jboss cache listener methods) of the new area. It then selects a node in the cluster for processing. | * The working (event sending) threads are notified of the new area and starts sending events. | The problem appears, as I originally wrote, when the events arrive at their deignated node and a data gravitation occurs. And as far as I have seen only on the backup node. For example: | * Node A is coordinator and adds an area (X). | * Node B becomes buddy backup for X. | * The message bus assigns X to node C. | * When messages arrive at node C a data gravitation occurs. | * Node B receives two consecutive data gravitation cleanup commands, one of which doesn't seem to let go of its identity lock. | * After "buddyCommunicationTimeout" (logged as debug) node C get null from the cache and fails. | This is loosely modeled in the UnitTest. In the test worker threads are assigned areas statically, areas are added in one cache after which the workers are notified of their new area and attempts to access it. If any of the worker threads gets a null from the cache the test fails as we know the area should exist (ie. the worker is not notified of its new area until it has already been added). View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4095516#4095516 Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4095516 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBossCache] - Re: Data gravitation synchronization issue
Sorry for not responding sooner. Keep in mind that session affinity is a requirement when you use buddy replication and data gravitation. I wouldn't expect a gravitation event to occur if the primary (or even the backup!) is still being written to (i.e., has a write lock on either the primary or buddy backup node). Could you tell me a little more about the access patterns involved? How do you choose which instances in the cluster to speak to? View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4095287#4095287 Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4095287 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBossCache] - Re: Data gravitation synchronization issue
* bump * It would be nice to know if anyone has been able to duplicate this problem. This is a major blocker for our development at the moment. View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4095161#4095161 Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4095161 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
[jboss-user] [JBossCache] - Re: Data gravitation synchronization issue
Unit test can be downloaded here: http://www.cubeia.com/files/cache-lock-test.tar.gz Regards /Lars J. Nilsson www.cubeia.com View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4091939#4091939 Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4091939 ___ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
