Re: [infinispan-dev] weekly questions
some general comments: - preserving some subset of the existing transactional guarantees is all well and good BUT if the user is relying on additional 'guarantees' that are not preserved in all cases then they'll be in trouble. Therefore, it's essential to document not just what the minimum expected guarantees for a given config are, but also that no additional properties that may coincidently be observed are actually guaranteed. Some vendors go further and explicitly document the bad things that may happen with given settings, since in some cases these are not easy to reproduce in testing. - aquiring locks in defined (e.g. HC) order is kind of obvious and I'm surprised it's not default behaviour. It probably should be as soon as the implementation is stable. Additionally, rather than making consecutive RPCs to a node for each of the keys it owns, why not locally walk the full set of objects in the tx first and batch them by owning node, then make one call to each node with the whole batch of lock requests? - I'm rather suspicions of any discussion that includes phrases like 'it will have to rollback its running transactions'. A node in an XA transaction that has already prepared CAN'T rollback, the protocol does not allow it. So, optimisations using non-replicated locks for replicated data should be disabled if the tx is under external control e.g. XA rather than local to infinispan. For that matter I think you'll have trouble with internal transactions too in some failure cases, but at least they won't affect me :-) Jonathan. On 05/25/2011 12:55 PM, Mircea Markus wrote: Hi Jonathan, We're thinking about doing some significant changes/improvements around transactions[1]. I'd like to pick your brain especially on improvement #1, as this changes the way we acquire locks quite a bit. With this new locking approach, we don't acquire WL for a transaction until prepare phase. On each write we keep a copy of the written key on the node where transaction is executed so that each subsequent access, in the same transaction, would read/update this copy. Same goes for reads. When transaction commits we acquire WL during prepare time. This looks to me very similar to the optimistic CC, just that there's no conflict verification before commit. Also, it seems to preserve repeatable read transaction isolation correctly. This is a very significant change in our transaction model - do you see any problems/suggestions around it? Cheers, Mircea [1] http://community.jboss.org/wiki/PossibleLockingImprovements -- Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom. Registered in UK and Wales under Company Registration No. 3798903 Directors: Michael Cunningham (USA), Charlie Peters (USA), Matt Parsons (USA) and Brendan Lane (Ireland). ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] weekly questions
2011/6/1 Jonathan Halliday jonathan.halli...@redhat.com: some general comments: - preserving some subset of the existing transactional guarantees is all well and good BUT if the user is relying on additional 'guarantees' that are not preserved in all cases then they'll be in trouble. Therefore, it's essential to document not just what the minimum expected guarantees for a given config are, but also that no additional properties that may coincidently be observed are actually guaranteed. Some vendors go further and explicitly document the bad things that may happen with given settings, since in some cases these are not easy to reproduce in testing. Agreed on documenting it properly, as I'm still confused myself. But just to clarify this thread is not about preserving a subset of existing transactional guarantees, it's more about having the same guarantees on each node: we observed that some locking patterns have a different semantic when run on the node owning the locked keys, compared to run them on nodes which need to acquire the same locks by using a remote call. - aquiring locks in defined (e.g. HC) order is kind of obvious and I'm surprised it's not default behaviour. It probably should be as soon as the implementation is stable. Additionally, rather than making consecutive RPCs to a node for each of the keys it owns, why not locally walk the full set of objects in the tx first and batch them by owning node, then make one call to each node with the whole batch of lock requests? It was surprising to me as well to find they where not acquired in order, but the main use case of a key value store is to operate on single keys; also if we don't postpone the lock acquisition to the prepare phase (as with the current proposal), it all depends in the order of execution of the user operations so there was no point in having such a logic. It would be nice to be able to batch the lock commands in single invocations per owner, but it's going to be tricky to implement that if you also want to acquire them all in a specific order. We'll need to make sure that the defined order in which locks are acquired matches the order of keys on the wheel: we first define the sequence of locks, and then split this sequence in chunks of commands only when contiguous keys are owned by the same node. Another good reason to keep a single lock owner per key, that would make this much simpler. - I'm rather suspicions of any discussion that includes phrases like 'it will have to rollback its running transactions'. A node in an XA transaction that has already prepared CAN'T rollback, the protocol does not allow it. So, optimisations using non-replicated locks for replicated data should be disabled if the tx is under external control e.g. XA rather than local to infinispan. For that matter I think you'll have trouble with internal transactions too in some failure cases, but at least they won't affect me :-) What do you mean by me ? Thanks for the very valuable feedback. Sanne Jonathan. On 05/25/2011 12:55 PM, Mircea Markus wrote: Hi Jonathan, We're thinking about doing some significant changes/improvements around transactions[1]. I'd like to pick your brain especially on improvement #1, as this changes the way we acquire locks quite a bit. With this new locking approach, we don't acquire WL for a transaction until prepare phase. On each write we keep a copy of the written key on the node where transaction is executed so that each subsequent access, in the same transaction, would read/update this copy. Same goes for reads. When transaction commits we acquire WL during prepare time. This looks to me very similar to the optimistic CC, just that there's no conflict verification before commit. Also, it seems to preserve repeatable read transaction isolation correctly. This is a very significant change in our transaction model - do you see any problems/suggestions around it? Cheers, Mircea [1] http://community.jboss.org/wiki/PossibleLockingImprovements -- Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom. Registered in UK and Wales under Company Registration No. 3798903 Directors: Michael Cunningham (USA), Charlie Peters (USA), Matt Parsons (USA) and Brendan Lane (Ireland). ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
[infinispan-dev] Quicker builds
For those of you that are constantly building Infinispan, I've gone and moved all non-essential build plugins to an 'extras' profile that can be disabled at build time, i..e.: mvn install -P-extras More info in: http://community.jboss.org/docs/DOC-15189 Cheers, -- Galder Zamarreño Sr. Software Engineer Infinispan, JBoss Cache ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Failure looking up the river marshaller under AS 7 environment
What do you mean by TopologyView's Externalizer not being picked? If it's not, that's a bug (=JIRA). On Jun 1, 2011, at 6:29 AM, 이희승 (Trustin Lee) wrote: Thanks for a hint. TopologyView has its own Externalizer, but it was not picked up by Infinispan and thus fell back to Serializable. Adding more services=import attributes to the relevant depepdencies did the trick. On 06/01/2011 12:56 PM, David M. Lloyd wrote: This issue is caused by users who put a writeObject() or readObject() method on their class, and never write/read the class' serializable fields. This is a violation of spec [2][3] which causes undefined behavior in some cases and thus it has always been forbidden with JBMAR. However, it has been pointed out that a lot of classes actually rely on the specific behavior of the Sun JDK (which is to read/write fields in the default way and hope for the best), hence I have a JIRA issue [1] which is not yet resolved. Unfortunately I don't currently have time to devote to this and won't for a while. If anyone is sufficiently annoyed by this that they want to volunteer to fix it, I can walk someone through the code and test suite. [1] https://issues.jboss.org/browse/JBMAR-120 [2] http://download.oracle.com/javase/6/docs/platform/serialization/spec/output.html#861 [3] http://download.oracle.com/javase/6/docs/platform/serialization/spec/input.html#2971 On 05/31/2011 10:39 PM, 이희승 (Trustin Lee) wrote: With Marshalling.getProvidedMarshallerFactory(), I was able to look up the RiverMarshallerFactory. However, another exception is raised while exchanging the topology view (java.io.NotActiveException: Fields were never written): http://pastebin.com/5zVj2FwG It seems like it has a problem with marshalling an ImmortalCacheEntry? On 06/01/2011 11:38 AM, 이희승 (Trustin Lee) wrote: Using Marshalling.getProvidedMarshallerFactory() sounds like a better solution. Let me create a JIRA issue and send a pull request soon. On 06/01/2011 01:47 AM, David M. Lloyd wrote: Easiest solution is: Marshalling.getProvidedMarshallerFactory(river); This will always use the JBMAR class loader. Otherwise you would have to make sure that the module from which you load the river protocol imports the river module like this: dependencies ... module name=org.jboss.marshalling.river services=import/ ... /dependencies ... and then use your own class loader. On 05/31/2011 10:13 AM, Galder Zamarreño wrote: The marshaller is a global component, so it'd have to be at the GlobalConfiguration. Let's bounce some ideas tomorrow on IRC. On May 31, 2011, at 5:07 PM, 이희승 (Trustin Lee) wrote: The problem is that the marshaller lookup code exists in Infinispan unlike other subsystems. We need to make it configurable. Where would be the best place to expose the property in Infinispan? For EmbeddedCachaManager, it is so simple because we have Configuration and GlobalConfiguration, but I'm not sure about the others. On 05/31/2011 11:38 PM, Galder Zamarreño wrote: In fact, a quick grep on AS7 shows: ./host-controller/src/main/java/org/jboss/as/host/controller/ManagedServer.java: MARSHALLER_FACTORY = Marshalling.getMarshallerFactory(river, Module.getModuleFromCallerModuleLoader(ModuleIdentifier.fromString(org.jboss.marshalling.river)).getClassLoader()); ./protocol/src/main/java/org/jboss/as/protocol/ProtocolUtils.java: MARSHALLER_FACTORY = Marshalling.getMarshallerFactory(river, ProtocolUtils.class.getClassLoader()); ./server/src/main/java/org/jboss/as/server/DomainServerMain.java: final MarshallerFactory factory = Marshalling.getMarshallerFactory(river, DomainServerMain.class.getClassLoader()); We might need different mechanisms depending on the env. On May 31, 2011, at 11:28 AM, Galder Zamarreño wrote: They're in diff jars and most likely in different JBoss Modules. Trustin, AS7 uses JBoss Marshalling, so maybe have a look in their code to see how they instantiate the marshaller factory - that might give you the clues here :) Cheers, On May 31, 2011, at 10:20 AM, Dan Berindei wrote: We don't use the TCCL to load the MarshallerFactory: factory = Marshalling.getMarshallerFactory(river, Marshalling.class.getClassLoader()); Maybe the Marshalling class and the RiverMarshallerFactory classes are in different modules? Dan On Tue, May 31, 2011 at 10:14 AM, 이희승 (Trustin Lee)trus...@gmail.com wrote: Hi folks, I'm trying to run Infinispan under AS 7 (i.e. JBoss Modules). I succeeded to run an EmbeddedCacheManager, HotRodServer, and MemcachedServer. However, it fails when a new node joins the cluser: http://pastebin.com/pGfxSWJP The root cause of the failure is that GenericJBossMarshaller fails to find the RiverMarshallerFactory. So, I set the TCCL, but it didn't help at all. Even setting the TCCL to RiverMarshallerFactory.class.getClassLoader() doesn't seem to help. Any clues? Cheers --
[infinispan-dev] Partial state transfer in Infinispan
We currently use JGroups' partial state transfer to transfer individual caches from one Infinispan instance to another. Since I got rid of partial state transfer in JGroups 3.0, and don't like to add it back, I'd like to know whether this is still needed. I thought that we currently require the same set of caches to be available in all Infinispan instances, and the reason (IIRC) was that distribution wouldn't work if we have caches 1 and 2 available on instances A and B, but not on C, because consistent hashing distributes the data based on views, and we didn't want to have to keep track of individual caches... Why are we actually using JGroups' state transfer with replication, but use our own state transfer with distribution ? Opinions ? -- Bela Ban Lead JGroups / Clustering Team JBoss ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Partial state transfer in Infinispan
Hi Bela, 2011/6/1 Bela Ban b...@redhat.com: We currently use JGroups' partial state transfer to transfer individual caches from one Infinispan instance to another. Since I got rid of partial state transfer in JGroups 3.0, and don't like to add it back, I'd like to know whether this is still needed. I thought that we currently require the same set of caches to be available in all Infinispan instances, and the reason (IIRC) was that distribution wouldn't work if we have caches 1 and 2 available on instances A and B, but not on C, because consistent hashing distributes the data based on views, and we didn't want to have to keep track of individual caches... Well I really don't like this limitation in Infinispan and was actually hoping that we could remove it at some point. Imagine the scenario in which you have a running cluster, and at some point the new release of your application needs an additional cache: there's no way to start a new node having this new cache. Also right now when an application starts, it's possible with the proper timing that it joins the cluster before having defined and started all caches (starting the cachemanager and the caches is not an atomic operation), basically failing to start because of this limitation. Maybe it's still possible to build such a thing on top of non-partial state transfer? As it doesn't exist, we didn't design it. Why are we actually using JGroups' state transfer with replication, but use our own state transfer with distribution ? I don't know, but guess it's because each node has a different set of keys so no node has the same state as another ? Cheers, Sanne Opinions ? -- Bela Ban Lead JGroups / Clustering Team JBoss ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Partial state transfer in Infinispan
On 6/1/11 4:21 PM, Sanne Grinovero wrote: Hi Bela, 2011/6/1 Bela Banb...@redhat.com: We currently use JGroups' partial state transfer to transfer individual caches from one Infinispan instance to another. Since I got rid of partial state transfer in JGroups 3.0, and don't like to add it back, I'd like to know whether this is still needed. I thought that we currently require the same set of caches to be available in all Infinispan instances, and the reason (IIRC) was that distribution wouldn't work if we have caches 1 and 2 available on instances A and B, but not on C, because consistent hashing distributes the data based on views, and we didn't want to have to keep track of individual caches... Well I really don't like this limitation in Infinispan and was actually hoping that we could remove it at some point. Imagine the scenario in which you have a running cluster, and at some point the new release of your application needs an additional cache: there's no way to start a new node having this new cache. Yes, I fully agree. Another example is HTTP web sessions: currently 1 webapp == 1 cache, so we currently require the same webapps to be deployed in all JBoss instances *if* we use replication (distribution is different) ! Also right now when an application starts, it's possible with the proper timing that it joins the cluster before having defined and started all caches (starting the cachemanager and the caches is not an atomic operation), basically failing to start because of this limitation. Yep Maybe it's still possible to build such a thing on top of non-partial state transfer? As it doesn't exist, we didn't design it. Yes. Well, Infinispan already uses its own state transfer for distribution, I wonder why this isn't the case for replication. Why are we actually using JGroups' state transfer with replication, but use our own state transfer with distribution ? I don't know, but guess it's because each node has a different set of keys so no node has the same state as another ? You could still use JGroups state transfer; getState() would list the state provider as target node. In general, partial state transfer involves transferring (1) the partial state and (2) the digest, which is a vector of highest seqnos seen for every member. When we get a partial state, we always overwrite our own digest with the one received, and update our state accordingly. However, when this is done a couple of times, for different partial states, I'm not sure that we won't receive a few messages multiple times, due to the digest overwriting... I think the cleanest solution would be for you guys to reuse the state transfer you already use for state transfer in distribution mode. -- Bela Ban Lead JGroups / Clustering Team JBoss ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Partial state transfer in Infinispan
Why are we actually using JGroups' state transfer with replication, but use our own state transfer with distribution ? I don't know, but guess it's because each node has a different set of keys so no node has the same state as another ? You could still use JGroups state transfer; getState() would list the state provider as target node. In general, partial state transfer involves transferring (1) the partial state and (2) the digest, which is a vector of highest seqnos seen for every member. When we get a partial state, we always overwrite our own digest with the one received, and update our state accordingly. However, when this is done a couple of times, for different partial states, I'm not sure that we won't receive a few messages multiple times, due to the digest overwriting... I think the cleanest solution would be for you guys to reuse the state transfer you already use for state transfer in distribution mode. +1. There's also transaction logic* related to state transfer, which would need to be maintained in two implementations - not good! * actually the transaction-failover logic needs to be revisited especially after the new rebalancing code. Dan and I will go over it next week - we working at his place and this will make things easier. ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
[infinispan-dev] Possible Bug in Distribution Manager
Hi, My name is Pedro and I am working in CloudTM project. I think that I may have encountered a possible bug in the consistent hash function. I am working on Infinispan 'Pagoa' 5.0.0-SNAPSHOT with JGroups 3.0.0 Alpha1 and I have a total of 100 000 keys distributed by 11 nodes. I am using Radargun, that was modified to execute the maximum number of transactions in a 5 minutes run. All transactions updates at least one key, ie, I didn't have read-only transactions. At the end of the test, I am printing all the keys and their location (keys' owners), and, as you can see for instance here [1], different nodes have different opinions concerning which replicas are in charge of storing some keys. Note that I didn't change anything in the distribution code and my tests only update keys, never delete them. Part of the log can be found in [2] and the config file is in [3]. In the log file we have a lot of 'WARN [DistLockingInterceptor] xxx entry commit warmup_key(...) = Thread(...)'. Ignore this entries. Unfortunately, the problem does not always show up, in my tests this shows up, say, 5% of the times. Did this ever happen to you, or am I hitting a known issue? I hope that the information that I am providing can be sufficient to reproduce the bug, let me know if there is anything else that I can do to help Cheers Pedro [1] - http://pastebin.com/Pp47ctj9 [2] - http://pastebin.com/tpSXqVmV [3] - http://pastebin.com/vjVP48L2 -- INESC-ID Lisboa, sala 511 gsd.inesc-id.pt/~pruivo ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] weekly questions
On 1 Jun 2011, at 08:26, Jonathan Halliday wrote: some general comments: - preserving some subset of the existing transactional guarantees is all well and good BUT if the user is relying on additional 'guarantees' that are not preserved in all cases then they'll be in trouble. Therefore, it's essential to document not just what the minimum expected guarantees for a given config are, but also that no additional properties that may coincidently be observed are actually guaranteed. Some vendors go further and explicitly document the bad things that may happen with given settings, since in some cases these are not easy to reproduce in testing. Agreed: https://issues.jboss.org/browse/ISPN-1146 - aquiring locks in defined (e.g. HC) order is kind of obvious and I'm surprised it's not default behaviour. It is the default behaviour indeed. We are considering an option to reorder lock acquisition in order to avoid deadlocks[1]. Enablable through a configuration. It probably should be as soon as the implementation is stable. Additionally, rather than making consecutive RPCs to a node for each of the keys it owns, why not locally walk the full set of objects in the tx first and batch them by owning node, then make one call to each node with the whole batch of lock requests? The futures[2] API does something similar, but not quite that though. Interesting optimisation for the Future API. - I'm rather suspicions of any discussion that includes phrases like 'it will have to rollback its running transactions'. A node in an XA transaction that has already prepared CAN'T rollback, the protocol does not allow it. yes, totally agree. And since 5.0 we don't do that anymore. So, optimisations using non-replicated locks for replicated data should be disabled if the tx is under external control e.g. XA rather than local to infinispan. For that matter I think you'll have trouble with internal transactions too in some failure cases, but at least they won't affect me :-) [1] http://community.jboss.org/wiki/PossibleLockingImprovements, see section #3. [2] http://community.jboss.org/wiki/AsynchronousAPI___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] 5.0.0.CR4 for 1st June
I'll hold on ISPN-1107 which is about measuring performance and switch to doing fixes. I'll also take some issues from Manik's queue. On 31 May 2011, at 22:42, Vladimir Blagojevic wrote: I'll close ISPN-83 if I do not hear any negative feedback from Paul about Infinispan flush tests that were failing before in AS codebase. On 11-05-31 4:26 PM, Galder Zamarreño wrote: Guys, There're still 25 issues open for CR4: http://goo.gl/EZj3e What's the situation in them? Cheers, On May 25, 2011, at 3:17 PM, Galder Zamarreño wrote: Hi all, I'm planning to release CR4 next Wednesday, 1st of June, and between now and then I'll be trying to narrow down more testsuite failures and fix the suggestions on the Klockwork report. Cheers, -- Galder Zamarreño Sr. Software Engineer Infinispan, JBoss Cache ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Galder Zamarreño Sr. Software Engineer Infinispan, JBoss Cache ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
[infinispan-dev] discussion about impact of using TransactionSynchronizationRegistry in AS7...
I posted a message on the as7-dev ml (http://lists.jboss.org/pipermail/jboss-as7-dev/2011-May/002254.html), about switching to use the TransactionSynchronizationRegistry. Does Infinispan currently register Transaction synchronization objects? Does Infinispan currently register synchronizations via TransactionSynchronizationRegistry (TSR)? I'm trying to get a sense for, what would happen if container managed (AS7) session beans were registered with the active JTA transaction via the TSR. If AS7 switches to use the TSR, I think that Infinispan might need to ensure that it doesn't attempt to register with the TX too late. See http://pastie.org/1836698 for an example of what would happen if a TSR synchronization object is already present and someone tries to register a TX synchronization after tx.commit has been started. ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Partial state transfer in Infinispan
On 6/1/11 6:05 PM, Mircea Markus wrote: Why are we actually using JGroups' state transfer with replication, but use our own state transfer with distribution ? I don't know, but guess it's because each node has a different set of keys so no node has the same state as another ? You could still use JGroups state transfer; getState() would list the state provider as target node. In general, partial state transfer involves transferring (1) the partial state and (2) the digest, which is a vector of highest seqnos seen for every member. When we get a partial state, we always overwrite our own digest with the one received, and update our state accordingly. However, when this is done a couple of times, for different partial states, I'm not sure that we won't receive a few messages multiple times, due to the digest overwriting... I think the cleanest solution would be for you guys to reuse the state transfer you already use for state transfer in distribution mode. +1. There's also transaction logic* related to state transfer, which would need to be maintained in two implementations - not good! * actually the transaction-failover logic needs to be revisited especially after the new rebalancing code. Dan and I will go over it next week - we working at his place and this will make things easier. Excellent ! Let us know what the outcome is, naturally I'm especially interested in the new rebalancing code since I wrote it ! :-) Note that I'll be on vacation until June 11th. Cheers, -- Bela Ban Lead JGroups / Clustering Team JBoss ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
