Re: [infinispan-dev] Grouping API (ISPN-312) WAS: Generated keys affected by rehash Was: https://issues.jboss.org/browse/ISPN-977
If you return the same group id for a set of keys consistently, then they will always be colocated, even across a rehash. I wasn't planning on adding any functionality that to this actually allows you to pin a specific group to a specific node though (which I think is what you are asking about?). You would need to use a magic key generator for this. At an impl level, my approach is to simply use a hash of group name to determine the addresses to which a key is hashed, rather than a hash of the key. On 17 May 2011, at 23:39, Erik Salter wrote: Hi Pete, A question about the mechanics. If I specify a dynamic group for a set of keys and send them to one of my cache nodes for processing, will the interceptor generate a group that will hash those keys to that local node? And those keys are guaranteed to be collocated across a rehash? 'Cause that would be really, really cool. Erik -Original Message- From: infinispan-dev-boun...@lists.jboss.org [mailto:infinispan-dev-boun...@lists.jboss.org] On Behalf Of Pete Muir Sent: Tuesday, May 17, 2011 6:08 PM To: infinispan -Dev List Subject: Re: [infinispan-dev] Grouping API (ISPN-312) WAS: Generated keys affected by rehash Was: https://issues.jboss.org/browse/ISPN-977 It's worth adding that if you are using any kind of dynamic group (i.e. one that isn't specified statically at compile time) it's up to you to ensure you always return the same group for the same key! I'll add a big fat warning on this one! On 17 May 2011, at 22:52, Pete Muir wrote: Hi Erik I've evolved the api a small bit from the JIRA. The group generator now behaves more like an interceptor than described in the jira (you can chain them with the initial value being provided by the annotation). No, you don't need to know the group name to access the cache. The group is simply used as a hint to Infinispan. On 17 May 2011, at 22:04, Erik Salter wrote: Hi Manik, I think we are in agreement that playing with hash codes was only a temporary measure. In my case, having 200 entries with the same hash code was worth it for knowing that I could handle transactions locally and reap the benefits of increased throughput. So I can now replace the hash code with @Group. Cool. The group generator interface looks interesting, since it closest reflects my situation. I now have requirements where an immutable key class will need to be saved within the same transaction as the scenario above (obviously, hashing to the same node is a plus) One thing isn't clear from the JIRA. If I wanted to get Employee SteveVai from the cache, do I need to know the group context is com.ibanez.SteveVai? My calling application only knows the key value, not the value with the key context. Erik From: infinispan-dev-boun...@lists.jboss.org [mailto:infinispan-dev-boun...@lists.jboss.org] On Behalf Of Manik Surtani Sent: Tuesday, May 17, 2011 1:34 PM To: infinispan -Dev List Subject: [infinispan-dev] Grouping API (ISPN-312) WAS: Generated keys affected by rehash Was: https://issues.jboss.org/browse/ISPN-977 Erik, Dan is correct that playing with hash codes is not the correct solution. ISPN-312 is the correct approach. Pete has been working on a first-cut of this and it should make 5.0.0.CR3. (Understood that release candidates aren't the place to add new features, but we're adding it as a preview, just to get feedback on the API and impl.) Have a look at the proposed API on https://issues.jboss.org/browse/ISPN-312 and let us know if it works for you. Cheers Manik On 13 May 2011, at 18:28, Erik Salter wrote: Hi Dan, I don't necessarily care about which server it's on, as long as the keys for my set of caches all remain collocated. I understand they will all end up in the same bucket, but for one hash code, that's at most 200 keys. I must acquire a lock for a subset of them during a transaction -- so I make liberal use of the eagerLockSingleNode option and redirecting my calling application to execute a transaction on the local node. Acquiring cluster-wide locks is an absolute throughput killer. I took a look at the KeyAffinityService a while ago (when it came out) and quickly realized it would not be suitable for my purposes. I was wondering if ISPN-977 would allow me to use it. But you're right. What I ultimately want is ISPN-312. Erik -Original Message- From: infinispan-dev-boun...@lists.jboss.org [mailto:infinispan-dev-boun...@lists.jboss.org] On Behalf Of Dan Berindei Sent: Friday, May 13, 2011 12:58 PM To: infinispan -Dev List Subject: Re: [infinispan-dev] Generated keys affected by rehash Was: https://issues.jboss.org/browse/ISPN-977 On Fri, May 13, 2011 at 6:38 PM, Erik Salter esal...@bnivideo.com wrote: Yes, collocation of all keys is a large concern of my application(s). Currently, I can handle keys I'm in control of
Re: [infinispan-dev] Grouping API (ISPN-312) WAS: Generated keys affected by rehash Was: https://issues.jboss.org/browse/ISPN-977
Just a curiosity - I'm not sure yet I would use it - would this work across different caches? So if I have cacheA and cacheB, both using some form of distribution, could I have some object stored in cacheA, some other stored in cacheB, and use grouping to have them stored in the same nodes ? Sanne 2011/5/18 Pete Muir pm...@redhat.com: If you return the same group id for a set of keys consistently, then they will always be colocated, even across a rehash. I wasn't planning on adding any functionality that to this actually allows you to pin a specific group to a specific node though (which I think is what you are asking about?). You would need to use a magic key generator for this. At an impl level, my approach is to simply use a hash of group name to determine the addresses to which a key is hashed, rather than a hash of the key. On 17 May 2011, at 23:39, Erik Salter wrote: Hi Pete, A question about the mechanics. If I specify a dynamic group for a set of keys and send them to one of my cache nodes for processing, will the interceptor generate a group that will hash those keys to that local node? And those keys are guaranteed to be collocated across a rehash? 'Cause that would be really, really cool. Erik -Original Message- From: infinispan-dev-boun...@lists.jboss.org [mailto:infinispan-dev-boun...@lists.jboss.org] On Behalf Of Pete Muir Sent: Tuesday, May 17, 2011 6:08 PM To: infinispan -Dev List Subject: Re: [infinispan-dev] Grouping API (ISPN-312) WAS: Generated keys affected by rehash Was: https://issues.jboss.org/browse/ISPN-977 It's worth adding that if you are using any kind of dynamic group (i.e. one that isn't specified statically at compile time) it's up to you to ensure you always return the same group for the same key! I'll add a big fat warning on this one! On 17 May 2011, at 22:52, Pete Muir wrote: Hi Erik I've evolved the api a small bit from the JIRA. The group generator now behaves more like an interceptor than described in the jira (you can chain them with the initial value being provided by the annotation). No, you don't need to know the group name to access the cache. The group is simply used as a hint to Infinispan. On 17 May 2011, at 22:04, Erik Salter wrote: Hi Manik, I think we are in agreement that playing with hash codes was only a temporary measure. In my case, having 200 entries with the same hash code was worth it for knowing that I could handle transactions locally and reap the benefits of increased throughput. So I can now replace the hash code with @Group. Cool. The group generator interface looks interesting, since it closest reflects my situation. I now have requirements where an immutable key class will need to be saved within the same transaction as the scenario above (obviously, hashing to the same node is a plus) One thing isn't clear from the JIRA. If I wanted to get Employee SteveVai from the cache, do I need to know the group context is com.ibanez.SteveVai? My calling application only knows the key value, not the value with the key context. Erik From: infinispan-dev-boun...@lists.jboss.org [mailto:infinispan-dev-boun...@lists.jboss.org] On Behalf Of Manik Surtani Sent: Tuesday, May 17, 2011 1:34 PM To: infinispan -Dev List Subject: [infinispan-dev] Grouping API (ISPN-312) WAS: Generated keys affected by rehash Was: https://issues.jboss.org/browse/ISPN-977 Erik, Dan is correct that playing with hash codes is not the correct solution. ISPN-312 is the correct approach. Pete has been working on a first-cut of this and it should make 5.0.0.CR3. (Understood that release candidates aren't the place to add new features, but we're adding it as a preview, just to get feedback on the API and impl.) Have a look at the proposed API on https://issues.jboss.org/browse/ISPN-312 and let us know if it works for you. Cheers Manik On 13 May 2011, at 18:28, Erik Salter wrote: Hi Dan, I don't necessarily care about which server it's on, as long as the keys for my set of caches all remain collocated. I understand they will all end up in the same bucket, but for one hash code, that's at most 200 keys. I must acquire a lock for a subset of them during a transaction -- so I make liberal use of the eagerLockSingleNode option and redirecting my calling application to execute a transaction on the local node. Acquiring cluster-wide locks is an absolute throughput killer. I took a look at the KeyAffinityService a while ago (when it came out) and quickly realized it would not be suitable for my purposes. I was wondering if ISPN-977 would allow me to use it. But you're right. What I ultimately want is ISPN-312. Erik -Original Message- From: infinispan-dev-boun...@lists.jboss.org [mailto:infinispan-dev-boun...@lists.jboss.org] On Behalf Of Dan Berindei Sent: Friday, May 13, 2011 12:58 PM To: infinispan
Re: [infinispan-dev] Grouping API (ISPN-312) WAS: Generated keys affected by rehash Was: https://issues.jboss.org/browse/ISPN-977
On 18 May 2011, at 11:38, Sanne Grinovero wrote: Just a curiosity - I'm not sure yet I would use it - would this work across different caches? So if I have cacheA and cacheB, both using some form of distribution, could I have some object stored in cacheA, some other stored in cacheB, and use grouping to have them stored in the same nodes ? Assuming they both have the same number of nodes, numOwners, ConsistentHash implementation and hash algorithm, etc. configured, then I presume so. - Manik -- Manik Surtani ma...@jboss.org twitter.com/maniksurtani Lead, Infinispan http://www.infinispan.org ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Grouping API (ISPN-312) WAS: Generated keys affected by rehash Was: https://issues.jboss.org/browse/ISPN-977
On 18 May 2011, at 11:43, Manik Surtani wrote: On 18 May 2011, at 11:38, Sanne Grinovero wrote: Just a curiosity - I'm not sure yet I would use it - would this work across different caches? So if I have cacheA and cacheB, both using some form of distribution, could I have some object stored in cacheA, some other stored in cacheB, and use grouping to have them stored in the same nodes ? Assuming they both have the same number of nodes, numOwners, ConsistentHash implementation and hash algorithm, etc. configured, then I presume so. I think so to. ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Grouping API (ISPN-312) WAS: Generated keys affected by rehash Was: https://issues.jboss.org/browse/ISPN-977
Hi Manik, I think we are in agreement that playing with hash codes was only a temporary measure. In my case, having 200 entries with the same hash code was worth it for knowing that I could handle transactions locally and reap the benefits of increased throughput. So I can now replace the hash code with @Group. Cool. The group generator interface looks interesting, since it closest reflects my situation. I now have requirements where an immutable key class will need to be saved within the same transaction as the scenario above (obviously, hashing to the same node is a plus) One thing isn't clear from the JIRA. If I wanted to get Employee SteveVai from the cache, do I need to know the group context is com.ibanez.SteveVai? My calling application only knows the key value, not the value with the key context. Erik From: infinispan-dev-boun...@lists.jboss.org [mailto:infinispan-dev-boun...@lists.jboss.org] On Behalf Of Manik Surtani Sent: Tuesday, May 17, 2011 1:34 PM To: infinispan -Dev List Subject: [infinispan-dev] Grouping API (ISPN-312) WAS: Generated keys affected by rehash Was: https://issues.jboss.org/browse/ISPN-977 Erik, Dan is correct that playing with hash codes is not the correct solution. ISPN-312 is the correct approach. Pete has been working on a first-cut of this and it should make 5.0.0.CR3. (Understood that release candidates aren't the place to add new features, but we're adding it as a preview, just to get feedback on the API and impl.) Have a look at the proposed API on https://issues.jboss.org/browse/ISPN-312 and let us know if it works for you. Cheers Manik On 13 May 2011, at 18:28, Erik Salter wrote: Hi Dan, I don't necessarily care about which server it's on, as long as the keys for my set of caches all remain collocated. I understand they will all end up in the same bucket, but for one hash code, that's at most 200 keys. I must acquire a lock for a subset of them during a transaction -- so I make liberal use of the eagerLockSingleNode option and redirecting my calling application to execute a transaction on the local node. Acquiring cluster-wide locks is an absolute throughput killer. I took a look at the KeyAffinityService a while ago (when it came out) and quickly realized it would not be suitable for my purposes. I was wondering if ISPN-977 would allow me to use it. But you're right. What I ultimately want is ISPN-312. Erik -Original Message- From: infinispan-dev-boun...@lists.jboss.orgmailto:infinispan-dev-boun...@lists.jboss.org [mailto:infinispan-dev-boun...@lists.jboss.org] On Behalf Of Dan Berindei Sent: Friday, May 13, 2011 12:58 PM To: infinispan -Dev List Subject: Re: [infinispan-dev] Generated keys affected by rehash Was: https://issues.jboss.org/browse/ISPN-977 On Fri, May 13, 2011 at 6:38 PM, Erik Salter esal...@bnivideo.commailto:esal...@bnivideo.com wrote: Yes, collocation of all keys is a large concern of my application(s). Currently, I can handle keys I'm in control of (like database-generated keys), where I can play around with the hash code. What I would love to do is collocate that data with keys I can't control (like UUIDs) so that all cache operations can be done in the same transaction on the data owner's node. By playing around with the hash code do you mean you set the hashcode for all the keys you want on a certain server to the same value? I imagine that would degrade performance quite a bit, because we have HashMaps everywhere and your keys will always end up in the same hash bucket. ISPN-312 seems to me like a much better fit for your use case than the KeyAffinityService. Even if you added a listener to change your keys when the topology changes, that would mean on a rehash the keys could get moved to the new server and then back to the old server, whereas with ISPN-312 they will either all stay on the old node or they will all move to the new node. Cheers Dan Erik -Original Message- From: infinispan-dev-boun...@lists.jboss.orgmailto:infinispan-dev-boun...@lists.jboss.org [mailto:infinispan-dev-boun...@lists.jboss.org] On Behalf Of Manik Surtani Sent: Friday, May 13, 2011 5:25 AM To: infinispan -Dev List Subject: [infinispan-dev] Generated keys affected by rehash Was: https://issues.jboss.org/browse/ISPN-977 On 11 May 2011, at 18:47, Erik Salter wrote: Wouldn't any rehash affect the locality of these generated keys, or am I missing something? It would. And hence ISPN-977, to address that. Or is your concern keys already generated before the rehash? The latter would certainly be a problem. Perhaps, if this was important to the application, on detecting a change in topology, re-generate keys and move data around? For other apps, move the session to the appropriate node? Cheers Manik -- Manik Surtani ma...@jboss.orgmailto:ma...@jboss.org twitter.com/maniksurtanihttp://twitter.com/maniksurtani Lead, Infinispan http://www.infinispan.org
