[infinispan-dev] HotRod server and Rolling Upgrades

2012-12-11 Thread Tristan Tarrant 

So,
I thought we had everything ready to go for HotRod rolling upgrades:

 * have HotRod server full of data (the "source")
 * configure a new HotRod server (the "target") with a RemoteCacheStore
   pointing to the "source" (using "rawValues")
 * clients switch over to the "target" server which on cache misses
   should seamlessly fetch entries from the "source"
 * issue a "dump keys" on the source
 * fetch the "dumped keys" from the target
 * disable the RCS on the target and switch off the "source" for good
 * PROFIT$$$

Unfortunately there is a teeny tiny flaw in the plan: entries in a 
HotRod-managed cache are ByteArrayKey/CacheValue pairs and 
unfortunately, when the "target" reads from the RCS they get unwrapped 
into their byte[] equivalents.


The solutions we have are:

1. have a special marshaller placed on the RemoteCacheStore's
   RemoteCacheManager which rewraps the entries. Unfortunately
   marshallers can't distinguish between keys and values, so this would
   probably require some horrid ThreadLocal trickery
2. Add a new option to RemoteCacheStore so that it rewraps entries in
   the ByteArrayKey/CacheValue format. Unfortunately the CacheValue
   class is part of server-core, but the dependency could be made
   optional, and in the context of the Rolling Upgrade scenario it is a
   non-issue, since it will be in the classpath
3. Introduce a new MigrationRemoteCacheStore which does the same as the
   above, but without changing RCS itself.

My personal favourite is number 2, but I trust your better judgement.

I think these are merely workarounds and we should have a better way for 
"entry wrappers" (such as the cache servers) to "localize" the entries 
for their own particular needs. Also I believe we need a better way to 
attach metadata to entries in a portable way so that we don't need these 
value wrappers.


Tristan

___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] HotRod server and Rolling Upgrades

2012-12-11 Thread Dan Berindei 
+1 to eliminate the value wrappers.

-1 to adding a dependency from core to server-core, if you feel creating
and maintaining a separate MigrationRemoteCacheStore is too much work I'd
rather we moved CacheValue to core.

If we move CacheValue to core, I think we can do the re-wrapping on the
rawValues branch and avoid adding another setting to the RCS configuration.



On Tue, Dec 11, 2012 at 3:02 PM, Tristan Tarrant wrote:

>  So,
> I thought we had everything ready to go for HotRod rolling upgrades:
>
>
>- have HotRod server full of data (the "source")
>- configure a new HotRod server (the "target") with a RemoteCacheStore
>pointing to the "source" (using "rawValues")
>- clients switch over to the "target" server which on cache misses
>should seamlessly fetch entries from the "source"
>- issue a "dump keys" on the source
>- fetch the "dumped keys" from the target
>- disable the RCS on the target and switch off the "source" for good
> - PROFIT$$$
>
> Unfortunately there is a teeny tiny flaw in the plan: entries in a
> HotRod-managed cache are ByteArrayKey/CacheValue pairs and unfortunately,
> when the "target" reads from the RCS they get unwrapped into their byte[]
> equivalents.
>
> The solutions we have are:
>
>1. have a special marshaller placed on the RemoteCacheStore's
>RemoteCacheManager which rewraps the entries. Unfortunately marshallers
>can't distinguish between keys and values, so this would probably require
>some horrid ThreadLocal trickery
>2. Add a new option to RemoteCacheStore so that it rewraps entries in
>the ByteArrayKey/CacheValue format. Unfortunately the CacheValue class is
>part of server-core, but the dependency could be made optional, and in the
>context of the Rolling Upgrade scenario it is a non-issue, since it will be
>in the classpath
> 3. Introduce a new MigrationRemoteCacheStore which does the same as
>the above, but without changing RCS itself.
>
> My personal favourite is number 2, but I trust your better judgement.
>
> I think these are merely workarounds and we should have a better way for
> "entry wrappers" (such as the cache servers) to "localize" the entries for
> their own particular needs. Also I believe we need a better way to attach
> metadata to entries in a portable way so that we don't need these value
> wrappers.
>
> Tristan
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] HotRod server and Rolling Upgrades

2012-12-11 Thread Tristan Tarrant 
On 12/11/2012 02:35 PM, Dan Berindei wrote:
> +1 to eliminate the value wrappers.
>
> -1 to adding a dependency from core to server-core, if you feel 
> creating and maintaining a separate MigrationRemoteCacheStore is too 
> much work I'd rather we moved CacheValue to core.
The dependency would only be on hotrod-client and optional at that.

Tristan
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] HotRod server and Rolling Upgrades

2012-12-11 Thread Dennis Reed 
I don't like #1.  Seems more complicated, harder to maintain & debug 
than the others.


In my opinion the best option would be #4 (eliminate the different 
formats), but that probably can't be done in a minor release?


Between 2 and 3, I'd prefer #2, handling it in the base class so it's 
automatically inherited by any custom classes that extend it.
Since the use case isn't limited to rolling upgrades; you could have a 
HotRod cache with a full-time RemoteCacheStore.


-Dennis

On 12/11/2012 07:02 AM, Tristan Tarrant wrote:

So,
I thought we had everything ready to go for HotRod rolling upgrades:

* have HotRod server full of data (the "source")
* configure a new HotRod server (the "target") with a
  RemoteCacheStore pointing to the "source" (using "rawValues")
* clients switch over to the "target" server which on cache misses
  should seamlessly fetch entries from the "source"
* issue a "dump keys" on the source
* fetch the "dumped keys" from the target
* disable the RCS on the target and switch off the "source" for good
* PROFIT$$$

Unfortunately there is a teeny tiny flaw in the plan: entries in a 
HotRod-managed cache are ByteArrayKey/CacheValue pairs and 
unfortunately, when the "target" reads from the RCS they get unwrapped 
into their byte[] equivalents.


The solutions we have are:

   1. have a special marshaller placed on the RemoteCacheStore's
  RemoteCacheManager which rewraps the entries. Unfortunately
  marshallers can't distinguish between keys and values, so this
  would probably require some horrid ThreadLocal trickery
   2. Add a new option to RemoteCacheStore so that it rewraps entries
  in the ByteArrayKey/CacheValue format. Unfortunately the
  CacheValue class is part of server-core, but the dependency
  could be made optional, and in the context of the Rolling
  Upgrade scenario it is a non-issue, since it will be in the
  classpath
   3. Introduce a new MigrationRemoteCacheStore which does the same as
  the above, but without changing RCS itself.

My personal favourite is number 2, but I trust your better judgement.

I think these are merely workarounds and we should have a better way 
for "entry wrappers" (such as the cache servers) to "localize" the 
entries for their own particular needs. Also I believe we need a 
better way to attach metadata to entries in a portable way so that we 
don't need these value wrappers.


Tristan


___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] HotRod server and Rolling Upgrades

2012-12-14 Thread Manik Surtani 
Even then, #2 would only be a temporary solution until we have #4, right?  
Would https://issues.jboss.org/browse/ISPN-2281 help in any way?

- M

On 11 Dec 2012, at 17:37, Dennis Reed  wrote:

> I don't like #1.  Seems more complicated, harder to maintain & debug than the 
> others.
> 
> In my opinion the best option would be #4 (eliminate the different formats), 
> but that probably can't be done in a minor release?
> 
> Between 2 and 3, I'd prefer #2, handling it in the base class so it's 
> automatically inherited by any custom classes that extend it.
> Since the use case isn't limited to rolling upgrades; you could have a HotRod 
> cache with a full-time RemoteCacheStore.
> 
> -Dennis
> 
> On 12/11/2012 07:02 AM, Tristan Tarrant wrote:
>> 
>> So,
>> I thought we had everything ready to go for HotRod rolling upgrades:
>> 
>> have HotRod server full of data (the "source")
>> configure a new HotRod server (the "target") with a RemoteCacheStore 
>> pointing to the "source" (using "rawValues")
>> clients switch over to the "target" server which on cache misses should 
>> seamlessly fetch entries from the "source"
>> issue a "dump keys" on the source
>> fetch the "dumped keys" from the target
>> disable the RCS on the target and switch off the "source" for good
>> PROFIT$$$
>> Unfortunately there is a teeny tiny flaw in the plan: entries in a 
>> HotRod-managed cache are ByteArrayKey/CacheValue pairs and 
>> unfortunately, when the "target" reads from the RCS they get unwrapped into 
>> their byte[] equivalents.
>> The solutions we have are:
>> have a special marshaller placed on the RemoteCacheStore's 
>> RemoteCacheManager which rewraps the entries. Unfortunately marshallers 
>> can't distinguish between keys and values, so this would probably require 
>> some horrid ThreadLocal trickery
>> Add a new option to RemoteCacheStore so that it rewraps entries in the 
>> ByteArrayKey/CacheValue format. Unfortunately   the CacheValue class 
>> is part of server-core, but the dependency could be made optional, and in 
>> the context of the Rolling Upgrade scenario it is a non-issue, since it will 
>> be in the classpath
>> Introduce a new MigrationRemoteCacheStore which does the same as the above, 
>> but without changing RCS itself.
>> My personal favourite is number 2, but I trust your better judgement.
>> I think these are merely workarounds and we should have a better way for 
>> "entry wrappers" (such as the cache servers) to "localize" the entries for 
>> their own particular needs. Also I believe we need a better way to attach 
>> metadata to entries in a portable way so that we don't need these value 
>> wrappers.
>> Tristan
>> 
>> ___
>> infinispan-dev mailing list
>> infinispan-dev@lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

--
Manik Surtani
ma...@jboss.org
twitter.com/maniksurtani

Platform Architect, JBoss Data Grid
http://red.ht/data-grid

___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] HotRod server and Rolling Upgrades

2012-12-14 Thread Tristan Tarrant 
I would really like ISPN-2281 to be solved by the proper solution, i.e. 
attach any additional metadata required by the servers (or any other 
data "enricher") to the InternalCacheEntry and not as part of the value 
itself. Metadata would be some kind of sparse structure whose values 
could also be inherited implicitly by Cache-level metadata (to save memory).


Let's think about data typing: "a string is a string is a string", and 
not some weirdly marshalled byte array (which may be influenced by 
Marshaller, protocol value wrapper, phase of the moon, etc). Protocols 
would say "this is a string" (e.g. REST via Content-type: text/plain, 
InVM by just storing a java.lang.String) and the server would store it 
in an "as-native-as-possible" format, recording the type along with it. 
When someone retrieves it it needs to be translated to something the 
client understands. Obviously if I'm going to have a Cache of Strings, 
that metadata would be "global" to the cache and entries would not have 
that information.


Was this ever planned ? Canned ? Banned ?

Tristan

On 12/14/2012 01:14 PM, Manik Surtani wrote:
Even then, #2 would only be a temporary solution until we have #4, 
right?  Would https://issues.jboss.org/browse/ISPN-2281 help in any way?


- M

On 11 Dec 2012, at 17:37, Dennis Reed > wrote:


I don't like #1. Seems more complicated, harder to maintain & debug 
than the others.


In my opinion the best option would be #4 (eliminate the different 
formats), but that probably can't be done in a minor release?


Between 2 and 3, I'd prefer #2, handling it in the base class so it's 
automatically inherited by any custom classes that extend it.
Since the use case isn't limited to rolling upgrades; you could have 
a HotRod cache with a full-time RemoteCacheStore.


-Dennis

On 12/11/2012 07:02 AM, Tristan Tarrant wrote:

So,
I thought we had everything ready to go for HotRod rolling upgrades:

  * have HotRod server full of data (the "source")
  * configure a new HotRod server (the "target") with a
RemoteCacheStore pointing to the "source" (using "rawValues")
  * clients switch over to the "target" server which on cache misses
should seamlessly fetch entries from the "source"
  * issue a "dump keys" on the source
  * fetch the "dumped keys" from the target
  * disable the RCS on the target and switch off the "source" for good
  * PROFIT$$$

Unfortunately there is a teeny tiny flaw in the plan: entries in a 
HotRod-managed cache are ByteArrayKey/CacheValue pairs and 
unfortunately, when the "target" reads from the RCS they get 
unwrapped into their byte[] equivalents.


The solutions we have are:

 1. have a special marshaller placed on the RemoteCacheStore's
RemoteCacheManager which rewraps the entries. Unfortunately
marshallers can't distinguish between keys and values, so this
would probably require some horrid ThreadLocal trickery
 2. Add a new option to RemoteCacheStore so that it rewraps entries
in the ByteArrayKey/CacheValue format. Unfortunately the
CacheValue class is part of server-core, but the dependency
could be made optional, and in the context of the Rolling
Upgrade scenario it is a non-issue, since it will be in the
classpath
 3. Introduce a new MigrationRemoteCacheStore which does the same as
the above, but without changing RCS itself.

My personal favourite is number 2, but I trust your better judgement.

I think these are merely workarounds and we should have a better way 
for "entry wrappers" (such as the cache servers) to "localize" the 
entries for their own particular needs. Also I believe we need a 
better way to attach metadata to entries in a portable way so that 
we don't need these value wrappers.


Tristan


___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org 
https://lists.jboss.org/mailman/listinfo/infinispan-dev


--
Manik Surtani
ma...@jboss.org 
twitter.com/maniksurtani 

Platform Architect, JBoss Data Grid
http://red.ht/data-grid



___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev


___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] HotRod server and Rolling Upgrades

2012-12-17 Thread Galder Zamarreño 

On Dec 14, 2012, at 2:56 PM, Tristan Tarrant  wrote:

> I would really like ISPN-2281 to be solved by the proper solution, i.e. 
> attach any additional metadata required by the servers (or any other data 
> "enricher") to the InternalCacheEntry and not as part of the value itself. 
> Metadata would be some kind of sparse structure whose values could also be
>inherited implicitly by Cache-level metadata (to save memory).

^ +1 - I think that would get rid of unnecessary wrapper values (which get 
wrapped again in ICE…)  

> Let's think about data typing: "a string is a string is a string", and not 
> some weirdly marshalled byte array (which may be influenced by Marshaller, 
> protocol value wrapper, phase of the moon, etc). Protocols would say "this is 
> a string" (e.g. REST via Content-type: text/plain, InVM by just storing a 
> java.lang.String) and the server would store it in an "as-native-as-possible" 
> format, recording the type along with it. When someone retrieves it it needs 
> to be translated to something the client understands. Obviously if I'm going 
> to have a Cache of Strings, that metadata would be "global" to the cache and 
> entries would not have that information.

^ Somehow related to this, Tristan made an interesting suggestion last week wrt 
Hot Rod key type. Right now this is a ByteArrayKey, which wraps an array and 
provides content based equals() implementation.

It would be interesting to allow byte[] to be stored as keys directly, and 
instead do the comparison via some Comparator implementation. This would get 
rid of the wrapper class.

I will add a note to ISPN-2281 on this.

Cheers,

> Was this ever planned ? Canned ? Banned ?
> 
> Tristan
> 
> On 12/14/2012 01:14 PM, Manik Surtani wrote:
>> Even then, #2 would only be a temporary solution until we have #4, right?  
>> Would https://issues.jboss.org/browse/ISPN-2281 help in any way?
>> 
>> - M
>> 
>> On 11 Dec 2012, at 17:37, Dennis Reed  wrote:
>> 
>>> I don't like #1.  Seems more complicated, harder to maintain & debug than 
>>> the others.
>>> 
>>> In my opinion the best option would be #4 (eliminate the different 
>>> formats), but that probably can't be done in a minor release?
>>> 
>>> Between 2 and 3, I'd prefer #2, handling it in the base class so it's 
>>> automatically inherited by any custom classes that extend it.
>>> Since the use case isn't limited to rolling upgrades; you could have a 
>>> HotRod cache with a full-time RemoteCacheStore.
>>> 
>>> -Dennis
>>> 
>>> On 12/11/2012 07:02 AM, Tristan Tarrant wrote:
 So,
 I thought we had everything ready to go for HotRod rolling upgrades:
 
• have HotRod server full of data (the "source")
• configure a new HotRod server (the "target") with a RemoteCacheStore 
 pointing to the "source" (using "rawValues")
• clients switch over to the "target" server which on cache misses 
 should seamlessly fetch entries from the "source"
• issue a "dump keys" on the source
• fetch the "dumped keys" from the target
• disable the RCS on the target and switch off the "source" for good
• PROFIT$$$
 Unfortunately there is a teeny tiny flaw in the plan: entries in a 
 HotRod-managed cache are ByteArrayKey/CacheValue pairs and unfortunately, 
 when the "target" reads from the RCS they get unwrapped into their byte[] 
 equivalents.
 The solutions we have are:
• have a special marshaller placed on the RemoteCacheStore's 
 RemoteCacheManager which rewraps the entries. Unfortunately marshallers 
 can't distinguish between keys and values, so this would probably require 
 some horrid ThreadLocal trickery
• Add a new option to RemoteCacheStore so that it rewraps entries in 
 the ByteArrayKey/CacheValue format. Unfortunately the CacheValue class is 
 part of server-core, but the dependency could be made optional, and in the 
 context of the Rolling Upgrade scenario it is a non-issue, since it will 
 be in the classpath
• Introduce a new MigrationRemoteCacheStore which does the same as the 
 above, but without changing RCS itself.
 My personal favourite is number 2, but I trust your better judgement.
 I think these are merely workarounds and we should have a better way for 
 "entry wrappers" (such as the cache servers) to "localize" the entries for 
 their own particular needs. Also I believe we need a better way to attach 
 metadata to entries in a portable way so that we don't need these value 
 wrappers.
 Tristan
 
 ___
 infinispan-dev mailing list
 
 infinispan-dev@lists.jboss.org
 https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>> ___
>>> infinispan-dev mailing list
>>> infinispan-dev@lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> 
>> --
>> Manik Surtani

Re: [infinispan-dev] HotRod server and Rolling Upgrades

2012-12-19 Thread Manik Surtani 

On 14 Dec 2012, at 13:56, Tristan Tarrant  wrote:

> 
> Let's think about data typing: "a string is a string is a string", and not 
> some weirdly marshalled byte array (which may be influenced by Marshaller, 
> protocol value wrapper, phase of the moon, etc). Protocols would say "this is 
> a string" (e.g. REST via Content-type: text/plain, InVM by just storing a 
> java.lang.String) and the server would store it in an "as-native-as-possible" 
> format, recording the type along with it. When someone retrieves it it needs 
> to be translated to something the client understands. Obviously if I'm going 
> to have a Cache of Strings, that metadata would be "global" to the cache and 
> entries would not have that information.
> 
> Was this ever planned ? Canned ? Banned ?

Never planned.  But makes plenty of sense.  We should discuss this as a part of 
ISPN-2281, which addresses format portability.

--
Manik Surtani
ma...@jboss.org
twitter.com/maniksurtani

Platform Architect, JBoss Data Grid
http://red.ht/data-grid

___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev