Re: [infinispan-dev] Infinispan client/server architecture based on gRPC

[Sanne Grinovero]

On 29 May 2018 at 13:45, Vittorio Rigamonti  wrote:

> Thanks Adrian,
>
> of course there's a marshalling work under the cover and that is reflected
> into the generated code (specially the accessor methods generated from the
> oneof clause).
>
> My opinion is that on the client side this could be accepted, as long as
> the API are well defined and documented: application developer can build an
> adhoc decorator on the top if needed. The alternative to this is to develop
> a protostream equivalent for each supported language and it doesn't seem
> really feasible to me.
>

​This might indeed be reasonable for some developers, some languages.

Just please make sure it's not the only option, as many other developers
will not expect to need a compiler at hand in various stages of the
application lifecycle.

For example when deploying a JPA model into an appserver, or just booting
Hibernate in JavaSE as well, there is a strong expectation that we'll be
able - at runtime - to inspect the listed Java POJOs via reflection and
automatically generate whatever Infinispan will need.

Perhaps a key differentiator is between invoking Infinispan APIs (RPC) vs
defining the object models and related CODECs for keys, values, streams and
query results? It might get a bit more fuzzy to differentiate them for
custom functions but I guess we can draw a line somewhere.

Thanks,
Sanne



>
> On the server side (java only) the situation is different: protobuf is
> optimized for streaming not for storing so probably a Protostream layer is
> needed.
>
> On Mon, May 28, 2018 at 4:47 PM, Adrian Nistor  wrote:
>
>> Hi Vittorio,
>> thanks for exploring gRPC. It seems like a very elegant solution for
>> exposing services. I'll have a look at your PoC soon.
>>
>> I feel there are some remarks that need to be made regarding gRPC. gRPC
>> is just some nice cheesy topping on top of protobuf. Google's
>> implementation of protobuf, to be more precise.
>> It does not need handwritten marshallers, but the 'No need for
>> marshaller' does not accurately describe it. Marshallers are needed and are
>> generated under the cover by the library and so are the data objects and
>> you are unfortunately forced to use them. That's both the good news and the
>> bad news:) The whole thing looks very promising and friendly for many uses
>> cases, especially for demos and PoCs :))). Nobody wants to write those
>> marshallers. But it starts to become a nuisance if you want to use your own
>> data objects.
>> There is also the ugliness and excessive memory footprint of the
>> generated code, which is the reason Infinispan did not adopt the
>> protobuf-java library although it did adopt protobuf as an encoding format.
>> The Protostream library was created as an alternative implementation to
>> solve the aforementioned problems with the generated code. It solves this
>> by letting the user provide their own data objects. And for the marshallers
>> it gives you two options: a) write the marshaller yourself (hated), b)
>> annotated your data objects and the marshaller gets generated (loved).
>> Protostream does not currently support service definitions right now but
>> this is something I started to investigate recently after Galder asked me
>> if I think it's doable. I think I'll only find out after I do it:)
>>
>> Adrian
>>
>>
>> On 05/28/2018 04:15 PM, Vittorio Rigamonti wrote:
>>
>> Hi Infinispan developers,
>>
>> I'm working on a solution for developers who need to access Infinispan
>> services  through different programming languages.
>>
>> The focus is not on developing a full featured client, but rather
>> discover the value and the limits of this approach.
>>
>> - is it possible to automatically generate useful clients in different
>> languages?
>> - can that clients interoperate on the same cache with the same data
>> types?
>>
>> I came out with a small prototype that I would like to submit to you and
>> on which I would like to gather your impressions.
>>
>>  You can found the project here [1]: is a gRPC-based client/server
>> architecture for Infinispan based on and EmbeddedCache, with very few
>> features exposed atm.
>>
>> Currently the project is nothing more than a poc with the following
>> interesting features:
>>
>> - client can be generated in all the grpc supported language: java, go,
>> c++ examples are provided;
>> - the interface is full typed. No need for marshaller and clients build
>> in different language can cooperate on the same cache;
>>
>> The second item is my preferred one beacuse it frees the developer from
>> data marshalling.
>>
>> What do you think about?
>> Sounds interesting?
>> Can you see any flaw?
>>
>> There's also a list of issues for the future [2], basically I would like
>> to investigate these questions:
>> How far this architecture can go?
>> Topology, events, queries... how many of the Infinispan features can be
>> fit in a grpc architecture?
>>
>> Thank you
>> Vittorio
>>
>> [1] https://github.com/rigazilla/ispn-

Re: [infinispan-dev] Infinispan client/server architecture based on gRPC

[Sanne Grinovero]

On 30 May 2018 at 13:26, Adrian Nistor  wrote:
> Yest, the client needs that hash but that does not necessarily mean it
> has to compute it itself.
> The hash should be applied to the storage format which might be
> different from the format the client sees. So hash computation could be
> done on the server, just a thought.

Unless we want to explore some form of hybrid gRPC which benefits from
Hot Rod intelligence level 3?

In which case the client will need to compute the hash before it can
hint the network layer were to connect to.

Thanks,
Sanne

>
> On 05/30/2018 02:47 PM, Radim Vansa wrote:
>> On 05/30/2018 12:46 PM, Adrian Nistor wrote:
>>> Thanks for clarifying this Galder.
>>> Yes, the network layer is indeed the culprit and the purpose of this
>>> experiment.
>>>
>>> What is the approach you envision regarding the IDL? Should we strive
>>> for a pure IDL definition of the service? That could be an interesting
>>> approach that would make it possible for a third party to generate
>>> their own infinispan grpc client in any new language that we do not
>>> already offer support, just based on the IDL. And maybe using a
>>> different grpc implementation if they do not find suitable the one
>>> from google.
>>>
>>> I was not suggesting we should do type transformation or anything on
>>> the client side that would require an extra layer of code on top of
>>> what grpc generates for the client, so maybe a pure IDL based service
>>> definition would indeed be possible, without extra helpers. No type
>>> transformation, just type information. Exposing the type info that
>>> comes from the server would be enough, a lot better than dumbing
>>> everything down to a byte[].
>> I may be wrong but key transformation on client is necessary for correct
>> hash-aware routing, isn't it? We need to get byte array for each key and
>> apply murmur hash there (IIUC even when we use protobuf as the storage
>> format, segment is based on the raw protobuf bytes, right?).
>>
>> Radim
>>
>>> Adrian
>>>
>>> On 05/30/2018 12:16 PM, Galder Zamarreno wrote:
 On Tue, May 29, 2018 at 8:57 PM Adrian Nistor >>> > wrote:

  Vittorio, a few remarks regarding your statement "...The
  alternative to this is to develop a protostream equivalent for
  each supported language and it doesn't seem really feasible to me."

  No way! That's a big misunderstanding. We do not need to
  re-implement the protostream library in C/C++/C# or any new
  supported language.
  Protostream is just for Java and it is compatible with Google's
  protobuf lib we already use in the other clients. We can continue
  using Google's protobuf lib for these clients, with or without gRPC.
  Protostream does not handle protobuf services as gRPC does, but
  we can add support for that with little effort.

  The real problem here is if we want to replace our hot rod
  invocation protocol with gRPC to save on the effort of
  implementing and maintaining hot rod in all those clients. I
  wonder why the obvious question is being avoided in this thread.


 ^ It is not being avoided. I stated it quite clearly when I replied
 but maybe not with enough detail. So, I said:

>   The biggest problem I see in our client/server architecture is the
 ability to quickly deliver features/APIs across multiple language
 clients. Both Vittorio and I have seen how long it takes to implement
 all the different features available in Java client and port them to
 Node.js, C/C++/C#...etc. This effort lead by Vittorio is trying to
 improve on that by having some of that work done for us. Granted, not
 all of it will be done, but it should give us some good foundations
 on which to build.

 To expand on it a bit further: the reason it takes us longer to get
 different features in is because each client implements its own
 network layer, parses the protocol and does type transformations
 (between byte[] and whatever the client expects).

 IMO, the most costly things there are getting the network layer right
 (from experience with Node.js, it has taken a while to do so) and
 parsing work (not only parsing itself, but doing it in a efficient
 way). Network layer also includes load balancing, failover, cluster
 failover...etc.

  From past experience, transforming from byte[] to what the client
 expects has never really been very problematic for me. What's been
 difficult here is coming up with encoding architecture that Gustavo
 lead, whose aim was to improve on the initial compatibility mode.
 But, with that now clear, understood and proven to solve our issues,
 the rest in this area should be fairly straightforward IMO.

 Type transformation, once done, is a constant. As we add more Hot Rod
 operations,

Re: [infinispan-dev] Infinispan client/server architecture based on gRPC

[Sanne Grinovero]

Thanks Manik! Great to hear some feedback from you, especially as you
have way more experience with gRPC.

Beyond helping to develop (and maintain!) clients for a wider range of
programming languages - it would also help to provide both a
"traditional" and a non-blocking client for each such language, while
having to maintain just an async server implementation.

Sanne

On 6 June 2018 at 07:13, Manik Surtani  wrote:
> Hello everyone! (Wow, it's been a while since I dropped by and said hello...
> )
>
> Super-interesting discussion.
>
> Adrian: "What is the approach you envision regarding the IDL? Should we
> strive for a pure IDL definition of the service? That could be an
> interesting approach that would make it possible for a third party to
> generate their own infinispan grpc client in any new language that we do not
> already offer support, just based on the IDL. And maybe using a different
> grpc implementation if they do not find suitable the one from google."
>
> This is spot-on, and where I see value in gRPC being incorporated into
> Infinispan: making use of open(-ish) standards of RPC communication and
> applying that to how we do client/server. Good discussion on handling
> payload types in the interface definition. I've made use of some of the
> ideas discussed here when creating a proto-defined IDL to look up ... more
> serialized proto definitions for given types! Keen to see what your PoC
> looks like.
>
> - M
>
>
> On Wed, 30 May 2018 at 08:19 Galder Zamarreno  wrote:
>>
>> On Wed, May 30, 2018 at 5:00 PM Radim Vansa  wrote:
>>>
>>> On 05/30/2018 02:53 PM, Sanne Grinovero wrote:
>>> > On 30 May 2018 at 13:26, Adrian Nistor  wrote:
>>> >> Yest, the client needs that hash but that does not necessarily mean it
>>> >> has to compute it itself.
>>> >> The hash should be applied to the storage format which might be
>>> >> different from the format the client sees. So hash computation could
>>> >> be
>>> >> done on the server, just a thought.
>>> > Unless we want to explore some form of hybrid gRPC which benefits from
>>> > Hot Rod intelligence level 3?
>>>
>>> Since Tristan said that gRPC is viable only if the performance is
>>> comparable - I concluded that this involves the smart routing. I was
>>> hoping that gRPC networking layer would provide some hook to specify the
>>> destination.
>>
>>
>> It does, via SubchannelPicker implementations. It requires key to be sent
>> as HTTP header down the stack so that the SubchannelPicker can extract it.
>>
>> SubchannelPicker impl can then apply hash on it and decide based on
>> available channels.
>>
>>>
>>> An alternative would be a proxy hosted on the same node
>>> that would do the routing.
>>>
>>>
>>> If we're to replace Hot Rod I was expecting the (generated) gRPC client
>>> to be extensible enough to allow us add client-side features (like near
>>> cache, maybe listeners would need client-side code too) but saving us
>>> most of the hassle with networking and parsing, while providing basic
>>> client in languages we don't embrace without additional cost.
>>>
>>> R.
>>>
>>> >
>>> > In which case the client will need to compute the hash before it can
>>> > hint the network layer were to connect to.
>>> >
>>> > Thanks,
>>> > Sanne
>>> >
>>> >> On 05/30/2018 02:47 PM, Radim Vansa wrote:
>>> >>> On 05/30/2018 12:46 PM, Adrian Nistor wrote:
>>> >>>> Thanks for clarifying this Galder.
>>> >>>> Yes, the network layer is indeed the culprit and the purpose of this
>>> >>>> experiment.
>>> >>>>
>>> >>>> What is the approach you envision regarding the IDL? Should we
>>> >>>> strive
>>> >>>> for a pure IDL definition of the service? That could be an
>>> >>>> interesting
>>> >>>> approach that would make it possible for a third party to generate
>>> >>>> their own infinispan grpc client in any new language that we do not
>>> >>>> already offer support, just based on the IDL. And maybe using a
>>> >>>> different grpc implementation if they do not find suitable the one
>>> >>>> from google.
>>> >>>>
>>> >>>> I was not suggesting we should do type transformation or an

[infinispan-dev] Upgrade to JTA 1.2 API?

[Sanne Grinovero]

Would it be possible to upgrade
 - 
org.jboss.spec.javax.transaction:jboss-transaction-api_1.1_spec:jar:1.0.1.Final
to
 - 
org.jboss.spec.javax.transaction:jboss-transaction-api_1.2_spec:jar:1.0.1.Final

?

Thanks!
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] Upgrade to JTA 1.2 API?

[Sanne Grinovero]

If I knew I would have opened a JIRA and sent a PR ;)

The most important point IMO is that other projects such as WildFly and
Hibernate are using the 1.2 version since a long time already so Infinispan
is effectively testing a version which is not the one we run it with, thus
might hide some issue.


On Tue, 26 Jun 2018, 09:56 Galder Zamarreno,  wrote:

> Are the backwards compatible? Any gotchas we should be aware of?
>
> On Mon, Jun 25, 2018 at 10:07 PM Sanne Grinovero 
> wrote:
>
>> Would it be possible to upgrade
>>  -
>> org.jboss.spec.javax.transaction:jboss-transaction-api_1.1_spec:jar:1.0.1.Final
>> to
>>  -
>> org.jboss.spec.javax.transaction:jboss-transaction-api_1.2_spec:jar:1.0.1.Final
>>
>> ?
>>
>> Thanks!
>> ___
>> infinispan-dev mailing list
>> infinispan-dev@lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] Upgrade to JTA 1.2 API?

[Sanne Grinovero]

On Tue, 26 Jun 2018 at 10:20, Jonathan Halliday
 wrote:
>
>
> 1.2 added the @Transactional and @TransactionScoped annotations, but did
> not change the existing interfaces. It's a low risk upgrade.

Thanks Jonathan!
 - https://issues.jboss.org/browse/ISPN-9324

>
> Jonathan.
>
> On 26/06/18 10:08, Sanne Grinovero wrote:
> > If I knew I would have opened a JIRA and sent a PR ;)
> >
> > The most important point IMO is that other projects such as WildFly and
> > Hibernate are using the 1.2 version since a long time already so
> > Infinispan is effectively testing a version which is not the one we run
> > it with, thus might hide some issue.
> >
> >
> > On Tue, 26 Jun 2018, 09:56 Galder Zamarreno,  > <mailto:gal...@redhat.com>> wrote:
> >
> > Are the backwards compatible? Any gotchas we should be aware of?
> >
> > On Mon, Jun 25, 2018 at 10:07 PM Sanne Grinovero
> > mailto:sa...@infinispan.org>> wrote:
> >
> > Would it be possible to upgrade
> >   -
> > 
> > org.jboss.spec.javax.transaction:jboss-transaction-api_1.1_spec:jar:1.0.1.Final
> > to
> >   -
> > 
> > org.jboss.spec.javax.transaction:jboss-transaction-api_1.2_spec:jar:1.0.1.Final
> >
> > ?
> >
> > Thanks!
> > ___
> > infinispan-dev mailing list
> > infinispan-dev@lists.jboss.org
> > <mailto:infinispan-dev@lists.jboss.org>
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >
> > ___
> > infinispan-dev mailing list
> > infinispan-dev@lists.jboss.org <mailto:infinispan-dev@lists.jboss.org>
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >
> >
> >
> > ___
> > infinispan-dev mailing list
> > infinispan-dev@lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >
>
> --
> Registered in England and Wales under Company Registration No. 03798903
> Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

[infinispan-dev] Build broken

[Sanne Grinovero]

Radim, Dan, your last commit 58aa3b2185 broke master.

Please fix or revert ?

Thanks
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] Build broken

[Sanne Grinovero]

Dan fixed it via PR #6094

Thanks!
On Tue, 26 Jun 2018 at 12:45, Sanne Grinovero  wrote:
>
> Radim, Dan, your last commit 58aa3b2185 broke master.
>
> Please fix or revert ?
>
> Thanks
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

[infinispan-dev] Dependency hygiene: 2LC for Hibernate to depend on RxJava ?

[Sanne Grinovero]

Hi all,

I noticed the latest versions of infinispan-core now depend on
io.reactivex.rxjava2:rxjava aka RxJava.

Would it be possible to make this an optional dependency, at least for
those only looking for a simple JPA Cache?

I tried to exclude it but that's not going to work, apparently many
crucial code paths depend on it.

Thanks,
Sanne
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] Obtaining cache via CDI on WildFly

[Sanne Grinovero]

On Wed, 5 Dec 2018 at 11:16, Tristan Tarrant  wrote:
>
> On 12/5/18 9:44 AM, Gunnar Morling wrote:
> > Hey,
> >
> > I was trying to configure and inject an Infinispan cache through CDI,
> > running on WildFly 14, using the Infinispan modules provided by the
> > server.
> >
> > While I'm not sure whether that's something supported or recommended,
> > I found this preferable over adding Infinispan another time as part of
> > the deployment. I couldn't find any recent info on doing this (would
> > love any pointers, though), so here's my findings, in case it's
> > interesting for others:
>
> You should not be using the Infinispan subsystem that comes with WildFly
> as its configuration capabilities are a bit limited, but the modules we
> supply:
>
> http://infinispan.org/docs/stable/user_guide/user_guide.html#infinispan_modules_for_wildfly_eap

Hi Gunnar,
to automate such things during integration testing, have a look at the
provisioning plugins for Maven.

 - 
https://github.com/infinispan/infinispan/tree/master/integrationtests/wildfly-modules
 - 
https://github.com/infinispan/infinispan/blob/master/integrationtests/wildfly-modules/pom.xml#L222-L249
 - 
https://github.com/infinispan/infinispan/blob/master/integrationtests/wildfly-modules/server-provisioning.xml

One the custom WildFly is built, you can use it "as usual" via Arquillian.

Don't use the Infinispan jars included within Wildfly as those are
"implementation detail": don't have all the bits necessary (e.g. CDI
integration) and are not tested nor supportable for such use case.

Hibernate OGM does something similar:
 - 
https://github.com/hibernate/hibernate-ogm/blob/master/integrationtest/server-provisioning.xml
It's not listing the Infinispan modules explicitly as some specific
OGM modules depend on them transitively, but you get the idea ;)

-- Sanne

>
> > Btw. I also couldn't find an example for configuring a cache through
> > jboss-cli.sh, perhaps that's something to consider, too?
>
> Yes, that should be added.
>
> Tristan
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] Obtaining cache via CDI on WildFly

[Sanne Grinovero]

Hi Gunnar,

if all you want is to invalidate Hibernate ORM's 2LC caches, Hibernate
exposes a specific API to do just that.

Sanne

On Wed, 5 Dec 2018 at 14:30, Gunnar Morling  wrote:
>
> Hey all,
>
> Thanks a lot for the quick replies!
>
> To give some background on what I was trying to do: my intention was
> to use a simple cache within my app for a demo + blog post I'm
> creating on invalidating the JPA 2nd-level cache after external data
> changes (i.e. bypassing the app) via Debezium:
>
> 
> https://github.com/debezium/debezium-examples/tree/master/cache-invalidation
> https://github.com/debezium/debezium.github.io/pull/230
>
> For that I need a simple in-app cache to keep track of all
> transactions run by the app itself, so to keep them apart from
> transactions run by external clients (as I need to invalidate the 2L
> cache items only for the latter).
>
> So the questions around support are not too much of a concern for my
> purpose. Using the modules coming with the server seemed so easy in
> comparison to putting the modules in place :) I'll try and have a look
> at how this could be done in my Dockerfile (this btw. could be an
> interesting example for you to have, too). Regarding CDI, I gave up on
> this and just obtained a cache via the API. Seemed simpler in the end.
>
> Thanks again,
>
> --Gunnar
>
>
> Am Mi., 5. Dez. 2018 um 13:02 Uhr schrieb Wolf Fink :
> >
> > As Tristan said, the infinispan bits shipped with WildFly and its 
> > configuration will not have all ISPN features. It might change over the 
> > time as there is no test which ensure that any feature beside those which 
> > are used from the WF container.
> > The configuration for the subsystem is different and will not allow all 
> > features.
> > Also if there is a plan to move to the supported products this is not 
> > supported!
> >
> > The best option is to use the infinispan modules and configure it in 
> > server-mode, in this case the cache lifecycle is bound to the WF instance 
> > and can be shared/injected to all deployed applications (sharing the cache 
> > between application in embedded mode will not work)
> > Note that you might use the infinispan endpoints here, but if there is a 
> > plan to use the products the use of endpoints is not supported (as it will 
> > make WF a hybrid server for both)
> >
> > Wolf
> >
> > On Wed, Dec 5, 2018 at 12:17 PM Tristan Tarrant  wrote:
> >>
> >> On 12/5/18 9:44 AM, Gunnar Morling wrote:
> >> > Hey,
> >> >
> >> > I was trying to configure and inject an Infinispan cache through CDI,
> >> > running on WildFly 14, using the Infinispan modules provided by the
> >> > server.
> >> >
> >> > While I'm not sure whether that's something supported or recommended,
> >> > I found this preferable over adding Infinispan another time as part of
> >> > the deployment. I couldn't find any recent info on doing this (would
> >> > love any pointers, though), so here's my findings, in case it's
> >> > interesting for others:
> >>
> >> You should not be using the Infinispan subsystem that comes with WildFly
> >> as its configuration capabilities are a bit limited, but the modules we
> >> supply:
> >>
> >> http://infinispan.org/docs/stable/user_guide/user_guide.html#infinispan_modules_for_wildfly_eap
> >>
> >> > Btw. I also couldn't find an example for configuring a cache through
> >> > jboss-cli.sh, perhaps that's something to consider, too?
> >>
> >> Yes, that should be added.
> >>
> >> Tristan
> >> ___
> >> infinispan-dev mailing list
> >> infinispan-dev@lists.jboss.org
> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

[infinispan-dev] BoundedConcurrentHashMap

[Sanne Grinovero]

Hi all,

does anyone remember where BoundedConcurrentHashMap was copied from?

we have a copy in Hibernate ORM; the comments state:
 - copied from Infinispan
 - original author Doug Lea

but I don't see any similar implementation in JSR166, nor any
reference to this classname on their archives:
 - 
http://jsr166-concurrency.10961.n7.nabble.com/template/NamlServlet.jtp?macro=search_page&node=2&query=BoundedConcurrentHashMap

The comments looks like suspiciously like this was originally a copy
of ConcurrentHashMap... I'm wondering which fixes we're missing out,
and if I should plan to get rid of this liability since Infinispan
also seems to have removed it.

Thanks,
Sanne
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] BoundedConcurrentHashMap

[Sanne Grinovero]

Thanks Pedro and Tristan!

I've been digging a bit more; found a comment in the sources pointing
also to [1], which nowadays is also known as "Caffeine" ... :)

But indeed seems most credit goes to Infinispan; too bad it's no
longer maintained.

Thanks,
Sanne

1 - 
http://code.google.com/p/concurrentlinkedhashmap/source/browse/trunk/src/test/java/com/googlecode/concurrentlinkedhashmap/caches/LirsMap.java


On Wed, 10 Jul 2019 at 20:16, Tristan Tarrant  wrote:
>
> The bounding part was never part of Doug's code, but was written by us
>
> On Wed, 10 Jul 2019, 20:25 Sanne Grinovero,  wrote:
>>
>> Hi all,
>>
>> does anyone remember where BoundedConcurrentHashMap was copied from?
>>
>> we have a copy in Hibernate ORM; the comments state:
>>  - copied from Infinispan
>>  - original author Doug Lea
>>
>> but I don't see any similar implementation in JSR166, nor any
>> reference to this classname on their archives:
>>  - 
>> http://jsr166-concurrency.10961.n7.nabble.com/template/NamlServlet.jtp?macro=search_page&node=2&query=BoundedConcurrentHashMap
>>
>> The comments looks like suspiciously like this was originally a copy
>> of ConcurrentHashMap... I'm wondering which fixes we're missing out,
>> and if I should plan to get rid of this liability since Infinispan
>> also seems to have removed it.
>>
>> Thanks,
>> Sanne
>> ___
>> infinispan-dev mailing list
>> infinispan-dev@lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] L1 Data Container

[Sanne Grinovero]

On 19 June 2013 19:00, Dan Berindei  wrote:
>
>
>
> On Wed, Jun 19, 2013 at 5:19 PM, Sanne Grinovero 
> wrote:
>>
>> On 19 June 2013 13:44, William Burns  wrote:
>> > All the L1 data for a DIST cache is stored in the same data container as
>> > the
>> > actual distributed data itself.  I wanted to propose breaking this out
>> > so
>> > there is a separate data container for the L1 cache as compared to the
>> > distributed data.
>> >
>> > I thought of a few quick benefits/drawbacks:
>> >
>> > Benefits:
>> > 1. L1 cache can be separately tuned - L1 maxEntries for example
>>
>> -1!
>>   I don't think thats a benefit actually, from the point of view of a
>> user:
>> as a user I only know I have a certain amount of memory available on
>> each node, and the application is going to use certain data way more
>> often than others.
>> The eviction strategy should be put in condition to be able to make an
>> optimal choice about which entries - among all - are better kept in
>> memory vs. passivated.
>> I don't see a specific reason to "favour" keeping in memory owned
>> entries over an L1 entry: the L1 entry might be very hot, and the
>> owned entry might be almost-never read.
>>
>> Considering that even serving a Get operation to another node (as
>> owners of the entry) makes the entry less likely to be passivated (it
>> counts as a "hit"), the current design naturally provides an optimal
>> balance for memory usage.
>>
>> At the opposite site, I don't see how - as a user - I could optimally
>> tune a separate container.
>>
>> > 2. L1 values will not cause eviction of real data
>>
>> -1
>> That's not a benefit, as I explained above. "Real Data" is not less
>> important, especially if it's never used.
>> Granted I'm making some assumptions about the application having some
>> hot-data and some less hot data, and not being able to take advantage
>> of node pinning or affinity strategies.. but that is another way to
>> say that I'm assuming the user needs L1: if it was possible to apply
>> these more advanced strategies I'd disable L1 altogether.
>>
>
> You're also assuming that data can always be recreated from another source,
> but I don't think that's always the case. If Infinispan is the canonical
> data store and there is no cache store, you can't enable eviction for the
> "real data" and with a single container you can't enable eviction for the L1
> entries either.

Well spotted. True we are making many assumptions, I guess each of us
is having some use case in mind and designing for it.
While our configuration options are flexible, we can design to provide
an optimal structure for all combinations, especially as some
configuration options can be mixed in ways which would never be used
in a real-world solution.

In the specific situation, if I where to use Infinispan as a
"canonical data store", and without having a cache store, I think I
would at the very least isolate it from its clients by using Hot Rod;
in such case L1 would be disabled for obvious reasons.

The case I'm personally thinking of is of course the one of a Lucene
index storage; in such case I expect you to use passivation.

Would any deployment consider running without a cachestore? I doubt
it, unless it's being used as a cache.
So let's not optimize the internal structure for configurations which
would never be used.

Cheers,
Sanne


>
>>
>> > 3. Would make https://issues.jboss.org/browse/ISPN-3229 an easy fix
>> > 4. Could add a new DataContainer implementation specific to L1 with
>> > additional optimizations
>>
>> You have some example of what you have in mind?
>> Considering you would need to consider the optimal balance usage of
>> the available heap space, I suspect that would be quite hard.
>>
>> > 5. Help with some concurrency issues with L1 without requiring wider
>> > locking
>> > (such as locking a key for an entire ClusteredGet rpc call) -
>> > https://issues.jboss.org/browse/ISPN-3197.
>>
>> I don't understand this. L1 entries require the same level of
>> consistency than any other entry so I suspect you would need the same
>> locking patterns replicated. The downside would be that you get
>> duplication of the same logic.
>> Remember also that L1 is having some similarities with entries still
>> "hanging around" when they where previously stored in this node after
>> a state transfer.. today these are 

Re: [infinispan-dev] Big commit coming up

[Sanne Grinovero]

You could add a checkstyle rule to verify there is no copyright on any file;
that will help in short term during rebase but also in the longer term
as I do have IDE templates setup for other projects.. I'm often afraid
to mix them up ;-)

Sanne

On 21 June 2013 16:52, Manik Surtani  wrote:
> Guys,
>
> I'm about to put in a big commit on master, to switch to the Apache license.  
> The patch will contain:
>
> * Updated README.mkdn
> * Updated LICENSE.txt.vm
> * A new COPYRIGHT.mkdn in the project root
> * Removal of *all* copyright headers on all source code files
>
> Point #4 will mean that all outstanding pull requests will need to be rebased 
> before being merged in.
>
> Also, as a note to reviewers, please ensure there are NO copyright headers on 
> any src code files after this patch is in.
>
> Now before I submit the patch - what does everyone think of this?
>
> Cheers
> Manik
> --
> Manik Surtani
> ma...@jboss.org
> twitter.com/maniksurtani
>
> Platform Architect, JBoss Data Grid
> http://red.ht/data-grid
>
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] TeamCity posting a message in pull req if any test failed?

[Sanne Grinovero]

hibernate.ci would notify you on IRC, and on your desktop if you have
Gnome3 running.

https://extensions.gnome.org/extension/399/jenkins-ci-server-indicator/



On 21 June 2013 17:55, Galder Zamarreño  wrote:
> Hi all,
>
> Can we have a message commented in each pull request whenever teamcity has 
> completed the build if there has been any failures?
>
> Otherwise, developers are expected to check back on the pull request to see 
> if any regressions were added or not…
>
> i.e. https://github.com/infinispan/infinispan/pull/1914
>
> Cheers,
> --
> Galder Zamarreño
> gal...@redhat.com
> twitter.com/galderz
>
> Project Lead, Escalante
> http://escalante.io
>
> Engineer, Infinispan
> http://infinispan.org
>
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] release name for Infinispan 6.0.0

[Sanne Grinovero]

+1 for "Hop Rod Rye" : fabulously fitting name :)

but this is not where we vote right? so no point in adding +1s

BTW *far* more important to discuss: when 5.3.0.Final is released we
should get together and drink some Tactical Nuclear Penguin. Even
though I don't like beer..

Cheers,
Sanne

On 21 June 2013 18:06, Galder Zamarreño  wrote:
> +1 :)
>
> On Jun 21, 2013, at 8:14 AM, Navin Surtani  wrote:
>
>> I'm gonna suggest "Hopocalypse" again since it was tied last time.
>>
>> 
>> Navin Surtani
>>
>> Software Engineer
>> JBoss SET
>> JBoss EAP
>>
>> Twitter: @navssurtani
>> Blog: navssurtani.blogspot.com
>>
>> From: "Jeff Ramsdale" 
>> To: "infinispan -Dev List" 
>> Sent: Thursday, June 20, 2013 12:52:43 AM
>> Subject: Re: [infinispan-dev] release name for Infinispan 6.0.0
>>
>> I'm gonna suggest Hop Rod Rye, for obvious reasons: 
>> http://beeradvocate.com/beer/profile/610/3158
>>
>> -j
>>
>>
>> On Mon, Jun 17, 2013 at 1:37 PM, Mircea Markus  wrote:
>> Hi,
>>
>> Following the tradition, each Infinispan release is code is a beer.
>> Suggestions?
>>
>> I'll start:
>> - Infinium
>>
>>
>> Cheers,
>> --
>> Mircea Markus
>> Infinispan lead (www.infinispan.org)
>>
>>
>>
>>
>> ___
>> infinispan-dev mailing list
>> infinispan-dev@lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>>
>> ___
>> infinispan-dev mailing list
>> infinispan-dev@lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>> ___
>> infinispan-dev mailing list
>> infinispan-dev@lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> --
> Galder Zamarreño
> gal...@redhat.com
> twitter.com/galderz
>
> Project Lead, Escalante
> http://escalante.io
>
> Engineer, Infinispan
> http://infinispan.org
>
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] Introducing builder subpackages

[Sanne Grinovero]

Please don't break the configuration API again :-(

On 26 June 2013 06:29, Navin Surtani  wrote:
> While working through ISPN-2463, and the sub-tasks I was wondering about the
> organisation of the ConfigurationBuilder classes.
>
> Currently, they are located in org.infinispan.configuration.cache.etc or
> org.infinispan.configuration.global.etc. The actual Configuration classes
> are within the same package already as well. To me, this sounds a little bit
> cluttered and perhaps not very intuitive and I was wondering if it might be
> a better idea to have something like:
>
> org.infinispan.configuration.cache.builders.ConfigurationBuilder (and
> others)
> org.infinispan.configuration.global.builders.GlobalConfigurationBuilder (etc
> etc)
>
> Another suggestion could be:
>
> org.infinispan.configuration.builders.cache.etc
> org.infinispan.configuration.builders.glocal.etc
>
> The only problem with that would be breaking backward compatibility, but
> from ISPN 6.x onwards I think that there are a fair few classes and packages
> being moved around anyway. It's just an idea that might make the API seem a
> little bit cleaner as to where it's located.
>
> Thoughts?
>
> 
> Navin Surtani
>
> Software Engineer
> JBoss SET
> JBoss EAP
>
> Twitter: @navssurtani
> Blog: navssurtani.blogspot.com
>
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] RELAY2 using shorts as site-ids

[Sanne Grinovero]

It doesn't look like an issue, but you could encode the (deprecated)
short configuration option as a string right?

On 1 July 2013 15:33, Bela Ban  wrote:
> This is re [1].
>
> I want to replace shorts as site-ids with strings. This would make EAP
> and JDG configuration simpler, as both configure the mapping from site
> string (e.g. "sfo") to ID (e.g. 1) on the fly. If we have sites sfo, lon
> and nyc in 1 config, we might end up with sfo=0, lon=1 and nyc=2. If we
> only configure lon and nyc in a differemt config, we have lon=0, nyc=1.
> This means that the second site interprets 0 as lon, although it was
> effectively sent by sfo.
>
> However, this will make RELAY2 incompatible with previous releases as
> SiteUUID is now different. The change would be in 3.4 (not 3.3.x), and
> as it will be used by Infinispan 6.0, I don't think this should be an
> issue...
>
> Thoughts ?
>
> [1] https://issues.jboss.org/browse/JGRP-1640
>
> --
> Bela Ban, JGroups lead (http://www.jgroups.org)
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] Cachestores performance

[Sanne Grinovero]

It would be nice to have a deck of "cheat sheets" on the expected use
cases and guarantees: to me it looks like everyone is trying to solve
a different problem / having a different problem in mind.

My own take on it:

Scenario #1
I'll primarily use Infinispan with DIST, and I don't care much for
other options. Reliability is guaranteed via numOwners>1, NOT by
persisting to disk: if a node fails, I kill the VM (the machine, not
the Java process) and start new ones to compensate: I'm assuming cloud
nodes, so it's likely that when a failed node is gone, the disk is
gone as well, with all the carefully stored data.
I will use Infinispan primarily to absorb write spikes - so a "synch
flush" is no good for me - and to boost read performance by as much
memory I can throw at it.
CacheStore is used for two reasons:
 - overflow (LIRS+passivation) for then the memory is not enough
 - clean shutdown: you can think of it as a way to be able to upgrade
some component in the system (Infinispan or my own); I would expect
some kind of "JMX flush" operation to do a clean shutdown without data
loss.

Given such a scenario, I am not interested at all in synchronous
storage. Before we commit into a design which is basically assuming
the need for synchronous storage guarantees, I'd like to understand
what kind of use case it's aiming to solve.

It would be great to document each such use case and put down a table
of things which can be expected, which features should not be expected
(be very explicit on the limitations), and how basic operations are
expected to be performed in the scenario: like how do you do a rolling
upgrade in Scenario 1# ? How do you do a backup? And of course some
configurations & code examples.

Only then we would be able to pick a design (or multiple ones); for my
use case the proposal from Karsten seems excellent, so I'm wondering
why I should be looking for alternatives, and wondering why everyone
is still wasting time on different discussions :-D

I'm pretty sure there is people looking forward for a synch-CacheStore
too: if you could nail down such a scenario however I'm pretty sure
that some other considerations would not be taken into account (like
consistency of data when reactivating a dormant node), so I suspect
that just implementing such a component would actually not make any
new architecture possible, as you would get blocked by other problems
which need to be solved too.. better define all expectations asap!

To me this thread smells of needing the off-heap Direct Memory buffers
which I suggested [long time ago] to efficiently offload internal
buffers, but failing to recognise this we're pushing responsibility to
an epic level complex CacheStore.. guys let's not forget that a mayor
bottleneck of CacheStores today is the SPI it has to implement, we
identified several limitations in the contract in the past which
prevent a superior efficiency: we're working towards a mayor release
now so I'd rather focus on the API changes which will make it possible
to get decent performance even without changing any storage engine..
I'm pretty sure Cassandra (to pick one) doesn't scale too bad.

Cheers,
Sanne



On 2 July 2013 10:09, Radim Vansa  wrote:
> Hi,
>
> I've written down this proposal for the implementation of new cache store.
>
> https://community.jboss.org/wiki/BrnoCacheStoreDesignProposal
>
> WDYT?
>
> Radim
>
> - Original Message -
> | From: "Radim Vansa" 
> | To: "infinispan -Dev List" 
> | Sent: Thursday, June 27, 2013 2:37:43 PM
> | Subject: Re: [infinispan-dev] Cachestores performance
> |
> |
> |
> | - Original Message -
> | | From: "Galder Zamarreño" 
> | | To: "infinispan -Dev List" 
> | | Sent: Thursday, June 27, 2013 1:52:11 PM
> | | Subject: Re: [infinispan-dev] Cachestores performance
> | |
> | | > As for Karsten's FCS implementation, I too have issues with the key set
> | | > and
> | | > value offsets being solely in memory.  However I think that could be
> | | > improved by storing only a certain number of keys/offsets in memory, and
> | | > flushing the rest to disk again into an index file.
> | |
> | | ^ Karsten's implementation makes this relatively easy to achieve because 
> it
> | | already keeps this mapping in a LinkedHashMap (with a given max entries
> | | limit [1]) assuming removeEldestEntry() is overriden to flush to disk 
> older
> | | entries. Some extra logic would be needed to bring back data from the disk
> | | too… but your suggestion below is also quite interesting...
> |
> | I certainly wouldn't call this easy task, because the most problematic part
> | is what we will do when the whole entry (both key and value) are gone from
> | memory and we want to read them - that requires keeping some searchable
> | structure on-disk. And that's the hard stuff.
> |
> | |
> | | > I believe LevelDB follows a similar design, but I think Karsten's FCS
> | | > will
> | | > perform better than LevelDB since it doesn't attempt to maintain a 
> sorted
> | | > struc

Re: [infinispan-dev] L1 consistency for transactional caches.

[Sanne Grinovero]

+1 for considering it a BUG

Didn't we decide a year ago that GET operations should be sent to a
single node only (the primary) ?

On 2 July 2013 15:59, Pedro Ruivo  wrote:
> Hi all,
>
> simple question: What are the consistency guaranties that is supposed to
> be ensured?
>
> I have the following scenario (happened in a test case):
>
> NonOwner: remote get key
> BackupOwner: receives the remote get and replies (with the correct value)
> BackupOwner: put in L1 the value
> PrimaryOwner: [at the same time] is committing a transaction that will
> update the key.
> PrimaryOwer: receives the remote get after sending the commit. The
> invalidation for L1 is not sent to NonOwner.
>
> The test finishes and I perform a check for the key value in all the
> caches. The NonOwner returns the L1 cached value (==test fail).
>
> IMO, this is bug (or not) depending what guaranties we provide.
>
> wdyt?
>
> Pedro
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] L1 consistency for transactional caches.

[Sanne Grinovero]

I see, so we keep the wrong implementation because it's faster?

:D

On 2 July 2013 16:38, Dan Berindei  wrote:
>
>
>
> On Tue, Jul 2, 2013 at 6:36 PM, Pedro Ruivo  wrote:
>>
>>
>>
>> On 07/02/2013 04:21 PM, Sanne Grinovero wrote:
>> > +1 for considering it a BUG
>> >
>> > Didn't we decide a year ago that GET operations should be sent to a
>> > single node only (the primary) ?
>>
>> +1 :)
>>
>
> Manik had a patch for staggering remote GET calls, but it was slowing down
> reads by 25%: http://markmail.org/message/vsx46qbfzzxkkl4w
>
>>
>> >
>> > On 2 July 2013 15:59, Pedro Ruivo  wrote:
>> >> Hi all,
>> >>
>> >> simple question: What are the consistency guaranties that is supposed
>> >> to
>> >> be ensured?
>> >>
>> >> I have the following scenario (happened in a test case):
>> >>
>> >> NonOwner: remote get key
>> >> BackupOwner: receives the remote get and replies (with the correct
>> >> value)
>> >> BackupOwner: put in L1 the value
>> >> PrimaryOwner: [at the same time] is committing a transaction that will
>> >> update the key.
>> >> PrimaryOwer: receives the remote get after sending the commit. The
>> >> invalidation for L1 is not sent to NonOwner.
>> >>
>> >> The test finishes and I perform a check for the key value in all the
>> >> caches. The NonOwner returns the L1 cached value (==test fail).
>> >>
>> >> IMO, this is bug (or not) depending what guaranties we provide.
>> >>
>> >> wdyt?
>> >>
>> >> Pedro
>> >> ___
>> >> infinispan-dev mailing list
>> >> infinispan-dev@lists.jboss.org
>> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> > ___
>> > infinispan-dev mailing list
>> > infinispan-dev@lists.jboss.org
>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >
>> ___
>> infinispan-dev mailing list
>> infinispan-dev@lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] L1 consistency for transactional caches.

[Sanne Grinovero]

On 2 July 2013 17:24, Dan Berindei  wrote:
> It's not wrong, sending the invalidation only from the primary owner is
> wrong :)

Agreed, sending a GET operation to multiple nodes might not be wrong
per-se but is the root cause of such race conditions, and other subtle
complexities we might not even be aware of yet.

I don't know why it was slower, but since the result doesn't make
sense we should look at it a second time rather than throwing the code
away.

Sending invalidations from a non-primary owner is an interesting
approach, but then we're having each owner to maintain an independent
list of nodes who have read the value.
For each write, the primary node would send an invalidation to each
registered node, plus the copy to the secondary nodes, which in turn
sends more L1 invalidation nodes to each of their registered nodes..
what's the likelihood of duplication of invalidation messages here?
Sounds like a big network traffic amplifier, lots of network traffic
triggered for each write.

It also implies that we don't have reliability on the list of
registered nodes, as each owner will be maintaining a different set.
In this case we should also have each node invalidate its L1 stored
entries when the node from which they got these entries has left the
cluster.

Having it all dealt by the primary owner makes for a much simpler
design and also makes it more likely that a single L1 invalidate
message is sent via multicast, or at least with less duplication.

Cheers,
Sanne




>
>
>
> On Tue, Jul 2, 2013 at 7:14 PM, Sanne Grinovero 
> wrote:
>>
>> I see, so we keep the wrong implementation because it's faster?
>>
>> :D
>>
>> On 2 July 2013 16:38, Dan Berindei  wrote:
>> >
>> >
>> >
>> > On Tue, Jul 2, 2013 at 6:36 PM, Pedro Ruivo 
>> > wrote:
>> >>
>> >>
>> >>
>> >> On 07/02/2013 04:21 PM, Sanne Grinovero wrote:
>> >> > +1 for considering it a BUG
>> >> >
>> >> > Didn't we decide a year ago that GET operations should be sent to a
>> >> > single node only (the primary) ?
>> >>
>> >> +1 :)
>> >>
>> >
>> > Manik had a patch for staggering remote GET calls, but it was slowing
>> > down
>> > reads by 25%: http://markmail.org/message/vsx46qbfzzxkkl4w
>> >
>> >>
>> >> >
>> >> > On 2 July 2013 15:59, Pedro Ruivo  wrote:
>> >> >> Hi all,
>> >> >>
>> >> >> simple question: What are the consistency guaranties that is
>> >> >> supposed
>> >> >> to
>> >> >> be ensured?
>> >> >>
>> >> >> I have the following scenario (happened in a test case):
>> >> >>
>> >> >> NonOwner: remote get key
>> >> >> BackupOwner: receives the remote get and replies (with the correct
>> >> >> value)
>> >> >> BackupOwner: put in L1 the value
>> >> >> PrimaryOwner: [at the same time] is committing a transaction that
>> >> >> will
>> >> >> update the key.
>> >> >> PrimaryOwer: receives the remote get after sending the commit. The
>> >> >> invalidation for L1 is not sent to NonOwner.
>> >> >>
>> >> >> The test finishes and I perform a check for the key value in all the
>> >> >> caches. The NonOwner returns the L1 cached value (==test fail).
>> >> >>
>> >> >> IMO, this is bug (or not) depending what guaranties we provide.
>> >> >>
>> >> >> wdyt?
>> >> >>
>> >> >> Pedro
>> >> >> ___
>> >> >> infinispan-dev mailing list
>> >> >> infinispan-dev@lists.jboss.org
>> >> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >> > ___
>> >> > infinispan-dev mailing list
>> >> > infinispan-dev@lists.jboss.org
>> >> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >> >
>> >> ___
>> >> infinispan-dev mailing list
>> >> infinispan-dev@lists.jboss.org
>> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >
>> >
>> >
>> > ___
>> > infinispan-dev mailing list
>> > infinispan-dev@lists.jboss.org
>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> ___
>> infinispan-dev mailing list
>> infinispan-dev@lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] Cachestores performance

[Sanne Grinovero]

On 2 July 2013 14:32, Radim Vansa  wrote:
> | Given such a scenario, I am not interested at all in synchronous
> | storage. Before we commit into a design which is basically assuming
> | the need for synchronous storage guarantees, I'd like to understand
> | what kind of use case it's aiming to solve.
>
> You're right that I may miss the wider perspective and that I am too much 
> focused on storing stuff reliably when we have reliability assured by 
> multiple owners.

Your approach might be perfectly suited for someone, I just would like
to define what kind of user that is, and how you would suggest - in
such a configuration (?) to perform the essential administration
operations. I am literally looking forward to make a set of cheat
sheets, one page per use case, to clearly summarize our different use
cases.

>
> |
> | Only then we would be able to pick a design (or multiple ones); for my
> | use case the proposal from Karsten seems excellent, so I'm wondering
> | why I should be looking for alternatives, and wondering why everyone
> | is still wasting time on different discussions :-D
>
> Karsten implementation has one major drawback: all the keys are stored 
> in-memory, and there's no search structure for overflow. For use case we've 
> had with bigger keys and very small values this is simply not an option. 
> However, we have the LevelDB JNI implementation for this purpose and the 
> numbers are not bad in non-syncing mode. Do we want to have some efficient 
> no-dependency cache-store? If the question is no, okay, let use recommend 
> LevelDB JNI for this purposes.

Great points. In my case, I'm totally fine with that as the ratio
key/value size is about 1/10^7 : obviously "keeping in memory" just
sounds like "optimal".


> | I'm pretty sure there is people looking forward for a synch-CacheStore
> | too: if you could nail down such a scenario however I'm pretty sure
> | that some other considerations would not be taken into account (like
> | consistency of data when reactivating a dormant node), so I suspect
> | that just implementing such a component would actually not make any
> | new architecture possible, as you would get blocked by other problems
> | which need to be solved too.. better define all expectations asap!
> |
> | To me this thread smells of needing the off-heap Direct Memory buffers
> | which I suggested [long time ago] to efficiently offload internal
> | buffers,
>
> Yes, when it comes to reducing GC pauses, that is the best option. However, 
> you can't have enough RAM to handle everything in-memory (off-heap), 
> persistent storage for overflow is a must-have.

Memory mapped buffers don't need to be in RAM, they can be pointers to
disk, and no engine will ever beat their performance.


> | but failing to recognise this we're pushing responsibility to
> | an epic level complex CacheStore.. guys let's not forget that a mayor
> | bottleneck of CacheStores today is the SPI it has to implement, we
> | identified several limitations in the contract in the past which
> | prevent a superior efficiency: we're working towards a mayor release
> | now so I'd rather focus on the API changes which will make it possible
> | to get decent performance even without changing any storage engine..
> | I'm pretty sure Cassandra (to pick one) doesn't scale too bad.
>
> The issues common to all local cache-stores are definitely something to do. I 
> just don't feel enough into the problematics to handle them myself - that's 
> why I jumped onto implementing just another cache store.

Sorry Radim my rant was not directed to your attempt. I'm just
concerned that the general discussion is revolving about resolving too
many things at once: your approach might be very welcome to some
problem, I just have a hard time figuring out which one as it depends
on how you expect people to be using it - likely different than my
typical use case.

Cheers,
Sanne

>
> Radim
>
> |
> | Cheers,
> | Sanne
> |
> |
> |
> | On 2 July 2013 10:09, Radim Vansa  wrote:
> | > Hi,
> | >
> | > I've written down this proposal for the implementation of new cache store.
> | >
> | > https://community.jboss.org/wiki/BrnoCacheStoreDesignProposal
> | >
> | > WDYT?
> | >
> | > Radim
> | >
> | > - Original Message -
> | > | From: "Radim Vansa" 
> | > | To: "infinispan -Dev List" 
> | > | Sent: Thursday, June 27, 2013 2:37:43 PM
> | > | Subject: Re: [infinispan-dev] Cachestores performance
> | > |
> | > |
> | > |
> | > | - Original Message -
> | > | | From: "Galder Zamarreño" 
> | > | | To: "infinispan -Dev List" 
> | > | | Sent: Thursday, June 27, 2013 1:52:11 PM
> | > | | Subject: Re: [infinispan-dev] Cachestores performance
> | > | |
> | > | | > As for Karsten's FCS implementation, I too have issues with the key
> | > | | > set
> | > | | > and
> | > | | > value offsets being solely in memory.  However I think that could be
> | > | | > improved by storing only a certain number of keys/offsets in memory,
> | > | | > and
> | > |

Re: [infinispan-dev] Cachestores performance

[Sanne Grinovero]

Thanks Erik ! exactly the kind of use-case description I'd like to collect.

If I could summarise the main differences with my use case, it's that
you are implying a non-shared (local) CacheStore right?

I wouldn't mind having that on my own wishlist too, but as you also
point out it's unclear how to restore the grid state from the various
pieces of stored data. In my case I'm assuming a single shared
CacheStore as that's the only scenario in which consistency in the
CacheStore isn't puzzling me too much.

On to your recovery approach: I don't think collecting a copy of all
SST files will save you: many nodes might have older copies of the
same entry which where not cleaned up after they lost the role of
"owners" on that particular entry.

You could even enter the messy situation of nodes which
 - are owners for K1
 - store some value K1->V1
 - is no longer an owner
 - K1 is deleted by other activity
 - you become an owner again (no state transfer happens as the entry is gone)
 - you receive a GET and serve it from your CacheStore

.. and I could make up more horror stories like this ;-)
Bottom-line: I would not advice DIST + local cachestores as long as
your app has some peculiarity (like never reusing a key) which avoids
such situations. At least, this is why I can't use it.
This is just an example of the kinds of limitations I would like to
collect on a per-configuration basis.

Sanne


On 2 July 2013 18:39, Erik Salter  wrote:
> I concur with part of the below, but with a few changes:
>
> - The cache is the primary storage, similar to Sanne's case. (DIST mode)
> - My customers are not interested in extra components to the system, like 
> databases or Cassandra nodes.  They wonder why they can't simply use the 
> existing file system on the nodes they have.
> - I'm only going to be using the filesystem to recover in the case of 
> upgrades and catastrophic failures.  So during normal operation, flushes to 
> disk cannot impact cluster performance.
> - Most importantly, there needs to be a way, scripted or otherwise, to 
> recover the keys from local storage in a DIST-mode cache.  I cannot guarantee 
> anything regarding node ordering, so anything about persisting segment 
> info/previous CHs are out.  If that means copying all LevelDB SST files to 
> all nodes and restarting them, that's fine.
>
> At the executive levels of my customer, they don't see (or really care about) 
> the differentiation between data grids and MySQL -- only that one has 
> file-based persistence and the other doesn't.
>
> In production, we've already taken a massive outage where a unbelievable 
> series of coincidences occurred to reveal a JBoss AS bug that ended up 
> deadlocking all threads in the cluster and we had to restart all nodes.   And 
> I'm sure it'll happen again.
>
> Hope this offers some user perspective.
>
> Erik
>
> -Original Message-
> From: infinispan-dev-boun...@lists.jboss.org 
> [mailto:infinispan-dev-boun...@lists.jboss.org] On Behalf Of Sanne Grinovero
> Sent: Tuesday, July 02, 2013 8:47 AM
> To: infinispan -Dev List
> Subject: Re: [infinispan-dev] Cachestores performance
>
> It would be nice to have a deck of "cheat sheets" on the expected use cases 
> and guarantees: to me it looks like everyone is trying to solve a different 
> problem / having a different problem in mind.
>
> My own take on it:
>
> Scenario #1
> I'll primarily use Infinispan with DIST, and I don't care much for other 
> options. Reliability is guaranteed via numOwners>1, NOT by persisting to 
> disk: if a node fails, I kill the VM (the machine, not the Java process) and 
> start new ones to compensate: I'm assuming cloud nodes, so it's likely that 
> when a failed node is gone, the disk is gone as well, with all the carefully 
> stored data.
> I will use Infinispan primarily to absorb write spikes - so a "synch flush" 
> is no good for me - and to boost read performance by as much memory I can 
> throw at it.
> CacheStore is used for two reasons:
>  - overflow (LIRS+passivation) for then the memory is not enough
>  - clean shutdown: you can think of it as a way to be able to upgrade some 
> component in the system (Infinispan or my own); I would expect some kind of 
> "JMX flush" operation to do a clean shutdown without data loss.
>
> Given such a scenario, I am not interested at all in synchronous storage. 
> Before we commit into a design which is basically assuming the need for 
> synchronous storage guarantees, I'd like to understand what kind of use case 
> it's aiming to solve.
>
> It would be great to document each such use case and put down a table of 
> things w

Re: [infinispan-dev] Cachestores performance

[Sanne Grinovero]

Hi Galder,
you make many interesting points but I am not interested in discussing
my specific design ideas in detail, I just sketched that quickly as an
example of requirements description.
My intervention in this thread is all about demanding what the use
case is for such a synchronous cachestore.

It is my understanding that lots of people are currently discussing
how to best make a synchronous cachestore mostly efficient, but I yet
have to see what the requirements are. My use case - although
expecting an asynchronous one - is just an example of how I would like
to see the general architecture described first before we waste time
on a useless component.

I am not stating that nobody wants a strictly synchronous CacheStore,
but I would like to challenge the arguments who made someone (who?)
ask for this, as I believe there are many other things that would need
to be addressed.

Therefore, my suggestion for describing the main use cases we expect
to support is not off-topic at all, it's the first thing any engineer
would have requested, and I would not spend a minute more of our
engineers in coding without a clear description of the expected
architecture, expected reliability, expected operations to be used.

Sanne

On 3 July 2013 14:59, Galder Zamarreño  wrote:
> Sanne, let me comment on some of the points you raised that didn't comment on 
> in an earlier comment...
>
> On Jul 2, 2013, at 2:47 PM, Sanne Grinovero  wrote:
>
>> It would be nice to have a deck of "cheat sheets" on the expected use
>> cases and guarantees: to me it looks like everyone is trying to solve
>> a different problem / having a different problem in mind.
>>
>> My own take on it:
>>
>> Scenario #1
>> I'll primarily use Infinispan with DIST, and I don't care much for
>> other options. Reliability is guaranteed via numOwners>1, NOT by
>> persisting to disk: if a node fails, I kill the VM (the machine, not
>> the Java process) and start new ones to compensate: I'm assuming cloud
>> nodes, so it's likely that when a failed node is gone, the disk is
>> gone as well, with all the carefully stored data.
>> I will use Infinispan primarily to absorb write spikes - so a "synch
>> flush" is no good for me - and to boost read performance by as much
>> memory I can throw at it.
>> CacheStore is used for two reasons:
>> - overflow (LIRS+passivation) for then the memory is not enough
>> - clean shutdown: you can think of it as a way to be able to upgrade
>> some component in the system (Infinispan or my own); I would expect
>> some kind of "JMX flush" operation to do a clean shutdown without data
>> loss.
>
> ^ Should this really be implemented at the Infinispan level? In the 
> AS/EAP/Wildfly case, they take care that all transactions have finished 
> before shutting down, and Infinispan benefits from that.
>
>> Given such a scenario, I am not interested at all in synchronous
>> storage. Before we commit into a design which is basically assuming
>> the need for synchronous storage guarantees, I'd like to understand
>> what kind of use case it's aiming to solve.
>
> Sanne, any **strict** synchronous storage guarantees (e.g. to force or to 
> not) will be configurable and most likely they'll be disabled, just like 
> Level DB JNI, or Karsten's file cache store by default. A case where someone 
> might want to enable this is when it just has a local cache and wants to 
> persist data for recovery. Of course, the whole node and the disk could die…, 
> but this is not so far fetched IMO.
>
> The whole discussion about **strict** synchronous storage guarantees in this 
> thread is to make sure we're comparing apples with apples. IOW, it doesn't 
> make sense to compare performance when each has different **strict** 
> synchronous storage guarantee settings.
>
>> It would be great to document each such use case and put down a table
>> of things which can be expected, which features should not be expected
>> (be very explicit on the limitations), and how basic operations are
>> expected to be performed in the scenario: like how do you do a rolling
>> upgrade in Scenario 1# ? How do you do a backup? And of course some
>> configurations & code examples.
>
> ^ Hmmm, these operations are not really specific to the file cache store 
> per-se. They are valid points, for sure, but out of the scope of this IMO.
>
>> Only then we would be able to pick a design (or multiple ones); for my
>> use case the proposal from Karsten seems excellent, so I'm wondering
>> why I should be looking for alternatives, and wondering why everyone
>> is still wasting time on differen

Re: [infinispan-dev] 6.0.0.Alpha1 scheduled for 12th July (Friday)

[Sanne Grinovero]

I'm puzzled by the pace.
We just forked from 5.3, we get no time to do all the refactoring and
API changes we had in mind?

-1 to do those after the Alpha1 tag

Also we had discussed to base 6.0 on newer Hibernate Search 5.0, but
we barely started that work. With this pace it means we'll switch to
Search 5.0 (and new API) in a late phase of Infinispan 6.0 releases.


On 4 July 2013 09:01, Mircea Markus  wrote:
> And here is the list of issues scheduled for this release: http://goo.gl/45Osp
>
> Cheers,
> --
> Mircea Markus
> Infinispan lead (www.infinispan.org)
>
>
>
>
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] RemoteCache vs BasicCache

[Sanne Grinovero]

+1 for #2 or #3

Having multiple interfaces sounds interesting, as "traits" but I'm not
too convinced.. would need to play with it.
But I definitely think we need at least one common API so that people
can easily replace embedded/remote usage.

If we had a sane roadmap I would say go for #3, and we eventually back
off if it doesn't look too good in examples.

Sanne

On 4 July 2013 13:10, Tristan Tarrant  wrote:
> Dear all,
>
> during my latest destructive commits, I have liberated
> infinispan-client-hotrod from infinispan-core.
> One of the things I did was remove the inheritance of RemoteCache from
> BasicCache, since my argument was that our base API contract was the
> ConcurrentMap interface and that would suffice, since it implied that a
> remote Cache could match a core Cache in functionality (which is
> somewhat true). Now, I'm not convinced it was a good choice after all,
> since there are indeed a ton of common methods (the async API for one).
>
>
> I would like people's opinion on the above, and propose one of the
> following:
>
> 1. we leave it as it is
> 2. we move org.infinispan.api to infinispan-commons and make RemoteCache
> inherit from BasicCache
> 3. we go even further and split the concept of BasicCache into multiple
> interfaces: AsyncCache, TransactionalCache, QueryableCache, etc and add
> them to the RemoteCache as we will in the blanks, since we are aiming at
> feature parity. This could also mix well with the ideal of having the
> JCache API as our public API.
>
> Tristan
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] Cachestores performance

[Sanne Grinovero]

On 4 July 2013 13:56, Manik Surtani  wrote:
> Interesting and convoluted thread.  I'll try and summarise my thoughts on one 
> email:
>
> * I agree with Sanne that a cheat sheet will be extremely useful.  Sanne, 
> feel like kicking this off with your use case as described here, on the wiki? 
>  I like your ideas but just ranting about them won't help anybody.  ;) Maybe 
> then others can follow your template for more use cases, such as the one Erik 
> has detailed.

+1 and sorry for the rant. But still even after all my whining nobody
explained why we want to prioritise synchronous cachestore :-)
My point is not being against doing it, but to identify eagerly what
else is needed to accomodate such a use case: having the CacheStore
implementation is not enough to make such a configuration usable.

> * An SPI redesign has been on the cards for a while.  I believe Mircea is 
> working on this, a part of this redesign will involve directly involving 
> cache stores in JTA transactions where possible (e.g., JDBC cache stores).  I 
> recommend kicking off a separate thread to discuss that SPI.  Galder, I think 
> you can consider he JSR 107 CacheLoader/Writer interfaces as stable by now.
>
> * I agree with Galder as well, that we do need a filesystem-based cache store 
> that performs decently well OOTB when used both synchronously and 
> asynchronously, regardless of the outcome of any use case analysis.  Sanne, I 
> don't think Cassandra is the right answer here - too much setup/maintenance.  
> Maybe Karsten's FCS impl for smaller caches and then LevelDB/JNI is a valid 
> combination.

I wasn't serious on Cassandra being the ultimate solution, as I said I
just sketched an example use case.

>
> * Sanne, re: mmapped files, yes, very nice, but I'm holding back on that for 
> a number of reasons (see an earlier thread on the subject), one of them being 
> some rework in JGroups to be able to pass handles to direct buffers from the 
> network stack to Infinispan.

I thought we had agreed to hold back until the mayor version change..
!? That's the better solution, would be sad to waste time on temporary
compromises.

Sanne

>
> - M
>
> On 3 Jul 2013, at 15:26, Sanne Grinovero  wrote:
>
>> Hi Galder,
>> you make many interesting points but I am not interested in discussing
>> my specific design ideas in detail, I just sketched that quickly as an
>> example of requirements description.
>> My intervention in this thread is all about demanding what the use
>> case is for such a synchronous cachestore.
>>
>> It is my understanding that lots of people are currently discussing
>> how to best make a synchronous cachestore mostly efficient, but I yet
>> have to see what the requirements are. My use case - although
>> expecting an asynchronous one - is just an example of how I would like
>> to see the general architecture described first before we waste time
>> on a useless component.
>>
>> I am not stating that nobody wants a strictly synchronous CacheStore,
>> but I would like to challenge the arguments who made someone (who?)
>> ask for this, as I believe there are many other things that would need
>> to be addressed.
>>
>> Therefore, my suggestion for describing the main use cases we expect
>> to support is not off-topic at all, it's the first thing any engineer
>> would have requested, and I would not spend a minute more of our
>> engineers in coding without a clear description of the expected
>> architecture, expected reliability, expected operations to be used.
>>
>> Sanne
>>
>> On 3 July 2013 14:59, Galder Zamarreño  wrote:
>>> Sanne, let me comment on some of the points you raised that didn't comment 
>>> on in an earlier comment...
>>>
>>> On Jul 2, 2013, at 2:47 PM, Sanne Grinovero  wrote:
>>>
>>>> It would be nice to have a deck of "cheat sheets" on the expected use
>>>> cases and guarantees: to me it looks like everyone is trying to solve
>>>> a different problem / having a different problem in mind.
>>>>
>>>> My own take on it:
>>>>
>>>> Scenario #1
>>>> I'll primarily use Infinispan with DIST, and I don't care much for
>>>> other options. Reliability is guaranteed via numOwners>1, NOT by
>>>> persisting to disk: if a node fails, I kill the VM (the machine, not
>>>> the Java process) and start new ones to compensate: I'm assuming cloud
>>>> nodes, so it's likely that when a failed node is gone, the disk is
>>>> gone as well, with all the carefully stored data.
>>>>

Re: [infinispan-dev] 6.0.0.Alpha1 scheduled for 12th July (Friday)

[Sanne Grinovero]

On 4 July 2013 23:25, Mircea Markus  wrote:
> On 4 Jul 2013, at 14:23, Sanne Grinovero  wrote:
>
>> I'm puzzled by the pace.
>> We just forked from 5.3,
>
> Just forked ? :-)
> By the time we release Alpha1 (12 July), the 6.0 branch will be running for 
> almost one month on its own branch (cut in 17 Jul). Also the work of 6.0 
> features has been started way before the actual tagging took place (e.g. 
> remote querying).

Sorry I'm just not used to this approach. It might be my own habits, I
would generally consider it way too short but have no specific
problems with it. All cool things we planned where moved out anyway..
I just hope we'll not have to wait too long for some of those.

Sanne

>
>> we get no time to do all the refactoring and
>> API changes we had in mind?
>
> Why do we need to release all the refactoring in the first alpha? There are 
> three 
> Alphas(https://issues.jboss.org/secure/Dashboard.jspa?selectPageId=12315590) 
> scheduled for 6.0 in which we'll have time to incrementally add more API 
> changes.
>
>>
>> -1 to do those after the Alpha1 tag
>
> why?
> Alpha don't require API freezes. OTOH we have some features that can be 
> released to the community already:
> https://issues.jboss.org/browse/ISPN-3109
> https://issues.jboss.org/browse/ISPN-2861
> https://issues.jboss.org/browse/ISPN-761
>
> Bear in mind that we are targeting 3-4 months release cycles for a Final, so 
> we should keep releasing every 2 weeks or so (preferably shorter release 
> cycles).
>
>>
>> Also we had discussed to base 6.0 on newer Hibernate Search 5.0, but
>> we barely started that work. With this pace it means we'll switch to
>> Search 5.0 (and new API) in a late phase of Infinispan 6.0 releases.
>
> Again I don't see how these are related. There are more alphas on the way in 
> which we can tackle that.
>
> Cheers,
> --
> Mircea Markus
> Infinispan lead (www.infinispan.org)
>
>
>
>
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] L1 consistency for transactional caches.

[Sanne Grinovero]

On 3 July 2013 10:26, Dan Berindei  wrote:
>
>
>
> On Tue, Jul 2, 2013 at 8:41 PM, Sanne Grinovero 
> wrote:
>>
>> On 2 July 2013 17:24, Dan Berindei  wrote:
>> > It's not wrong, sending the invalidation only from the primary owner is
>> > wrong :)
>>
>> Agreed, sending a GET operation to multiple nodes might not be wrong
>> per-se but is the root cause of such race conditions, and other subtle
>> complexities we might not even be aware of yet.
>>
>> I don't know why it was slower, but since the result doesn't make
>> sense we should look at it a second time rather than throwing the code
>> away.
>>
>
> It does make sense: statistically, the backup owner will sometimes reply
> faster than the primary owner.
>
> http://markmail.org/message/qmpn7yueym4tbnve
>
> http://www.bailis.org/blog/doing-redundant-work-to-speed-up-distributed-queries/

Of course, I remember the discussion but you can put may questions
marks on this decision. First off, this is doubling the load on
network, which is supposedly our most precious resource, so I highly
question how we're measuring the "benefit" of the second request.
If you read the articles you linked to you'll see google applies such
a strategy to improve the tail latency, but only sends the second
request when the first one is not getting a fast answer, and in their
tests this seems to pay off to a mere 5% of increased network usage..
I would say that's a significantly different level of trade off.
Also, opposed to Google's BigTable and Apache Cassandra which use
these techniques, Infinispan is not supporting an eventually
consistent model which makes it far more dangerous to read the
slightly out of date value from the non-owner... sure we can resolve
those things, but it gets hairy.

In this case specifically, reading from the primary owner only seems a
much "cleaner" solution, as IMHO it's a move towards simplification
rather than adding yet another special case in the codebase.

>
>>
>> Sending invalidations from a non-primary owner is an interesting
>> approach, but then we're having each owner to maintain an independent
>> list of nodes who have read the value.
>> For each write, the primary node would send an invalidation to each
>> registered node, plus the copy to the secondary nodes, which in turn
>> sends more L1 invalidation nodes to each of their registered nodes..
>> what's the likelihood of duplication of invalidation messages here?
>> Sounds like a big network traffic amplifier, lots of network traffic
>> triggered for each write.
>>
>
> The likelihood of duplication is very near to 100%, indeed, and in non-tx
> caches it would add another RPC to the critical path.
>
> As always, it's a compromise: if we do something to speed up writes, it will
> slow down reads. Perhaps we could send the request to the primary owners
> only when L1 is enabled, as the number of remote gets should be smaller, and
> send the request to all the owners when L1 is disabled, and the number of
> remote gets is higher.
>
> Pedro's suggestion to send the request to all the owners, but only write the
> value to L1 if the first reply was from the primary owner, sounds like it
> should work just as well. It would make L1 slightly less efficient, but it
> wouldn't have latency spikes caused by a delay on the primary owner.

That's far from an ideal solution; we don't have a clue on how to
measure what "slightly less efficient" means: that might reveal to be
"unbearably worse" for some usage pattern.

While we have no clue how worse it can be, it will definitely always
provide a worse cache hit/miss ratio, so it's easily proven that it's
going to be sub optimal in all cases.

If you really go for something like that, at least take the value from
the primary owners when it arrives (second, third, .. whatever but at
some point you might get it) and then store it in L1: will cost you a
second unmarshalling operation but that's far better than causing
(several?) cache misses.

>
>>
>> It also implies that we don't have reliability on the list of
>> registered nodes, as each owner will be maintaining a different set.
>> In this case we should also have each node invalidate its L1 stored
>> entries when the node from which they got these entries has left the
>> cluster.
>>
>
> Right now we invalidate from L1 all the keys for which the list of owners
> changed, whether they're still alive or not, because we don't keep track of
> the node we got each entry from.
>
> If we only sent remote get commands to the primary owner, we'd have to
> invalidate

Re: [infinispan-dev] Integrating Karsten's FCS

[Sanne Grinovero]

+1 to integrate it.

Also, +1 on Radim's comment to remove the existing one: just yesterday
I had a problem with Infinispan blowing up with weird Externalizer
exceptions because I was attempting to activate data from a
FilesystemCacheStore which was stored with version 5.2.x, and I was
loading it from 5.3.0 : that doesn't work as apparently some encoding
changed, so unless we want to fix that there is no point in keeping
the existing CacheStore around.

Sanne

On 9 July 2013 10:07, Radim Vansa  wrote:
>
>
> - Original Message -
> | From: "Galder Zamarreño" 
> | To: "infinispan -Dev List" 
> | Sent: Tuesday, July 9, 2013 10:32:52 AM
> | Subject: [infinispan-dev] Integrating Karsten's FCS
> |
> | Hi,
> |
> | I think we all agree that Karsten's file cache store is a good base
> | replacement for the current file cache store, particularly for caches with
> | relatively small keys, or not a huge amount of them.
> |
> | I'm working with Radim to try to figure out what would be the tipping point
> | at which LevelDB based cache store behaves better compared to Karsten's
> | cache store.
> |
> | In the mean time, I think we should integrate Karsten's FCS into master and
> | have it ready for people to try from Alpha1. There's one or two things that
> | do require a bit of review, but I can pin point them in the pull req.
> |
> | The question is, how should we name it? And what should we do about the
> | current FCS implementation?
> |
> | Shall we keep the current FCS implementation, deprecate it, and get rid of 
> it
> | in the next minor/major version? Some users might have data stored in the
> | current FCS and would be quite abrupt to just get rid of it right now.
>
> I'd remove it in Final, it cannot be seriously used either. Users can migrate 
> the data using rolling upgrades.
>
> |
> | If so, what do we name karsten's file cache store? AppendOnlyFileCacheStore?
>
> You must have mixed something, Karsten is far from append-only, it writes to 
> first big enough empty space in the data file.
> I'd emphasise that the keys are still in memory, therefore something like 
> InMemoryKeysFCS or MemKeysFCS. Or, we will not be as descriptive and call it 
> HotFCS or NextFCS...
>
> |
> | Cheers,
> | --
> | Galder Zamarreño
> | gal...@redhat.com
> | twitter.com/galderz
> |
> | Project Lead, Escalante
> | http://escalante.io
> |
> | Engineer, Infinispan
> | http://infinispan.org
> |
> |
> | ___
> | infinispan-dev mailing list
> | infinispan-dev@lists.jboss.org
> | https://lists.jboss.org/mailman/listinfo/infinispan-dev
> |
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] RemoteCache vs BasicCache

[Sanne Grinovero]

On 8 July 2013 20:37, Mircea Markus  wrote:
> TBH I think having the RemoteCache implementing Cache wasn't such a good idea 
> as it caused confusion through users: e.g. people trying to use transactions 
> with a remote cache. I like the multiple interfaces idea as it clearly 
> indicates what's actually supported in both remote and embedded modes.

Right that has been confusing, but now we also have evidence of people
specifically demanding to have the same query functionality (same =
exact same API) in remote as in embedded.

Yes we have different functionality, but also there is a good value in
being able to easily switch a client from embedded vs remote .. what
about we maintain two different interfaces but having a shared parent?

We would then try to define as much as possible in the common Cache
interface, and regularly promote features from the lower interfaces to
the upper ones when the functionality is covered by both.

In terms of practical development it's self-evident that we'll always
be a step ahead in embedded mode, so we should design for it rather
than trying to prevent it.

Sanne

>
> On 4 Jul 2013, at 14:10, Tristan Tarrant  wrote:
>
>> during my latest destructive commits, I have liberated
>> infinispan-client-hotrod from infinispan-core.
>> One of the things I did was remove the inheritance of RemoteCache from
>> BasicCache, since my argument was that our base API contract was the
>> ConcurrentMap interface and that would suffice, since it implied that a
>> remote Cache could match a core Cache in functionality (which is
>> somewhat true). Now, I'm not convinced it was a good choice after all,
>> since there are indeed a ton of common methods (the async API for one).
>>
>>
>> I would like people's opinion on the above, and propose one of the
>> following:
>>
>> 1. we leave it as it is
>> 2. we move org.infinispan.api to infinispan-commons and make RemoteCache
>> inherit from BasicCache
>> 3. we go even further and split the concept of BasicCache into multiple
>> interfaces: AsyncCache, TransactionalCache, QueryableCache, etc and add
>> them to the RemoteCache as we will in the blanks, since we are aiming at
>> feature parity. This could also mix well with the ideal of having the
>> JCache API as our public API.
>
> Cheers,
> --
> Mircea Markus
> Infinispan lead (www.infinispan.org)
>
>
>
>
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] RemoteCache vs BasicCache

[Sanne Grinovero]

Indeed that's an excellent idea. But I would add also a common parent
interface having the basic functionality common to all.

On 9 July 2013 11:39, Mircea Markus  wrote:
>
> On 9 Jul 2013, at 13:15, Sanne Grinovero  wrote:
>
>> On 8 July 2013 20:37, Mircea Markus  wrote:
>>> TBH I think having the RemoteCache implementing Cache wasn't such a good 
>>> idea as it caused confusion through users: e.g. people trying to use 
>>> transactions with a remote cache. I like the multiple interfaces idea as it 
>>> clearly indicates what's actually supported in both remote and embedded 
>>> modes.
>>
>> Right that has been confusing, but now we also have evidence of people
>> specifically demanding to have the same query functionality (same =
>> exact same API) in remote as in embedded.
>>
>> Yes we have different functionality, but also there is a good value in
>> being able to easily switch a client from embedded vs remote .. what
>> about we maintain two different interfaces but having a shared parent?
>>
>> We would then try to define as much as possible in the common Cache
>> interface, and regularly promote features from the lower interfaces to
>> the upper ones when the functionality is covered by both.
>>
>> In terms of practical development it's self-evident that we'll always
>> be a step ahead in embedded mode, so we should design for it rather
>> than trying to prevent it.
>
> That would work but I think it would be nicer to have separate interfaces to 
> define the functionality, as suggested by Tristan:
>
> 3. we go even further and split the concept of BasicCache into multiple
> interfaces: AsyncCache, TransactionalCache, QueryableCache, etc and add
> them to the RemoteCache as we will in the blanks, since we are aiming at
> feature parity. This could also mix well with the ideal of having the
> JCache API as our public API.
>
> Cheers,
> --
> Mircea Markus
> Infinispan lead (www.infinispan.org)
>
>
>
>
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] Integrating Karsten's FCS

[Sanne Grinovero]

Hi Randall,
I would agree with you that this should be a priority, but keep in
mind that just migrating data from a CacheStore to another won't be
enough: as I pointed out in my previous mail, binary encoding also
changed, making it impossible to deserialize the values.

I'm not sure if the encoding change was meant to happen, but
apparently there is currently no effort in place to test for this kind
of backward compatibility.

If you need such a thing for ModeShape it would likely be easier for
you to provide such a tool in ModeShape to extract all data and dump
it to some external file, than to provide the hooks in Infinispan as
the transcoding tool would need to depend on multiple Infinispan
versions.

Sanne


On 9 July 2013 14:37, Randall Hauch  wrote:
>
> On Jul 9, 2013, at 4:07 AM, Radim Vansa  wrote:
>
>> - Original Message -
>> | From: "Galder Zamarreño" 
>> | Shall we keep the current FCS implementation, deprecate it, and get rid of 
>> it
>> | in the next minor/major version? Some users might have data stored in the
>> | current FCS and would be quite abrupt to just get rid of it right now.
>>
>> I'd remove it in Final, it cannot be seriously used either. Users can 
>> migrate the data using rolling upgrades.
>
> Can you describe this process, especially for how it can be accomplished with 
> a single (local) cache?
>
> A migration mechanism is absolutely a must. There are ModeShape users that 
> have used the FCS simply because it suits their needs - primarily because 
> there are no extra dependencies and no additional "system" underneath. There 
> HAS to be a migration path, especially if the old FCS is going to be removed.
>
> Why does Infinispan not have a general-purpose mechanism for converting from 
> one (offline) cache store to another (offline) cache store?
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] RemoteCache vs BasicCache

[Sanne Grinovero]

On 9 July 2013 18:01, Mircea Markus  wrote:
>
> On 9 Jul 2013, at 13:47, Sanne Grinovero  wrote:
>
>> Indeed that's an excellent idea. But I would add also a common parent
>> interface having the basic functionality common to all.
>
> If ConcurrentMap doesn't have what we need, of course :-)

ConcurrentMap adds a load of pitfalls which should not be there: all
the methods whose result is depending on the node they are run on.

I know we make it clear in javadocs and several warnings on the doc,
but experience tells that this is not good enough :-(

I'd propose to remove them all, and provide an adapter layer
(optional) for those needing to replace a ConcurrentMap in their code.
Methods like "size()" - as discussed before - are possibly useful for
statistics and diagnostics, and therefore should be accessed in a
different explicit way.

cache.diagnosticsStatistics().localContainerSize()

-- Sanne




>
>>
>> On 9 July 2013 11:39, Mircea Markus  wrote:
>>>
>>> On 9 Jul 2013, at 13:15, Sanne Grinovero  wrote:
>>>
>>>> On 8 July 2013 20:37, Mircea Markus  wrote:
>>>>> TBH I think having the RemoteCache implementing Cache wasn't such a good 
>>>>> idea as it caused confusion through users: e.g. people trying to use 
>>>>> transactions with a remote cache. I like the multiple interfaces idea as 
>>>>> it clearly indicates what's actually supported in both remote and 
>>>>> embedded modes.
>>>>
>>>> Right that has been confusing, but now we also have evidence of people
>>>> specifically demanding to have the same query functionality (same =
>>>> exact same API) in remote as in embedded.
>>>>
>>>> Yes we have different functionality, but also there is a good value in
>>>> being able to easily switch a client from embedded vs remote .. what
>>>> about we maintain two different interfaces but having a shared parent?
>>>>
>>>> We would then try to define as much as possible in the common Cache
>>>> interface, and regularly promote features from the lower interfaces to
>>>> the upper ones when the functionality is covered by both.
>>>>
>>>> In terms of practical development it's self-evident that we'll always
>>>> be a step ahead in embedded mode, so we should design for it rather
>>>> than trying to prevent it.
>>>
>>> That would work but I think it would be nicer to have separate interfaces 
>>> to define the functionality, as suggested by Tristan:
>>>
>>> 3. we go even further and split the concept of BasicCache into multiple
>>> interfaces: AsyncCache, TransactionalCache, QueryableCache, etc and add
>>> them to the RemoteCache as we will in the blanks, since we are aiming at
>>> feature parity. This could also mix well with the ideal of having the
>>> JCache API as our public API.
>>>
>>> Cheers,
>>> --
>>> Mircea Markus
>>> Infinispan lead (www.infinispan.org)
>>>
>>>
>>>
>>>
>>>
>>> ___
>>> infinispan-dev mailing list
>>> infinispan-dev@lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>> ___
>> infinispan-dev mailing list
>> infinispan-dev@lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> Cheers,
> --
> Mircea Markus
> Infinispan lead (www.infinispan.org)
>
>
>
>
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] RemoteCache vs BasicCache

[Sanne Grinovero]

On 9 July 2013 18:13, Mircea Markus  wrote:
>
> On 9 Jul 2013, at 20:07, Sanne Grinovero  wrote:
>
>> On 9 July 2013 18:01, Mircea Markus  wrote:
>>>
>>> On 9 Jul 2013, at 13:47, Sanne Grinovero  wrote:
>>>
>>>> Indeed that's an excellent idea. But I would add also a common parent
>>>> interface having the basic functionality common to all.
>>>
>>> If ConcurrentMap doesn't have what we need, of course :-)
>>
>> ConcurrentMap adds a load of pitfalls which should not be there: all
>> the methods whose result is depending on the node they are run on.
>>
>> I know we make it clear in javadocs and several warnings on the doc,
>> but experience tells that this is not good enough :-(
>>
>> I'd propose to remove them all, and provide an adapter layer
>> (optional) for those needing to replace a ConcurrentMap in their code.
>> Methods like "size()" - as discussed before - are possibly useful for
>> statistics and diagnostics, and therefore should be accessed in a
>> different explicit way.
>>
>> cache.diagnosticsStatistics().localContainerSize()
>
> +1. That would be part of the migration towards to JCache API and implemented 
> in 7.0. I think Tristan was looking for something now.

I see. But since we plan a change anyway, why not doing the right
thing right away? Ok we can change interfaces on mayor versions, still
the less you change them the better: any change is a big pain for
other projects.

Unless Tristan was looking for something backwards compatible right
now? Of course I'd prefer that, I was making suggestions assuming the
interface chain would change now in some non-compabile way.

Cheers,
Sanne

>
> Cheers,
> --
> Mircea Markus
> Infinispan lead (www.infinispan.org)
>
>
>
>
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] RemoteCache vs BasicCache

[Sanne Grinovero]

On 10 July 2013 11:32, Mircea Markus  wrote:
>
> On 9 Jul 2013, at 20:22, Sanne Grinovero  wrote:
>
>>>> ConcurrentMap adds a load of pitfalls which should not be there: all
>>>> the methods whose result is depending on the node they are run on.
>>>>
>>>> I know we make it clear in javadocs and several warnings on the doc,
>>>> but experience tells that this is not good enough :-(
>>>>
>>>> I'd propose to remove them all, and provide an adapter layer
>>>> (optional) for those needing to replace a ConcurrentMap in their code.
>>>> Methods like "size()" - as discussed before - are possibly useful for
>>>> statistics and diagnostics, and therefore should be accessed in a
>>>> different explicit way.
>>>>
>>>> cache.diagnosticsStatistics().localContainerSize()
>>>
>>> +1. That would be part of the migration towards to JCache API and 
>>> implemented in 7.0. I think Tristan was looking for something now.
>>
>> I see. But since we plan a change anyway, why not doing the right
>> thing right away?
>
> That would making Cache not implement the j.u.ConcurrentMap interface but a 
> new interface, potentially the one exposed by JSR 107. Whilst the timing is 
> good (new major) I think that is a rather large change in the context of the 
> already very busy 6.0 and not as important as other features or stability. 
> There's also JDG backward compatibility. I guess Tristan can comment more on 
> the effort needed for implementing such a change.

Fair enough. I just assumed you where discussing about breaking the
backwards compatibility, if not that's very welcome.

>
>> Ok we can change interfaces on mayor versions, still
>> the less you change them the better: any change is a big pain for
>> other projects.
>
> This is not that much a change in method signature but only moving methods 
> around to separate interfaces.

That sounds like it still breaks backwards compatibility.. possibly I
have not understood how you envision that.

Cheers,
Sanne

>
>>
>> Unless Tristan was looking for something backwards compatible right
>> now? Of course I'd prefer that, I was making suggestions assuming the
>> interface chain would change now in some non-compabile way.
>
> Cheers,
> --
> Mircea Markus
> Infinispan lead (www.infinispan.org)
>
>
>
>
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] Integrating Karsten's FCS

[Sanne Grinovero]

This is the requested stacktrace; it's easy to reproduce in case you need it:
 - checkout Hibernate Search at tag 4.3.0.Final and run
org.hibernate.search.infinispan.StoredIndexTest
 - switch to commit id dfc8bff and run the same test again

Cheers,
Sanne

org.hibernate.search.SearchException: HSEARCH000103: Unable to
initialize IndexManager emails
at 
org.hibernate.search.indexes.impl.IndexManagerHolder.createIndexManager(IndexManagerHolder.java:230)
at 
org.hibernate.search.indexes.impl.IndexManagerHolder.buildEntityIndexBinding(IndexManagerHolder.java:101)
at 
org.hibernate.search.spi.SearchFactoryBuilder.initDocumentBuilders(SearchFactoryBuilder.java:413)
at 
org.hibernate.search.spi.SearchFactoryBuilder.buildNewSearchFactory(SearchFactoryBuilder.java:222)
at 
org.hibernate.search.spi.SearchFactoryBuilder.buildSearchFactory(SearchFactoryBuilder.java:146)
at 
org.hibernate.search.event.impl.FullTextIndexEventListener.initialize(FullTextIndexEventListener.java:130)
at 
org.hibernate.search.hcore.impl.HibernateSearchIntegrator.integrate(HibernateSearchIntegrator.java:83)
at org.hibernate.internal.SessionFactoryImpl.(SessionFactoryImpl.java:303)
at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:1769)
at 
org.hibernate.search.test.util.FullTextSessionBuilder.build(FullTextSessionBuilder.java:181)
at 
org.hibernate.search.infinispan.StoredIndexTest.startNode(StoredIndexTest.java:132)
at 
org.hibernate.search.infinispan.StoredIndexTest.testRestartingNode(StoredIndexTest.java:53)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at 
org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at 
org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at 
org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at 
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at 
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at 
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at 
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
Caused by: org.infinispan.CacheException:
org.infinispan.loaders.CacheLoaderException: Error while reading from
file
at 
org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:351)
at org.infinispan.CacheImpl.get(CacheImpl.java:391)
at org.infinispan.DecoratedCache.get(DecoratedCache.java:415)
at 
org.infinispan.lucene.impl.FileListOperations.getFileList(FileListOperations.java:66)
at org.infinispan.lucene.InfinispanDirectory.list(InfinispanDirectory.java:168)
at 
org.infinispan.lucene.InfinispanDirectory.listAll(InfinispanDirectory.java:328)
at 
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:602)
at 
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:554)
at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:359)
at org.apache.lucene.index.IndexReader.indexExists(IndexReader.java:1099)
at 
org.hibernate.search.store.impl.DirectoryProviderHelper.initializeIndexIfNeeded(DirectoryProviderHelper.java:155)
at 
org.hibernate.search.infinispan.impl.InfinispanDirectoryProvider.start(InfinispanDirectoryProvider.java:103)
at 
org.hibernate.search.indexes.impl.DirectoryBasedIndexManager.initialize(DirectoryBasedIndexManager.java:104)
at 
org.hibernate.search.indexes.impl.IndexManagerHolder.createIndexManager(IndexManagerHolder.java:226)
... 36 more
Caused by: org.infinispan.loaders.CacheLoaderException: Error while
reading from file
at 
org.infinispan.loaders.file.FileCacheSt

Re: [infinispan-dev] Integrating Karsten's FCS

[Sanne Grinovero]

Thanks Galder for opening the JIRA; but I wonder if we actually have a way out.

5.3.0.Final was released already, so if you want to "fix" its format
to be compatible with 5.2 in a minor patch version - probably 5.3.1 -
then that would break compatibility with 5.3.0 which is maybe even
worse.

Probably the lesser evil is to clearly warn about this and suggest an
external dump; this way we also save some work as we could drop the
existing CacheStore implementation and move on.. the saved dev time
could be spent on tests to make sure this problem won't occur again as
it clearly is a major problem if it slips into a released version
without being spotted on time.

Cheers,
Sanne


On 11 July 2013 10:53, Mircea Markus  wrote:
>
> On 11 Jul 2013, at 10:22, Galder Zamarreño  wrote:
>
>>> |
>>> | Hi,
>>> |
>>> | I think we all agree that Karsten's file cache store is a good base
>>> | replacement for the current file cache store, particularly for caches with
>>> | relatively small keys, or not a huge amount of them.
>>> |
>>> | I'm working with Radim to try to figure out what would be the tipping 
>>> point
>>> | at which LevelDB based cache store behaves better compared to Karsten's
>>> | cache store.
>>> |
>>> | In the mean time, I think we should integrate Karsten's FCS into master 
>>> and
>>> | have it ready for people to try from Alpha1. There's one or two things 
>>> that
>>> | do require a bit of review, but I can pin point them in the pull req.
>>> |
>>> | The question is, how should we name it? And what should we do about the
>>> | current FCS implementation?
>>> |
>>> | Shall we keep the current FCS implementation, deprecate it, and get rid 
>>> of it
>>> | in the next minor/major version? Some users might have data stored in the
>>> | current FCS and would be quite abrupt to just get rid of it right now.
>>>
>>> I'd remove it in Final, it cannot be seriously used either. Users can 
>>> migrate the data using rolling upgrades.
>>
>> ^ I don't think you can remove it in 6.0 Final. If you wanna do any kind of 
>> migration or rolling upgrade from the curren FCS to the new one, you need to 
>> have the current FCS around, to be able to read data in that format/setup 
>> and then write it to the new cache store. The moment you can remove it is 
>> the moment you've decided that no more migrations are gonna happen.
>
> You can have the logic around but don't expose it to the users as a shipped 
> cache store. E.g. move it to a different package and rename it to whatever 
> you want.
>
>
> Cheers,
> --
> Mircea Markus
> Infinispan lead (www.infinispan.org)
>
>
>
>
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] Keeping formats compatible

[Sanne Grinovero]

On 11 July 2013 17:26, Galder Zamarreño  wrote:
> Too many paralell discussions in this email thread… changing title…

+1

> On Jul 11, 2013, at 12:14 PM, Sanne Grinovero  wrote:
>
>> Thanks Galder for opening the JIRA; but I wonder if we actually have a way 
>> out.
>>
>> 5.3.0.Final was released already, so if you want to "fix" its format
>> to be compatible with 5.2 in a minor patch version - probably 5.3.1 -
>> then that would break compatibility with 5.3.0 which is maybe even
>> worse.
>
> Hold on till we've figured out the cause… and if you're a rigt, a potential 
> fix could be investigated to make 5.3.0 and 5.3.1 compatible...

Right.

>
>> Probably the lesser evil is to clearly warn about this and suggest an
>> external dump; this way we also save some work as we could drop the
>> existing CacheStore implementation and move on.. the saved dev time
>> could be spent on tests to make sure this problem won't occur again as
>> it clearly is a major problem if it slips into a released version
>> without being spotted on time.
>
> When we started investigating rolling upgrades we played with the idea of 
> having a serialized format that's readable between different versions and we 
> created [1]. I actually created some branches to investigate the feasibility 
> of adding such tests: [2, 3, 4, 5, 6], and of course found incompatibilities. 
> This idea was abandoned and instead we decided on a Hot Rod based mechanism 
> to transfer the data from one cluster to the other.

That was my understanding as well, and why I suggest to not give too
much importance to the CacheStore: we somehow agreed that this wasn't
expected to work. To be clear, I was not agreeing on that but since
that was the consensus based on what we can practically do I'm just
sticking to it now: we decided it would be incompatible.

> Such tests could be added, but there needs to be a limit on how far we want 
> to be backwards compatible with. IOW, trying for Infinispan 5.3 to be 
> backwards compatible with 4.0 would be a lot of work. The minimum req would 
> possibly be that we're backwards compatible between minors.

+1 If we make this as an external tool released with each version to
migrate from the previous version, then people can "chain" them up to
migrate across longer version gaps. This could eventually evolve in a
set of helper scripts which help migrate from any version (since we
start this practice) to any other version.

>
> This is something we can try to start imposing from 6.0 onwards. It's a good 
> moment to start doing it, at the beginning of a major release.

+1 Which again implies dropping older version's compatibility.

Cheers,
Sanne

>
> [1] https://issues.jboss.org/browse/ISPN-2034
> [2] https://github.com/galderz/infinispan/tree/archive/t_2034_40x
> [3] https://github.com/galderz/infinispan/tree/archive/t_2034_41x
> [4] https://github.com/galderz/infinispan/tree/archive/t_2034_42x
> [5] https://github.com/galderz/infinispan/tree/archive/t_2034_50x
> [6] https://github.com/galderz/infinispan/tree/archive/t_2034_51x
>
>
>>
>> Cheers,
>> Sanne
>
> --
> Galder Zamarreño
> gal...@redhat.com
> twitter.com/galderz
>
> Project Lead, Escalante
> http://escalante.io
>
> Engineer, Infinispan
> http://infinispan.org
>
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] staggered_get Question

[Sanne Grinovero]

Hi Erik,
could you elaborate on how "proximate" would look like please? I'm not
understanding how X-Site is relating to this.

Mircea,
how is a secondary owner going to behave when setting MAIN_OWNER ?

 -Sanne

On 12 July 2013 17:21, Erik Salter  wrote:
> - my boss and others.
>
> Why not proximate?  I may only want to get data from the same machine/site
> and retry if necessary.
>
> Erik
>
> -Original Message-
> From: infinispan-dev-boun...@lists.jboss.org
> [mailto:infinispan-dev-boun...@lists.jboss.org] On Behalf Of Mircea Markus
> Sent: Friday, July 12, 2013 11:56 AM
> To: Hammad Said; infinispan -Dev List
> Cc: Bryan Perrotti (bperrott); Erik Salter (esalter); Michael Walker; Balaji
> Rajam
> Subject: Re: [infinispan-dev] staggered_get Question
>
>
> On 12 Jul 2013, at 18:22, Mircea Markus  wrote:
>
>> (Adding -dev)
>>
>> On 12 Jul 2013, at 05:04, Hammad Said  wrote:
>>
>>> I have implemented the change for address ordering in and created the
> topic branch optimize_staggered-get in:
>>> https://github.com/hsaid4327/infinispan.git
>>
>> which branch do you want me to look at?
>> Or better can you please issue a pull request with your change?
>>
>>>
>>> The next part is the configuration change. For the configuration change,
> there are certain design decisions that need to be made:
>>> a) We need to introduce two configuration params staggered_get_flag and
> staggered_get_timeout. Where exactly are these params specified in the cache
> configuration file. At the global or the cache level and under what element.
> Are they be implemented as attributes of clustering element, separate
> element as a child of clustering or cache.
>>
>> What about something like:
>> 
>>  
>
> to clarify there are 3 ways of doing remote reads: all owners (like we
> currently do), staggered (as discussed) and only to the main owner.
>
>> 
>>
>> What others think?
>>
>>> b) once we read the properties second big question is how to pass them on
> to the dispatcher in question i.e CommandAwareRpcDispatacher. There are two
> choices here:
>>>  1- We change the method signature of invokeRemoteCommands and pass on
> these two params. This choice involves making up the calling stack in the
> classes that invoke this method  namele JGroupsTransport and possible up the
> calling stream.
>>> 2- Another option is to set it on the ResponseFilter which is passed to
> CommandAwareRpcDispatcher.
>>
>> I'm not sure the CARD needs to be aware of the staggering. I'd think it is
> the responsibility of the caller (DistributionInterceptor) to orchestrate
> staggering..
>>
>>>
>>> Also, would the changes in configuration would require making a change in
> XSD schema file?
>>>
>>> Thanks!
>>> Hammad
>>>
>>>
>>> `
>>
>> Cheers,
>> --
>> Mircea Markus
>> Infinispan lead (www.infinispan.org)
>>
>>
>>
>>
>
> Cheers,
> --
> Mircea Markus
> Infinispan lead (www.infinispan.org)
>
>
>
>
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] staggered_get Question

[Sanne Grinovero]

Thanks, got it now. Definitely makes sense.
This order could even be resolved just once at each view change, and
so store the pre-calculated "preferred node" in the metadata we have
for each segment.

 -Sanne

On 12 July 2013 19:40, Erik Salter  wrote:
> Hi Sanne,
>
> By "proximate", I'd like to fetch from an owner on the same
> machine/rack/site (as defined by the TACH) as the originator.The idea is
> to minimize the network traffic and any assorted latency as much as
> possible.  This doesn't involve XSite.  FWIW, if I use XSite, it won't be
> for reading.
>
> Thanks,
>
> Erik
>
>
> -Original Message-
> From: infinispan-dev-boun...@lists.jboss.org
> [mailto:infinispan-dev-boun...@lists.jboss.org] On Behalf Of Sanne Grinovero
> Sent: Friday, July 12, 2013 12:54 PM
> To: infinispan -Dev List
> Subject: Re: [infinispan-dev] staggered_get Question
>
> Hi Erik,
> could you elaborate on how "proximate" would look like please? I'm not
> understanding how X-Site is relating to this.
>
> Mircea,
> how is a secondary owner going to behave when setting MAIN_OWNER ?
>
>  -Sanne
>
> On 12 July 2013 17:21, Erik Salter  wrote:
>> - my boss and others.
>>
>> Why not proximate?  I may only want to get data from the same
>> machine/site and retry if necessary.
>>
>> Erik
>>
>> -Original Message-
>> From: infinispan-dev-boun...@lists.jboss.org
>> [mailto:infinispan-dev-boun...@lists.jboss.org] On Behalf Of Mircea
>> Markus
>> Sent: Friday, July 12, 2013 11:56 AM
>> To: Hammad Said; infinispan -Dev List
>> Cc: Bryan Perrotti (bperrott); Erik Salter (esalter); Michael Walker;
>> Balaji Rajam
>> Subject: Re: [infinispan-dev] staggered_get Question
>>
>>
>> On 12 Jul 2013, at 18:22, Mircea Markus  wrote:
>>
>>> (Adding -dev)
>>>
>>> On 12 Jul 2013, at 05:04, Hammad Said  wrote:
>>>
>>>> I have implemented the change for address ordering in and created
>>>> the
>> topic branch optimize_staggered-get in:
>>>> https://github.com/hsaid4327/infinispan.git
>>>
>>> which branch do you want me to look at?
>>> Or better can you please issue a pull request with your change?
>>>
>>>>
>>>> The next part is the configuration change. For the configuration
>>>> change,
>> there are certain design decisions that need to be made:
>>>> a) We need to introduce two configuration params staggered_get_flag
>>>> and
>> staggered_get_timeout. Where exactly are these params specified in the
>> cache configuration file. At the global or the cache level and under what
> element.
>> Are they be implemented as attributes of clustering element, separate
>> element as a child of clustering or cache.
>>>
>>> What about something like:
>>> 
>>>  
>>
>> to clarify there are 3 ways of doing remote reads: all owners (like we
>> currently do), staggered (as discussed) and only to the main owner.
>>
>>> 
>>>
>>> What others think?
>>>
>>>> b) once we read the properties second big question is how to pass
>>>> them on
>> to the dispatcher in question i.e CommandAwareRpcDispatacher. There
>> are two choices here:
>>>>  1- We change the method signature of invokeRemoteCommands and pass
>>>> on
>> these two params. This choice involves making up the calling stack in
>> the classes that invoke this method  namele JGroupsTransport and
>> possible up the calling stream.
>>>> 2- Another option is to set it on the ResponseFilter which is passed
>>>> to
>> CommandAwareRpcDispatcher.
>>>
>>> I'm not sure the CARD needs to be aware of the staggering. I'd think
>>> it is
>> the responsibility of the caller (DistributionInterceptor) to
>> orchestrate staggering..
>>>
>>>>
>>>> Also, would the changes in configuration would require making a
>>>> change in
>> XSD schema file?
>>>>
>>>> Thanks!
>>>> Hammad
>>>>
>>>>
>>>> `
>>>
>>> Cheers,
>>> --
>>> Mircea Markus
>>> Infinispan lead (www.infinispan.org)
>>>
>>>
>>>
>>>
>>
>> Cheers,
>> --
>> Mircea Markus
>> Infinispan lead (www.infinispan.org)
>>
>>
>>
>>
>>
>> ___
>> infinispan-dev mailing list
>> infinispan-dev@lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>> ___
>> infinispan-dev mailing list
>> infinispan-dev@lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

[infinispan-dev] Something wrong in the build order?

[Sanne Grinovero]

I was doing a clean build on an empty system, so I suspect the Maven
order is having some weirdness causing this:

[INFO] Reactor Summary:
[INFO]
[INFO] Infinispan BOM  SUCCESS [0.485s]
[INFO] Infinispan Common Parent .. SUCCESS [3.339s]
[INFO] Infinispan Commons  SUCCESS [12.924s]
[INFO] Infinispan Core ... SUCCESS [51.556s]
[INFO] Infinispan Extended Statistics  SUCCESS [11.652s]
[INFO] Parent pom for server modules . SUCCESS [3.136s]
[INFO] Infinispan Server - Core Components ... SUCCESS [50.031s]
[INFO] Parent pom for cachestore modules . SUCCESS [1.068s]
[INFO] Infinispan JDBC CacheStore  SUCCESS [3.142s]
[INFO] Parent pom for the Lucene integration modules . SUCCESS [0.766s]
[INFO] Infinispan integration with Lucene v.3.x .. SUCCESS [2.477s]
[INFO] Infinispan integration with Lucene v.4.x .. SUCCESS [1.646s]
[INFO] Infinispan Lucene Directory Implementation  SUCCESS [1.896s]
[INFO] Infinispan Query API .. SUCCESS [5.505s]
[INFO] Infinispan Tools .. SUCCESS [2.873s]
[INFO] Infinispan JDBM CacheStore  SUCCESS [3.151s]
[INFO] Infinispan Tree API ... SUCCESS [3.035s]
[INFO] Infinispan BDBJE CacheStore ... SUCCESS [2.281s]
[INFO] Infinispan CloudCacheStore  SUCCESS [4.656s]
[INFO] Infinispan Hot Rod Server . SUCCESS [58.436s]
[INFO] Infinispan Hot Rod Client . FAILURE [6.072s]
[INFO] Infinispan remote CacheStore .. SKIPPED
[INFO] Infinispan CassandraCacheStore  SKIPPED
[INFO] Infinispan HBaseCacheStore  SKIPPED
[INFO] Infinispan JPA CacheStore . SKIPPED
[INFO] Infinispan LevelDB CacheStore . SKIPPED
[INFO] Infinispan Memcached Server ... SKIPPED
[INFO] Infinispan WebSocket Server ... SKIPPED
[INFO] Infinispan REST Server  SKIPPED
[INFO] Infinispan RHQ Plugin . SKIPPED
[INFO] Infinispan Rolling Upgrade Tooling  SKIPPED
[INFO] Infinispan Spring Integration . SKIPPED
[INFO] Infinispan QL Interpreter . SKIPPED
[INFO] Infinispan CLI Client . SKIPPED
[INFO] Infinispan GUI Demo ... SKIPPED
[INFO] Infinispan EC2 Demo ... SKIPPED
[INFO] Infinispan Distributed Executors and Map/Reduce Demo  SKIPPED
[INFO] Infinispan EC2 Demo UI  SKIPPED
[INFO] Infinispan Directory Demo . SKIPPED
[INFO] Infinispan Lucene Directory Demo .. SKIPPED
[INFO] Infinispan GridFileSystem WebDAV interface  SKIPPED
[INFO] Infinispan Near Cache Demo  SKIPPED
[INFO] Infinispan JCACHE (JSR-107) implementation  SKIPPED
[INFO] Infinispan CDI support  SKIPPED
[INFO] Infinispan Near Cache Demo Client . SKIPPED
[INFO] Infinispan AS/EAP modules . SKIPPED
[INFO] Integration tests: Lucene Directory with Infinispan Query  SKIPPED
[INFO] Integration tests: AS Module Integration Tests  SKIPPED
[INFO] Integration tests: Infinispan compatibility mode .. SKIPPED
[INFO] Infinispan Distribution ... SKIPPED
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 3:52.887s
[INFO] Finished at: Tue Jul 16 10:17:58 BST 2013
[INFO] Final Memory: 94M/1003M
[INFO] 
[ERROR] Failed to execute goal on project infinispan-client-hotrod:
Could not resolve dependencies for project
org.infinispan:infinispan-client-hotrod:bundle:6.0.0-SNAPSHOT: Failed
to collect dependencies for
[org.infinispan:infinispan-commons:jar:6.0.0-SNAPSHOT (compile),
org.jboss.marshalling:jboss-marshalling-river:jar:1.3.15.GA (compile),
commons-pool:commons-pool:jar:1.6 (compile),
org.apache.hadoop:avro:jar:1.3.3 (provided),
org.infinispan:infinispan-core:jar:tests:6.0.0-SNAPSHOT (test),
org.infinispan:infinispan-server-hotrod:jar:6.0.0-SNAPSHOT (test),
org.infinispan:infinispan-server-hotrod:jar:tests:6.0.0-SNAPSHOT
(test), org.infinispan:infinispan-server-core:jar:tests:6.0.0-SNAPSHOT
(test), org.scala-lang:scala-library:jar:2.10.2 (test),
org.infinispan.protostream:protostream:jar:1.0.0.Alpha1 (compile),
org.infinispan.protostream:sample-domain-definition:jar:1.0.0.Alpha1
(test), com.thoughtworks.xstream:xst

Re: [infinispan-dev] Something wrong in the build order?

[Sanne Grinovero]

Thanks! working fine now.

On 16 July 2013 12:20, Adrian Nistor  wrote:
> It was in repo, but not all of it :(
> org.infinispan.protostream:parent:pom:1.0.0.Alpha1 was lost on the way
> from staging to releases repo, but is fixed now.
>
> On 07/16/2013 01:11 PM, Mircea Markus wrote:
>> Caused the fact that protostream is not in the maven repo yet, should be 
>> sorted soon by Adrian.
>>
>> On 16 Jul 2013, at 11:04, Sanne Grinovero  wrote:
>>
>>> I was doing a clean build on an empty system, so I suspect the Maven
>>> order is having some weirdness causing this:
>>>
>>> [INFO] Reactor Summary:
>>> [INFO]
>>> [INFO] Infinispan BOM  SUCCESS [0.485s]
>>> [INFO] Infinispan Common Parent .. SUCCESS [3.339s]
>>> [INFO] Infinispan Commons  SUCCESS [12.924s]
>>> [INFO] Infinispan Core ... SUCCESS [51.556s]
>>> [INFO] Infinispan Extended Statistics  SUCCESS [11.652s]
>>> [INFO] Parent pom for server modules . SUCCESS [3.136s]
>>> [INFO] Infinispan Server - Core Components ... SUCCESS [50.031s]
>>> [INFO] Parent pom for cachestore modules . SUCCESS [1.068s]
>>> [INFO] Infinispan JDBC CacheStore  SUCCESS [3.142s]
>>> [INFO] Parent pom for the Lucene integration modules . SUCCESS [0.766s]
>>> [INFO] Infinispan integration with Lucene v.3.x .. SUCCESS [2.477s]
>>> [INFO] Infinispan integration with Lucene v.4.x .. SUCCESS [1.646s]
>>> [INFO] Infinispan Lucene Directory Implementation  SUCCESS [1.896s]
>>> [INFO] Infinispan Query API .. SUCCESS [5.505s]
>>> [INFO] Infinispan Tools .. SUCCESS [2.873s]
>>> [INFO] Infinispan JDBM CacheStore  SUCCESS [3.151s]
>>> [INFO] Infinispan Tree API ... SUCCESS [3.035s]
>>> [INFO] Infinispan BDBJE CacheStore ... SUCCESS [2.281s]
>>> [INFO] Infinispan CloudCacheStore  SUCCESS [4.656s]
>>> [INFO] Infinispan Hot Rod Server . SUCCESS [58.436s]
>>> [INFO] Infinispan Hot Rod Client . FAILURE [6.072s]
>>> [INFO] Infinispan remote CacheStore .. SKIPPED
>>> [INFO] Infinispan CassandraCacheStore  SKIPPED
>>> [INFO] Infinispan HBaseCacheStore  SKIPPED
>>> [INFO] Infinispan JPA CacheStore . SKIPPED
>>> [INFO] Infinispan LevelDB CacheStore . SKIPPED
>>> [INFO] Infinispan Memcached Server ... SKIPPED
>>> [INFO] Infinispan WebSocket Server ... SKIPPED
>>> [INFO] Infinispan REST Server  SKIPPED
>>> [INFO] Infinispan RHQ Plugin . SKIPPED
>>> [INFO] Infinispan Rolling Upgrade Tooling  SKIPPED
>>> [INFO] Infinispan Spring Integration . SKIPPED
>>> [INFO] Infinispan QL Interpreter . SKIPPED
>>> [INFO] Infinispan CLI Client . SKIPPED
>>> [INFO] Infinispan GUI Demo ... SKIPPED
>>> [INFO] Infinispan EC2 Demo ... SKIPPED
>>> [INFO] Infinispan Distributed Executors and Map/Reduce Demo  SKIPPED
>>> [INFO] Infinispan EC2 Demo UI  SKIPPED
>>> [INFO] Infinispan Directory Demo . SKIPPED
>>> [INFO] Infinispan Lucene Directory Demo .. SKIPPED
>>> [INFO] Infinispan GridFileSystem WebDAV interface  SKIPPED
>>> [INFO] Infinispan Near Cache Demo  SKIPPED
>>> [INFO] Infinispan JCACHE (JSR-107) implementation  SKIPPED
>>> [INFO] Infinispan CDI support  SKIPPED
>>> [INFO] Infinispan Near Cache Demo Client . SKIPPED
>>> [INFO] Infinispan AS/EAP modules . SKIPPED
>>> [INFO] Integration tests: Lucene Directory with Infinispan Query  SKIPPED
>>> [INFO] Integration tests: AS Module Integration Tests  SKIPPED
>>> [INFO] Integration tests: Infinispan compatibility mode .. SKIPPED
>>> [INFO] Infinispan Distribution ... SKIPPED
>>> [INFO] 
>>> ---

Re: [infinispan-dev] storeAsBinary keeps both the object and the byte[] - why?

[Sanne Grinovero]

Also the implementation of the new "defensive copy" feature is not
correct. The reference to the unmarshalled instance is cached in a
field of MarshalledValue and there are no barriers preventing multiple
GET operations to retrieve the same instance.

Should we reopen the issue or create a new one?

On 18 July 2013 14:48, Mircea Markus  wrote:
>
> On 18 Jul 2013, at 14:08, Dan Berindei  wrote:
>
>> Are you sure both representations are kept?
>>
>> MarshalledValueInterceptor.visitGetKeyValueCommand() should call 
>> MarshalledValue.compact just before returning, which should keep only one 
>> representation.
>
> MarshalledValueInterceptor.visitGetKeyValueCommand() invokes 
> MarshalledValueInterceptor.compact(mv). The *mv* parameter , at least for 
> values, is null.
>
> Cheers,
> --
> Mircea Markus
> Infinispan lead (www.infinispan.org)
>
>
>
>
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] Infinispan 6.0.0.Alpha1 is out!

[Sanne Grinovero]

congratulations, and for first tag of Protostream as well!

Should I assume it's licensed under ASL2 ?
Would be nice to add at least a readme, now that you're leading a new
OSS project you should start looking at these other aspects.. maybe a
project on JIRA? ;-)

Cheers,
Sanne

On 17 July 2013 19:50, Adrian Nistor  wrote:
> Dear Infinispan community,
>
> We're proud to announce the first Alpha release of Infinispan 6.0.0.
> Starting with this release, Infinispan license is moving to the terms of the
> Apache Software Licence version 2.0.
>
> Besides increased stability (about 30 bug fixes) this release also brings
> several new features:A more efficient FileCacheStore implementation
> (courtesy Karsten Blees)
>
> A new set of usage and performance statistics developed within the scope of
> the CloudTM project
> A new (experimental) marshaller for Hot Rod based on protobuf, which will be
> primarily used by the upcoming remote querying feature. Since this has reuse
> potential in other projects it was promoted to an independent project named
> protostream under the Infinispan umbrella
>
> For a complete list of features and fixes included in this release please
> refer to the release notes.
> Visit our downloads section to find the latest release and if you have any
> questions please check our forums, our mailing lists or ping us directly on
> IRC.
>
> Thanks to everyone for their involvement and contribution!
>
> Cheers,
> Adrian
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] requirements for the new CacheStore API

[Sanne Grinovero]

Also:
 - support to efficiently preload only the owned entries
 - expiry of entries without loading them in memory
 - pass one (or more) existing CacheStore instance(s) into a starting
CacheManager to be used

New one, after recent talking to myself :-) :
 - Allow the CacheStore instance to get a reference to the Cache
itself: I need to perform a get() operation from the CacheStore to
implement "write behind" to a relational database, in case you have
mapped entities -> OGM -> Infinispan -> ORM -> relational database and
want to respect the original mapping.
Otherwise entries might be incomplete, or get written in the wrong
order, violating referential integrity rules.

In other words, just allow me to inject a reference to the
CacheManager in the CacheStore instance.

On 24 July 2013 16:45, Mircea Markus  wrote:
>
> On 24 Jul 2013, at 16:42, Galder Zamarreño  wrote:
>
>>> Hi,
>>>
>>> Starting from the original document Manik rolled out a while ago [1], 
>>> here's the list of requirements I'm currently aware of in the context of 
>>> the new CacheStore API:
>>> - better integration with the fluent API (CacheStore.init() is horrendous)
>>> - support for non-distributed transaction cache stores (1PC) and support 
>>> for XA capable cache store
>>> - support iteration over all the keys/entries in the store
>>> - needed for efficient Map/Reduce integration
>>> - needed for efficient implementation of Cache.keySet(), Cache.entrySet(), 
>>> Cache.values() methods
>>> - a simple read(k) + write(k,v) interface to be implemented by users that 
>>> just want to position ISPN as a cache between an app and a legacy system 
>>> and which don't need/want to be bothered with all the other complex features
>>> - support for expiration notification (ISPN-3064)
>>> - support for size (efficient implementation of the cache.size() method)
>>>
>>> Re: JSR-107 integration, I don't think we should depend on the JSR-107 API 
>>> as it forces us to use JSR-107 internal structures[2] but we should at 
>>> least provide an adapter layer.
>>
>> I agree on not depending on it, but design it in such way that's easy to 
>> write such adapter. IOW, make sure the functionality in the JSR-107 API is 
>> covered easily, without having add boiler-plate code :)
>
> +1
>
> Cheers,
> --
> Mircea Markus
> Infinispan lead (www.infinispan.org)
>
>
>
>
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] requirements for the new CacheStore API

[Sanne Grinovero]

On 24 July 2013 18:42, Mircea Markus  wrote:
>
> On 24 Jul 2013, at 17:16, Sanne Grinovero  wrote:
>
>> Also:
>> - support to efficiently preload only the owned entries
>
> We'd still need to iterate over all the stored keys to decide which maps to 
> the local node, but we won't have to load the entries. Is that what you have 
> in mind?

Ideally you wouldn't want to iterate on the keys: some storage engines
could run the CHM function as a selection criteria (among others,
those supporting Map/Reduce can do it; some of these run in Java
making this relatively easy - also for HotRod we'll likely have to
maintain CHM implementations in other languages anyway).

I realize that's complex to implement but the important change is to
not disallow such optimisations by using a limiting SPI: if the
CacheStore SPI would allow that, then people would be able to try
implement such smarter stores.. otherwise it's impossible.

Today we load all keys first, a good start would be to load them all
as a stream rather than as a single Set.

>
>> - expiry of entries without loading them in memory
>
> This is something we do already with CacheStore.purgeExpired

right, forgot :-)

>> - pass one (or more) existing CacheStore instance(s) into a starting
>> CacheManager to be used
>
> Something to consider but out of the scope of this change.
>
>> New one, after recent talking to myself :-) :
>> - Allow the CacheStore instance to get a reference to the Cache
>> itself: I need to perform a get() operation from the CacheStore to
>> implement "write behind" to a relational database, in case you have
>> mapped entities -> OGM -> Infinispan -> ORM -> relational database and
>> want to respect the original mapping.
>> Otherwise entries might be incomplete, or get written in the wrong
>> order, violating referential integrity rules.
>
> The CacheLoader.init(..) method does this already, I'll make sure it doesn't 
> go away in this release :-)

thanks

Sanne

>
>>
>> In other words, just allow me to inject a reference to the
>> CacheManager in the CacheStore instance.
>>
>
> Cheers,
> --
> Mircea Markus
> Infinispan lead (www.infinispan.org)
>
>
>
>
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] CacheStore redesign: no XA cache stores

[Sanne Grinovero]

My experience with transactions is limited, so I likely am missing on
some base concept, but I don't understand why the fact that it's
running on a different process is limiting in any form. We do that
regularly from appservers, queues, RDBMS's, ... ?

In this case the Infinispan node N1 needs to be coordinated by the TM
on N2, and control its "owned" resource C1. I realize that this is
possibly not trivial but somehow I expected that this was implemented
already.. isn't that a component you need for XA anyway? I'm quite
sure Narayana supports this setup as the application server does.

AFAIK a similar design was discussed in 2010 in the Transactions over
Hot Rod design meeting; how would this be fundamentally different, if
that's possible to explain to a non-expert ?


On 30 July 2013 21:20, Mircea Markus  wrote:
> Hi,
>
> I don't think can support XA (JTA) enabled cache stores. Here's why:
> - C1 (cache store instance) runs on node N1
> - an JTA tx is started on N2 which writes to(has a key that maps to) N1 - 
> both to the DataContainer and C1.
> - the JTA transaction manager running the transaction resides on N2, so it's 
> impossible for C1@N1 (different process) to enlist within that transaction
>
> This limitation doesn't exist for local caches configured with a cache store, 
> but that's rather a particular case.
>
> Our current recovery mechanism supports situations when writing to a cache 
> store fails during commit. If the commit fails, the user is given the 
> opportunity to re-apply transaction's changed and that includes both memory 
> and local storage.
>
> Cheers,
> --
> Mircea Markus
> Infinispan lead (www.infinispan.org)
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] CacheStore redesign: no XA cache stores

[Sanne Grinovero]

On 31 July 2013 16:01, Mircea Markus  wrote:
>
> On 30 Jul 2013, at 23:12, Sanne Grinovero  wrote:
>
>> My experience with transactions is limited, so I likely am missing on
>> some base concept, but I don't understand why the fact that it's
>> running on a different process is limiting in any form. We do that
>> regularly from appservers, queues, RDBMS's, ... ?
>
> It's limiting in the sense that we would have to build an XAResource bridge 
> on the node where the tx manager runs that delegates/bridges xa instructions 
> received from the transaction manager as RPCs to the xa cache store residing 
> on the remote node(s). This is a rather complicated approach and might 
> involve all the bridging RPCs to be sent in sequence which would hurt the 
> performance drastically.
> I think it's much nicer experience for the user to interact with ISPN as 
> whole as an XAResource instead of Infinispan exposing the fact that its store 
> might be XA as well.

+1
Bottomline it's not impossible right?

___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] Design session today

[Sanne Grinovero]

On 9 August 2013 11:35, Manik Surtani  wrote:
> We should actually move all of this to infinispan-dev - cc'ing infinispan-dev 
> on my response.
>
> On 9 Aug 2013, at 11:19, Mircea Markus  wrote:
>
>> Hi,
>>
>> I've been giving some thought last evening and here are some second-day 
>> thoughts:
>>
>> 1. parallel processing is a great idea and I think its really something that 
>> would make a difference compared to our competition
>
> +1.  We should consider the JDK 8 collections APIs as a reference, as I 
> mentioned.
>
>>
>> 2. using a two interfaces(CacheLoader,CacheWriter) over one. I'm still not 
>> totally bought by the idea
>> Pros: cleaner design (interface segregation principle[1]) which would allow 
>> users to only implement what they need
>> Cons: the difference between cache loader and cache store (or writer) has 
>> been a source of confusion through the users, as most of the users[2] only 
>> use the combined version
>> I'll continue the discussion on the public list
>
> Also, JSR 107 (and from that, most other data grids) will also follow a 
> separate CacheLoader/CacheWriter.  I think people will get used to the 
> separation of interfaces.

+1 to separate them, especially as people will get familiar with the
JSR terminology.
Not going to dig in the mailing lists, but I'm quite sure we already
discussed this in the past, and AFAIR everyone was agreeing on this.

>
>>
>> 3. allowing the cache loader to expose unserialised data directly 
>> (ValueHolder.getBytes[]).
>
> I used the name ValueHolder but this is a really poor term - how about 
> ContentsProxy?  It is a proxy for the contents of the entry, exposing methods:
>
> interface ContentsProxy {
>   ByteBuffer getValueBuffer();
>   ByteBuffer getInternalCacheValueBuffer();
>   InternalCacheValue getInternalCacheValue();
>
>   // Same as above except this method only deserializes timestamps and 
> metadata.  Not the actual value.
>   InternalCacheValue getSparseInternalCacheValue();
> }

+1 for the name "ContentsProxy".
On exposing ByteBuffer .. I'm wondering if we shouldn't have our own
interface, potentially shared with JGroups. For example, I'd like to
see an appendToEnd capability and an insertToHead which don't require
to eagerly resize the underlying arrays.

>
>> The use cases we had for this are:
>> a) streaming data during rolling upgrades. This works for scenarios where 
>> the data format (user classes) haven't changed and the data is written 
>> directly to a persistent store in the destination cluster
>> b) backups. This can be a generic and efficient (no serialisation) way of 
>> creating a backup tool.
>
> There are two more:
> c) Pre-populating a cache store from an external resource.
> d) Exposing the underlying byte buffers directly for placement into, say, a 
> native data container or directly onto the network stack for transmission 
> (once JGroups has moved to JDK 7).
>
>> I haven't thought a) entirely, but seems to me that only applies in to a 
>> rather specific rolling upgrade scenario.
>> Re: b) there might be some mere efficient ways of backing up data: take a 
>> database dump(jdbc cache store), copy the files (file cache store) etc. Also 
>> I'm not sure that the speed with which you take the dump is critical - i.e. 
>> even if you serialise/deserialize data might just work.
>
> It's not just the performance hit we take on serialisation/de-serialisation, 
> but also the additional CPU load we place on the system which should be 
> running, performing transactions!
>
>> Also in order to solve a) and b) I don't think ValueHolder.getBytes[] is the 
>> way to go. E.g. for the bucket cache stores use as read(and serialisation) 
>> unit an entire bucket so forcing them to returns the byte on an per entry 
>> basis would mean:
>> - read the bucket as byte[]
>> - deserialize the bucket structure
>> - iterate over entries in the bucket and serialise them again in order to 
>> satisfy ValueHolder.getBytes[]
>
> That's just the way buckets are currently designed.  If, for example, each 
> bucket has a header with a structure that looks like: 
> [key1][position1][key2][position2][end-of-keys marker][value1][value2], then 
> just by reading the header part of the bucket, we can grab chunks based on 
> the position information for the values without deserializing them.  Of 
> course, this is an "efficient" implementation.  A naive one could do what you 
> said above and still comply with the contract.
>
>> A better approach for this is to have toStream and fromStream methods 
>> similar to what we currently have CacheStore, so that the whole 
>> marshalling/unmarshalling business is delegated to the CacheStore itself. 
>> Also now that we're here, the CacheStore.toStream/fromStream API was added 
>> with the intention of solving the same problem some 4 years ago and are not 
>> used at all at this stage, though implemented by all the existing store.
>
> Yes, but I think we can do better than the toStream/fromStream API.
>

Re: [infinispan-dev] Design session today

[Sanne Grinovero]

On 9 August 2013 12:39, Manik Surtani  wrote:
>
> On 9 Aug 2013, at 12:06, Sanne Grinovero  wrote:
>
>>>> 3. allowing the cache loader to expose unserialised data directly 
>>>> (ValueHolder.getBytes[]).
>>>
>>> I used the name ValueHolder but this is a really poor term - how about 
>>> ContentsProxy?  It is a proxy for the contents of the entry, exposing 
>>> methods:
>>>
>>> interface ContentsProxy {
>>>  ByteBuffer getValueBuffer();
>>>  ByteBuffer getInternalCacheValueBuffer();
>>>  InternalCacheValue getInternalCacheValue();
>>>
>>>  // Same as above except this method only deserializes timestamps and 
>>> metadata.  Not the actual value.
>>>  InternalCacheValue getSparseInternalCacheValue();
>>> }
>>
>> +1 for the name "ContentsProxy".
>> On exposing ByteBuffer .. I'm wondering if we shouldn't have our own
>> interface, potentially shared with JGroups. For example, I'd like to
>> see an appendToEnd capability and an insertToHead which don't require
>> to eagerly resize the underlying arrays.
>
> https://docs.jboss.org/infinispan/5.2/apidocs/index.html?org/infinispan/io/ByteBuffer.html

Perfect :-)

Cheers,
Sanne

>
> --
> Manik Surtani
>
>
>
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] Recommended Cluster Size for Replication Mode

[Sanne Grinovero]

Hi,
it's best to start by understanding how a read and a write operation
are different.

For a non-transactional synch read, if the entry happens to have a
copy on the local node the read is immediately satisfied with no need
for network round-trips.
That implies several things:
 - read performance in the order of millions/second (assuming small entries)
 - a hit-ratio of local cache of 100% for replication
 - a hit-ratio of 66.6% for distribution (assuming 2 owners: each key
is stored on 2 nodes out of 3; this ratio improves by setting more
owners and gets worse when adding more nodes)

A non-transactional synch write however always requires at least one
network RPC (if not 2) to confirm safe storage on at least one
additional node (when not 2):
 - you have 33% change to be the primary owner for each key: needs one
RPC to the secondary owner
 - 33% to be the secondary owner: needs one RPC to the primary owner
 - 33% (remaining cases): needs 2 RPCs: one to the primary and one to
the secondary
Each of these cases have slightly different expected performance
metrics, but whichever you pick, you should be in a range of about two
orders of magnitude higher latency than a read operation.

Now in your scenario, assuming your application does a balanced mix of
reads and writes, and having understood that write performance doesn't
change when switching from DIST to REPL, it's possible the read
operations are somewhat faster (33% in average), but their latency is
extremely lower than write operations latency so unless you do a lot
more reads than writes, it's possible that you don't perceive any
measurable difference.

The odds can change dramatically if you change some parameters
 - if you have 100s of nodes then REPL will still have a local
hit-ratio of 100% while DIST tends to zero, making a read similar to a
write in term of performance.
 - if you change the ratio of reads vs writes
 - if you set number of owners to 3, and have 3 nodes, you essentially
get the same behaviour as REPL.

Also to keep in mind: DIST can store more data in memory and scales
writes linearly, while with REPL write operations require a network
RPC for each node in the grid.
So yes it's possible that in 3 nodes the difference isn't noticeable,
but picking the right cache mode depends on statistical information
about the behaviour of your application, or more simply if you have
latency requirements on reads which require REPL.

--Sanne

On 13 August 2013 05:29, Faseela K  wrote:
> Hi,
>
>I am using infinispan 5.2.3.
>My configuration is non-transactional, synchronous.
>With this configuration, is my replication supposed to perform better than 
> distribution, for both reads and writes?
>My Cluster Size requirement is 4 nodes.
>And my application involves both reads and writes heavily.
>For better performance, are there any suggestions on the clustering 
> modes/configurations?
>All my tests show, replication having better performance than distribution 
> for reads as well as writes, with 4 nodes.
>
> Thanks,
> Faseela
>
> -Original Message-
> From: infinispan-dev-boun...@lists.jboss.org 
> [mailto:infinispan-dev-boun...@lists.jboss.org] On Behalf Of Radim Vansa
> Sent: Monday, August 12, 2013 6:13 PM
> To: infinispan-dev@lists.jboss.org
> Subject: Re: [infinispan-dev] Recommended Cluster Size for Replication Mode
>
> Hi,
>
> which version exactly do you use, 5.2.x, 5.3.x or 6.0.x? In 5.2 the 
> replication mode was implemented separately from distribution mode and 
> depending on your configuration (is it non-transactional synchronous?) the 
> message control flow could differ. Since 5.3 replication mode is implemented 
> in the same manner and the results should be more comparable.
> I may be wrong here, but in 5.2.x concurrent writes to single key in 
> non-transactional mode could result in entries being out of sync on some 
> nodes (the writes could arrive at two nodes in different order). I think this 
> cannot happen in >= 5.3 anymore.
>
> Radim
>
> On 08/12/2013 09:04 AM, Faseela K wrote:
>> Hi,
>>
>>With a 3 node cluster, even for "WRITES" my replication performance is 
>> better than distribution.
>>That's why I came across this doubt.
>>Could some body please clarify, why the behaviour is like this?
>>
>> Thanks,
>> Faseela
>>
>> -Original Message-
>> From: infinispan-dev-boun...@lists.jboss.org
>> [mailto:infinispan-dev-boun...@lists.jboss.org] On Behalf Of Mircea
>> Markus
>> Sent: Friday, August 09, 2013 7:14 PM
>> To: infinispan -Dev List
>> Subject: Re: [infinispan-dev] Recommended Cluster Size for Replication
>> Mode
>>
>> On 6 Aug 2013, at 15:19, Faseela K  wrote:
>>
>>> What is the recommended cluster size for Replication Mode?
>>> Given 3 nodes, My replication configuration performs better than my 
>>> distributed configuration.
>>> Just wanted to know, at what cluster size, distribution will perform 
>>> better than replication.
>> There's no straight 

Re: [infinispan-dev] Granularity of cache store Git repositories

[Sanne Grinovero]

On 13 August 2013 11:48, Mircea Markus  wrote:
>
> On 13 Aug 2013, at 11:40, Galder Zamarreño  wrote:
>
>> We now have 8 separate git repositories for other cache stores in [1]. 
>> Doesn't this make it a royal PITA if we wanna change something for all cache 
>> stores? You'd have to send 9 pull requests!!
>
> Each cache store depends on a specific ISPN core version so they can be 
> upgraded and released in isolation. I think both these can be achieved with 
> some mvn tricks if all of them reside in the same repo - Tristan?

mvn tricks? Just keep the project poms separate, but dumping them in
the same repository.

>
> Cheers,
> --
> Mircea Markus
> Infinispan lead (www.infinispan.org)
>
>
>
>
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] cache loader: purgerThreads and purgeSynchronously

[Sanne Grinovero]

On 14 August 2013 12:27, Radim Vansa  wrote:
> It seems to me that these attributes may not be common to all cache
> stores. For some stores, the purging can be executed as part of other
> process anyway and it is not desired to purge the store on-demand. Some
> stores may not be able to purge in parallel at all (for example some
> single-file store where you just can't jump in the middle of file) or
> would be highly inefficient. Rather than emitting warning upon improper
> configuration, I'd let these parameters to implementation and make the
> purging service optional.
>
> Radim
>
> On 08/14/2013 12:59 PM, Mircea Markus wrote:
>> Hi,
>>
>> I'm not sure these config attributes are needed.
>>
>> - *purgeThreads*  configures the number of threads that run the storage 
>> purging (removal of expired entries from the storage). The more threads the 
>> faster the purging processes. Is there a reason for putting effort in making 
>> the purging fast (and parallel) though? The cache store implementations 
>> check if an entry is expired before returning it anyway.

That's not enough.You might have lots of large entries which need to
be cleaned up in the CacheStore which are never hit by a get
operation, you still need to delete them regularly.

On a similar reasoning, multiple threads might be needed if the
cleanup is slower than the amount of garbage the system is able to
store - agreed that's quite an extreme case, but still a possibility
for example on efficient append-only systems.. so I tend to agree with
Radim in saying that these could be CacheStore specific details.

>> Actually I'll move the code in CacheLoader interceptor to make sure this 
>> happens for all stores.
>> - *purgeSynchronously*. I think the reason for this parameter is that the 
>> purging is invoked by the eviction thread. If the purging takes long then 
>> eviction is delayed. This is false by default (I doubt users change this 
>> btw) so there's a different thread than the eviction thread that runs the 
>> purging. I'd rather remove the config option and always run the purging in 
>> its own thread.
>>
>> Opinions?
>>
>> Cheers,
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] cache loader/preload

[Sanne Grinovero]

On 14 August 2013 11:15, Mircea Markus  wrote:
> Currently the "preload" attribute is configured at *loaders* level and the 
> first cache loader in the chain is used for preloading.
> A better place for preload might be the individual loader element: would 
> indicate clearly which loader is used for preloading and also consistent with 
> the fetchPeristentState which is again defined on a per loader basis.
>
> How does it sound?

Very confusing actually :-)
Are you going to prevent me to enable "preload" on multiple Cache
Loaders elements? And what if I'm actually storing entries in
different CacheStore instanced based on some discriminator, like the
value (key) type?

To make a practical example, the Lucene Directory CacheLoader only
loads some specific types, and only from named indexes which match a
configured name. You could want multiple instances using different
configurations, so that for each index name stored in the same cache
you actually store them in different cachestores.

Similar situation, with the JPA CacheStore you might want to use
multiple independent instances, one per mapped type. Not necessarily
needed (one could improve the CacheStore implementation as well) but
it gives lot of flexibility, like connecting to different RDBMs
instances or with different credentials/schemas, etc...

Cheers,
Sanne

>
> Cheers,
> --
> Mircea Markus
> Infinispan lead (www.infinispan.org)
>
>
>
>
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] cache loader: purgerThreads and purgeSynchronously

[Sanne Grinovero]

On 14 August 2013 15:39, Manik Surtani  wrote:
> I agree with you, Sanne.

+1 always good :P

> But I think from a config perspective, this does need an overhaul.

+1

> I think the correct approach - given some of the newly forming ideas on the 
> CacheLoader/CacheStore SPI we've been having - is to not specify whether 
> purging is synchronous or not

+1 that is implementation specific, some might even not need a purger
at all as - for example - a remote Infinispan store does understand
how to cleanup by itself.

> and not to specify a number of threads, but instead to point to a named 
> executor.
> E.g., the way we define executors for async notification, async transport, 
> etc.

+1 sounds good and consistent, but let's improve even more on consistency:

If we're reorganizing the CacheLoader configuration schema, I would
very much like it to finally have named CacheStore/Loader elements
like we do with Executors, and being able to refer to them from Caches
rather than nesting their configuration in a Cache element.
If you ever configured - for example - a JDBCCacheStore for at least 3
caches in the same CacheManager, you know how much this is desirable.

Mircea: we've spoken about that option several times, AFAIR your main
point to not do that (yet) was to not introduce changes in the
configuration schema. Since it seems you're restructuring it all,
let's do it all :-)

-- Sanne


>
> - M
>
> On 14 Aug 2013, at 15:05, Sanne Grinovero  wrote:
>
>> On 14 August 2013 12:27, Radim Vansa  wrote:
>>> It seems to me that these attributes may not be common to all cache
>>> stores. For some stores, the purging can be executed as part of other
>>> process anyway and it is not desired to purge the store on-demand. Some
>>> stores may not be able to purge in parallel at all (for example some
>>> single-file store where you just can't jump in the middle of file) or
>>> would be highly inefficient. Rather than emitting warning upon improper
>>> configuration, I'd let these parameters to implementation and make the
>>> purging service optional.
>>>
>>> Radim
>>>
>>> On 08/14/2013 12:59 PM, Mircea Markus wrote:
>>>> Hi,
>>>>
>>>> I'm not sure these config attributes are needed.
>>>>
>>>> - *purgeThreads*  configures the number of threads that run the storage 
>>>> purging (removal of expired entries from the storage). The more threads 
>>>> the faster the purging processes. Is there a reason for putting effort in 
>>>> making the purging fast (and parallel) though? The cache store 
>>>> implementations check if an entry is expired before returning it anyway.
>>
>> That's not enough.You might have lots of large entries which need to
>> be cleaned up in the CacheStore which are never hit by a get
>> operation, you still need to delete them regularly.
>>
>> On a similar reasoning, multiple threads might be needed if the
>> cleanup is slower than the amount of garbage the system is able to
>> store - agreed that's quite an extreme case, but still a possibility
>> for example on efficient append-only systems.. so I tend to agree with
>> Radim in saying that these could be CacheStore specific details.
>>
>>>> Actually I'll move the code in CacheLoader interceptor to make sure this 
>>>> happens for all stores.
>>>> - *purgeSynchronously*. I think the reason for this parameter is that the 
>>>> purging is invoked by the eviction thread. If the purging takes long then 
>>>> eviction is delayed. This is false by default (I doubt users change this 
>>>> btw) so there's a different thread than the eviction thread that runs the 
>>>> purging. I'd rather remove the config option and always run the purging in 
>>>> its own thread.
>>>>
>>>> Opinions?
>>>>
>>>> Cheers,
>>>
>>> ___
>>> infinispan-dev mailing list
>>> infinispan-dev@lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>> ___
>> infinispan-dev mailing list
>> infinispan-dev@lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> --
> Manik Surtani
>
>
>
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] cache loader/preload

[Sanne Grinovero]

On 14 August 2013 17:59, Mircea Markus  wrote:
>
> On 14 Aug 2013, at 15:26, Sanne Grinovero  wrote:
>
>> On 14 August 2013 11:15, Mircea Markus  wrote:
>>> Currently the "preload" attribute is configured at *loaders* level and the 
>>> first cache loader in the chain is used for preloading.
>>> A better place for preload might be the individual loader element: would 
>>> indicate clearly which loader is used for preloading and also consistent 
>>> with the fetchPeristentState which is again defined on a per loader basis.
>>>
>>> How does it sound?
>>
>> Very confusing actually :-)
>> Are you going to prevent me to enable "preload" on multiple Cache
>> Loaders elements?
>
> I'm not going to take anything from you, this is something you're not allowed 
> to do ATM :-)

That's exactly my point. I'm not allowed to do it ATM, but if you
change the position of this attribute to *each* loader it will look
like I could enable/disable it differently on a per-instance.

>
>> And what if I'm actually storing entries in
>> different CacheStore instanced based on some discriminator, like the
>> value (key) type?
>>
>> To make a practical example, the Lucene Directory CacheLoader only
>> loads some specific types, and only from named indexes which match a
>> configured name. You could want multiple instances using different
>> configurations, so that for each index name stored in the same cache
>> you actually store them in different cachestores.
>>
>> Similar situation, with the JPA CacheStore you might want to use
>> multiple independent instances, one per mapped type. Not necessarily
>> needed (one could improve the CacheStore implementation as well) but
>> it gives lot of flexibility, like connecting to different RDBMs
>> instances or with different credentials/schemas, etc...
>
> Indeed user might want this in the future. Moving "preload" at "loader" level 
> would only facilitate that.

Right. I'm making those examples to highlight the benefit of moving
the attribute to each cachestore, as you proposed, but this only makes
sense if you actually allow the option to be applied differently for
each instance.

>
>>
>> Cheers,
>> Sanne
>>
>>>
>>> Cheers,
>>> --
>>> Mircea Markus
>>> Infinispan lead (www.infinispan.org)
>>>
>>>
>>>
>>>
>>>
>>> ___
>>> infinispan-dev mailing list
>>> infinispan-dev@lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> ___
>> infinispan-dev mailing list
>> infinispan-dev@lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> Cheers,
> --
> Mircea Markus
> Infinispan lead (www.infinispan.org)
>
>
>
>
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

[infinispan-dev] to stop swallowing exceptions in testsuite?

[Sanne Grinovero]

It took me quite some time to figure out about a resource leak in the
Query testsuite.

The Search Engine properly warns about unclosed resources, but the
automated test "cleanup" phase of the Infinispan testing framework
swallows and ignores any exception happening during a
CacheManager#close.

I realize there is a need for a "safe" cleanup but ignoring exceptions
is very tricky, could we change that to have the test fail at least?

Cheers,
Sanne
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] to stop swallowing exceptions in testsuite?

[Sanne Grinovero]

Some of these warnings - which where ignored so far - are pointing out
to badly written tests which are leaking resources.
In particular if the Query engine isn't closed appropriately, file
handles to index segments are not cleared up, and these are at the
core of some of the intermittent failures we've recently seen cropping
up in the Query module.

I can't blame the test contributors having forgot some #close()
methods; for sure the intention of
TestingUtil.killCacheManagers(cacheManagers) - to make sure we
properly cleanup - is not fulfilled this way: to the contrary it
caused leaks to stay unnoticed.
Tests should really fail eagerly rather than silently continue.


On 14 August 2013 23:39, Sanne Grinovero  wrote:
> It took me quite some time to figure out about a resource leak in the
> Query testsuite.
>
> The Search Engine properly warns about unclosed resources, but the
> automated test "cleanup" phase of the Infinispan testing framework
> swallows and ignores any exception happening during a
> CacheManager#close.
>
> I realize there is a need for a "safe" cleanup but ignoring exceptions
> is very tricky, could we change that to have the test fail at least?
>
> Cheers,
> Sanne
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] CacheLoader and CacheStore

[Sanne Grinovero]

Could we generate the parser and/or XSD as JGroups does with
org.jgroups.annotations.Property ?


On 15 August 2013 16:52, Ray Tsang  wrote:
>
>
>
> On Thu, Aug 15, 2013 at 11:45 AM, Radim Vansa  wrote:
>>
>> On 08/15/2013 03:57 PM, Ray Tsang wrote:
>>
>>
>>
>>
>> On Thu, Aug 15, 2013 at 9:48 AM, Galder Zamarreño 
>> wrote:
>>>
>>>
>>> On Aug 14, 2013, at 11:14 PM, Ray Tsang  wrote:
>>>
>>> > I actually did not enjoy writing parsers and config builders etc…
>>>
>>
>> i felt that writing the parsers are error prone - there are many places to
>> touch when adding a property - maybe I just didn't follow the best
>> example...
>>
>> i'm just thinking out loud but perhaps there is a way to consolidate the
>> configuration builder, configuration, configuration parser, and xsd with a
>> single set of metadata - after all, I think they are all trying to set the
>> same set of properties.
>>
>>
>> Well, it's true that the Maven build process could generate at least the
>> XSD automatically, and automatic parsing is not that complicated - in fact
>> I've done this in RadarGun, changing Mircea's original setter-way.
>> But in order to be able to use the programmatic configuration, you have
>> either write the builder yourself or generate it with some tool (I don't
>> think it would be right to generate further Java sources during the Maven
>> build, even if it is possible). And using tool for generation of some Java
>> code based on another code seems to me as another nasty complication.
>> Alltogether, writing *Configuration and *ConfigurationBuilder should be
>> enough, all the rest can be managed by runtime/build.
>>
>
> +1.  It just feel very redundant the way it is now.  condensing into
> Configuration and ConfigurationBuilder sounds great too, at least it's
> completely typesafe at this point.
>
> Otherwise, the way it is now, we have to keep property names in sync, and
> documentation in sync, and default values in sync, etc.. and 52 parser vs
> as7 parser? =(
>
>>
>>
>>
>>
>>>
>>> ^ Why not? Too verbose? Too many classes?
>>>
>>> I'd rather improve this than carry on supporting property-based
>>> configuration (any other type safety advocates?)
>>
>>
>> +1, the fact that I had to test both was a nuance...
>>
>> +1 as well
>>
>> Radim
>>
>> ___
>> infinispan-dev mailing list
>> infinispan-dev@lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] cache loader/preload

[Sanne Grinovero]

On 19 August 2013 11:02, Pedro Ruivo  wrote:
>
>
> On 08/16/2013 07:51 PM, Ray Tsang wrote:
>>
>>
>> On Fri, Aug 16, 2013 at 5:05 AM, Galder Zamarreño > <mailto:gal...@redhat.com>> wrote:
>>
>>
>> On Aug 15, 2013, at 4:25 AM, Ray Tsang > <mailto:saturn...@gmail.com>> wrote:
>>
>>  > Btw how do we deal with preloading in a cluster?
>>
>> Each node preloads locally only.
>>
>>
>> How is this done?  Is it through loadAllKeys first, and then determining
>> which keys are local?
>> Or do you mean it preloads locally from a non-shared cache store?  What
>> if the cache store is shared?
>
> the behaviour is the same for non-shared/shared caches. Each node loads
> all the data from the cache store and puts it in their local container
> (without any replication/distribution neither any communication with
> other nodes).

Let's note that Pedro and Galder are correctly replying to "how is
this done", but this is not functionally complete and we're aware of
it.

The problem is that if you intend to stop the whole grid to restart it
and reload the existing state from the CacheStore(s), the nodes get
likely assigned a different position in the view, and therefore in the
hashwheel, and the content of each local CacheStore won't match the
expected keys.
Today the only viable configuration is to use a single shared CacheStore..

further design discussion is on
 
-https://community.jboss.org/wiki/ControlledClusterShutdownWithDataRestoreFromPersistentStorage

-- Sanne


>
>>
>>
>>  > Eg how do we prevent every node from preloading the same data?
>>
>> Preloaded data is not replicated/distributed to other nodes, in
>> order to speed up process. For replicated caches, each node loads
>> its own stuff (from local or shared). For distributed caches, I
>> don't think we're intelligent enough to only load data that belongs
>> to a particular node (I think there's some jiras on this topic…)
>>
>>
>> For replicated cache, when it loads the data, does it put to the local
>> cache container only? (i.e., avoid unnecessarily replicating to other
>> nodes of course).
>
> yes
>
> Cheers,
> Pedro
>
>>
>>
>> Cheers,
>>
>>  >
>>  > On Aug 14, 2013, at 16:09, Mircea Markus > <mailto:mmar...@redhat.com>> wrote:
>>  >
>>  >>
>>  >> On 14 Aug 2013, at 20:53, Sanne Grinovero > <mailto:sa...@infinispan.org>> wrote:
>>  >>
>>  >>> On 14 August 2013 17:59, Mircea Markus > <mailto:mmar...@redhat.com>> wrote:
>>  >>>>
>>  >>>> On 14 Aug 2013, at 15:26, Sanne Grinovero
>> mailto:sa...@infinispan.org>> wrote:
>>  >>>>
>>  >>>>> On 14 August 2013 11:15, Mircea Markus > <mailto:mmar...@redhat.com>> wrote:
>>  >>>>>> Currently the "preload" attribute is configured at *loaders*
>> level and the first cache loader in the chain is used for preloading.
>>  >>>>>> A better place for preload might be the individual loader
>> element: would indicate clearly which loader is used for preloading
>> and also consistent with the fetchPeristentState which is again
>> defined on a per loader basis.
>>  >>>>>>
>>  >>>>>> How does it sound?
>>  >>>>>
>>  >>>>> Very confusing actually :-)
>>  >>>>> Are you going to prevent me to enable "preload" on multiple Cache
>>  >>>>> Loaders elements?
>>  >>>>
>>  >>>> I'm not going to take anything from you, this is something
>> you're not allowed to do ATM :-)
>>  >>>
>>  >>> That's exactly my point. I'm not allowed to do it ATM, but if you
>>  >>> change the position of this attribute to *each* loader it will look
>>  >>> like I could enable/disable it differently on a per-instance.
>>  >>
>>  >> Gotcha, I'll either add a check to allow specifying preload on a
>> single store, or (if time allows) I'll implement the multiple
>> preload feature.
>>  >>
>>  >>>
>>  >>>>
>>  >>>>> And what if I'm actually storing entries in
>>  >&

Re: [infinispan-dev] Consistent hashes and Hot Rod

[Sanne Grinovero]

+1 nice idea, but we need to consider that this could be limiting for
certain use cases.

It's fine for when we control the hash function, but when the hash
function can not be trusted, or the hash code of keys is forged, we
might want to use a prime number of buckets; I don't think this
applies to us in practice?

On 29 August 2013 13:35, Manik Surtani  wrote:
> The Hot Rod protocol current passes the size of the hash space used to 
> clients, so that clients are able to create a consistent hash and perform 
> smart routing.
>
> https://docs.jboss.org/author/display/ISPN/Hot+Rod+Protocol+-+Version+1.0#HotRodProtocol-Version1.0-HashDistributionAwareClientTopologyChangeHeader
> https://docs.jboss.org/author/display/ISPN/Hot+Rod+Protocol+-+Version+1.2#HotRodProtocol-Version1.2-HashDistributionAwareClientTopologyChangeHeader
>
> However there are no rules as to what this hash space is, and this forces 
> clients to use the remainder operation when performing modular arithmetic.  
> This has been proven to be inefficient [1] [2] [3], and can be greatly 
> improved if we made a simple assumption that the hash space will always be a 
> power of two.
>
> Perhaps something we can add in v1.3 of the protocol, so that clients talking 
> to a server using Hot Rod 1.3 can make the assumption that the hash space is 
> a power of 2?
>
> WDYT?
>
> - Manik
>
>
> [1] 
> http://dhruba.name/2011/07/12/performance-pattern-modulo-and-powers-of-two/
> [2] 
> http://scicomp.stackexchange.com/questions/187/why-is-division-so-much-more-complex-than-other-arithmetic-operations
> [3] http://disruptor.googlecode.com/files/Disruptor-1.0.pdf (Page 5, last 
> paragraph)
>
>
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

[infinispan-dev] Sparkey: new OSS on disk hastable from Spotify

[Sanne Grinovero]

It's heavily tuned for read-mostly systems, still it could be
something to consider for an efficient on disk CacheStore?

https://github.com/spotify/sparkey

Cheers,
Sanne
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] Indexing for remote query

[Sanne Grinovero]

On 4 September 2013 12:01, Adrian Nistor  wrote:
> Hi devs,
>
> Currently remote caches have byte[] values. With the introduction of
> remote query these byte arrays are no longer opaque for search-enabled
> caches, they are Protobuf encoded and their schema is known and we have
> to somehow help hibernate-search extract the indexable data and index
> it. As a side note, the indexing for embedded mode is done by
> QueryInterceptor which relies on hibernate-search annotated cache
> values. This definitely does not work directly for byte[].
>
> One option is to write another interceptor similar to QueryInterceptor
> but capable of making sense of the byte[], but that would duplicate
> functionality (and maintenance) and would have to use Lucene directly
> because in order to use hibernate-search you would need annotations on
> your entities. And if we go this route we might need to also duplicate
> MassIndexer.
>
> A second more preferable option is to wrap the byte[] values into a
> hsearch FieldBridge or ClassBridge annotated entity so we can continue
> to use existing QueryInterceptor. This was implemented in the PR [1],
> the wrapping entity being
> org.infinispan.query.remote.indexing.ProtobufValueWrapper and the place
> where the wrapping happens is a new interceptor,
> org.infinispan.query.remote.indexing.RemoteValueWrapperInterceptor.
> RemoteValueWrapperInterceptor uses a mechanism similar to
> TypeConverterInterceptor (of compat mode), so a common base class was
> extracted for them.
>
> The wrapping approach has received favourable review so far. But I'm
> asking myself (and you) maybe there's an even simpler solution that I've
> overlooked? Or maybe we can do the wrapping somehow without introducing
> a new interceptor?

Current solution seems quite simple and maintainable to me, +1 to keep
it. I'd also advise to stay away from low-level Lucene APIs: they
change significantly and you don't want to rewrite it all for each
update, unless you read the lucene-dev mailing list at breakfast
everyday.

Still, in this first round we favored quick development and there is
room for improvement in several areas; for example, since you
mentioned specifically the QueryInterceptor, I would prefer to see a
specialized edition of it (maybe sharing some code) to skip all the
overhead it has in checking if the indexed type is a well-known
indexed class, this logic is quite tricky and needs some reflection
while it's all unnecessary burden if you are able to register the
one-and-only indexed valueholder.

I'm confident that you'll find more simplifications on this road, just
pointing you to a potential starting point.. also if you're running
out of cool tasks I could point you to some open issues, there are
some other feature requests that I would consider quite important, for
example:

- https://issues.jboss.org/browse/ISPN-951
- https://issues.jboss.org/browse/ISPN-2610

>
> Thanks!
>
> [1] https://github.com/infinispan/infinispan/pull/2022
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] ISPN-3051 configuration

[Sanne Grinovero]

On 9 September 2013 10:37, Pedro Ruivo  wrote:
>
>
> On 09/09/2013 10:18 AM, Dan Berindei wrote:
>> Hi guys
>>
>> As you know, I'm working on ISPN-3051, allowing each node to take a
>> higher or lower proportion of the entries in the cache. I've implemented
>> this by adding a float "loadFactor" setting in each node's
>> configuration, with 1 being the default and any positive value being
>> accepted (including 0).
>>
>> There are two questions I wanted to ask you about the configuration:
>>
>> 1. What do you think about the "loadFactor" name? I started having
>> doubts about it, since it has a very different meaning in HashMap. I
>> have come up with a couple alternatives, but I don't love any of them:
>> "proportionalLoad" and "proportionalCapacity".
>
> I think capacity is a good name...

+1
"capacityFactor"

>
>>
>> 2. Where should we put this setting? I have added it as
>> CacheConfiguration.clustering().hash().loadFactor(), but I can't think
>> of a reason for having different values for each cache, so we might as
>> well put it in the global configuration.
>
> My vote is to having it per cache, so we can support use cases like this:
> having 2 caches in the same node but one is more prioritized/requested
> than the other and you may want to keep more data for the prioritized cache.

+1 to have it cache specific. A node could be interested in
participating with some caches but not needing a high degree of
locality, while being more interested in keeping as much as possible
in memory for a different Cache.

>
>>
>> Cheers
>> Dan
>>
>>
>>
>> ___
>> infinispan-dev mailing list
>> infinispan-dev@lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] persistence thread pool

[Sanne Grinovero]

+1 to add configuration

I'm not sure if it should be a global component. Having a single
instance would help to reuse some threads and keep the overall amount
of threads under control, but also one would not be able to cap
processing on a single store, and in some use cases if a job is
loading from one store and storing to other stores it could deadlock.

On 11 September 2013 21:33, Mircea Markus  wrote:
> Hi,
>
> With the new store API it is possible to iterate over the entries in a store 
> in parallel. Also the purging of the expired entries happen in parallel as 
> well.
> I' like to add a persistenceExecutor (global component) to define the thread 
> pool that would be used for these parallel operations.
> How does that sound?
>
> Cheers,
> --
> Mircea Markus
> Infinispan lead (www.infinispan.org)
>
>
>
>
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] SNAPSHOT dependencies on protostream

[Sanne Grinovero]

+1
and maybe you want SNAPSHOT dependencies of Hibernate Search ?

On 12 September 2013 16:08, Mircea Markus  wrote:
> Hi,
>
> Whilst having dependencies on SNAPSHOT is not generally good, we do allow 
> SNAPSHOT dependencies between our own components: e.g. cache stores' upstream 
> depend on an SNAPSHOT of Infinispan core. I think the rule should still apply 
> for protostream, i.e. on master, the query should be allowed to depend on 
> protostream-SNAPSHOT, assuming protostream doesn't have external SNAPSHOT 
> dependencies (it doesn't). Otherwise we end up releasing a new protostream 
> whenever e.g. a bug gets fixed and query needs it - way too often.
>
> Of course we shouldn't allow any SNAPSHOT dependencies during the release.
>
> Cheers,
> --
> Mircea Markus
> Infinispan lead (www.infinispan.org)
>
>
>
>
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] SNAPSHOT dependencies on protostream

[Sanne Grinovero]

On 12 September 2013 16:37, Mircea Markus  wrote:
> +1 for Hibernate Search as well, seems like making Adrian's life easier.

Thanks!

> On Sep 12, 2013, at 3:27 PM, Adrian Nistor  wrote:
>
>> When issues like "HSEARCH-1396 A FieldBridge should be able to easily
>> acquire services/resources" or "HSEARCH-1397 Ability to offer the notion
>> of composite bridge" get fixed I'd like to be able to update infinispan
>> code immediately rather than waiting for the next hsearch alpha release,
>> even if that happens in the same day. Or is it more manageable for
>> hsearch to make piecemeal alpha releases after each issue is fixed?

Not too eager to make "piecemeal alpha release" even if we definitely
should release more often. Glad it's not necessary.

Sanne

>>
>> On 09/12/2013 05:11 PM, Sanne Grinovero wrote:
>>> +1
>>> and maybe you want SNAPSHOT dependencies of Hibernate Search ?
>>>
>>> On 12 September 2013 16:08, Mircea Markus  wrote:
>>>> Hi,
>>>>
>>>> Whilst having dependencies on SNAPSHOT is not generally good, we do allow 
>>>> SNAPSHOT dependencies between our own components: e.g. cache stores' 
>>>> upstream depend on an SNAPSHOT of Infinispan core. I think the rule should 
>>>> still apply for protostream, i.e. on master, the query should be allowed 
>>>> to depend on protostream-SNAPSHOT, assuming protostream doesn't have 
>>>> external SNAPSHOT dependencies (it doesn't). Otherwise we end up releasing 
>>>> a new protostream whenever e.g. a bug gets fixed and query needs it - way 
>>>> too often.
>>>>
>>>> Of course we shouldn't allow any SNAPSHOT dependencies during the release.
>>>>
>>>> Cheers,
>>>> --
>>>> Mircea Markus
>>>> Infinispan lead (www.infinispan.org)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> ___
>>>> infinispan-dev mailing list
>>>> infinispan-dev@lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>> ___
>>> infinispan-dev mailing list
>>> infinispan-dev@lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>> ___
>> infinispan-dev mailing list
>> infinispan-dev@lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> Cheers,
> --
> Mircea Markus
> Infinispan lead (www.infinispan.org)
>
>
>
>
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] blog on new cache store API

[Sanne Grinovero]

Thanks for the write-up! Good to have a recap.
I'm very happy with these improvements; I've been tinkering around the
new API - just playing so far - but it definitely is very powerful:
all SPI restrictions are taken away while still not exposing too much
complexity; there is a lot I can improve in my implementations now and
also it looks like it will be quite easy.

Cheers,
Sanne

On 16 September 2013 14:05, Mircea Markus  wrote:
> http://infinispan.blogspot.co.uk/2013/09/new-persistence-api-in-infinispan.html
>
> Cheers,
> --
> Mircea Markus
> Infinispan lead (www.infinispan.org)
>
>
>
>
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] Deprecated API removal

[Sanne Grinovero]

It would be nice to have a documentation (wiki?) page tracking all
mayor code migration needs between versions,
for example: https://community.jboss.org/wiki/HibernateSearchMigrationGuide

On 18 September 2013 13:58, Pedro Ruivo  wrote:
> Hello,
>
> The following API is going to be removed in the next release (6.0.0.Beta1):
>
> classes:
> * org.infinispan.loaders.remote.wrapper.EntryWrapper
> * org.infinispan.context.FlagContainer
>
> methods:
> * StreamingMarshaller.startObjectOutput
> * AdvancedCache.getCacheEntry
> * AtomicMapLookup.getAtomicMap
> * DistributionManager.isLocal
> * TxInterceptor.setStatisticsEnabled
> * GridFile.delete
> * ModuleLifecycle.cacheManagerStarting
> * ModuleLifecycle.cacheStarting
> * DeadlockDetectingLockManager.getLocallyInterruptedTransactions
>
> packages:
> * org.infinispan.config
>
> Please update your code accordingly.
>
> Thank you.
>
> Regards,
> Pedro Ruivo
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

[infinispan-dev] Query and dynamic discovery of types

[Sanne Grinovero]

For Infinispan 6.0 we decided that the following issue is a bloker:

https://issues.jboss.org/browse/ISPN-2143 - Improve how different
indexed caches sync up on new indexed types

It really is important, and I'm a bit concerned that it was moved to
CR1 as it might not be trivial: for embedded query it needs to store a
reference to class definitions (potentially with classloader
information too), and for remote queries it needs to distribute the
indexing configuration schema.

Also I'm wondering if we shouldn't go even further: in Hibernate
Search the internal complexity of handling newly appearing types
(rather than statically configured types) is very hard.

In the next version we might need to drop this, so it would be great
if we could stabilize the Infinispan Query API right now in such a way
that it won't need this feature anymore.

I proposed it years ago but it was rejected because of usability
concerns.. would you still reject the idea to simply list the indexed
types at configuration time? If so, I think we need to explore
alternative solutions.

Note that I only want to remove the dependency to dynamic _Class_
definitions being added at runtime; we will still support dynamically
defined types so in case you need extreme flexibility like with remote
queries that will work, as long as you provide a smart bridge able to
reconfigure itself based on the dynamic metadata; I think that's a
cleaner approach as you would be directly in control of it rather than
having to workaround a design which was thought for a different
purpose.

Sanne
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] Read Committed Distributed Cache Concerns

[Sanne Grinovero]

On 19 September 2013 09:06, Radim Vansa  wrote:
> I think that Read Committed isolation level is not obliged to present
> you with up-to-date committed data - the only fact is that it can, but
> application must not rely on that. It's lower isolation level.
> Nevertheless, I think that lower isolation level should mean better
> performance. I would be strongly against imposing any additional
> overhead that could slow it down. I should give it some test, but if it
> currently does not offer any advantage against RR, I don't see any
> reason for this to exist.

+1
I also think the lower level of isolation is not necessarily a
promise, more something people can select when they are able to handle
it in their apps / business requirements.
As long as this is properly documented it should be fine.

I agree with your point that it's undesirable that depending on the
node topology (i.e. on which node you happen to run a test) the
experienced isolation might seem different, but again let's just
document it clearly.

>
> Regarding the bug: Thanks a lot for pointing this out, I should adapt
> the testsuite to use different object types as values (if I understand
> the scenario correctly).
>
> Radim
>
> On 09/19/2013 12:03 AM, William Burns wrote:
>> I was recently refactoring code dealing with isolation levels and
>> found how ReadCommitted is implemented and I have a few concerns I
>> wanted to bring up.
>>
>> ReadCommitted read operations work by storing a reference to the value
>> from the data store in its caller's context.  Thus whenever another
>> transaction is committed that updates the data store value any context
>> that has that reference now sees the latest committed value.  This
>> works well for Local and Replicated caches since all data stores are
>> updated with the latest value upon completion of the transaction.
>> However Distributed caches only the owners see the update in their
>> data store and thus any non owner will still have the old value they
>> previously read before the commit occurred.
>>
>> This seems quite inconsistent that Distributed caches run in a mix of
>> Repeatable Read/Read Committed depending on what node and what key you
>> are using.
>>
>> To operate properly we could track requests similar to how it works
>> for L1 so we can tell non owners to clear out their context values for
>> values they read remotely that they haven't updated (since Read
>> Committed writes should return the same written value).  That seems
>> like quite a bit of additional overhead though.
>>
>> I am wondering is it worth it to try to keep Read Committed isolation
>> level though?  It seems that Repeatable Read would be simpler and most
>> likely more performant as you wouldn't need all the additional remote
>> calls to get it to work properly.  Or is it okay that we have
>> different isolation levels for some keys on some nodes?  This could be
>> quite confusing if a user was using a local and remote transaction and
>> a transaction may not see the other's committed changes when they
>> expect to.
>>
>> What do you guys think?
>>
>>   - Will
>>
>> P.S.
>>
>> I also found a bug with Read Committed for all caches where if you do
>> a write that changes the underlying InternalCacheEntry to a new type,
>> that reads won't see subsequent committed values.  This is caused
>> because the underlying data is changed to a new reference and a read
>> would still be holding onto a reference of the old InternalCacheEntry.
>>   This can happen when using the various overridden put methods for
>> example.  We should have a good solution for it, but may not be
>> required if we find that Read Committed itself is flawed beyond
>> saving.
>> ___
>> infinispan-dev mailing list
>> infinispan-dev@lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> --
> Radim Vansa 
> JBoss DataGrid QA
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] Query and dynamic discovery of types

[Sanne Grinovero]

On 20 September 2013 11:19, Mircea Markus  wrote:
> Thanks for the heads up!
>
> It's not clear for me what the functional impact of ISPN-2143 is: incomplete 
> query results?

yes

>
> On Sep 19, 2013, at 11:42 AM, Sanne Grinovero  wrote:
>
>> For Infinispan 6.0 we decided that the following issue is a bloker:
>>
>> https://issues.jboss.org/browse/ISPN-2143 - Improve how different
>> indexed caches sync up on new indexed types
>>
>> It really is important, and I'm a bit concerned that it was moved to
>> CR1 as it might not be trivial: for embedded query it needs to store a
>> reference to class definitions (potentially with classloader
>> information too), and for remote queries it needs to distribute the
>> indexing configuration schema.
>>
>> Also I'm wondering if we shouldn't go even further: in Hibernate
>> Search the internal complexity of handling newly appearing types
>> (rather than statically configured types) is very hard.
>>
>> In the next version we might need to drop this, so it would be great
>> if we could stabilize the Infinispan Query API right now in such a way
>> that it won't need this feature anymore.
>>
>> I proposed it years ago but it was rejected because of usability
>> concerns.. would you still reject the idea to simply list the indexed
>> types at configuration time? If so, I think we need to explore
>> alternative solutions.
>>
>> Note that I only want to remove the dependency to dynamic _Class_
>> definitions being added at runtime; we will still support dynamically
>> defined types so in case you need extreme flexibility like with remote
>> queries that will work, as long as you provide a smart bridge able to
>> reconfigure itself based on the dynamic metadata; I think that's a
>> cleaner approach as you would be directly in control of it rather than
>> having to workaround a design which was thought for a different
>> purpose.
>>
>> Sanne
>> ___
>> infinispan-dev mailing list
>> infinispan-dev@lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> Cheers,
> --
> Mircea Markus
> Infinispan lead (www.infinispan.org)
>
>
>
>
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] Read Committed Distributed Cache Concerns

[Sanne Grinovero]

On 19 September 2013 18:29, Mircea Markus  wrote:
> (Adding Jonathan who knows a thing or two about transactions.)
>
> Given that READ_COMMITTED (RC) is less performant than REPEATABLE_READ (RR)
> I don't see any value in keeping RC around. I don't think users rely on
> exact RC semantics (i.e. if an entry has been committed then an ongoing
> tx requires the most up 2 date value between reads) - that actually
> is not the case with DIST caches as you've mentioned.

I don't think you can generalize from the specific example William
made; there will still be cases in which READ_COMMITTED will be more
efficient than REPEATABLE_READ,
especially if you avoid "fixing" this, as suggested by Radim and
myself in the two previous emails [not sure you if saw them, since you
forking the conversation ignoring those mails]:
if we agree that the current semantics is acceptable, it will
consistently be faster than REPEATABLE_READ.

Sanne

> I think RC is only preferred to RR because of performance, but if the 
> performance
> is the same (or even worse) I think we should only provide RR. Jonathan, care 
> to comment?
>
>
> On Sep 18, 2013, at 11:03 PM, William Burns  wrote:
>
>> I was recently refactoring code dealing with isolation levels and
>> found how ReadCommitted is implemented and I have a few concerns I
>> wanted to bring up.
>>
>> ReadCommitted read operations work by storing a reference to the value
>> from the data store in its caller's context.  Thus whenever another
>> transaction is committed that updates the data store value any context
>> that has that reference now sees the latest committed value.  This
>> works well for Local and Replicated caches since all data stores are
>> updated with the latest value upon completion of the transaction.
>> However Distributed caches only the owners see the update in their
>> data store and thus any non owner will still have the old value they
>> previously read before the commit occurred.
>>
>> This seems quite inconsistent that Distributed caches run in a mix of
>> Repeatable Read/Read Committed depending on what node and what key you
>> are using.
>>
>> To operate properly we could track requests similar to how it works
>> for L1 so we can tell non owners to clear out their context values for
>> values they read remotely that they haven't updated (since Read
>> Committed writes should return the same written value).  That seems
>> like quite a bit of additional overhead though.
>>
>> I am wondering is it worth it to try to keep Read Committed isolation
>> level though?  It seems that Repeatable Read would be simpler and most
>> likely more performant as you wouldn't need all the additional remote
>> calls to get it to work properly.  Or is it okay that we have
>> different isolation levels for some keys on some nodes?  This could be
>> quite confusing if a user was using a local and remote transaction and
>> a transaction may not see the other's committed changes when they
>> expect to.
>>
>> What do you guys think?
>>
>> - Will
>>
>> P.S.
>>
>> I also found a bug with Read Committed for all caches where if you do
>> a write that changes the underlying InternalCacheEntry to a new type,
>> that reads won't see subsequent committed values.  This is caused
>> because the underlying data is changed to a new reference and a read
>> would still be holding onto a reference of the old InternalCacheEntry.
>> This can happen when using the various overridden put methods for
>> example.  We should have a good solution for it, but may not be
>> required if we find that Read Committed itself is flawed beyond
>> saving.
>> ___
>> infinispan-dev mailing list
>> infinispan-dev@lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> Cheers,
> --
> Mircea Markus
> Infinispan lead (www.infinispan.org)
>
>
>
>
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] replacing the (FineGrained)AtomicMap with grouping

[Sanne Grinovero]

On 20 September 2013 23:19, William Burns  wrote:
> Responses inline
>
> Also want to preface this with: If you haven't seen in other mailing
> list Read Committed is going away as it doesn't work properly in DIST
> (in fact AM is is really badly bugged with RC with DIST);

We didn't agree on dropping it, in fact I think that would be a big
mistake considering the overhead of RR when dealing with large entries
/ many entries.
For example in all Search use cases we really don't need any RR
guarantee and would be wise to handle each operation in the most
efficient strategy.
[Technically it would be awesome to be able to be able to rely on RR
but it doesn't work as in databases - it doesn't snapshot the version
of entries not touched yet - so we have to compensate at a higher
layer..]

> On Fri, Sep 20, 2013 at 9:56 AM, Emmanuel Bernard
>  wrote:
>> I sort of see how we could replace it but we do make use of the FGAM to
>> represent an Hibernate OGM dehydrated entity with one property per map
>> entry.
>> >From what you are describing we would get an alternative solution but
>> that would mean more memory strain and object creation. That will
>> negatively impact the Infinispan backend.
>
> Object creation overhead shouldn't be that bad, the only real addition
> is each node would keep a map containing the group name pointing to
> the keys that are tied to that group (for entrySet etc).  This allows
> for much better serialization performance, detailed below, since you
> can optimize only reading the key(s) you care about.

Isn't FGAM also more efficient storage-wise? Technically it's storing
a single entry so I would expect it to be "better packed" somehow.

>>
>> Also I don't remember if we use the key lock but at some point we will.
>> I imagine a workaround is to lock the id property.

+1
Also we already include the id property consistently in any other
query to guarantee ordering of writes, so that would be a simple
change.

> Yeah just using a separate but shared lock would cover that pretty
> easily, but also opens to missed locking as it isn't implicit anymore.

For OGM specifically locking shouldn't be a problem.

>> OGM could live with it but it seems the usage is rendered more
>> complicated and users having the same style of requirements would need
>> to be more expert (more complex APIs).

+1

>>
>> Emmanuel
>
>
> On Fri, Sep 20, 2013 at 1:38 PM, Emmanuel Bernard
>  wrote:
>> Well, I have always wanted a way to only read some keys from a FGAM. Sort of 
>> like a projection.
> +1
>>
>>> On 20 sept. 2013, at 21:14, Randall Hauch  wrote:
>>>
>>> IMO, the primary benefit of the FGAM is that you can aggregate your entries 
>>> into a single entry that is a real aggregate: read the map and you get all 
>>> the FGAM's entries in one fell swoop. IIUC, your proposal loses this 
>>> capability for a single read of all aggregate parts. Is that right?

I think Randall nailed it: that was my first thought as well, that's not nice.
The good news is that I had already opened - a long time ago - a
feature request for multi-Get: something like "Value[] get(Key...) ",
if we had such a feature then FGAM would be in a better position to be
deprecated.

> That is one benefit.  However this also very costly when you are
> performing any operation on the AtomicMap from a node that doesn't own
> that value as it will have to retrieve the entire contents on every
> operation remotely if not owner (only once inside a given batch/tx).

But the proposed alternatives doesn't improve on that either ;-)

>
> The current grouping API doesn't allow for aggregated keys and values,
> but Mircea is proposing to add the Cache.getGroup method.  In that
> case you can control what keys you bring back if you want 1 or all for
> example.

What if I don't know the keys? We're not always able to list them,
currently I can iterate the keyset from FGAM.

>
>>>
 On Sep 20, 2013, at 11:49 AM, Mircea Markus  wrote:

 Hi,

 Most of the FGAM functionality can be achieved with grouping, by using the 
 FGAM key as a grouping key.
 The single bit that seems to be missing from grouping to equivalent the 
 functionality of FGAM is obtaining all the entries under a single group. 
 IOW a method like:

 Map groupedKeys = Cache.getGroup(groupingKey, KeyFilter);

looks good!
And I assume KeyFilter could be implemented as "accept all" to
degenerate in the FGAM-like semantics?
Specifically what I'd like to confirm is that KeyFilter doesn't have
to be able to enumerate the keys, or in other words that I can
construct one without having a clue about which keys might be stored
in the group.

>
> Just to be clear this is only for the purpose of retrieving multiple
> values, right?  If a user is doing operations on a single key they
> would still use the existing get, remove, replace, etc methods on
> Cache right?  They would just do a put based on the key of the "inner
> map" and they would just have to have a @Group 

Re: [infinispan-dev] Query and dynamic discovery of types

[Sanne Grinovero]

I already explained this ... just 30 months ago ;-)

http://lists.jboss.org/pipermail/infinispan-dev/2011-April/008000.html

Sanne

On 20 September 2013 17:31, Mircea Markus  wrote:
>
> On Sep 20, 2013, at 4:44 PM, Adrian Nistor  wrote:
>
>> Could you detail please?
>
> +1 :-)
>
>>
>> On 09/20/2013 05:43 PM, Sanne Grinovero wrote:
>>> On 20 September 2013 11:19, Mircea Markus  wrote:
>>>> Thanks for the heads up!
>>>>
>>>> It's not clear for me what the functional impact of ISPN-2143 is: 
>>>> incomplete query results?
>>> yes
>>>
>>>> On Sep 19, 2013, at 11:42 AM, Sanne Grinovero  wrote:
>>>>
>>>>> For Infinispan 6.0 we decided that the following issue is a bloker:
>>>>>
>>>>> https://issues.jboss.org/browse/ISPN-2143 - Improve how different
>>>>> indexed caches sync up on new indexed types
>>>>>
>>>>> It really is important, and I'm a bit concerned that it was moved to
>>>>> CR1 as it might not be trivial: for embedded query it needs to store a
>>>>> reference to class definitions (potentially with classloader
>>>>> information too), and for remote queries it needs to distribute the
>>>>> indexing configuration schema.
>>>>>
>>>>> Also I'm wondering if we shouldn't go even further: in Hibernate
>>>>> Search the internal complexity of handling newly appearing types
>>>>> (rather than statically configured types) is very hard.
>>>>>
>>>>> In the next version we might need to drop this, so it would be great
>>>>> if we could stabilize the Infinispan Query API right now in such a way
>>>>> that it won't need this feature anymore.
>>>>>
>>>>> I proposed it years ago but it was rejected because of usability
>>>>> concerns.. would you still reject the idea to simply list the indexed
>>>>> types at configuration time? If so, I think we need to explore
>>>>> alternative solutions.
>>>>>
>>>>> Note that I only want to remove the dependency to dynamic _Class_
>>>>> definitions being added at runtime; we will still support dynamically
>>>>> defined types so in case you need extreme flexibility like with remote
>>>>> queries that will work, as long as you provide a smart bridge able to
>>>>> reconfigure itself based on the dynamic metadata; I think that's a
>>>>> cleaner approach as you would be directly in control of it rather than
>>>>> having to workaround a design which was thought for a different
>>>>> purpose.
>>>>>
>>>>> Sanne
>>>>> ___
>>>>> infinispan-dev mailing list
>>>>> infinispan-dev@lists.jboss.org
>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>> Cheers,
>>>> --
>>>> Mircea Markus
>>>> Infinispan lead (www.infinispan.org)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> ___
>>>> infinispan-dev mailing list
>>>> infinispan-dev@lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>> ___
>>> infinispan-dev mailing list
>>> infinispan-dev@lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>> ___
>> infinispan-dev mailing list
>> infinispan-dev@lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> Cheers,
> --
> Mircea Markus
> Infinispan lead (www.infinispan.org)
>
>
>
>
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] Read Committed Distributed Cache Concerns

[Sanne Grinovero]

On 22 September 2013 13:22, Mircea Markus  wrote:
>
>> On 21 Sep 2013, at 23:07, Sanne Grinovero  wrote:
>>
>>> On 19 September 2013 18:29, Mircea Markus  wrote:
>>> (Adding Jonathan who knows a thing or two about transactions.)
>>>
>>> Given that READ_COMMITTED (RC) is less performant than REPEATABLE_READ (RR)
>>> I don't see any value in keeping RC around. I don't think users rely on
>>> exact RC semantics (i.e. if an entry has been committed then an ongoing
>>> tx requires the most up 2 date value between reads) - that actually
>>> is not the case with DIST caches as you've mentioned.
>>
>> I don't think you can generalize from the specific example William
>> made;
>
> William was reffering to the general case.
>
>> there will still be cases in which READ_COMMITTED will be more
>> efficient than REPEATABLE_READ,
>
> Looking at the implementation, (also as described by William) RC is't faster 
> than RR in the general case. Curious why you think it would be though.

William is describing a potential "fix" for the semantics which would
make it slower than RR, we're arguing that this "fix" is not desired.
Also, I'm not interested in counting method invocations needed to
achieve this: I'm just thinking about the theoretical memory
consumption of RR, which I'd consider more critical.

>> especially if you avoid "fixing" this, as suggested by Radim and
>> myself in the two previous emails [not sure you if saw them, since you
>> forking the conversation ignoring those mails]:
>> if we agree that the current semantics is acceptable, it will
>> consistently be faster than REPEATABLE_READ.
>
> Radim's suggestion was to drop RC after running some tests to validate that 
> RR provides the same performance. You +1 that so I don't understand why you 
> say the conversation was forked.

By "forking" I meant that you asked Jonathan's opinion but without
including our response, so forking the conversation in two parallel
discussions. I assume that was unintentional, but looked like you
might not have seen our responses yet at time of writing yours.
Also, I did "+1" a full paragraph of Radim's comments, not just his
last sentence. Personally I find the initial part more important, so
I'll quote it again:

~   On 19 September 2013 09:06, Radim Vansa  wrote:
~   I think that Read Committed isolation level is not obliged to present
~   you with up-to-date committed data - the only fact is that it can, but
~   application must not rely on that. It's lower isolation level.
~   Nevertheless, I think that lower isolation level should mean better
~   performance. I would be strongly against imposing any additional
~   overhead that could slow it down [...]

+1 for this. As mentioned above, if you're storing data blocks of
non-irrelevant size, and my code is happy reading an older version
even in the same transaction, I don't wish to incur in the performance
penalty imposed by RR.

--Sanne

>
>>
>> Sanne
>>
>>> I think RC is only preferred to RR because of performance, but if the 
>>> performance
>>> is the same (or even worse) I think we should only provide RR. Jonathan, 
>>> care to comment?
>>>
>>>
>>>> On Sep 18, 2013, at 11:03 PM, William Burns  wrote:
>>>>
>>>> I was recently refactoring code dealing with isolation levels and
>>>> found how ReadCommitted is implemented and I have a few concerns I
>>>> wanted to bring up.
>>>>
>>>> ReadCommitted read operations work by storing a reference to the value
>>>> from the data store in its caller's context.  Thus whenever another
>>>> transaction is committed that updates the data store value any context
>>>> that has that reference now sees the latest committed value.  This
>>>> works well for Local and Replicated caches since all data stores are
>>>> updated with the latest value upon completion of the transaction.
>>>> However Distributed caches only the owners see the update in their
>>>> data store and thus any non owner will still have the old value they
>>>> previously read before the commit occurred.
>>>>
>>>> This seems quite inconsistent that Distributed caches run in a mix of
>>>> Repeatable Read/Read Committed depending on what node and what key you
>>>> are using.
>>>>
>>>> To operate properly we could track requests similar to how it works
>>>> for L1 so we can tell non owners to clear out their context values for
>>

Re: [infinispan-dev] replacing the (FineGrained)AtomicMap with grouping

[Sanne Grinovero]

On 22 September 2013 18:05, William Burns  wrote:
> Whoops, I put Oracle instead of MySQL sorry about that.  And to be
> more specific,  Infinispan RR with Pessimistic locking is like MySQL
> RR in that you have non locking repeatable reads with locking writes.
>
> But rereading MySQL docs again I am not sure if it does a db level
> snapshot or only read data (haven't worked with it in quite a long
> while, let alone RR) [1]
>
> I think I put Oracle on accident since I was thinking about it,
> because it doesn't have a RR isolation level :)
>
> Either way I wonder if isolation levels should be named after the SQL
> counterparts or have something more specific to their behavior.
> Coherence for example [2]

Spot on! We should avoid using the same names of the SQL world as they
are often misleading.
Would be nice if Infinispan could propose some better suited names,
then in the documentation where they are explained they could of
course refer to similarities (and differences) with the traditional
ones.
The Coherence docs you linked to seem to do a better job in this case.

--Sanne

>
> [1] http://dev.mysql.com/doc/refman/5.6/en/set-transaction.html
> [2] 
> http://docs.oracle.com/cd/E18686_01/coh.37/e18677/api_transactionslocks.htm#CIHHDHBC
>
> On Sun, Sep 22, 2013 at 6:05 AM, Emmanuel Bernard
>  wrote:
>>
>>
>> On 21 sept. 2013, at 18:32, William Burns  wrote:
>>
 [Technically it would be awesome to be able to be able to rely on RR
 but it doesn't work as in databases - it doesn't snapshot the version
 of entries not touched yet - so we have to compensate at a higher
 layer..]
>>>
>>> The repeatable read implementation is the same as Oracle using multi
>>> versioned entries.  It sounds like what you are talking about is
>>> Serializable which doesn't scale
>>
>> You mean Oracle DB?
>> Then no I don't think you are correct unless something has changed in this 
>> area in Infinispan. Oracle achieves ANSI RR with snapshot isolation - which 
>> offers a contract between RR and SERIALIZABLE.
>> But that's not their default isolation level.
>>
>> Regardless of that my understanding of Infinispan's implementation is that 
>> in the group approach - and I believe FGAM-, I could update different 
>> "columns" of the same "row" - in OGM terms - in parallel provided that the 
>> columns are different. This is not the case of RDBMS RR as I understand it. 
>> To work around that your application must have a version column that you 
>> update for each change.
>>
>> I am a bit rusty on these subjects so feel free to correct me.
>> ___
>> infinispan-dev mailing list
>> infinispan-dev@lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

[infinispan-dev] Hibernate HQL Parser 1.0.0.Alpha4 released

[Sanne Grinovero]

We just tagged and published another alpha for the new ANTLR3 based HQL parser.

Best,
Sanne
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

[infinispan-dev] ISPN-3557: interactions between a clear() operation and a Transaction

[Sanne Grinovero]

I'd love to brainstorm about the clear() operation and what it means
on Infinispan.

I'm not sure to what extent, but it seems that clear() is designed to
work in a TX, or even create an implicit transaction if needed, but
I'm not understanding how that can work.

Obviously a clear() operation isn't listing all keys explicitly. Which
implies that it's undefined on which keys it's going to operate when
it's fired.. that seems like terribly wrong in a distributed key/value
store as we can't possibly freeze the global state and somehow define
a set of keys which are going to be affected, while an explicit
enumeration is needed to acquire the needed locks.

It might give a nice safe feeling that, when invoking a clear()
operation in a transaction, I can still abort the transaction to make
it cancel the operation; that's the only good part I can think of: we
can cancel it.

I don't think it has anything to do with consistency though? To make
sure you're effectively involving all replicas of all entries in a
consistent way, a lock would need to be acquired on each affected key,
which again implies a need to enumerate all keys, including the
unknown keys which might be hiding in a CacheStore: it's not enough to
broadcast the clear() operation to all nodes and have them simply wipe
their local state as that's never going to deal correctly
(consistently) with in-flight transactions working on different nodes
at different times (I guess enabling Total Order could help but you'd
need to make it mandatory).

So let's step back a second and consider what is the use case for
clear() ? I suspect it's primarily a method needed during testing, or
maybe cleanup before a backup is restored (operations), maybe a
manually activated JMX operation to clear the cache in exceptional
cases.

I don't think there would ever be a need for a clear() operation to
interact with other transactions, so I'd rather make it illegal to
invoke a clear() inside a transaction, or simply ignore the
transactional scope and have an immediate and distributed effect.

I'm likely missing something. What terrible consequences would this have?

Cheers,
Sanne
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

[infinispan-dev] Which QueryBuilder ?

[Sanne Grinovero]

It seems we have now 2 different interfaces both names "QueryBuilder"
when using Infinispan Query.
One is coming from Hibernate Search, and represents the "classic" way
to build queries for Infinispan Query in embedded mode.

The other one is new, and represents the simplified approach, also
implemented for remote queries.

Could we find an alternative name for the new API?

It's certainly going to be confusing, even more when we'll have to
document the differences, and which one is more suited for one use
cases vs. another.

Cheers,
Sanne
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] ISPN-3557: interactions between a clear() operation and a Transaction

[Sanne Grinovero]

On 3 October 2013 10:29, Dan Berindei  wrote:
>
> On Wed, Oct 2, 2013 at 12:48 PM, Pedro Ruivo  wrote:
>>
>>
>>
>> On 10/02/2013 12:03 AM, Sanne Grinovero wrote:
>> > I'd love to brainstorm about the clear() operation and what it means
>> > on Infinispan.
>> >
>> > I'm not sure to what extent, but it seems that clear() is designed to
>> > work in a TX, or even create an implicit transaction if needed, but
>> > I'm not understanding how that can work.
>
>
> I think it's pretty straight-forward to define the semantics of operations
> after clear(): they should assume that every key in the cache was removed.
>
> True, like Pedro mentioned, the current implementation allows subsequent
> operations in the same tx to see keys inserted by other txs. But I don't
> think fixing that would be harder than ensuring that if a tx inserts key k
> owned by nodes A and B, clear() from outside the tx either deletes k on A
> and B or doesn't delete k on either node.
>
>
>>
>> >
>> > Obviously a clear() operation isn't listing all keys explicitly.
>>
>> (offtopic) like entrySet(), keySet(), values() and iterator() when
>> executed in a transactional context. however, I think that this
>> operations are more difficult to handle, because they have to return
>> consistent data...
>
>
> I agree, defining how entrySet() and the others should work in a tx seems
> harder than defining clear().

I agree, those other nasty beasts are next on my list: see below..
But we're not arguing about which method deserves the gold medal for
complex design ;-)


> Will, I'm guessing your fix for ISPN-3560 still allows entrySet() after
> clear() to see entries inserted by another tx in the meantime?
>
>>
>> > Which
>> > implies that it's undefined on which keys it's going to operate when
>> > it's fired.. that seems like terribly wrong in a distributed key/value
>> > store as we can't possibly freeze the global state and somehow define
>> > a set of keys which are going to be affected, while an explicit
>> > enumeration is needed to acquire the needed locks.
>>
>> you may not need to explicit acquire the lock for the affected key. you
>> can use a global lock that is share acquired for write transactions
>> without clear operation and exclusively acquire for transactions with
>> clear operation (exactly what I use for Total Order). also, the clear
>> transactions (i.e. the transactions with clear operation) can also only
>> acquired the global lock.
>
>
> +1 for the global lock. It may slow things down a bit in the general case,
> because every tx will have to acquire it in shared mode, but it will
> guarantee consistency for clear().

I had thought of that too, but it will slow down all operations, at
which benefit?
To have a nice transactional clear() operation which guarantees consistency?

The goal of this thread is to challenge if that makes sense for real use cases,
it seems in fact we would highly prefer a non transactional version,
and considering that we can't run a single Cache in both modes anymore
that's a problem.

Side note on the implementation complexity:
even the AsyncCacheStore decorator code is made extremely more tricky
just to handle clear() appropriately, to keep it in the correct order
of execution
compared to the other operations, while still allowing
"coaleshing" of other operations, which essentially implies we allow
reordering of operations as long as they affect independent keys.. where clear()
is of course a jolly which requires it's own execution path, its own global lock
and generally makes it far more complex.

>
> It still won't help entrySet() and its brethren...
>
>>
>> >
>> > It might give a nice safe feeling that, when invoking a clear()
>> > operation in a transaction, I can still abort the transaction to make
>> > it cancel the operation; that's the only good part I can think of: we
>> > can cancel it.
>>
>> yep
>
>
> Wouldn't it be harder to integrate it with Total Order if it didn't have a
> tx?
>
> But most of all, I think we have to keep it because it's a part of the Map
> interface and a part of JCache's Cache interface.
>
>>
>>
>> >
>> > I don't think it has anything to do with consistency though? To make
>> > sure you're effectively involving all replicas of all entries in a
>> > consistent way, a lock would need to be acquired on each affected key,
>> > which again implies a need to enumerate all keys, 

Re: [infinispan-dev] Which QueryBuilder ?

[Sanne Grinovero]

On 3 October 2013 14:10, Adrian Nistor  wrote:
> I know, was just joking. Anyway, I don't see any confusion having two
> classes with the same name.

It's going to be hard enough to explain to people why we are providing
two different approaches, if we can't even think of a different name
to properly highlight the different usage then we have a problem.

Try figuring the forum / support question "I'm using QueryBuilder on
Infinispan 7.3 and this happens... "

The javadoc index will have it listed twice -> annoying.

Google for "QueryBuilder Infinispan" -> annoying

Or try figuring out the documentation:

# Chapter 5: Queries.
There are two approaches to run Queries in Infinispan. Either you use
the QueryBuilder, which provides simple domain oriented properties and
can work both in embedded and remote mode, or you use the more
powerfull QueryBuilder.

# 5.1 QueryBuilder
blah blah

# 5.2 QueryBuilder
blah blah


If they are different, the should really have different names, even
just to avoid confusion among ourselves when talking about hem. If you
feel they're the same, the interesting alternative is to literally
merge them in one single interface, potentially exposing multiple
methods.

Sanne

>
> On 10/03/2013 02:29 PM, Emmanuel Bernard wrote:
>> It's already productized code.
>>
>> On Thu 2013-10-03 14:16, Adrian Nistor wrote:
>>> I would suggest renaming the old one :))
>>>
>>> On 10/02/2013 11:13 PM, Sanne Grinovero wrote:
>>>> It seems we have now 2 different interfaces both names "QueryBuilder"
>>>> when using Infinispan Query.
>>>> One is coming from Hibernate Search, and represents the "classic" way
>>>> to build queries for Infinispan Query in embedded mode.
>>>>
>>>> The other one is new, and represents the simplified approach, also
>>>> implemented for remote queries.
>>>>
>>>> Could we find an alternative name for the new API?
>>>>
>>>> It's certainly going to be confusing, even more when we'll have to
>>>> document the differences, and which one is more suited for one use
>>>> cases vs. another.
>>>>
>>>> Cheers,
>>>> Sanne
>>>> ___
>>>> infinispan-dev mailing list
>>>> infinispan-dev@lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>> ___
>>> infinispan-dev mailing list
>>> infinispan-dev@lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> ___
>> infinispan-dev mailing list
>> infinispan-dev@lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

[infinispan-dev] Infinispan Query API changes

[Sanne Grinovero]

Hi,
just a heads up that I've been moving some APIs around from
org.infinispan.query:

https://github.com/infinispan/infinispan/pull/2131

In particular I think that CapeDwarf is using some methods from

org.infinispan.query.SearchManager

which are no longer exposed; the fix is simple, cast any SearchManager
you might have into a
org.infinispan.query.spi.SearchManagerImplementor

SearchManagerImplementor is meant to be an SPI, so a relatively stable
interface for projects like CD but not expected to be used by every
day users.

Sanne
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

[infinispan-dev] Configuration compatibility expectations ?

[Sanne Grinovero]

Hello,
I just tried to undust an old test I had shelved. When I run it I got
an exception about the configuration having an unexpected header:

http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="urn:infinispan:config:5.2
http://www.infinispan.org/schemas/infinispan-config-5.2.xsd";
xmlns="urn:infinispan:config:5.2">

Everything works fine after I change it to:

http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="urn:infinispan:config:6.0
http://www.infinispan.org/schemas/infinispan-config-6.0.xsd";
xmlns="urn:infinispan:config:6.0">

Is that expected?

Sanne
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] Configuration compatibility expectations ?

[Sanne Grinovero]

Ok, as long as it's expected I guess it's fine.
Thanks!
Sanne

On 3 October 2013 18:32, Tristan Tarrant  wrote:
> You need the 52x compatibility jar.
>
> Tristan
>
> On 10/03/2013 06:53 PM, Sanne Grinovero wrote:
>> Hello,
>> I just tried to undust an old test I had shelved. When I run it I got
>> an exception about the configuration having an unexpected header:
>>
>> >  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
>>  xsi:schemaLocation="urn:infinispan:config:5.2
>> http://www.infinispan.org/schemas/infinispan-config-5.2.xsd";
>>  xmlns="urn:infinispan:config:5.2">
>>
>> Everything works fine after I change it to:
>>
>> >  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
>>  xsi:schemaLocation="urn:infinispan:config:6.0
>> http://www.infinispan.org/schemas/infinispan-config-6.0.xsd";
>>  xmlns="urn:infinispan:config:6.0">
>>
>> Is that expected?
>>
>> Sanne
>> ___
>> infinispan-dev mailing list
>> infinispan-dev@lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>>
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

[infinispan-dev] Warnings vs. Fail Fast

[Sanne Grinovero]

Currently if a cache is configured with indexing enabled, but the
Query module isn't on classpath, you get a simple warning.

I think this should fail with a configuration validation error; it's
not just safer but also consistent with many other validations.

I've created ISPN-3583 and patch is ready.. any good reason to not apply it?

Sanne
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] Warnings vs. Fail Fast

[Sanne Grinovero]

Good point, will change that.

I've actually been experimenting with making a CheckStyle rule which
will detect such violations, but there are so many that I won't send a
pull with that :-)
At some point when we'll have a more relaxed pace I'll send a
checkstyle proposal to log warnings rather than fail the build, so we
can then change them gradually.

Sanne

On 4 October 2013 12:03, Dan Berindei  wrote:
> +1 to apply it, but maybe we should use the occasion to move the error
> message to the Log interface as well.
>
>
> On Fri, Oct 4, 2013 at 2:43 AM, Sanne Grinovero 
> wrote:
>>
>> Currently if a cache is configured with indexing enabled, but the
>> Query module isn't on classpath, you get a simple warning.
>>
>> I think this should fail with a configuration validation error; it's
>> not just safer but also consistent with many other validations.
>>
>> I've created ISPN-3583 and patch is ready.. any good reason to not apply
>> it?
>>
>> Sanne
>> ___
>> infinispan-dev mailing list
>> infinispan-dev@lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] Need help

[Sanne Grinovero]

Hi Pedro,
looks like you're diving in some good fun :-)
BTW please keep the dev discussions on the mailing list, adding it.

inline :

On 4 October 2013 22:01, Pedro Ruivo  wrote:
> Hi,
>
> Sanne I need your expertise in here. I'm afraid that the problem is in
> FileListOperations :(
> I think the FileListOperations implementation needs a transactional cache
> with strong consistency...
>
> I'm 99% sure that it is originating the java.lang.AssertionError: file XPTO
> does not exist. I find out that we have multiple threads adding and removing
> files from the list. The scenario in [1] we see 2 threads loading the key
> from the cache loader and one thread adds a file and other removes. the
> thread that removes is the last one to commit and the file list is updated
> to an old state. When it tries to updat an index, I got the assertion error.

Nice, looks like you're on something.
I've never seen specifically an AssertionError, looks like you have a
new test. Could you share it?

Let's step back a second and consider the Cache usage from the point
of view of FileListOperations.
Note that even if you have two threads writing at the same time, as
long as they are on the same node they will be adding/removing
elements from the same instance of a ConcurrentHashMap.
Since it's the same instance, it doesn't matter which thread will do
the put operation as last: it will push the correct state.
(there is an assumptions here, but we can forget about those for the
sake of this debugging: same node -> fine as there is an external
lock, no other node is allowed to write at the same time)

But you are focusing on CacheStore operations, I can see how that
might be different in terms of implementation but it is not acceptable
that the CacheStore is storing a different state than what we have in
memory.
I don't expect to need a Transaction for that ? Writes need *always*
to be applied in the right order so that the CacheStore content
matches the in-memory content.

So -1 for the problem being in FileListOperations, it's in the
CacheStore. Also, I've run plenty of stress tests on in-memory Caches
and never hit problems: if Infinispan changes the semantics by
enabling a CacheStore, that's a critical issue.

Also, this needs to work correctly with async cachestores.


> Also, I was able to "reproduce" the EOF. This was the first problem I found
> and it is related to DefaultCacheManager.startCaches(String... cacheName),
> that is starting the caches in separated threads. The SingleFileStore is
> failing to start but the exception in "swallow" by the thread. So, Hibernate
> Search is not notified and it uses the cache anyway. To make it worst, the
> cache accepts the requests but it is not persisting the data. This creates
> the EOF in the restart... I will open a JIRA about it to discuss it (maybe
> throw an exception in startCaches? and throw exception if any operation is
> invoked in a non-successfull started cache?)

+1 on the exception on startCaches, should not be swallowed!

But why is SingleFileStore throwing an exception?

Thanks a lot, very interesting.

Sanne

>
> Any comments?
>
> If I was not clear let me know :)
>
> Thanks!
> Pedro
>
> [1]https://gist.github.com/pruivo/93edeb82a21e9827d2c9
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

[infinispan-dev] Running stress tests on CI ?

[Sanne Grinovero]

Hi all,
the following change introduced a critical issue in the Lucene Directory:

   final Set filesList = fileOps.getFileList();
-  String[] array = filesList.toArray(new String[0]);
-  return array;
+ return filesList.toArray(new String[filesList.size()]);

I'll leave it as a puzzler to figure why the change is able to cause trouble ;-)

This generates a NPE in just a single second of running one of the
stress tests or performance tests, but I'm guilty of not being able to
make a normal unit test for this case.

That module contains such limited code, that in the very rare
occasions in which I apply some changes I re-run the included
benchmarks; I realize I can't expect that from all of you, so..

Should we enable some stress tests on CI?

As a side warning consequence of this, the Lucene Directory in release
6.0.0.CR1 is very unreliable [ISPN-3592].

--Sanne
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] Need help

[Sanne Grinovero]

On 6 October 2013 00:01, Pedro Ruivo  wrote:
> Hi Sanne.
>
> Thanks for your comments. please see inline...
>
> Cheers,
> Pedro
>
>
> On 10/05/2013 09:15 PM, Sanne Grinovero wrote:
>>
>> Hi Pedro,
>> looks like you're diving in some good fun :-)
>> BTW please keep the dev discussions on the mailing list, adding it.
>>
>> inline :
>>
>> On 4 October 2013 22:01, Pedro Ruivo  wrote:
>>>
>>> Hi,
>>>
>>> Sanne I need your expertise in here. I'm afraid that the problem is in
>>> FileListOperations :(
>>> I think the FileListOperations implementation needs a transactional cache
>>> with strong consistency...
>>>
>>> I'm 99% sure that it is originating the java.lang.AssertionError: file
>>> XPTO
>>> does not exist. I find out that we have multiple threads adding and
>>> removing
>>> files from the list. The scenario in [1] we see 2 threads loading the key
>>> from the cache loader and one thread adds a file and other removes. the
>>> thread that removes is the last one to commit and the file list is
>>> updated
>>> to an old state. When it tries to updat an index, I got the assertion
>>> error.
>>
>>
>> Nice, looks like you're on something.
>> I've never seen specifically an AssertionError, looks like you have a
>> new test. Could you share it?
>
>
> yes of course:
> https://github.com/pruivo/infinispan/blob/a4483d08b92d301350823c7fd42725c339a65c7b/query/src/test/java/org/infinispan/query/cacheloaders/CacheStoreTest.java
>
> so far, only the tests with eviction are failing...

Some of the failures you're seeing are caused by internal "assert"
keyword in Lucene's code, which have the purpose of verifying the
"filesystem" is going to be synched properly.
These assertions don't apply when using our storage: we don't need
this synch to happen: in fact if it weren't because of the assertions
the whole method would be a no-op as it finally delegates all logic to
a method in the Infinispan Directory which is a no-op.

In other words, these are misleading failures and we'd need to avoid
the TestNG "feature" of enabling assertions in this case.

Still, even if the stacktrace is misleading, I agree on your diagnosis below.

Could you reproduce the problem without involving also the Query framework?
I'd hope that such a test could be independent and live solely in the
lucene-directory module; in practice if you can only reproduce it with
the query module it makes me suspicious that we're actually debugging
a race condition in the initialization of the two services: a race
between the query initialization thread needing to check the index
state (so potentially triggering a load from cachestore), and the
thread performing the cachestore preload.
(I see your test also fails without preload, but just wondering if
that might be an additional complexity).

>>
>> Let's step back a second and consider the Cache usage from the point
>> of view of FileListOperations.
>> Note that even if you have two threads writing at the same time, as
>> long as they are on the same node they will be adding/removing
>> elements from the same instance of a ConcurrentHashMap.
>> Since it's the same instance, it doesn't matter which thread will do
>> the put operation as last: it will push the correct state.
>> (there is an assumptions here, but we can forget about those for the
>> sake of this debugging: same node -> fine as there is an external
>> lock, no other node is allowed to write at the same time)
>>
>
> 100% agreed with you but with cache store, we no longer ensure that 2 (or
> more) threads are pointing to the same instance of Concurrent Hash Set.
>
> With eviction, the entries are removed from in-memory container and
> persisted in the cache store. The scenario I've described, 2 threads are
> trying to add/remove a file and the file list does not exist in-memory. So,
> each thread will read from cache store and deserialize the byte array. In
> the end, each thread will have a pointer for different instances of
> ConcurrentHashSet but with the same elements. And when this happens, we lost
> one of the operation.

I'm seeing more than a couple of different smelly behaviors interacting here:

1## The single instance ConcurrentHashSet
I guess this could be re-thought as it's making some strong
assumptions, but considering this service can't be transactional I'd
rather explore other solutions first as I think the following changes
should be good enough.

2## Eviction not picking the right entry
This single key is literall

Re: [infinispan-dev] Need help

[Sanne Grinovero]

On 7 October 2013 10:44, Dan Berindei  wrote:
>
>
>
> On Mon, Oct 7, 2013 at 2:30 AM, Sanne Grinovero 
> wrote:
>>
>> On 6 October 2013 00:01, Pedro Ruivo  wrote:
>> > Hi Sanne.
>> >
>> > Thanks for your comments. please see inline...
>> >
>> > Cheers,
>> > Pedro
>> >
>> >
>> > On 10/05/2013 09:15 PM, Sanne Grinovero wrote:
>> >>
>> >> Hi Pedro,
>> >> looks like you're diving in some good fun :-)
>> >> BTW please keep the dev discussions on the mailing list, adding it.
>> >>
>> >> inline :
>> >>
>> >> On 4 October 2013 22:01, Pedro Ruivo  wrote:
>> >>>
>> >>> Hi,
>> >>>
>> >>> Sanne I need your expertise in here. I'm afraid that the problem is in
>> >>> FileListOperations :(
>> >>> I think the FileListOperations implementation needs a transactional
>> >>> cache
>> >>> with strong consistency...
>> >>>
>> >>> I'm 99% sure that it is originating the java.lang.AssertionError: file
>> >>> XPTO
>> >>> does not exist. I find out that we have multiple threads adding and
>> >>> removing
>> >>> files from the list. The scenario in [1] we see 2 threads loading the
>> >>> key
>> >>> from the cache loader and one thread adds a file and other removes.
>> >>> the
>> >>> thread that removes is the last one to commit and the file list is
>> >>> updated
>> >>> to an old state. When it tries to updat an index, I got the assertion
>> >>> error.
>> >>
>> >>
>> >> Nice, looks like you're on something.
>> >> I've never seen specifically an AssertionError, looks like you have a
>> >> new test. Could you share it?
>> >
>> >
>> > yes of course:
>> >
>> > https://github.com/pruivo/infinispan/blob/a4483d08b92d301350823c7fd42725c339a65c7b/query/src/test/java/org/infinispan/query/cacheloaders/CacheStoreTest.java
>> >
>> > so far, only the tests with eviction are failing...
>>
>> Some of the failures you're seeing are caused by internal "assert"
>> keyword in Lucene's code, which have the purpose of verifying the
>> "filesystem" is going to be synched properly.
>> These assertions don't apply when using our storage: we don't need
>> this synch to happen: in fact if it weren't because of the assertions
>> the whole method would be a no-op as it finally delegates all logic to
>> a method in the Infinispan Directory which is a no-op.
>>
>> In other words, these are misleading failures and we'd need to avoid
>> the TestNG "feature" of enabling assertions in this case.
>>
>> Still, even if the stacktrace is misleading, I agree on your diagnosis
>> below.
>>
>> Could you reproduce the problem without involving also the Query
>> framework?
>> I'd hope that such a test could be independent and live solely in the
>> lucene-directory module; in practice if you can only reproduce it with
>> the query module it makes me suspicious that we're actually debugging
>> a race condition in the initialization of the two services: a race
>> between the query initialization thread needing to check the index
>> state (so potentially triggering a load from cachestore), and the
>> thread performing the cachestore preload.
>> (I see your test also fails without preload, but just wondering if
>> that might be an additional complexity).
>>
>> >>
>> >> Let's step back a second and consider the Cache usage from the point
>> >> of view of FileListOperations.
>> >> Note that even if you have two threads writing at the same time, as
>> >> long as they are on the same node they will be adding/removing
>> >> elements from the same instance of a ConcurrentHashMap.
>> >> Since it's the same instance, it doesn't matter which thread will do
>> >> the put operation as last: it will push the correct state.
>> >> (there is an assumptions here, but we can forget about those for the
>> >> sake of this debugging: same node -> fine as there is an external
>> >> lock, no other node is allowed to write at the same time)
>> >>
>> >
>> > 100% agreed with you but with cache store, we no longer ensure that 2
>> > (or
&

Re: [infinispan-dev] Need help

[Sanne Grinovero]

On 7 October 2013 11:12, Pedro Ruivo  wrote:
>
>
> On 10/07/2013 12:30 AM, Sanne Grinovero wrote:
>>
>> On 6 October 2013 00:01, Pedro Ruivo  wrote:
>>>
>>> Hi Sanne.
>>>
>>> Thanks for your comments. please see inline...
>>>
>>> Cheers,
>>> Pedro
>>>
>>>
>>> On 10/05/2013 09:15 PM, Sanne Grinovero wrote:
>>>>
>>>>
>>>> Hi Pedro,
>>>> looks like you're diving in some good fun :-)
>>>> BTW please keep the dev discussions on the mailing list, adding it.
>>>>
>>>> inline :
>>>>
>>>> On 4 October 2013 22:01, Pedro Ruivo  wrote:
>>>>>
>>>>>
>>>>> Hi,
>>>>>
>>>>> Sanne I need your expertise in here. I'm afraid that the problem is in
>>>>> FileListOperations :(
>>>>> I think the FileListOperations implementation needs a transactional
>>>>> cache
>>>>> with strong consistency...
>>>>>
>>>>> I'm 99% sure that it is originating the java.lang.AssertionError: file
>>>>> XPTO
>>>>> does not exist. I find out that we have multiple threads adding and
>>>>> removing
>>>>> files from the list. The scenario in [1] we see 2 threads loading the
>>>>> key
>>>>> from the cache loader and one thread adds a file and other removes. the
>>>>> thread that removes is the last one to commit and the file list is
>>>>> updated
>>>>> to an old state. When it tries to updat an index, I got the assertion
>>>>> error.
>>>>
>>>>
>>>>
>>>> Nice, looks like you're on something.
>>>> I've never seen specifically an AssertionError, looks like you have a
>>>> new test. Could you share it?
>>>
>>>
>>>
>>> yes of course:
>>>
>>> https://github.com/pruivo/infinispan/blob/a4483d08b92d301350823c7fd42725c339a65c7b/query/src/test/java/org/infinispan/query/cacheloaders/CacheStoreTest.java
>>>
>>> so far, only the tests with eviction are failing...
>>
>>
>> Some of the failures you're seeing are caused by internal "assert"
>> keyword in Lucene's code, which have the purpose of verifying the
>> "filesystem" is going to be synched properly.
>> These assertions don't apply when using our storage: we don't need
>> this synch to happen: in fact if it weren't because of the assertions
>> the whole method would be a no-op as it finally delegates all logic to
>> a method in the Infinispan Directory which is a no-op.
>>
>> In other words, these are misleading failures and we'd need to avoid
>> the TestNG "feature" of enabling assertions in this case.
>>
>> Still, even if the stacktrace is misleading, I agree on your diagnosis
>> below.
>>
>> Could you reproduce the problem without involving also the Query
>> framework?
>
>
> yes, I think I could
>
>
>> I'd hope that such a test could be independent and live solely in the
>> lucene-directory module; in practice if you can only reproduce it with
>> the query module it makes me suspicious that we're actually debugging
>> a race condition in the initialization of the two services: a race
>> between the query initialization thread needing to check the index
>> state (so potentially triggering a load from cachestore), and the
>> thread performing the cachestore preload.
>> (I see your test also fails without preload, but just wondering if
>> that might be an additional complexity).
>
>
> the assertion is triggered during the putData() method. but I believe that
> the same problem can happen in the cache restart without preload.

Ok I was just trying to figure out why you might have needed the test
to include the Query module.
Looking forward to a test needing Directory only then :-)


>>>> Let's step back a second and consider the Cache usage from the point
>>>> of view of FileListOperations.
>>>> Note that even if you have two threads writing at the same time, as
>>>> long as they are on the same node they will be adding/removing
>>>> elements from the same instance of a ConcurrentHashMap.
>>>> Since it's the same instance, it doesn't matter which thread will do
>>>> the put operation as last: it will push the correct state.
>>>> (there is an assumptions here, but w

Re: [infinispan-dev] Running stress tests on CI ?

[Sanne Grinovero]

Nice!
Is that going to work out with our hardware?

We'll need to be careful now with the configured duration of each such test.

On 7 October 2013 13:48, Dan Berindei  wrote:
> I've created a build in CI for the stress tests:
>
> http://ci.infinispan.org/viewType.html?buildTypeId=Infinispan_StressHotspotJdk6
>
> Cheers
> Dan
>
>
>
> On Sun, Oct 6, 2013 at 12:07 AM, Sanne Grinovero 
> wrote:
>>
>> Hi all,
>> the following change introduced a critical issue in the Lucene Directory:
>>
>>final Set filesList = fileOps.getFileList();
>> -  String[] array = filesList.toArray(new String[0]);
>> -  return array;
>> + return filesList.toArray(new String[filesList.size()]);
>>
>> I'll leave it as a puzzler to figure why the change is able to cause
>> trouble ;-)
>>
>> This generates a NPE in just a single second of running one of the
>> stress tests or performance tests, but I'm guilty of not being able to
>> make a normal unit test for this case.
>>
>> That module contains such limited code, that in the very rare
>> occasions in which I apply some changes I re-run the included
>> benchmarks; I realize I can't expect that from all of you, so..
>>
>> Should we enable some stress tests on CI?
>>
>> As a side warning consequence of this, the Lucene Directory in release
>> 6.0.0.CR1 is very unreliable [ISPN-3592].
>>
>> --Sanne
>> ___
>> infinispan-dev mailing list
>> infinispan-dev@lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] Need help

[Sanne Grinovero]

On 7 October 2013 13:20, Pedro Ruivo  wrote:
>
>
> On 10/07/2013 11:43 AM, Sanne Grinovero wrote:
>>
>>
>> Proposal:
>> let's be more aggressive on validation.
>>   - make preload mandatory (for the metadata only)
>>   - eviction not allowed (for the metadata only)
>>
>> Note that I don't think we're putting much of a restriction to users,
>> we're more likely helping
>> with good guidance on how these caches are supposed to work.
>> I think it's a good thing to narrow down the possible configuration
>> options
>> and make it clear what we expect people to use for this purpose.
>
>
> it's good enough for me. :)
>
>
>>
>>
>>>> 3## The CacheLoader loading the same entry multiple times in parallel
>>>> Kudos for finding out that there are situations in which we deal with
>>>> multiple different instances of ConcurrentHashSet! Still, I think that
>>>> Infinispan core is terribly wrong in this case:
>>>> from the client code POV a new CHS is created with a put-if-absent
>>>> atomic operation, and I will assume there that core will check/load
>>>> any enabled cachestore as well.
>>>> To handle multiple GET operations in parallel, or even in parallel
>>>> with preload or the client's put-if-absent operation, I would *not*
>>>> expect Infinispan core to storm the CacheStore implementation with
>>>> multiple load operations on the same put: a lock should be hold on the
>>>> potential key during such a load operation.
>>>
>>>
>>>
>>> I didn't understand what you mean here ^^. Infinispan only tries to load
>>> from a cache store once per operation (get, put, remove, etc...).
>>>
>>> however, I think we have a big windows in which multiple operations can
>>> load
>>> the same key from the store. This happens because we only write to the
>>> data
>>> container after the operation end.
>>
>>
>> I think that's a very important consideration.
>>
>> Imagine a REST cache is built on Infinispan, using a filesystem based
>> CacheStore, and we receive a million requests per second
>> for a key "X1", which doesn't exist.
>> Since each request is not going to find it in the data container, each
>> request is triggering a CacheStore load.
>> Do you want to enqueue a million IO operations to the disk?
>> I'd much rather have some locking in memory, that will prevent nut
>> just disks from struggling but also allow the CPUs to context switch
>> to perform more useful tasks: being blocked on a lock is a good thing
>> in this case, rather than allowing "parallel processing of pointless
>> requests". That's just parallel energy burning, and potentially DOSing
>> yourself regarding more interesting requests which could be handled
>> instead.
>>
>> To translate it into an implementation design, you need to lock the
>> "X1" key before attempting to load it from the CacheStore, and when
>> the lock is acquired the data container needs to be re-checked as it
>> might have been loaded by a parallel thread.
>> Or alternatively, lock the key before attempting to read from the data
>> container: that's not too bad, it's a very brief local only lock.
>
>
> you have good point here, but IMO this is a high risk operation because we
> are getting close in the final release. and I don't want to make the cache
> store unusable...

I understand there is risk, but this should have been done long ago.
AFAIR one of the goals for 6.0 is to improve performance of CacheStores,
this is one of the many changes going in that direction.
Besides, I wouldn't be too worried as the CacheStores API is entirely
new and most people will only provide feedback after the Final release,
so this is not an entirely unfortunate timing.

>
> but, let's see what others think about it.
>
> my 1 cent pseudo-code suggestion:
>
> lock(key) {
>   value = dataContainer.get(key)
>   if value is not null; then return value
>   loaded = cacheLoader.load(key)
>   if loaded is null; then return null
>   existing = dataContainer.putIfAbsent(key, loaded)
>   if existing is null; then return loaded; otherwise existing
>
> }

+1 for something similar. Not sure why you need a putIfAbsent since
you have the lock, but that's a detail.

An alternative with a lockfree fast-path:

value = dataContainer.get(key)
if (value!=null) return value
lock(key) {
value = dataContainer.get(key)
if value is not null; then return value
loaded = cacheLoader.load(key)
if loaded is null; then return null
existing = dataContainer.put(key, loaded)
return loaded
}

Wouldn't be much of a benefit for the example I made above until we
switch to using a placeholder token (tombstone) for null values,
but there is great value in this pattern for existing entries.

Sanne
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] Which QueryBuilder ?

[Sanne Grinovero]

On 9 October 2013 13:30, Adrian Nistor  wrote:
> Hi Sanne and Martin,
>
> these come from different products.

No they come from different projects, but are part of the same product.

> Would they really appear in the same documentation and javadoc?

As far as I understood, and unless you do something about it, yes.

> Are usually used in different contexts too,
> although there might be users wanting to use both HS QB and ISPN QB in
> the same context, so to ease their pain I suggest using DslQueryBuilder
> in ISPN. Would that fit?

I don't think we should name things explicitly as DSL.

> But if we start on this route should we also do something about
> org.infinispan.query.dsl.Query?

Right, that's probably the culprit. If we rename that to something
which better expresses the simplified/limited form, it might directly
solve the problem of naming its builder.
Something on the lines of Criteria / Filtering ?
I primarily see it as a Filter Stack, and if we keep it conceptually
close to that, it makes it easier for the planned retro-fitting in the
MR framework.

> I wish we could do something about
> org.infinispan.configuration.cache.ConfigurationBuilder /
> org.infinispan.client.hotrod.configuration.ConfigurationBuilder and
> several other examples of colliding names but I guess it's too late for
> them.

You still have the power to address it.
It's quite late yes, but it's too late only after it's released.
Releasing in this shape is imho very bad.

Sanne

>
> Adrian
>
> On 10/08/2013 06:32 PM, Martin Gencur wrote:
>> Right Sanne,
>> it's already starting to be a documentation problem for the product.
>> It's really confusing unless you developed one of these APIs and
>> precisely know the differences :) Any time there's a code snippet, it
>> must contain the package (usually this is not needed), but even then
>> it's easy to get confused.
>>
>> Martin
>>
>>
>> On 3.10.2013 16:48, Sanne Grinovero wrote:
>>> On 3 October 2013 14:10, Adrian Nistor  wrote:
>>>> I know, was just joking. Anyway, I don't see any confusion having two
>>>> classes with the same name.
>>> It's going to be hard enough to explain to people why we are providing
>>> two different approaches, if we can't even think of a different name
>>> to properly highlight the different usage then we have a problem.
>>>
>>> Try figuring the forum / support question "I'm using QueryBuilder on
>>> Infinispan 7.3 and this happens... "
>>>
>>> The javadoc index will have it listed twice -> annoying.
>>>
>>> Google for "QueryBuilder Infinispan" -> annoying
>>>
>>> Or try figuring out the documentation:
>>>
>>> # Chapter 5: Queries.
>>> There are two approaches to run Queries in Infinispan. Either you use
>>> the QueryBuilder, which provides simple domain oriented properties and
>>> can work both in embedded and remote mode, or you use the more
>>> powerfull QueryBuilder.
>>>
>>> # 5.1 QueryBuilder
>>> blah blah
>>>
>>> # 5.2 QueryBuilder
>>> blah blah
>>>
>>>
>>> If they are different, the should really have different names, even
>>> just to avoid confusion among ourselves when talking about hem. If you
>>> feel they're the same, the interesting alternative is to literally
>>> merge them in one single interface, potentially exposing multiple
>>> methods.
>>>
>>> Sanne
>>>
>>>> On 10/03/2013 02:29 PM, Emmanuel Bernard wrote:
>>>>> It's already productized code.
>>>>>
>>>>> On Thu 2013-10-03 14:16, Adrian Nistor wrote:
>>>>>> I would suggest renaming the old one :))
>>>>>>
>>>>>> On 10/02/2013 11:13 PM, Sanne Grinovero wrote:
>>>>>>> It seems we have now 2 different interfaces both names "QueryBuilder"
>>>>>>> when using Infinispan Query.
>>>>>>> One is coming from Hibernate Search, and represents the "classic" way
>>>>>>> to build queries for Infinispan Query in embedded mode.
>>>>>>>
>>>>>>> The other one is new, and represents the simplified approach, also
>>>>>>> implemented for remote queries.
>>>>>>>
>>>>>>> Could we find an alternative name for the new API?
>>>>>>>
>>>>>>> It's certainly going to be 

Re: [infinispan-dev] Running stress tests on CI ?

[Sanne Grinovero]

That works as long as someone runs the stress tests before tagging a release.

Sanne

On 9 October 2013 14:19, Dan Berindei  wrote:
> We don't have a configured maximum duration for the stress tests. The stress
> build take about 13 minutes now, but enabling all the stress tests will
> probably increase the duration.
>
> I didn't want to delay regular builds, I've scheduled it to run every night
> at 03:00 GMT instead of running after every commit.
>
>
> On Mon, Oct 7, 2013 at 4:09 PM, Sanne Grinovero 
> wrote:
>>
>> Nice!
>> Is that going to work out with our hardware?
>>
>> We'll need to be careful now with the configured duration of each such
>> test.
>>
>> On 7 October 2013 13:48, Dan Berindei  wrote:
>> > I've created a build in CI for the stress tests:
>> >
>> >
>> > http://ci.infinispan.org/viewType.html?buildTypeId=Infinispan_StressHotspotJdk6
>> >
>> > Cheers
>> > Dan
>> >
>> >
>> >
>> > On Sun, Oct 6, 2013 at 12:07 AM, Sanne Grinovero 
>> > wrote:
>> >>
>> >> Hi all,
>> >> the following change introduced a critical issue in the Lucene
>> >> Directory:
>> >>
>> >>final Set filesList = fileOps.getFileList();
>> >> -  String[] array = filesList.toArray(new String[0]);
>> >> -  return array;
>> >> + return filesList.toArray(new String[filesList.size()]);
>> >>
>> >> I'll leave it as a puzzler to figure why the change is able to cause
>> >> trouble ;-)
>> >>
>> >> This generates a NPE in just a single second of running one of the
>> >> stress tests or performance tests, but I'm guilty of not being able to
>> >> make a normal unit test for this case.
>> >>
>> >> That module contains such limited code, that in the very rare
>> >> occasions in which I apply some changes I re-run the included
>> >> benchmarks; I realize I can't expect that from all of you, so..
>> >>
>> >> Should we enable some stress tests on CI?
>> >>
>> >> As a side warning consequence of this, the Lucene Directory in release
>> >> 6.0.0.CR1 is very unreliable [ISPN-3592].
>> >>
>> >> --Sanne
>> >> ___
>> >> infinispan-dev mailing list
>> >> infinispan-dev@lists.jboss.org
>> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >
>> >
>> >
>> > ___
>> > infinispan-dev mailing list
>> > infinispan-dev@lists.jboss.org
>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> ___
>> infinispan-dev mailing list
>> infinispan-dev@lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] The windup of 6.0.0

[Sanne Grinovero]

Hi Erik,
thanks that's a very interesting research. Will you share your patch?

Best,
Sanne

On 11 October 2013 02:30, Erik Salter  wrote:
> Hi all,
>
> I'm interested in the performance regression, and since I saw a similar
> regression with JGroups 3.4.0 coupled with ISPN 5.2.x (until 6.0.x is
> ready), I decided to investigate:
>
> Here are some profiling snapshots that I shared with Pedro.  This is of my
> system running UDP on a 12 node DIST cluster
>
> https://dl.dropboxusercontent.com/u/50401510/pruvio/dg1_320_interpreted.jps
> https://dl.dropboxusercontent.com/u/50401510/pruvio/dg1_340_interpreted2.jps
>
> The big thing we noticed was that the building of a Message was now in the
> main UDP receive() thread instead of the thread pools.  To my poor brain
> (and since I haven't been involved in the latest ISPN/JGroups happenings),
> it appeared this would be the most likely culprit.  So I reworked the UDP
> receiver to be a bit more like 3.2.x with the Message construction done in
> the appropriate thread pool (main, OOB, internal), while quasi-punting on
> the new bundler implementation.  After 3 test runs, the average throughput
> (again, measured by my app) seems to have recaptured 50-80% of the
> performance degradation.   The variance is high enough, though, to give me
> pause that I'm on the right track.
>
> Take this FWIW.  Thanks.
>
> Erik
>
> -Original Message-
> From: infinispan-dev-boun...@lists.jboss.org
> [mailto:infinispan-dev-boun...@lists.jboss.org] On Behalf Of Mircea Markus
> Sent: Wednesday, October 09, 2013 3:35 PM
> To: infinispan -Dev List
> Subject: [infinispan-dev] The windup of 6.0.0
>
> Hi guys,
>
> - 6.0.0.CR2 was added for 16 Oct (Adrian) and 6.0.0.Final was moved to 23
> Oct (Dan)
> - we have some 20% performance regressions we need to look at before going
> final
> - I've updated JIRA:
> - added tasks for creating documentation and quickstarts
> - some JIRAs were moved here
> - please follow the JIRA or let me know if there's anything missing:
> http://goo.gl/y4Ky7t
>
> Cheers,
> --
> Mircea Markus
> Infinispan lead (www.infinispan.org)
>
>
>
>
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev