Re: [infinispan-dev] Threadpools in a large cluster

excellent, and we'll update it with our findings from the London meeting On 2/8/13 5:19 PM, Mircea Markus wrote: > I've created a JIRA in the scope of Infinispan 5.3 to track the thread > pool improvements discussed in this email thread: > ISPN-2808 - Make Infinispan use its own thread pool for se

Re: [infinispan-dev] Threadpools in a large cluster

yes, I mis-tagged it... :-( On 2/8/13 3:33 PM, Pedro Ruivo wrote: > Hi Bela, > > Alpha1 does not exist in GitHub. Instead you have 3.3.0.Alpha2 =) > > So, can I start to use it? > > Thanks > Pedro > > On 2/8/13 2:31 PM, Bela Ban wrote: >> On 2/8/13 3:23 PM, Mircea Markus wrote: >>> Hi Bela, >>> >>>

Re: [infinispan-dev] Threadpools in a large cluster

I've created a JIRA in the scope of Infinispan 5.3 to track the thread pool improvements discussed in this email thread: ISPN-2808 - Make Infinispan use its own thread pool for sending OOB messages in order to avoid thread deadlocks On 8 Feb 2013, at 14:44, Mircea Markus wrote: > > On 8 Feb 20

Re: [infinispan-dev] Threadpools in a large cluster

On 8 Feb 2013, at 14:31, Bela Ban wrote: > > On 2/8/13 3:23 PM, Mircea Markus wrote: >> Hi Bela, >> >> Do you think we can have this particular improvement in JGroups by 18 >> Feb? That week Pedro, Dan and I are going to start integrating TO >> protocols in ISPN and this is a requirement for

Re: [infinispan-dev] Threadpools in a large cluster

Hi Bela, Alpha1 does not exist in GitHub. Instead you have 3.3.0.Alpha2 =) So, can I start to use it? Thanks Pedro On 2/8/13 2:31 PM, Bela Ban wrote: > On 2/8/13 3:23 PM, Mircea Markus wrote: >> Hi Bela, >> >> Do you think we can have this particular improvement in JGroups by 18 >> Feb? That we

Re: [infinispan-dev] Threadpools in a large cluster

On 2/8/13 3:23 PM, Mircea Markus wrote: > Hi Bela, > > Do you think we can have this particular improvement in JGroups by 18 > Feb? That week Pedro, Dan and I are going to start integrating TO > protocols in ISPN and this is a requirement for it. It is done and part of the alpha1 release...

Re: [infinispan-dev] Threadpools in a large cluster

Hi Bela, Do you think we can have this particular improvement in JGroups by 18 Feb? That week Pedro, Dan and I are going to start integrating TO protocols in ISPN and this is a requirement for it. On 8 Feb 2013, at 12:41, Pedro Ruivo wrote: > On 2/8/13 12:26 PM, Mircea Markus wrote: >> >> >>

Re: [infinispan-dev] Threadpools in a large cluster

On 2/8/13 12:26 PM, Mircea Markus wrote: On 7 Feb 2013, at 11:36, Manik Surtani wrote: On 7 Feb 2013, at 11:29, Bela Ban > wrote: I meant to say that this is up to you guys to decide inwhich Infinispan release this will be used, but it will be available in JGroups 3

Re: [infinispan-dev] Threadpools in a large cluster

On 7 Feb 2013, at 11:36, Manik Surtani wrote: > On 7 Feb 2013, at 11:29, Bela Ban wrote: > >> I meant to say that this is up to you guys to decide inwhich Infinispan >> release this will be used, but it will be available in JGroups 3.3. >> >> What's the strategy/schedule for 6 and 5.3 anyway

Re: [infinispan-dev] Threadpools in a large cluster

| | after dealing with the large cluster for a while I find the way how | we use OOB threads in synchronous configuration non-robust. | Imagine a situation where node which is not an owner of the key calls | PUT. Then the a RPC is called to the primary owner of that key, | which reroutes the requ

Re: [infinispan-dev] Threadpools in a large cluster

t design no locks should be held when a RPC is >> called, it may be possible. >> >> Let's see what someone more informed (Dan?) would think about that. >> >> Thanks, Bela >> >> Radim >> >> - Original Message - >> | From: "

Re: [infinispan-dev] Threadpools in a large cluster

s > called, it may be possible. > > Let's see what someone more informed (Dan?) would think about that. > > Thanks, Bela > > Radim > > - Original Message - > | From: "Bela Ban" > | To: infinispan-dev@lists.jboss.org > | Sent: Friday, F

Re: [infinispan-dev] Threadpools in a large cluster

On 1 Feb 2013, at 08:04, Radim Vansa wrote: > Hi guys, > > after dealing with the large cluster for a while I find the way how we use > OOB threads in synchronous configuration non-robust. > Imagine a situation where node which is not an owner of the key calls PUT. > Then the a RPC is called t

Re: [infinispan-dev] Threadpools in a large cluster

On 2/7/13 2:55 PM, Pedro Ruivo wrote: > >> The default would be to invoke the old handle(Message) method. The >> dispatching mechanism could be changed to use the async method by >> setting an attribute in MessageDispatcher (which in turn sets it in >> RequestCorrelator). >> >> How would you do th

Re: [infinispan-dev] Threadpools in a large cluster

On Thu, Feb 7, 2013 at 3:55 PM, Pedro Ruivo wrote: > Hi, see inline > > Cheers, > Pedro > > On 2/7/13 11:42 AM, Bela Ban wrote: > > On 2/7/13 12:29 PM, Pedro Ruivo wrote: > >> Hi Bela > >> > >> On 2/7/13 4:53 AM, Bela Ban wrote: > >>> Hi Pedro, > >>> > >>> this is almost exactly what I wanted to

Re: [infinispan-dev] Threadpools in a large cluster

Hi, see inline Cheers, Pedro On 2/7/13 11:42 AM, Bela Ban wrote: > On 2/7/13 12:29 PM, Pedro Ruivo wrote: >> Hi Bela >> >> On 2/7/13 4:53 AM, Bela Ban wrote: >>> Hi Pedro, >>> >>> this is almost exactly what I wanted to implement ! >>> >>> Question: >>> - In RequestCorrelator.handleRequest(): >>>

Re: [infinispan-dev] Threadpools in a large cluster

On 2/7/13 12:36 PM, Manik Surtani wrote: > On 7 Feb 2013, at 11:29, Bela Ban wrote: > >> I meant to say that this is up to you guys to decide inwhich Infinispan >> release this will be used, but it will be available in JGroups 3.3. >> >> What's the strategy/schedule for 6 and 5.3 anyway ? > 5.3

Re: [infinispan-dev] Threadpools in a large cluster

On 2/7/13 1:38 PM, Dan Berindei wrote: > > > > > > No, I think it'll apply to all messages. A simple implementation could > dispatch OOB messages to the thread pool, as they don't need to be > ordered. Regular messages could be added to a queue where they are > processed sequential

Re: [infinispan-dev] Threadpools in a large cluster

On Thu, Feb 7, 2013 at 12:43 PM, Bela Ban wrote: > > On 2/7/13 11:09 AM, Dan Berindei wrote: > > > > > > A few changes I have in mind (need to think about it more): > > > > - I want to leave the existing RequestHandler interface in place, so > > current implementation continue to work

Re: [infinispan-dev] Threadpools in a large cluster

On 2/7/13 12:29 PM, Pedro Ruivo wrote: > Hi Bela > > On 2/7/13 4:53 AM, Bela Ban wrote: >> Hi Pedro, >> >> this is almost exactly what I wanted to implement ! >> >> Question: >> - In RequestCorrelator.handleRequest(): >> >> protected void handleRequest(Message req, Header hdr) { >> Object retval;

Re: [infinispan-dev] Threadpools in a large cluster

On 7 Feb 2013, at 11:29, Bela Ban wrote: > I meant to say that this is up to you guys to decide inwhich Infinispan > release this will be used, but it will be available in JGroups 3.3. > > What's the strategy/schedule for 6 and 5.3 anyway ? 5.3 will be relatively quick, just an incremental re

Re: [infinispan-dev] Threadpools in a large cluster

Hi Bela On 2/7/13 4:53 AM, Bela Ban wrote: > Hi Pedro, > > this is almost exactly what I wanted to implement ! > > Question: > - In RequestCorrelator.handleRequest(): > > protected void handleRequest(Message req, Header hdr) { > Object retval; > boolean threwException = false; > MessageRequest mes

Re: [infinispan-dev] Threadpools in a large cluster

I meant to say that this is up to you guys to decide inwhich Infinispan release this will be used, but it will be available in JGroups 3.3. What's the strategy/schedule for 6 and 5.3 anyway ? On 2/7/13 11:30 AM, Bela Ban wrote: > No, this *could* be in Infinispan 5.3 (it will be in JGroups 3.3).

Re: [infinispan-dev] Threadpools in a large cluster

On 7 Feb 2013, at 10:30, Bela Ban wrote: > No, this *could* be in Infinispan 5.3 (it will be in JGroups 3.3). > > A MessageDispatcher (RpcDispatcher) instance picks which dispatching > mechanism it wants to use. I have RequestHandler (the default) and a > sub-interface AsyncRequestHandler. Me

Re: [infinispan-dev] Threadpools in a large cluster

No, this *could* be in Infinispan 5.3 (it will be in JGroups 3.3). A MessageDispatcher (RpcDispatcher) instance picks which dispatching mechanism it wants to use. I have RequestHandler (the default) and a sub-interface AsyncRequestHandler. MessageDispatcher (and subclass RpcDispatcher) will imp

Re: [infinispan-dev] Threadpools in a large cluster

On 2/7/13 11:09 AM, Dan Berindei wrote: > > > A few changes I have in mind (need to think about it more): > > - I want to leave the existing RequestHandler interface in place, so > current implementation continue to work > - There will be a new AsyncRequestHandler interface (possib

Re: [infinispan-dev] Threadpools in a large cluster

On Thu, Feb 7, 2013 at 6:53 AM, Bela Ban wrote: > Hi Pedro, > > this is almost exactly what I wanted to implement ! > > Question: > - In RequestCorrelator.handleRequest(): > > protected void handleRequest(Message req, Header hdr) { > Object retval; > boolean threwException = false; > MessageReque

Re: [infinispan-dev] Threadpools in a large cluster

Very interesting. However I presume this would be something for Infinispan 6.0? Any thoughts on backward compat? On 7 Feb 2013, at 04:53, Bela Ban wrote: > Hi Pedro, > > this is almost exactly what I wanted to implement ! > > Question: > - In RequestCorrelator.handleRequest(): > > protecte

Re: [infinispan-dev] Threadpools in a large cluster

Hi Pedro, this is almost exactly what I wanted to implement ! Question: - In RequestCorrelator.handleRequest(): protected void handleRequest(Message req, Header hdr) { Object retval; boolean threwException = false; MessageRequest messageRequest = new MessageRequestImpl(req, hdr); try { retval=re

Re: [infinispan-dev] Threadpools in a large cluster

Hi all, Recently I came up with a solution that can help with the thread pool problem motivated by the following: In one of the first implementation of Total Order based commit protocol (TO), I had the requirement to move the PrepareCommand to another thread pool. In resume, the TO protocol

Re: [infinispan-dev] Threadpools in a large cluster

On 4 Feb 2013, at 07:46, Dan Berindei wrote: > Switching to a state machine approach would require rethinking and rewriting > all our interceptors, and I'm pretty sure the code would get more complex and > harder to debug (to say nothing about interpreting the logs). Are you sure > it's going

Re: [infinispan-dev] Threadpools in a large cluster

>into the FutureListener from the middle of interceptor stack - I >>am really not sure about that but as in current design no locks >>should be held when a RPC is called, it may be possible. >> >>Let's see what someone more informed (Dan?) would think about that

Re: [infinispan-dev] Threadpools in a large cluster

k execution path > > into the FutureListener from the middle of interceptor stack - I > > am really not sure about that but as in current design no locks > > should be held when a RPC is called, it may be possible. > > > > Let's see what someone more inf

Re: [infinispan-dev] Threadpools in a large cluster

If you send me the details, I'll take a look. I'm pretty busy with message batching, so I can't promise next week, but soon... On 2/1/13 11:08 AM, Pedro Ruivo wrote: > Hi, > > I had a similar problem when I tried GMU[1] in "large" cluster (40 vms), > because the remote gets and the commit message

Re: [infinispan-dev] Threadpools in a large cluster

that but as in current design no locks > should be held when a RPC is called, it may be possible. > > Let's see what someone more informed (Dan?) would think about that. > > Thanks, Bela > > Radim > > - Original Message ----- > | From: &

Re: [infinispan-dev] Threadpools in a large cluster

On Fri, Feb 1, 2013 at 12:13 PM, Manik Surtani wrote: > > On 1 Feb 2013, at 09:39, Dan Berindei wrote: > > > Radim, do these problems happen with the HotRod server, or only with > memcached? > > > > HotRod requests handled by non-owners should be very rare, instead the > vast majority should be

Re: [infinispan-dev] Threadpools in a large cluster

On Fri, Feb 1, 2013 at 12:40 PM, Radim Vansa wrote: > | > | Radim, do these problems happen with the HotRod server, or only with > | memcached? > > I didn't test memcached, only HotRod. The thing I was seing were many OOB > threads stuck when sending messages from handleRemoteWrite. > > | > | Hot

Re: [infinispan-dev] Threadpools in a large cluster

| | Radim, do these problems happen with the HotRod server, or only with | memcached? I didn't test memcached, only HotRod. The thing I was seing were many OOB threads stuck when sending messages from handleRemoteWrite. | | HotRod requests handled by non-owners should be very rare, instead | t

Re: [infinispan-dev] Threadpools in a large cluster

Hi, I had a similar problem when I tried GMU[1] in "large" cluster (40 vms), because the remote gets and the commit messages (I'm talking about ISPN commands) must wait for some conditions before being processed. I solved this problem by adding a feature in JGroups[2] that allows the request t

Re: [infinispan-dev] Threadpools in a large cluster

On 1 Feb 2013, at 09:39, Dan Berindei wrote: > Radim, do these problems happen with the HotRod server, or only with > memcached? > > HotRod requests handled by non-owners should be very rare, instead the vast > majority should be handled by the primary owner directly. So if this happens > wi

Re: [infinispan-dev] Threadpools in a large cluster

et's see what someone more informed (Dan?) would think about that. > > Thanks, Bela > > Radim > > - Original Message - > | From: "Bela Ban" > | To: infinispan-dev@lists.jboss.org > | Sent: Friday, February 1, 2013 9:39:43 AM > | Subject

Re: [infinispan-dev] Threadpools in a large cluster

Radim, do these problems happen with the HotRod server, or only with memcached? HotRod requests handled by non-owners should be very rare, instead the vast majority should be handled by the primary owner directly. So if this happens with HotRod, we should focus on fixing the HotRod routing instead

Re: [infinispan-dev] Threadpools in a large cluster

(Dan?) would think about that. Thanks, Bela Radim - Original Message - | From: "Bela Ban" | To: infinispan-dev@lists.jboss.org | Sent: Friday, February 1, 2013 9:39:43 AM | Subject: Re: [infinispan-dev] Threadpools in a large cluster | | It looks like the core problem is an

Re: [infinispan-dev] Threadpools in a large cluster

It looks like the core problem is an incoming RPC-1 which triggers another blocking RPC-2: the thread delivering RPC-1 is blocked waiting for the response from RPC-2, and can therefore not be used to serve other requests for the duration of RPC-2. If RPC-2 takes a while, e.g. waiting to acquire

[infinispan-dev] Threadpools in a large cluster

Hi guys, after dealing with the large cluster for a while I find the way how we use OOB threads in synchronous configuration non-robust. Imagine a situation where node which is not an owner of the key calls PUT. Then the a RPC is called to the primary owner of that key, which reroutes the reque