Re: [infinispan-dev] moving asyncMarshalling to jgroups
The last person I remember testing async marshalling extensively was Craig Bomba who was indeed the one to spot the re-ordering. I can't remember 100% cos we discussed this on IRC, but I think he used to see a nice performance improvement with async marshalling, but I think the reordering made it not usable for him. On Jan 20, 2012, at 10:14 AM, Galder Zamarreño wrote: On Jan 19, 2012, at 9:08 PM, Mircea Markus wrote: On 19 Jan 2012, at 19:42, Bela Ban wrote: On 1/19/12 6:36 PM, Galder Zamarreño wrote: On Jan 19, 2012, at 3:43 PM, Bela Ban wrote: This may not give you any performance increase: #1 In my experience, serialization is way faster than de-serialization. Unless you're doing something fancy in your serializer No. I think Mircea didn't explain this very well. What really happens here is that when asyncMarshalling is turned on (the name is confusing...), async transport sending is activated. What does this mean? When the request needs to be passed onto JGroups, this is done in a separate thread, which indirectly, results in marshalling happening in a different thread. How is this thread created ? I assume you use a thread pool with a *bounded* queue. yes, these[1] are the default initialisation values for the pool. Nope, the default value is not the one pointed out but rather the ones indicated in https://docs.jboss.org/author/display/ISPN/Default+Values+For+Property+Based+Attributes Which comes from: https://github.com/galderz/infinispan/blob/master/core/src/main/java/org/infinispan/factories/KnownComponentNames.java#L59 Returning to the original question, can't we achieve the same functionality (i.e. marshalling not to happen in caller's thread - that's for async replication *only*) by passing the marshaller(an adapter) to jgroups which would then serialize it in the (async) replication thread? [1]http://bit.ly/zUFIEb ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Galder Zamarreño Sr. Software Engineer Infinispan, JBoss Cache ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Galder Zamarreño Sr. Software Engineer Infinispan, JBoss Cache ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] moving asyncMarshalling to jgroups
This may not give you any performance increase: #1 In my experience, serialization is way faster than de-serialization. Unless you're doing something fancy in your serializer #2 Assuming that the serialization thread pool has a bounded queue/size, by pushing the serialization further down the stack, you'll be faster in Infinispan, but at some point, you'll block on a full queue / exhaused thread pool. #3 If you don't pass me a byte buffer, FRAG(2) will effectively not do fragmentation. If the serialized object later generates a larger byte buffer, it's too late for fragmentation. #4 Ditto for flow control which depends on the number of bytes sent to throttle a sender The only benefit I see here is to handle temporary performance spikes. I remember a couple of years ago, in Neuchatel, Manik, Jason and I looked at ehcache and why it was so much faster than JBossCache with async replication. We found out they had a replication buffer, into which they placed all replication tasks, and which was periodically flushed (causing real network traffic). Turns out they didn't flush it until the test was over, therefore getting super performance ! :-) Had they run it long enough, they would have paid the penalty by most threads getting blocked on a full replication queue... On 1/19/12 3:31 PM, Mircea Markus wrote: Hi Bela, ATM when asyncMarshalling is enabled[1], we use our own thread pool for a) serializing the object and then b) pass it to the jgroups transport to send it async. As per [1], this has a major limitation: requests might be re-ordered at infinispan's level, in the thread pool that handles serialization. A possible way to improve this is by sending both the marshaller and the object to the jgroups transport which would serialize it on the same thread used for sending it. This way we would avoid re-ordering and potentially reduce thread's context switching. Wdyt? [1] http://bit.ly/yNk6In -- Bela Ban Lead JGroups (http://www.jgroups.org) JBoss / Red Hat ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] moving asyncMarshalling to jgroups
On Jan 19, 2012, at 3:43 PM, Bela Ban wrote: This may not give you any performance increase: #1 In my experience, serialization is way faster than de-serialization. Unless you're doing something fancy in your serializer No. I think Mircea didn't explain this very well. What really happens here is that when asyncMarshalling is turned on (the name is confusing...), async transport sending is activated. What does this mean? When the request needs to be passed onto JGroups, this is done in a separate thread, which indirectly, results in marshalling happening in a different thread. This returns faster because we return the control earlier rather than waiting for JGroups to do his thing of marshalling and sending the command and not waiting for response. See the use of RpcManagerImpl.asyncExecutor for more info. #2 Assuming that the serialization thread pool has a bounded queue/size, by pushing the serialization further down the stack, you'll be faster in Infinispan, but at some point, you'll block on a full queue / exhaused thread pool. #3 If you don't pass me a byte buffer, FRAG(2) will effectively not do fragmentation. If the serialized object later generates a larger byte buffer, it's too late for fragmentation. #4 Ditto for flow control which depends on the number of bytes sent to throttle a sender The only benefit I see here is to handle temporary performance spikes. I remember a couple of years ago, in Neuchatel, Manik, Jason and I looked at ehcache and why it was so much faster than JBossCache with async replication. We found out they had a replication buffer, into which they placed all replication tasks, and which was periodically flushed (causing real network traffic). Turns out they didn't flush it until the test was over, therefore getting super performance ! :-) Had they run it long enough, they would have paid the penalty by most threads getting blocked on a full replication queue... On 1/19/12 3:31 PM, Mircea Markus wrote: Hi Bela, ATM when asyncMarshalling is enabled[1], we use our own thread pool for a) serializing the object and then b) pass it to the jgroups transport to send it async. As per [1], this has a major limitation: requests might be re-ordered at infinispan's level, in the thread pool that handles serialization. A possible way to improve this is by sending both the marshaller and the object to the jgroups transport which would serialize it on the same thread used for sending it. This way we would avoid re-ordering and potentially reduce thread's context switching. Wdyt? [1] http://bit.ly/yNk6In -- Bela Ban Lead JGroups (http://www.jgroups.org) JBoss / Red Hat ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Galder Zamarreño Sr. Software Engineer Infinispan, JBoss Cache ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] moving asyncMarshalling to jgroups
On 19 Jan 2012, at 17:36, Galder Zamarreño wrote: On Jan 19, 2012, at 3:43 PM, Bela Ban wrote: This may not give you any performance increase: #1 In my experience, serialization is way faster than de-serialization. Unless you're doing something fancy in your serializer No. I think Mircea didn't explain this very well. What really happens here is that when asyncMarshalling is turned on (the name is confusing...), async transport sending is activated. What does this mean? When the request needs to be passed onto JGroups, this is done in a separate thread, which indirectly, results in marshalling happening in a different thread. This returns faster because we return the control earlier rather than waiting for JGroups to do his thing of marshalling and sending the command and not waiting for response. See the use of RpcManagerImpl.asyncExecutor for more info. Thanks for the clarification Galder. So basically with asyncMarshalling=true (this only works for async x-casts): - user thread(1) passes control to marshalling thread(2). At this point user thread returns - marshalling thread serialises and builds the Message object then passed control to jgroups thread(3) that sends the response async My suggestion is about removing the middle man (2), as this one might reorder messages, and give (3) the responsibility of marshalling by passing in the Object to be marshalled together with the Marshaller itself. ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] moving asyncMarshalling to jgroups
On 1/19/12 6:36 PM, Galder Zamarreño wrote: On Jan 19, 2012, at 3:43 PM, Bela Ban wrote: This may not give you any performance increase: #1 In my experience, serialization is way faster than de-serialization. Unless you're doing something fancy in your serializer No. I think Mircea didn't explain this very well. What really happens here is that when asyncMarshalling is turned on (the name is confusing...), async transport sending is activated. What does this mean? When the request needs to be passed onto JGroups, this is done in a separate thread, which indirectly, results in marshalling happening in a different thread. How is this thread created ? I assume you use a thread pool with a *bounded* queue. Or are you spawning a new thread on every serialization ? If that's the case, then you're run out of threads if the serialization rate is high... -- Bela Ban Lead JGroups (http://www.jgroups.org) JBoss / Red Hat ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] moving asyncMarshalling to jgroups
On 20 Jan 2012, at 01:12, Bela Ban wrote: On 1/19/12 6:36 PM, Galder Zamarreño wrote: On Jan 19, 2012, at 3:43 PM, Bela Ban wrote: This may not give you any performance increase: #1 In my experience, serialization is way faster than de-serialization. Unless you're doing something fancy in your serializer No. I think Mircea didn't explain this very well. What really happens here is that when asyncMarshalling is turned on (the name is confusing...), async transport sending is activated. What does this mean? When the request needs to be passed onto JGroups, this is done in a separate thread, which indirectly, results in marshalling happening in a different thread. How is this thread created ? I assume you use a thread pool with a *bounded* queue. Or are you spawning a new thread on every serialization ? If that's the case, then you're run out of threads if the serialization rate is high… Yes, an executor with a configurable/tunable thread pool. And yes, if this queue fills we pay a price, and effectively starts to look like sync marshalling. But this isn't the default and users who use this feature need to understand that marshalling is a hotspot for them, and their transaction rate isn't that high that it overwhelms their marshalling queue otherwise this doesn't help them. I'm not entirely sure Mircea's and Galder's request makes sense though: with a GET_NONE, does JGroups use a separate thread to send a message when a message is cast using the MessageDispatcher? Or is this on the thread that calls cast() up until it hits the socket, but just doesn't wait for any response? Cheers Manik -- Manik Surtani ma...@jboss.org twitter.com/maniksurtani Lead, Infinispan http://www.infinispan.org ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] moving asyncMarshalling to jgroups
On 19 Jan 2012, at 19:42, Bela Ban wrote: On 1/19/12 6:36 PM, Galder Zamarreño wrote: On Jan 19, 2012, at 3:43 PM, Bela Ban wrote: This may not give you any performance increase: #1 In my experience, serialization is way faster than de-serialization. Unless you're doing something fancy in your serializer No. I think Mircea didn't explain this very well. What really happens here is that when asyncMarshalling is turned on (the name is confusing...), async transport sending is activated. What does this mean? When the request needs to be passed onto JGroups, this is done in a separate thread, which indirectly, results in marshalling happening in a different thread. How is this thread created ? I assume you use a thread pool with a *bounded* queue. yes, these[1] are the default initialisation values for the pool. Returning to the original question, can't we achieve the same functionality (i.e. marshalling not to happen in caller's thread - that's for async replication *only*) by passing the marshaller(an adapter) to jgroups which would then serialize it in the (async) replication thread? [1]http://bit.ly/zUFIEb ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] moving asyncMarshalling to jgroups
On Thu, Jan 19, 2012 at 7:46 PM, Mircea Markus mircea.mar...@jboss.com wrote: On 19 Jan 2012, at 17:36, Galder Zamarreño wrote: On Jan 19, 2012, at 3:43 PM, Bela Ban wrote: This may not give you any performance increase: #1 In my experience, serialization is way faster than de-serialization. Unless you're doing something fancy in your serializer No. I think Mircea didn't explain this very well. What really happens here is that when asyncMarshalling is turned on (the name is confusing...), async transport sending is activated. What does this mean? When the request needs to be passed onto JGroups, this is done in a separate thread, which indirectly, results in marshalling happening in a different thread. This returns faster because we return the control earlier rather than waiting for JGroups to do his thing of marshalling and sending the command and not waiting for response. See the use of RpcManagerImpl.asyncExecutor for more info. Thanks for the clarification Galder. So basically with asyncMarshalling=true (this only works for async x-casts): - user thread(1) passes control to marshalling thread(2). At this point user thread returns - marshalling thread serialises and builds the Message object then passed control to jgroups thread(3) that sends the response async Are you sure about this Mircea? AFAIK the message is always sent by JGroups from our thread (either the user thread or the async transport thread). The only difference between sync (GET_ALL) and async calls (GET_NONE) at the JGroups level is that with GET_NONE that thread doesn't wait for the response from the target nodes. Cheers Dan ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] moving asyncMarshalling to jgroups
Are you sure about this Mircea? AFAIK the message is always sent by JGroups from our thread (either the user thread or the async transport thread). The only difference between sync (GET_ALL) and async calls (GET_NONE) at the JGroups level is that with GET_NONE that thread doesn't wait for the response from the target nodes. Ah, I assumed that jgroups is using its own thread pool for that, seems like my assumption was wrong :) Thanks for the clarification Dan. ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
