Re: [infinispan-dev] Staggering remote GET calls
I'm not surprised that read performance suffers a bit actually. Which is why we broadcast the GETs originally. But once the staggering timeout becomes configurable, this should be something people can tune. - M On 21 Feb 2013, at 11:40, Radim Vansa rva...@redhat.com wrote: Hi, so I have re-ran (and checked this time!) both tests with the same setting. The result is not as miraculous as the previous one but it is still 500 compared to 200, which is good. However, read performance has dropped by 25%, but write performance has increased by 57%! See the charts (mainly the OOB maximum size distribution) in attachment. Config: library mode, non-transactional distributed cache with 2 owners and 512 segments, sync replication. Test: 10m warmup, 20m test, 10 stressing threads per node. Radim - Original Message - | From: Radim Vansa rva...@redhat.com | To: infinispan -Dev List infinispan-dev@lists.jboss.org | Sent: Wednesday, February 20, 2013 3:38:49 PM | Subject: Re: [infinispan-dev] Staggering remote GET calls | | Ouch, call me dumbass... I haven't checked the test results. | Something revoked my cluster allocation and the test was prematurely | stopped. | | I'll rerun it (and check!), and show performance numbers as well. | | Radim (the dumbass) | | - Original Message - | | From: Dan Berindei dan.berin...@gmail.com | | To: infinispan -Dev List infinispan-dev@lists.jboss.org | | Cc: Manik Surtani msurt...@redhat.com | | Sent: Wednesday, February 20, 2013 3:24:46 PM | | Subject: Re: [infinispan-dev] Staggering remote GET calls | | | | | | Radim, just to be sure, you are testing embedded mode with | | RadarGun, | | right? With HotRod most of the get operations should be initiated | | from the main owner, so Manik's changes shouldn't make a big | | difference in the number of active threads. | | | | How about throughput, has it also improved compared to 5.2.0.CR3, | | or | | is it the same? | | | | | | | | | | | | On Wed, Feb 20, 2013 at 2:15 PM, Radim Vansa rva...@redhat.com | | wrote: | | | | | | Hi Manik, | | | | so I have tried to compile this branch and issued a 20 minute | | stress | | test (preceded by 10 minute warmup) on 128 nodes, where each node | | has 10 stressor threads. | | While in 5.2.0.CR3 the maximum OOB threadpool size was 553 with | | this | | configuration, with t_825 it was 219. This looks good, but it's | | actually better :). When I looked on the per-node maximum, in t_825 | | there was only one node with the 219 threads (as the max), others | | were usually around 25, few around 40. On the contrary, in | | 5.2.0.CR3 | | all the nodes had maximum around 500! | | | | Glad to bring good news :) | | | | Radim | | | | | | | | - Original Message - | | | From: Manik Surtani msurt...@redhat.com | | | To: infinispan -Dev List infinispan-dev@lists.jboss.org , | | | Radim Vansa rva...@redhat.com | | | Sent: Tuesday, February 19, 2013 6:33:04 PM | | | Subject: Staggering remote GET calls | | | | | | Guys, | | | | | | I have a topic branch with a fix for ISPN-825, to stagger remote | | | GET | | | calls. (See the JIRA for details on this patch). | | | | | | This should have an interesting effect on greatly reducing the | | | pressure on the OOB thread pool. This isn't a *real* fix for the | | | problem that Radim reported (Pedro is working on that with Bela), | | | but reducing pressure on the OOB thread pool is a side effect of | | | this fix. | | | | | | It should generally make things faster too, with less traffic on | | | the | | | network. I'd be curious for you to give this branch a try, Radim | | | - | | | see how it impacts your tests. | | | | | | https://github.com/maniksurtani/infinispan/tree/t_825 | | | | | | Cheers | | | Manik | | | -- | | | Manik Surtani | | | ma...@jboss.org | | | twitter.com/maniksurtani | | | | | | Platform Architect, JBoss Data Grid | | | http://red.ht/data-grid | | | | | | | | ___ | | infinispan-dev mailing list | | infinispan-dev@lists.jboss.org | | https://lists.jboss.org/mailman/listinfo/infinispan-dev | | | | | | ___ | | infinispan-dev mailing list | | infinispan-dev@lists.jboss.org | | https://lists.jboss.org/mailman/listinfo/infinispan-dev | ___ | infinispan-dev mailing list | infinispan-dev@lists.jboss.org | https://lists.jboss.org/mailman/listinfo/infinispan-dev | t825.pdf___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Manik Surtani ma...@jboss.org twitter.com/maniksurtani Platform Architect, JBoss Data Grid http://red.ht/data-grid ___ infinispan-dev mailing
Re: [infinispan-dev] Staggering remote GET calls
On 26 Feb 2013, at 10:56, Manik Surtani wrote: I'm not surprised that read performance suffers a bit actually. Which is why we broadcast the GETs originally. But once the staggering timeout becomes configurable, this should be something people can tune. +1. Also the performance increase for writes is huge. I think this should be the default, as read efficiency can be improved by L1. Cheers, -- Mircea Markus Infinispan lead (www.infinispan.org) ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Staggering remote GET calls
Tune in real-time, of course ;) Erik From: infinispan-dev-boun...@lists.jboss.org [mailto:infinispan-dev-boun...@lists.jboss.org] On Behalf Of Mircea Markus Sent: Tuesday, February 26, 2013 7:36 AM To: infinispan -Dev List Subject: Re: [infinispan-dev] Staggering remote GET calls On 26 Feb 2013, at 10:56, Manik Surtani wrote: I'm not surprised that read performance suffers a bit actually. Which is why we broadcast the GETs originally. But once the staggering timeout becomes configurable, this should be something people can tune. +1. Also the performance increase for writes is huge. I think this should be the default, as read efficiency can be improved by L1. Cheers, -- Mircea Markus Infinispan lead (www.infinispan.org) ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Staggering remote GET calls
If you're really into self-tuning this parameter, I expect that a very simple gradient-descent mechanism would actually work pretty well in this case. We have done similar work in the Cloud-TM project (applied to both message batching and number of threads active per node), and if you're interested I may send more references on this. BTW, +1 for making this configurable: eager broadcasting read requests seemed to me to be a little too aggressive for the normal/failure-free behavior. Cheers, Paolo On 2/26/13 1:55 PM, Erik Salter wrote: Tune in real-time, of course ;) Erik *From:*infinispan-dev-boun...@lists.jboss.org [mailto:infinispan-dev-boun...@lists.jboss.org] *On Behalf Of *Mircea Markus *Sent:* Tuesday, February 26, 2013 7:36 AM *To:* infinispan -Dev List *Subject:* Re: [infinispan-dev] Staggering remote GET calls On 26 Feb 2013, at 10:56, Manik Surtani wrote: I'm not surprised that read performance suffers a bit actually. Which is why we broadcast the GETs originally. But once the staggering timeout becomes configurable, this should be something people can tune. +1. Also the performance increase for writes is huge. I think this should be the default, as read efficiency can be improved by L1. Cheers, -- Mircea Markus Infinispan lead (www.infinispan.org http://www.infinispan.org) ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Staggering remote GET calls
On 26 Feb 2013, at 14:12, Paolo Romano rom...@inesc-id.pt wrote: If you're really into self-tuning this parameter, I expect that a very simple gradient-descent mechanism would actually work pretty well in this case. We have done similar work in the Cloud-TM project (applied to both message batching and number of threads active per node), and if you're interested I may send more references on this. Yes, please do. :) BTW, +1 for making this configurable: eager broadcasting read requests seemed to me to be a little too aggressive for the normal/failure-free behavior. Cheers, Paolo On 2/26/13 1:55 PM, Erik Salter wrote: Tune in real-time, of course ;) Erik From: infinispan-dev-boun...@lists.jboss.org [mailto:infinispan-dev-boun...@lists.jboss.org] On Behalf Of Mircea Markus Sent: Tuesday, February 26, 2013 7:36 AM To: infinispan -Dev List Subject: Re: [infinispan-dev] Staggering remote GET calls On 26 Feb 2013, at 10:56, Manik Surtani wrote: I'm not surprised that read performance suffers a bit actually. Which is why we broadcast the GETs originally. But once the staggering timeout becomes configurable, this should be something people can tune. +1. Also the performance increase for writes is huge. I think this should be the default, as read efficiency can be improved by L1. Cheers, -- Mircea Markus Infinispan lead (www.infinispan.org) ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Manik Surtani ma...@jboss.org twitter.com/maniksurtani Platform Architect, JBoss Data Grid http://red.ht/data-grid ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Staggering remote GET calls
Hi, so I have re-ran (and checked this time!) both tests with the same setting. The result is not as miraculous as the previous one but it is still 500 compared to 200, which is good. However, read performance has dropped by 25%, but write performance has increased by 57%! See the charts (mainly the OOB maximum size distribution) in attachment. Config: library mode, non-transactional distributed cache with 2 owners and 512 segments, sync replication. Test: 10m warmup, 20m test, 10 stressing threads per node. Radim - Original Message - | From: Radim Vansa rva...@redhat.com | To: infinispan -Dev List infinispan-dev@lists.jboss.org | Sent: Wednesday, February 20, 2013 3:38:49 PM | Subject: Re: [infinispan-dev] Staggering remote GET calls | | Ouch, call me dumbass... I haven't checked the test results. | Something revoked my cluster allocation and the test was prematurely | stopped. | | I'll rerun it (and check!), and show performance numbers as well. | | Radim (the dumbass) | | - Original Message - | | From: Dan Berindei dan.berin...@gmail.com | | To: infinispan -Dev List infinispan-dev@lists.jboss.org | | Cc: Manik Surtani msurt...@redhat.com | | Sent: Wednesday, February 20, 2013 3:24:46 PM | | Subject: Re: [infinispan-dev] Staggering remote GET calls | | | | | | Radim, just to be sure, you are testing embedded mode with | | RadarGun, | | right? With HotRod most of the get operations should be initiated | | from the main owner, so Manik's changes shouldn't make a big | | difference in the number of active threads. | | | | How about throughput, has it also improved compared to 5.2.0.CR3, | | or | | is it the same? | | | | | | | | | | | | On Wed, Feb 20, 2013 at 2:15 PM, Radim Vansa rva...@redhat.com | | wrote: | | | | | | Hi Manik, | | | | so I have tried to compile this branch and issued a 20 minute | | stress | | test (preceded by 10 minute warmup) on 128 nodes, where each node | | has 10 stressor threads. | | While in 5.2.0.CR3 the maximum OOB threadpool size was 553 with | | this | | configuration, with t_825 it was 219. This looks good, but it's | | actually better :). When I looked on the per-node maximum, in t_825 | | there was only one node with the 219 threads (as the max), others | | were usually around 25, few around 40. On the contrary, in | | 5.2.0.CR3 | | all the nodes had maximum around 500! | | | | Glad to bring good news :) | | | | Radim | | | | | | | | - Original Message - | | | From: Manik Surtani msurt...@redhat.com | | | To: infinispan -Dev List infinispan-dev@lists.jboss.org , | | | Radim Vansa rva...@redhat.com | | | Sent: Tuesday, February 19, 2013 6:33:04 PM | | | Subject: Staggering remote GET calls | | | | | | Guys, | | | | | | I have a topic branch with a fix for ISPN-825, to stagger remote | | | GET | | | calls. (See the JIRA for details on this patch). | | | | | | This should have an interesting effect on greatly reducing the | | | pressure on the OOB thread pool. This isn't a *real* fix for the | | | problem that Radim reported (Pedro is working on that with Bela), | | | but reducing pressure on the OOB thread pool is a side effect of | | | this fix. | | | | | | It should generally make things faster too, with less traffic on | | | the | | | network. I'd be curious for you to give this branch a try, Radim | | | - | | | see how it impacts your tests. | | | | | | https://github.com/maniksurtani/infinispan/tree/t_825 | | | | | | Cheers | | | Manik | | | -- | | | Manik Surtani | | | ma...@jboss.org | | | twitter.com/maniksurtani | | | | | | Platform Architect, JBoss Data Grid | | | http://red.ht/data-grid | | | | | | | | ___ | | infinispan-dev mailing list | | infinispan-dev@lists.jboss.org | | https://lists.jboss.org/mailman/listinfo/infinispan-dev | | | | | | ___ | | infinispan-dev mailing list | | infinispan-dev@lists.jboss.org | | https://lists.jboss.org/mailman/listinfo/infinispan-dev | ___ | infinispan-dev mailing list | infinispan-dev@lists.jboss.org | https://lists.jboss.org/mailman/listinfo/infinispan-dev | t825.pdf Description: Adobe PDF document ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Staggering remote GET calls
Radim, just to be sure, you are testing embedded mode with RadarGun, right? With HotRod most of the get operations should be initiated from the main owner, so Manik's changes shouldn't make a big difference in the number of active threads. How about throughput, has it also improved compared to 5.2.0.CR3, or is it the same? On Wed, Feb 20, 2013 at 2:15 PM, Radim Vansa rva...@redhat.com wrote: Hi Manik, so I have tried to compile this branch and issued a 20 minute stress test (preceded by 10 minute warmup) on 128 nodes, where each node has 10 stressor threads. While in 5.2.0.CR3 the maximum OOB threadpool size was 553 with this configuration, with t_825 it was 219. This looks good, but it's actually better :). When I looked on the per-node maximum, in t_825 there was only one node with the 219 threads (as the max), others were usually around 25, few around 40. On the contrary, in 5.2.0.CR3 all the nodes had maximum around 500! Glad to bring good news :) Radim - Original Message - | From: Manik Surtani msurt...@redhat.com | To: infinispan -Dev List infinispan-dev@lists.jboss.org, Radim Vansa rva...@redhat.com | Sent: Tuesday, February 19, 2013 6:33:04 PM | Subject: Staggering remote GET calls | | Guys, | | I have a topic branch with a fix for ISPN-825, to stagger remote GET | calls. (See the JIRA for details on this patch). | | This should have an interesting effect on greatly reducing the | pressure on the OOB thread pool. This isn't a *real* fix for the | problem that Radim reported (Pedro is working on that with Bela), | but reducing pressure on the OOB thread pool is a side effect of | this fix. | | It should generally make things faster too, with less traffic on the | network. I'd be curious for you to give this branch a try, Radim - | see how it impacts your tests. | | https://github.com/maniksurtani/infinispan/tree/t_825 | | Cheers | Manik | -- | Manik Surtani | ma...@jboss.org | twitter.com/maniksurtani | | Platform Architect, JBoss Data Grid | http://red.ht/data-grid | | ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Staggering remote GET calls
Ouch, call me dumbass... I haven't checked the test results. Something revoked my cluster allocation and the test was prematurely stopped. I'll rerun it (and check!), and show performance numbers as well. Radim (the dumbass) - Original Message - | From: Dan Berindei dan.berin...@gmail.com | To: infinispan -Dev List infinispan-dev@lists.jboss.org | Cc: Manik Surtani msurt...@redhat.com | Sent: Wednesday, February 20, 2013 3:24:46 PM | Subject: Re: [infinispan-dev] Staggering remote GET calls | | | Radim, just to be sure, you are testing embedded mode with RadarGun, | right? With HotRod most of the get operations should be initiated | from the main owner, so Manik's changes shouldn't make a big | difference in the number of active threads. | | How about throughput, has it also improved compared to 5.2.0.CR3, or | is it the same? | | | | | | On Wed, Feb 20, 2013 at 2:15 PM, Radim Vansa rva...@redhat.com | wrote: | | | Hi Manik, | | so I have tried to compile this branch and issued a 20 minute stress | test (preceded by 10 minute warmup) on 128 nodes, where each node | has 10 stressor threads. | While in 5.2.0.CR3 the maximum OOB threadpool size was 553 with this | configuration, with t_825 it was 219. This looks good, but it's | actually better :). When I looked on the per-node maximum, in t_825 | there was only one node with the 219 threads (as the max), others | were usually around 25, few around 40. On the contrary, in 5.2.0.CR3 | all the nodes had maximum around 500! | | Glad to bring good news :) | | Radim | | | | - Original Message - | | From: Manik Surtani msurt...@redhat.com | | To: infinispan -Dev List infinispan-dev@lists.jboss.org , | | Radim Vansa rva...@redhat.com | | Sent: Tuesday, February 19, 2013 6:33:04 PM | | Subject: Staggering remote GET calls | | | | Guys, | | | | I have a topic branch with a fix for ISPN-825, to stagger remote | | GET | | calls. (See the JIRA for details on this patch). | | | | This should have an interesting effect on greatly reducing the | | pressure on the OOB thread pool. This isn't a *real* fix for the | | problem that Radim reported (Pedro is working on that with Bela), | | but reducing pressure on the OOB thread pool is a side effect of | | this fix. | | | | It should generally make things faster too, with less traffic on | | the | | network. I'd be curious for you to give this branch a try, Radim - | | see how it impacts your tests. | | | | https://github.com/maniksurtani/infinispan/tree/t_825 | | | | Cheers | | Manik | | -- | | Manik Surtani | | ma...@jboss.org | | twitter.com/maniksurtani | | | | Platform Architect, JBoss Data Grid | | http://red.ht/data-grid | | | | | ___ | infinispan-dev mailing list | infinispan-dev@lists.jboss.org | https://lists.jboss.org/mailman/listinfo/infinispan-dev | | | ___ | infinispan-dev mailing list | infinispan-dev@lists.jboss.org | https://lists.jboss.org/mailman/listinfo/infinispan-dev ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Staggering remote GET calls
So you are testing embedded mode with RadarGun? Or remotely over hotrod? On 20 Feb 2013, at 14:38, Radim Vansa wrote: Ouch, call me dumbass... I haven't checked the test results. Something revoked my cluster allocation and the test was prematurely stopped. I'll rerun it (and check!), and show performance numbers as well. Radim (the dumbass) - Original Message - | From: Dan Berindei dan.berin...@gmail.com | To: infinispan -Dev List infinispan-dev@lists.jboss.org | Cc: Manik Surtani msurt...@redhat.com | Sent: Wednesday, February 20, 2013 3:24:46 PM | Subject: Re: [infinispan-dev] Staggering remote GET calls | | | Radim, just to be sure, you are testing embedded mode with RadarGun, | right? With HotRod most of the get operations should be initiated | from the main owner, so Manik's changes shouldn't make a big | difference in the number of active threads. | | How about throughput, has it also improved compared to 5.2.0.CR3, or | is it the same? | | | | | | On Wed, Feb 20, 2013 at 2:15 PM, Radim Vansa rva...@redhat.com | wrote: | | | Hi Manik, | | so I have tried to compile this branch and issued a 20 minute stress | test (preceded by 10 minute warmup) on 128 nodes, where each node | has 10 stressor threads. | While in 5.2.0.CR3 the maximum OOB threadpool size was 553 with this | configuration, with t_825 it was 219. This looks good, but it's | actually better :). When I looked on the per-node maximum, in t_825 | there was only one node with the 219 threads (as the max), others | were usually around 25, few around 40. On the contrary, in 5.2.0.CR3 | all the nodes had maximum around 500! | | Glad to bring good news :) | | Radim | | | | - Original Message - | | From: Manik Surtani msurt...@redhat.com | | To: infinispan -Dev List infinispan-dev@lists.jboss.org , | | Radim Vansa rva...@redhat.com | | Sent: Tuesday, February 19, 2013 6:33:04 PM | | Subject: Staggering remote GET calls | | | | Guys, | | | | I have a topic branch with a fix for ISPN-825, to stagger remote | | GET | | calls. (See the JIRA for details on this patch). | | | | This should have an interesting effect on greatly reducing the | | pressure on the OOB thread pool. This isn't a *real* fix for the | | problem that Radim reported (Pedro is working on that with Bela), | | but reducing pressure on the OOB thread pool is a side effect of | | this fix. | | | | It should generally make things faster too, with less traffic on | | the | | network. I'd be curious for you to give this branch a try, Radim - | | see how it impacts your tests. | | | | https://github.com/maniksurtani/infinispan/tree/t_825 | | | | Cheers | | Manik | | -- | | Manik Surtani | | ma...@jboss.org | | twitter.com/maniksurtani | | | | Platform Architect, JBoss Data Grid | | http://red.ht/data-grid | | | | | ___ | infinispan-dev mailing list | infinispan-dev@lists.jboss.org | https://lists.jboss.org/mailman/listinfo/infinispan-dev | | | ___ | infinispan-dev mailing list | infinispan-dev@lists.jboss.org | https://lists.jboss.org/mailman/listinfo/infinispan-dev ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev Cheers, -- Mircea Markus Infinispan lead (www.infinispan.org) ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Staggering remote GET calls
On 20 Feb 2013, at 14:57, Radim Vansa wrote: Only library (embeded) mode, as the load is constant (10 threads issuing requests all the time) I believe that the results are more stable and comparable between runs. Thanks. As Dan suggested, the staggering of requests should ONLY have an (significant) impact over embedded client. Cheers, -- Mircea Markus Infinispan lead (www.infinispan.org) ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Staggering remote GET calls
Excellent ! On 2/20/13 3:15 PM, Radim Vansa wrote: Hi Manik, so I have tried to compile this branch and issued a 20 minute stress test (preceded by 10 minute warmup) on 128 nodes, where each node has 10 stressor threads. While in 5.2.0.CR3 the maximum OOB threadpool size was 553 with this configuration, with t_825 it was 219. This looks good, but it's actually better :). When I looked on the per-node maximum, in t_825 there was only one node with the 219 threads (as the max), others were usually around 25, few around 40. On the contrary, in 5.2.0.CR3 all the nodes had maximum around 500! Glad to bring good news :) Radim - Original Message - | From: Manik Surtani msurt...@redhat.com | To: infinispan -Dev List infinispan-dev@lists.jboss.org, Radim Vansa rva...@redhat.com | Sent: Tuesday, February 19, 2013 6:33:04 PM | Subject: Staggering remote GET calls | | Guys, | | I have a topic branch with a fix for ISPN-825, to stagger remote GET | calls. (See the JIRA for details on this patch). | | This should have an interesting effect on greatly reducing the | pressure on the OOB thread pool. This isn't a *real* fix for the | problem that Radim reported (Pedro is working on that with Bela), | but reducing pressure on the OOB thread pool is a side effect of | this fix. | | It should generally make things faster too, with less traffic on the | network. I'd be curious for you to give this branch a try, Radim - | see how it impacts your tests. | | https://github.com/maniksurtani/infinispan/tree/t_825 | | Cheers | Manik | -- | Manik Surtani | ma...@jboss.org | twitter.com/maniksurtani | | Platform Architect, JBoss Data Grid | http://red.ht/data-grid | | ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Bela Ban, JGroups lead (http://www.jgroups.org) ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Staggering remote GET calls
Awesome. :) Mircea, are you going to finish up my patch, make the staggering timeout configurable, etc? Sent from my mobile phone On 20 Feb 2013, at 15:15, Radim Vansa rva...@redhat.com wrote: Hi Manik, so I have tried to compile this branch and issued a 20 minute stress test (preceded by 10 minute warmup) on 128 nodes, where each node has 10 stressor threads. While in 5.2.0.CR3 the maximum OOB threadpool size was 553 with this configuration, with t_825 it was 219. This looks good, but it's actually better :). When I looked on the per-node maximum, in t_825 there was only one node with the 219 threads (as the max), others were usually around 25, few around 40. On the contrary, in 5.2.0.CR3 all the nodes had maximum around 500! Glad to bring good news :) Radim - Original Message - | From: Manik Surtani msurt...@redhat.com | To: infinispan -Dev List infinispan-dev@lists.jboss.org, Radim Vansa rva...@redhat.com | Sent: Tuesday, February 19, 2013 6:33:04 PM | Subject: Staggering remote GET calls | | Guys, | | I have a topic branch with a fix for ISPN-825, to stagger remote GET | calls. (See the JIRA for details on this patch). | | This should have an interesting effect on greatly reducing the | pressure on the OOB thread pool. This isn't a *real* fix for the | problem that Radim reported (Pedro is working on that with Bela), | but reducing pressure on the OOB thread pool is a side effect of | this fix. | | It should generally make things faster too, with less traffic on the | network. I'd be curious for you to give this branch a try, Radim - | see how it impacts your tests. | | https://github.com/maniksurtani/infinispan/tree/t_825 | | Cheers | Manik | -- | Manik Surtani | ma...@jboss.org | twitter.com/maniksurtani | | Platform Architect, JBoss Data Grid | http://red.ht/data-grid | | ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
Re: [infinispan-dev] Staggering remote GET calls
Manik, as I have stated in later mails, the test was not executed properly, therefore these good results. Actually, I think there is a bug in the branch, as you set expected responses in FutureCollator to dest.size() but still send only one message, if the first response is not successful but without exception set (the objectFuture.get() is interrupted, I am not sure why this happens, but it happens) the thread blocks infinitely. I have patched it and hope I'll get results soon. Radim - Original Message - | From: Manik Surtani msurt...@redhat.com | To: Radim Vansa rva...@redhat.com | Cc: infinispan -Dev List infinispan-dev@lists.jboss.org | Sent: Wednesday, February 20, 2013 6:41:49 PM | Subject: Re: Staggering remote GET calls | | Awesome. :) Mircea, are you going to finish up my patch, make the | staggering timeout configurable, etc? | | Sent from my mobile phone | | On 20 Feb 2013, at 15:15, Radim Vansa rva...@redhat.com wrote: | | Hi Manik, | | so I have tried to compile this branch and issued a 20 minute | stress test (preceded by 10 minute warmup) on 128 nodes, where | each node has 10 stressor threads. | While in 5.2.0.CR3 the maximum OOB threadpool size was 553 with | this configuration, with t_825 it was 219. This looks good, but | it's actually better :). When I looked on the per-node maximum, in | t_825 there was only one node with the 219 threads (as the max), | others were usually around 25, few around 40. On the contrary, in | 5.2.0.CR3 all the nodes had maximum around 500! | | Glad to bring good news :) | | Radim | | - Original Message - | | From: Manik Surtani msurt...@redhat.com | | To: infinispan -Dev List infinispan-dev@lists.jboss.org, | | Radim Vansa rva...@redhat.com | | Sent: Tuesday, February 19, 2013 6:33:04 PM | | Subject: Staggering remote GET calls | | | | Guys, | | | | I have a topic branch with a fix for ISPN-825, to stagger remote | | GET | | calls. (See the JIRA for details on this patch). | | | | This should have an interesting effect on greatly reducing the | | pressure on the OOB thread pool. This isn't a *real* fix for the | | problem that Radim reported (Pedro is working on that with Bela), | | but reducing pressure on the OOB thread pool is a side effect of | | this fix. | | | | It should generally make things faster too, with less traffic on | | the | | network. I'd be curious for you to give this branch a try, Radim | | - | | see how it impacts your tests. | | | | https://github.com/maniksurtani/infinispan/tree/t_825 | | | | Cheers | | Manik | | -- | | Manik Surtani | | ma...@jboss.org | | twitter.com/maniksurtani | | | | Platform Architect, JBoss Data Grid | | http://red.ht/data-grid | | | | | ___ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
