Re: high thread contention during load test on InternalComponentResourcesImpl.postRenderCleanup and org.apache.tapestry5.internal.util.NamedSet.getValues
On Wed, Jun 6, 2012 at 8:16 PM, Howard Lewis Ship hls...@gmail.com wrote: I'm writing up a little blog entry on this. I've also created 5.3.4-rc-5 with even more tweaks. I'm happy keeping with this process, even though were past the number of billable hours we agreed to. Here that is always the case btw. BTW, how much memory in your server? My most recent servers have from 8 to 12 gigs of memory and some share more then one tapestry applications (most ancient ones are 5.1.0.5-tweaked) but I still run servers with 4 Cheers -- Massimo - To unsubscribe, e-mail: users-unsubscr...@tapestry.apache.org For additional commands, e-mail: users-h...@tapestry.apache.org
Re: high thread contention during load test on InternalComponentResourcesImpl.postRenderCleanup and org.apache.tapestry5.internal.util.NamedSet.getValues
I didn't mean that message to go out to the mailing list. On Thu, Jun 7, 2012 at 3:11 AM, Massimo Lusetti mluse...@gmail.com wrote: On Wed, Jun 6, 2012 at 8:16 PM, Howard Lewis Ship hls...@gmail.com wrote: I'm writing up a little blog entry on this. I've also created 5.3.4-rc-5 with even more tweaks. I'm happy keeping with this process, even though were past the number of billable hours we agreed to. Here that is always the case btw. BTW, how much memory in your server? My most recent servers have from 8 to 12 gigs of memory and some share more then one tapestry applications (most ancient ones are 5.1.0.5-tweaked) but I still run servers with 4 Cheers -- Massimo - To unsubscribe, e-mail: users-unsubscr...@tapestry.apache.org For additional commands, e-mail: users-h...@tapestry.apache.org -- Howard M. Lewis Ship Creator of Apache Tapestry The source for Tapestry training, mentoring and support. Contact me to learn how I can get you up and productive in Tapestry fast! (971) 678-5210 http://howardlewisship.com - To unsubscribe, e-mail: users-unsubscr...@tapestry.apache.org For additional commands, e-mail: users-h...@tapestry.apache.org
Re: high thread contention during load test on InternalComponentResourcesImpl.postRenderCleanup and org.apache.tapestry5.internal.util.NamedSet.getValues
I'm writing up a little blog entry on this. I've also created 5.3.4-rc-5 with even more tweaks. I'm happy keeping with this process, even though were past the number of billable hours we agreed to. BTW, how much memory in your server? On Fri, May 11, 2012 at 6:07 AM, Robert Lentz rob...@teksolv.de wrote: Hi All, we want to rollout a Tapestry app very shortly, but we struggle with load testing issues. We are currently load testing on one Tomcat 6.0.32: - 500 worker threads, tapestry.production-mode=true - Intel(R) Xeon(R) CPU X7460 @ 2.66GHz - OpenJDK Runtime Environment (IcedTea6 1.7.10) (rhel-1.20.b17.el5-x86_64) OpenJDK 64-Bit Server VM (build 14.0-b16, mixed mode)) and 2 loadrunner test clients. After ramping up the concurrent requests (about 5min) we reach the maximum at about 450req/sec and get server busy errors. We see a high thread contention on InternalComponentResourcesImpl.postRenderCleanup currently with the Loop component, as there 10 Loop on the Index page. Is there a workaround possible without removing the Loop component from the page to increase the throughput? The thread dumps series looks like this: 1 thread locks 0xe3858990 and over 400 threads are waiting. This lock is persistent over a thread dumps series. I guess the private synchronized MapString, Object getRenderVariables(boolean create) call hits us. http-9080-188 daemon prio=10 tid=0x4d463000 nid=0x382a runnable [0x55b2f000] java.lang.Thread.State: RUNNABLE at org.apache.tapestry5.internal.transform.ParameterWorker$3$1.getState(ParameterWorker.java:206) at org.apache.tapestry5.internal.transform.ParameterWorker$3$1.reset(ParameterWorker.java:302) at org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl$1.work(InternalComponentResourcesImpl.java:136) at org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl$1.work(InternalComponentResourcesImpl.java:133) at org.apache.tapestry5.internal.util.NamedSet.eachValue(NamedSet.java:171) - locked 0xe3858990 (a org.apache.tapestry5.internal.util.NamedSet) at org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl.resetParameterConduits(InternalComponentResourcesImpl.java:546) - locked 0xe385c038 (a org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl) at org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl.postRenderCleanup(InternalComponentResourcesImpl.java:522) at org.apache.tapestry5.corelib.components.Loop.postRenderCleanup(Loop.java) at org.apache.tapestry5.internal.structure.ComponentPageElementImpl$1.run(ComponentPageElementImpl.java:85) at org.apache.tapestry5.internal.structure.ComponentPageElementImpl.invoke(ComponentPageElementImpl.java:956) at org.apache.tapestry5.internal.structure.ComponentPageElementImpl.access$1800(ComponentPageElementImpl.java:61) at org.apache.tapestry5.internal.structure.ComponentPageElementImpl$PostRenderCleanupPhase.render(ComponentPageElementImpl.java:443) at org.apache.tapestry5.internal.services.RenderQueueImpl.run(RenderQueueImpl.java:72) at org.apache.tapestry5.internal.services.PageRenderQueueImpl.render(PageRenderQueueImpl.java:124) at $PageRenderQueue_135675e1f6b934.render(Unknown Source) at $PageRenderQueue_135675e1f6b933.render(Unknown Source) at org.apache.tapestry5.internal.services.MarkupRendererTerminator.renderMarkup(MarkupRendererTerminator.java:37) ... http-9080-499 daemon prio=10 tid=0x4dffb000 nid=0x3b7d waiting for monitor entry [0x69063000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl.getRenderVariables(InternalComponentResourcesImpl.java:476) - waiting to lock 0xe385c038 (a org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl) at org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl.postRenderCleanup(InternalComponentResourcesImpl.java:517) at org.apache.tapestry5.corelib.components.Loop.postRenderCleanup(Loop.java) at org.apache.tapestry5.internal.structure.ComponentPageElementImpl$1.run(ComponentPageElementImpl.java:85) at org.apache.tapestry5.internal.structure.ComponentPageElementImpl.invoke(ComponentPageElementImpl.java:956) at org.apache.tapestry5.internal.structure.ComponentPageElementImpl.access$1800(ComponentPageElementImpl.java:61) at org.apache.tapestry5.internal.structure.ComponentPageElementImpl$PostRenderCleanupPhase.render(ComponentPageElementImpl.java:443) at org.apache.tapestry5.internal.services.RenderQueueImpl.run(RenderQueueImpl.java:72) at org.apache.tapestry5.internal.services.PageRenderQueueImpl.render(PageRenderQueueImpl.java:124) at $PageRenderQueue_135675e1f6b934.render(Unknown Source) at $PageRenderQueue_135675e1f6b933.render(Unknown Source) at
Re: high thread contention during load test on InternalComponentResourcesImpl.postRenderCleanup and org.apache.tapestry5.internal.util.NamedSet.getValues
Yes, I really should have asked you to verify that earlier. I'm running the final batch of tests on the 5.4 code, then will be backporting it as 5.3.4-rc-1. On Fri, May 11, 2012 at 9:20 PM, Robert Lentz rob...@teksolv.de wrote: Forgot to mention that we are using Tapestry 5.3.3. Regards Robert Am 11.05.2012 18:44, schrieb Robert Lentz: Howard, thank you very much for the quick and insightful response. I haven't yet understood the code completely, why that many synchronized methods are invoked like ( private synchronized void resetParameterConduits() etc) I would be really happy to test your shortly and crank out a 5.3.4-rc-1 snapshot version as soon as it will be available! Let me know, if I can assist you in someway. All the Best and Thx Robert Am 11.05.2012 18:34, schrieb Howard Lewis Ship: In case I wasn't clear; I'm waiting for a commitment that you will retest with updated Tapestry code before I make the changes. I'm willing to juggle my schedule to assist, if you're ready to do your part. - To unsubscribe, e-mail: users-unsubscr...@tapestry.apache.org For additional commands, e-mail: users-h...@tapestry.apache.org -- Howard M. Lewis Ship Creator of Apache Tapestry The source for Tapestry training, mentoring and support. Contact me to learn how I can get you up and productive in Tapestry fast! (971) 678-5210 http://howardlewisship.com - To unsubscribe, e-mail: users-unsubscr...@tapestry.apache.org For additional commands, e-mail: users-h...@tapestry.apache.org
high thread contention during load test on InternalComponentResourcesImpl.postRenderCleanup and org.apache.tapestry5.internal.util.NamedSet.getValues
Hi All, we want to rollout a Tapestry app very shortly, but we struggle with load testing issues. We are currently load testing on one Tomcat 6.0.32: - 500 worker threads, tapestry.production-mode=true - Intel(R) Xeon(R) CPU X7460 @ 2.66GHz - OpenJDK Runtime Environment (IcedTea6 1.7.10) (rhel-1.20.b17.el5-x86_64) OpenJDK 64-Bit Server VM (build 14.0-b16, mixed mode)) and 2 loadrunner test clients. After ramping up the concurrent requests (about 5min) we reach the maximum at about 450req/sec and get server busy errors. We see a high thread contention on InternalComponentResourcesImpl.postRenderCleanup currently with the Loop component, as there 10 Loop on the Index page. Is there a workaround possible without removing the Loop component from the page to increase the throughput? The thread dumps series looks like this: 1 thread locks 0xe3858990 and over 400 threads are waiting. This lock is persistent over a thread dumps series. I guess the private synchronized MapString, Object getRenderVariables(boolean create) call hits us. http-9080-188 daemon prio=10 tid=0x4d463000 nid=0x382a runnable [0x55b2f000] java.lang.Thread.State: RUNNABLE at org.apache.tapestry5.internal.transform.ParameterWorker$3$1.getState(ParameterWorker.java:206) at org.apache.tapestry5.internal.transform.ParameterWorker$3$1.reset(ParameterWorker.java:302) at org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl$1.work(InternalComponentResourcesImpl.java:136) at org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl$1.work(InternalComponentResourcesImpl.java:133) at org.apache.tapestry5.internal.util.NamedSet.eachValue(NamedSet.java:171) - locked 0xe3858990 (a org.apache.tapestry5.internal.util.NamedSet) at org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl.resetParameterConduits(InternalComponentResourcesImpl.java:546) - locked 0xe385c038 (a org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl) at org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl.postRenderCleanup(InternalComponentResourcesImpl.java:522) at org.apache.tapestry5.corelib.components.Loop.postRenderCleanup(Loop.java) at org.apache.tapestry5.internal.structure.ComponentPageElementImpl$1.run(ComponentPageElementImpl.java:85) at org.apache.tapestry5.internal.structure.ComponentPageElementImpl.invoke(ComponentPageElementImpl.java:956) at org.apache.tapestry5.internal.structure.ComponentPageElementImpl.access$1800(ComponentPageElementImpl.java:61) at org.apache.tapestry5.internal.structure.ComponentPageElementImpl$PostRenderCleanupPhase.render(ComponentPageElementImpl.java:443) at org.apache.tapestry5.internal.services.RenderQueueImpl.run(RenderQueueImpl.java:72) at org.apache.tapestry5.internal.services.PageRenderQueueImpl.render(PageRenderQueueImpl.java:124) at $PageRenderQueue_135675e1f6b934.render(Unknown Source) at $PageRenderQueue_135675e1f6b933.render(Unknown Source) at org.apache.tapestry5.internal.services.MarkupRendererTerminator.renderMarkup(MarkupRendererTerminator.java:37) ... http-9080-499 daemon prio=10 tid=0x4dffb000 nid=0x3b7d waiting for monitor entry [0x69063000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl.getRenderVariables(InternalComponentResourcesImpl.java:476) - waiting to lock 0xe385c038 (a org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl) at org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl.postRenderCleanup(InternalComponentResourcesImpl.java:517) at org.apache.tapestry5.corelib.components.Loop.postRenderCleanup(Loop.java) at org.apache.tapestry5.internal.structure.ComponentPageElementImpl$1.run(ComponentPageElementImpl.java:85) at org.apache.tapestry5.internal.structure.ComponentPageElementImpl.invoke(ComponentPageElementImpl.java:956) at org.apache.tapestry5.internal.structure.ComponentPageElementImpl.access$1800(ComponentPageElementImpl.java:61) at org.apache.tapestry5.internal.structure.ComponentPageElementImpl$PostRenderCleanupPhase.render(ComponentPageElementImpl.java:443) at org.apache.tapestry5.internal.services.RenderQueueImpl.run(RenderQueueImpl.java:72) at org.apache.tapestry5.internal.services.PageRenderQueueImpl.render(PageRenderQueueImpl.java:124) at $PageRenderQueue_135675e1f6b934.render(Unknown Source) at $PageRenderQueue_135675e1f6b933.render(Unknown Source) at org.apache.tapestry5.internal.services.MarkupRendererTerminator.renderMarkup(MarkupRendererTerminator.java:37) ... Additionally we experienced a similar issue when using a component with a mixin annotation in a loop, that was rendered more than 20 times on the page. The contention here was on the org.apache.tapestry5.internal.util.NamedSet.getValues call http-9080-79
Re: high thread contention during load test on InternalComponentResourcesImpl.postRenderCleanup and org.apache.tapestry5.internal.util.NamedSet.getValues
That's an interesting result. Why, oh why, did I ever implement render variables? So the conventional thinking on synchronized methods is that they where there is limited contention, and the method is short, synchronized is the best way to go. This method exists, and is synchronized, to avoid creating a PerThreadValue to store the render variable's Map on every component and mixin in the application. That adds up, given that only a tiny percentage of pages or components will ever use a render variable. I'd be willing to replace this code with something that uses an explicit per-instance Lock ... that would likely help with your high contention issues, as after the PerThreadValue is created, and the write lock is released, there would no longer be contention, as all threads could share the read lock. I can put that together shortly and crank out a 5.3.4-rc-1 snapshot, if you can then pick it up and test it. Further comments here: https://issues.apache.org/jira/browse/TAP5-1929 On Fri, May 11, 2012 at 6:07 AM, Robert Lentz rob...@teksolv.de wrote: Hi All, we want to rollout a Tapestry app very shortly, but we struggle with load testing issues. We are currently load testing on one Tomcat 6.0.32: - 500 worker threads, tapestry.production-mode=true - Intel(R) Xeon(R) CPU X7460 @ 2.66GHz - OpenJDK Runtime Environment (IcedTea6 1.7.10) (rhel-1.20.b17.el5-x86_64) OpenJDK 64-Bit Server VM (build 14.0-b16, mixed mode)) and 2 loadrunner test clients. After ramping up the concurrent requests (about 5min) we reach the maximum at about 450req/sec and get server busy errors. We see a high thread contention on InternalComponentResourcesImpl.postRenderCleanup currently with the Loop component, as there 10 Loop on the Index page. Is there a workaround possible without removing the Loop component from the page to increase the throughput? The thread dumps series looks like this: 1 thread locks 0xe3858990 and over 400 threads are waiting. This lock is persistent over a thread dumps series. I guess the private synchronized MapString, Object getRenderVariables(boolean create) call hits us. http-9080-188 daemon prio=10 tid=0x4d463000 nid=0x382a runnable [0x55b2f000] java.lang.Thread.State: RUNNABLE at org.apache.tapestry5.internal.transform.ParameterWorker$3$1.getState(ParameterWorker.java:206) at org.apache.tapestry5.internal.transform.ParameterWorker$3$1.reset(ParameterWorker.java:302) at org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl$1.work(InternalComponentResourcesImpl.java:136) at org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl$1.work(InternalComponentResourcesImpl.java:133) at org.apache.tapestry5.internal.util.NamedSet.eachValue(NamedSet.java:171) - locked 0xe3858990 (a org.apache.tapestry5.internal.util.NamedSet) at org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl.resetParameterConduits(InternalComponentResourcesImpl.java:546) - locked 0xe385c038 (a org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl) at org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl.postRenderCleanup(InternalComponentResourcesImpl.java:522) at org.apache.tapestry5.corelib.components.Loop.postRenderCleanup(Loop.java) at org.apache.tapestry5.internal.structure.ComponentPageElementImpl$1.run(ComponentPageElementImpl.java:85) at org.apache.tapestry5.internal.structure.ComponentPageElementImpl.invoke(ComponentPageElementImpl.java:956) at org.apache.tapestry5.internal.structure.ComponentPageElementImpl.access$1800(ComponentPageElementImpl.java:61) at org.apache.tapestry5.internal.structure.ComponentPageElementImpl$PostRenderCleanupPhase.render(ComponentPageElementImpl.java:443) at org.apache.tapestry5.internal.services.RenderQueueImpl.run(RenderQueueImpl.java:72) at org.apache.tapestry5.internal.services.PageRenderQueueImpl.render(PageRenderQueueImpl.java:124) at $PageRenderQueue_135675e1f6b934.render(Unknown Source) at $PageRenderQueue_135675e1f6b933.render(Unknown Source) at org.apache.tapestry5.internal.services.MarkupRendererTerminator.renderMarkup(MarkupRendererTerminator.java:37) ... http-9080-499 daemon prio=10 tid=0x4dffb000 nid=0x3b7d waiting for monitor entry [0x69063000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl.getRenderVariables(InternalComponentResourcesImpl.java:476) - waiting to lock 0xe385c038 (a org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl) at org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl.postRenderCleanup(InternalComponentResourcesImpl.java:517) at org.apache.tapestry5.corelib.components.Loop.postRenderCleanup(Loop.java) at
Re: high thread contention during load test on InternalComponentResourcesImpl.postRenderCleanup and org.apache.tapestry5.internal.util.NamedSet.getValues
Would it be better to put synchronized block inside null check block like:
if (renderVariables == null) { synchronized(this) { ...
If I'm not mistaken resetParametersConduits doesn't need to be synchronized it
should only change per thread value.
Denis
May 11, 2012 v 6:06 PM, Howard Lewis Ship:
That's an interesting result. Why, oh why, did I ever implement
render variables?
So the conventional thinking on synchronized methods is that they
where there is limited contention, and the method is short,
synchronized is the best way to go.
This method exists, and is synchronized, to avoid creating a
PerThreadValue to store the render variable's Map on every component
and mixin in the application. That adds up, given that only a tiny
percentage of pages or components will ever use a render variable.
I'd be willing to replace this code with something that uses an
explicit per-instance Lock ... that would likely help with your high
contention issues, as after the PerThreadValue is created, and the
write lock is released, there would no longer be contention, as all
threads could share the read lock.
I can put that together shortly and crank out a 5.3.4-rc-1 snapshot,
if you can then pick it up and test it.
Further comments here: https://issues.apache.org/jira/browse/TAP5-1929
On Fri, May 11, 2012 at 6:07 AM, Robert Lentz rob...@teksolv.de wrote:
Hi All,
we want to rollout a Tapestry app very shortly, but we struggle with
load testing issues. We are currently load testing on one Tomcat 6.0.32:
- 500 worker threads, tapestry.production-mode=true
- Intel(R) Xeon(R) CPU X7460 @ 2.66GHz
- OpenJDK Runtime Environment (IcedTea6 1.7.10)
(rhel-1.20.b17.el5-x86_64) OpenJDK 64-Bit Server VM (build 14.0-b16,
mixed mode))
and 2 loadrunner test clients.
After ramping up the concurrent requests (about 5min) we reach the
maximum at about 450req/sec and get server busy errors. We see a high
thread contention on InternalComponentResourcesImpl.postRenderCleanup
currently with the Loop component, as there 10 Loop on the Index page.
Is there a workaround possible without removing the Loop component from
the page to increase the throughput? The thread dumps series looks like
this: 1 thread locks 0xe3858990 and over 400 threads are
waiting. This lock is persistent over a thread dumps series. I guess the
private synchronized MapString, Object getRenderVariables(boolean
create) call hits us.
http-9080-188 daemon prio=10 tid=0x4d463000 nid=0x382a
runnable [0x55b2f000]
java.lang.Thread.State: RUNNABLE
at
org.apache.tapestry5.internal.transform.ParameterWorker$3$1.getState(ParameterWorker.java:206)
at
org.apache.tapestry5.internal.transform.ParameterWorker$3$1.reset(ParameterWorker.java:302)
at
org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl$1.work(InternalComponentResourcesImpl.java:136)
at
org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl$1.work(InternalComponentResourcesImpl.java:133)
at
org.apache.tapestry5.internal.util.NamedSet.eachValue(NamedSet.java:171)
- locked 0xe3858990 (a
org.apache.tapestry5.internal.util.NamedSet)
at
org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl.resetParameterConduits(InternalComponentResourcesImpl.java:546)
- locked 0xe385c038 (a
org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl)
at
org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl.postRenderCleanup(InternalComponentResourcesImpl.java:522)
at
org.apache.tapestry5.corelib.components.Loop.postRenderCleanup(Loop.java)
at
org.apache.tapestry5.internal.structure.ComponentPageElementImpl$1.run(ComponentPageElementImpl.java:85)
at
org.apache.tapestry5.internal.structure.ComponentPageElementImpl.invoke(ComponentPageElementImpl.java:956)
at
org.apache.tapestry5.internal.structure.ComponentPageElementImpl.access$1800(ComponentPageElementImpl.java:61)
at
org.apache.tapestry5.internal.structure.ComponentPageElementImpl$PostRenderCleanupPhase.render(ComponentPageElementImpl.java:443)
at
org.apache.tapestry5.internal.services.RenderQueueImpl.run(RenderQueueImpl.java:72)
at
org.apache.tapestry5.internal.services.PageRenderQueueImpl.render(PageRenderQueueImpl.java:124)
at $PageRenderQueue_135675e1f6b934.render(Unknown Source)
at $PageRenderQueue_135675e1f6b933.render(Unknown Source)
at
org.apache.tapestry5.internal.services.MarkupRendererTerminator.renderMarkup(MarkupRendererTerminator.java:37)
...
http-9080-499 daemon prio=10 tid=0x4dffb000 nid=0x3b7d waiting
for monitor entry [0x69063000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl.getRenderVariables(InternalComponentResourcesImpl.java:476)
- waiting to lock 0xe385c038 (a
Re: high thread contention during load test on InternalComponentResourcesImpl.postRenderCleanup and org.apache.tapestry5.internal.util.NamedSet.getValues
That's a common mistake. Although I've heard that future JDK/JVM may
address this common case, it is not valid to read a shared variable
outside a synchronized block, even to check for null-ness. This was a
common mistake propagated in books and blogs in the early 2000's.
Another alternative would be to make the field final and assign it in
the constructor, but that means you will have 1000's (or 10,000's) of
these floating around doing nothing. I've been working to make
Tapestry's runtime profile slimmer, so that would be a step backwards.
On Fri, May 11, 2012 at 9:16 AM, Denis Stepanov
denis.stepa...@gmail.com wrote:
Would it be better to put synchronized block inside null check block like:
if (renderVariables == null) { synchronized(this) { ...
If I'm not mistaken resetParametersConduits doesn't need to be synchronized
it should only change per thread value.
Denis
May 11, 2012 v 6:06 PM, Howard Lewis Ship:
That's an interesting result. Why, oh why, did I ever implement
render variables?
So the conventional thinking on synchronized methods is that they
where there is limited contention, and the method is short,
synchronized is the best way to go.
This method exists, and is synchronized, to avoid creating a
PerThreadValue to store the render variable's Map on every component
and mixin in the application. That adds up, given that only a tiny
percentage of pages or components will ever use a render variable.
I'd be willing to replace this code with something that uses an
explicit per-instance Lock ... that would likely help with your high
contention issues, as after the PerThreadValue is created, and the
write lock is released, there would no longer be contention, as all
threads could share the read lock.
I can put that together shortly and crank out a 5.3.4-rc-1 snapshot,
if you can then pick it up and test it.
Further comments here: https://issues.apache.org/jira/browse/TAP5-1929
On Fri, May 11, 2012 at 6:07 AM, Robert Lentz rob...@teksolv.de wrote:
Hi All,
we want to rollout a Tapestry app very shortly, but we struggle with
load testing issues. We are currently load testing on one Tomcat 6.0.32:
- 500 worker threads, tapestry.production-mode=true
- Intel(R) Xeon(R) CPU X7460 @ 2.66GHz
- OpenJDK Runtime Environment (IcedTea6 1.7.10)
(rhel-1.20.b17.el5-x86_64) OpenJDK 64-Bit Server VM (build 14.0-b16,
mixed mode))
and 2 loadrunner test clients.
After ramping up the concurrent requests (about 5min) we reach the
maximum at about 450req/sec and get server busy errors. We see a high
thread contention on InternalComponentResourcesImpl.postRenderCleanup
currently with the Loop component, as there 10 Loop on the Index page.
Is there a workaround possible without removing the Loop component from
the page to increase the throughput? The thread dumps series looks like
this: 1 thread locks 0xe3858990 and over 400 threads are
waiting. This lock is persistent over a thread dumps series. I guess the
private synchronized MapString, Object getRenderVariables(boolean
create) call hits us.
http-9080-188 daemon prio=10 tid=0x4d463000 nid=0x382a
runnable [0x55b2f000]
java.lang.Thread.State: RUNNABLE
at
org.apache.tapestry5.internal.transform.ParameterWorker$3$1.getState(ParameterWorker.java:206)
at
org.apache.tapestry5.internal.transform.ParameterWorker$3$1.reset(ParameterWorker.java:302)
at
org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl$1.work(InternalComponentResourcesImpl.java:136)
at
org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl$1.work(InternalComponentResourcesImpl.java:133)
at
org.apache.tapestry5.internal.util.NamedSet.eachValue(NamedSet.java:171)
- locked 0xe3858990 (a
org.apache.tapestry5.internal.util.NamedSet)
at
org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl.resetParameterConduits(InternalComponentResourcesImpl.java:546)
- locked 0xe385c038 (a
org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl)
at
org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl.postRenderCleanup(InternalComponentResourcesImpl.java:522)
at
org.apache.tapestry5.corelib.components.Loop.postRenderCleanup(Loop.java)
at
org.apache.tapestry5.internal.structure.ComponentPageElementImpl$1.run(ComponentPageElementImpl.java:85)
at
org.apache.tapestry5.internal.structure.ComponentPageElementImpl.invoke(ComponentPageElementImpl.java:956)
at
org.apache.tapestry5.internal.structure.ComponentPageElementImpl.access$1800(ComponentPageElementImpl.java:61)
at
org.apache.tapestry5.internal.structure.ComponentPageElementImpl$PostRenderCleanupPhase.render(ComponentPageElementImpl.java:443)
at
org.apache.tapestry5.internal.services.RenderQueueImpl.run(RenderQueueImpl.java:72)
at
Re: high thread contention during load test on InternalComponentResourcesImpl.postRenderCleanup and org.apache.tapestry5.internal.util.NamedSet.getValues
In case I wasn't clear; I'm waiting for a commitment that you will retest with updated Tapestry code before I make the changes. I'm willing to juggle my schedule to assist, if you're ready to do your part. On Fri, May 11, 2012 at 9:06 AM, Howard Lewis Ship hls...@gmail.com wrote: That's an interesting result. Why, oh why, did I ever implement render variables? So the conventional thinking on synchronized methods is that they where there is limited contention, and the method is short, synchronized is the best way to go. This method exists, and is synchronized, to avoid creating a PerThreadValue to store the render variable's Map on every component and mixin in the application. That adds up, given that only a tiny percentage of pages or components will ever use a render variable. I'd be willing to replace this code with something that uses an explicit per-instance Lock ... that would likely help with your high contention issues, as after the PerThreadValue is created, and the write lock is released, there would no longer be contention, as all threads could share the read lock. I can put that together shortly and crank out a 5.3.4-rc-1 snapshot, if you can then pick it up and test it. Further comments here: https://issues.apache.org/jira/browse/TAP5-1929 On Fri, May 11, 2012 at 6:07 AM, Robert Lentz rob...@teksolv.de wrote: Hi All, we want to rollout a Tapestry app very shortly, but we struggle with load testing issues. We are currently load testing on one Tomcat 6.0.32: - 500 worker threads, tapestry.production-mode=true - Intel(R) Xeon(R) CPU X7460 @ 2.66GHz - OpenJDK Runtime Environment (IcedTea6 1.7.10) (rhel-1.20.b17.el5-x86_64) OpenJDK 64-Bit Server VM (build 14.0-b16, mixed mode)) and 2 loadrunner test clients. After ramping up the concurrent requests (about 5min) we reach the maximum at about 450req/sec and get server busy errors. We see a high thread contention on InternalComponentResourcesImpl.postRenderCleanup currently with the Loop component, as there 10 Loop on the Index page. Is there a workaround possible without removing the Loop component from the page to increase the throughput? The thread dumps series looks like this: 1 thread locks 0xe3858990 and over 400 threads are waiting. This lock is persistent over a thread dumps series. I guess the private synchronized MapString, Object getRenderVariables(boolean create) call hits us. http-9080-188 daemon prio=10 tid=0x4d463000 nid=0x382a runnable [0x55b2f000] java.lang.Thread.State: RUNNABLE at org.apache.tapestry5.internal.transform.ParameterWorker$3$1.getState(ParameterWorker.java:206) at org.apache.tapestry5.internal.transform.ParameterWorker$3$1.reset(ParameterWorker.java:302) at org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl$1.work(InternalComponentResourcesImpl.java:136) at org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl$1.work(InternalComponentResourcesImpl.java:133) at org.apache.tapestry5.internal.util.NamedSet.eachValue(NamedSet.java:171) - locked 0xe3858990 (a org.apache.tapestry5.internal.util.NamedSet) at org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl.resetParameterConduits(InternalComponentResourcesImpl.java:546) - locked 0xe385c038 (a org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl) at org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl.postRenderCleanup(InternalComponentResourcesImpl.java:522) at org.apache.tapestry5.corelib.components.Loop.postRenderCleanup(Loop.java) at org.apache.tapestry5.internal.structure.ComponentPageElementImpl$1.run(ComponentPageElementImpl.java:85) at org.apache.tapestry5.internal.structure.ComponentPageElementImpl.invoke(ComponentPageElementImpl.java:956) at org.apache.tapestry5.internal.structure.ComponentPageElementImpl.access$1800(ComponentPageElementImpl.java:61) at org.apache.tapestry5.internal.structure.ComponentPageElementImpl$PostRenderCleanupPhase.render(ComponentPageElementImpl.java:443) at org.apache.tapestry5.internal.services.RenderQueueImpl.run(RenderQueueImpl.java:72) at org.apache.tapestry5.internal.services.PageRenderQueueImpl.render(PageRenderQueueImpl.java:124) at $PageRenderQueue_135675e1f6b934.render(Unknown Source) at $PageRenderQueue_135675e1f6b933.render(Unknown Source) at org.apache.tapestry5.internal.services.MarkupRendererTerminator.renderMarkup(MarkupRendererTerminator.java:37) ... http-9080-499 daemon prio=10 tid=0x4dffb000 nid=0x3b7d waiting for monitor entry [0x69063000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl.getRenderVariables(InternalComponentResourcesImpl.java:476) - waiting to lock 0xe385c038 (a
Re: high thread contention during load test on InternalComponentResourcesImpl.postRenderCleanup and org.apache.tapestry5.internal.util.NamedSet.getValues
Howard, thank you very much for the quick and insightful response. I haven't yet understood the code completely, why that many synchronized methods are invoked like ( private synchronized void resetParameterConduits() etc) I would be really happy to test your shortly and crank out a 5.3.4-rc-1 snapshot version as soon as it will be available! Let me know, if I can assist you in someway. All the Best and Thx Robert Am 11.05.2012 18:34, schrieb Howard Lewis Ship: In case I wasn't clear; I'm waiting for a commitment that you will retest with updated Tapestry code before I make the changes. I'm willing to juggle my schedule to assist, if you're ready to do your part. On Fri, May 11, 2012 at 9:06 AM, Howard Lewis Ship hls...@gmail.com wrote: That's an interesting result. Why, oh why, did I ever implement render variables? So the conventional thinking on synchronized methods is that they where there is limited contention, and the method is short, synchronized is the best way to go. This method exists, and is synchronized, to avoid creating a PerThreadValue to store the render variable's Map on every component and mixin in the application. That adds up, given that only a tiny percentage of pages or components will ever use a render variable. I'd be willing to replace this code with something that uses an explicit per-instance Lock ... that would likely help with your high contention issues, as after the PerThreadValue is created, and the write lock is released, there would no longer be contention, as all threads could share the read lock. I can put that together shortly and crank out a 5.3.4-rc-1 snapshot, if you can then pick it up and test it. Further comments here: https://issues.apache.org/jira/browse/TAP5-1929 On Fri, May 11, 2012 at 6:07 AM, Robert Lentz rob...@teksolv.de wrote: Hi All, we want to rollout a Tapestry app very shortly, but we struggle with load testing issues. We are currently load testing on one Tomcat 6.0.32: - 500 worker threads, tapestry.production-mode=true - Intel(R) Xeon(R) CPU X7460 @ 2.66GHz - OpenJDK Runtime Environment (IcedTea6 1.7.10) (rhel-1.20.b17.el5-x86_64) OpenJDK 64-Bit Server VM (build 14.0-b16, mixed mode)) and 2 loadrunner test clients. After ramping up the concurrent requests (about 5min) we reach the maximum at about 450req/sec and get server busy errors. We see a high thread contention on InternalComponentResourcesImpl.postRenderCleanup currently with the Loop component, as there 10 Loop on the Index page. Is there a workaround possible without removing the Loop component from the page to increase the throughput? The thread dumps series looks like this: 1 thread locks 0xe3858990 and over 400 threads are waiting. This lock is persistent over a thread dumps series. I guess the private synchronized MapString, Object getRenderVariables(boolean create) call hits us. http-9080-188 daemon prio=10 tid=0x4d463000 nid=0x382a runnable [0x55b2f000] java.lang.Thread.State: RUNNABLE at org.apache.tapestry5.internal.transform.ParameterWorker$3$1.getState(ParameterWorker.java:206) at org.apache.tapestry5.internal.transform.ParameterWorker$3$1.reset(ParameterWorker.java:302) at org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl$1.work(InternalComponentResourcesImpl.java:136) at org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl$1.work(InternalComponentResourcesImpl.java:133) at org.apache.tapestry5.internal.util.NamedSet.eachValue(NamedSet.java:171) - locked 0xe3858990 (a org.apache.tapestry5.internal.util.NamedSet) at org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl.resetParameterConduits(InternalComponentResourcesImpl.java:546) - locked 0xe385c038 (a org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl) at org.apache.tapestry5.internal.structure.InternalComponentResourcesImpl.postRenderCleanup(InternalComponentResourcesImpl.java:522) at org.apache.tapestry5.corelib.components.Loop.postRenderCleanup(Loop.java) at org.apache.tapestry5.internal.structure.ComponentPageElementImpl$1.run(ComponentPageElementImpl.java:85) at org.apache.tapestry5.internal.structure.ComponentPageElementImpl.invoke(ComponentPageElementImpl.java:956) at org.apache.tapestry5.internal.structure.ComponentPageElementImpl.access$1800(ComponentPageElementImpl.java:61) at org.apache.tapestry5.internal.structure.ComponentPageElementImpl$PostRenderCleanupPhase.render(ComponentPageElementImpl.java:443) at org.apache.tapestry5.internal.services.RenderQueueImpl.run(RenderQueueImpl.java:72) at org.apache.tapestry5.internal.services.PageRenderQueueImpl.render(PageRenderQueueImpl.java:124) at $PageRenderQueue_135675e1f6b934.render(Unknown Source) at $PageRenderQueue_135675e1f6b933.render(Unknown Source) at
Re: high thread contention during load test on InternalComponentResourcesImpl.postRenderCleanup and org.apache.tapestry5.internal.util.NamedSet.getValues
On Fri, May 11, 2012 at 6:33 PM, Howard Lewis Ship hls...@gmail.com wrote: That's a common mistake. Although I've heard that future JDK/JVM may address this common case, it is not valid to read a shared variable outside a synchronized block, even to check for null-ness. This was a common mistake propagated in books and blogs in the early 2000's. I believe that Java Memory Model was changed in Java 5 to fix this problem (JSR-133). You still need to mark a shared variable 'volatile', but checking for null-ness should be OK. - Cezary
Re: high thread contention during load test on InternalComponentResourcesImpl.postRenderCleanup and org.apache.tapestry5.internal.util.NamedSet.getValues
Forgot to mention that we are using Tapestry 5.3.3. Regards Robert Am 11.05.2012 18:44, schrieb Robert Lentz: Howard, thank you very much for the quick and insightful response. I haven't yet understood the code completely, why that many synchronized methods are invoked like ( private synchronized void resetParameterConduits() etc) I would be really happy to test your shortly and crank out a 5.3.4-rc-1 snapshot version as soon as it will be available! Let me know, if I can assist you in someway. All the Best and Thx Robert Am 11.05.2012 18:34, schrieb Howard Lewis Ship: In case I wasn't clear; I'm waiting for a commitment that you will retest with updated Tapestry code before I make the changes. I'm willing to juggle my schedule to assist, if you're ready to do your part. - To unsubscribe, e-mail: users-unsubscr...@tapestry.apache.org For additional commands, e-mail: users-h...@tapestry.apache.org
