Re: Memory consumption in T4.1.2 - Hard data
Hi Jesse -- Thanks for your work on Tapestry. I have noticed an issue around memory usage with our automated tests which extensively use hivemind. I was wondering if anyone has investigated hivemind memory leaks. I am going to do some further investigating as time permits after Tuesday. That being said, we have a publicly-accessible machine with the latest yourkit 7m2 release on it that could be used to get long-running memory information.Contact me off-list about getting access to the instance.This is our production machine - but since we are still in beta we don't have many users (i.e. no one will notice a little memory profiling). I would suggest the Tapestry community pitch in a little to help gather data to track down this issue - or determine it is a non-issue. One of open source's advantages is the power of the many to solve issues like this. I for one don't want to rely just on jesse to solve what sounds to be very serious barrier to my own personal success. -Pat Moore Amplafi On 8/28/07, Jesse Kuhnert [EMAIL PROTECTED] wrote: Umm yeah. I have looked in to it as I just said. The only thing left is for me to look at this youkit snapshot profile and play from there but I'm not capable of debugging issues that aren't reported properly. On 8/28/07, Peter Stavrinides [EMAIL PROTECTED] wrote: Hi Jessie I'm gonna try update my version of ognl, if that won't work I will downgrade. I have not filed any bugs yet, but I sent a couple of emails to the list. I don't get any exceptions during development, but that is because its not running for 3 or 4 days. I strongly suggest you consider looking into it, because I use the standard libraries provided by Tapestry through Maven. my configuration is as follows: JDK 6.01 32bit JVM (I have also tested on a 64 bit with no luck) Tomcat 5.5.20 Debian Linux (2.6.15.28 kernel) Tapestry 4.1.2-SNAPSHOT with ognl 2.7 regards, Peter Jesse Kuhnert wrote: I'd downgrade then if I were you. Extensive profiling with yourkit hasn't shed any new light on whatever problems others are having. The OGNL classes indeed aren't being kept in the javassist pool from what I can tell. I have another yourkit snapshot binary to look at so we'll see what that says. I don't remember - have you filed any bug reports or reported any problems? Are you getting ognl exceptions during development as well as production? Do these errors sound like umpotential errors that should be looked at further? On 8/28/07, Peter Stavrinides [EMAIL PROTECTED] wrote: Hi Jessie Any progress on this? sorry to bug you, but I have to take a decision soon, I have two production machines that will need to upgrade or downgrade Tapestry. Best wishes, Peter Jon Oakes wrote: Hi Bryan, I am a relative newbie but I was wondering if you are running with caching enabled or disabled? It might make sense to keep things around when caching is disabled whereas I think it would clearly be a bug to keep things around with it disabled. Jon Oakes Jesse Kuhnert wrote: Hmmm...well, I don't think I like the sound of any of that. I'm just going to pretend this problem doesn't exist. (just kidding) I had thought I was doing something special with the ognl compilations that would cause its generated classes to not hang around afterwards in any pools. I'll take a look at things this weekend and am sure a fix will appear in between now and Monday - if there is a reasonable fix to be found. (in 4.1.3 snapshot form) On 8/24/07, Bryan Dotzour [EMAIL PROTECTED] mailto: [EMAIL PROTECTED] wrote: I and another colleague of mine have been investigating what seems to be a memory leak in our Tapestry application for about a month since we upgraded to T4.1.2. I won't bore you with the saga of the last month, but I would like to present the data I've gathered and look to the list for a proposed solution. I was reading a recent thread in which Jesse said (08/09/2007): There is a map that grows as large as the system using it internally to javassist of various cached reflection info - but it doesn't leak in any way. This is precisely what I've found in profiling our application and it *appears* to be this map that is causing our applications to eventually run out of memory. The YourKit profiler shows me that, as time goes on, there is an instance of HiveMindClassPool that grows and grows as class instances are created. This class extends from javassist.ClassPool and is the map that Jesse is talking about in his quote above. And he's right, I wouldn't say that the class pool leaks either because it looks like it's designed to retain that memory until the class pool itself is no longer needed. Take this quote from the
RE: Memory consumption in T4.1.2 - Hard data
But that POM does use snapshots. You shouldn't need the repository http://people.apache.org/repo/m2-snapshot-repository at all. Probably, there are no very significant differences between the latest 4.1.2 snapshot and the release. But nevertheless, for a productive environment, I'd always go with a released version. -Original Message- From: Peter Stavrinides [mailto:[EMAIL PROTECTED] Sent: Wednesday, August 29, 2007 7:45 AM To: Tapestry users Subject: Re: Memory consumption in T4.1.2 - Hard data This is my POM: repositories repository releases updatePolicyalways/updatePolicy checksumPolicywarn/checksumPolicy /releases snapshots updatePolicyalways/updatePolicy checksumPolicyfail/checksumPolicy repository idapache.snapshots/id urlhttp://people.apache.org/repo/m2-snapshot-repository/url /repository /repositories dependency groupIdorg.apache.tapestry/groupId artifactIdtapestry-framework/artifactId version4.1.2-SNAPSHOT/version scopetest/scope /dependency dependency groupIdorg.apache.tapestry/groupId artifactIdtapestry-annotations/artifactId version4.1.2-SNAPSHOT/version scopetest/scope /dependency dependency groupIdorg.apache.tapestry/groupId artifactIdtapestry-contrib/artifactId version4.1.2-SNAPSHOT/version scopetest/scope /dependency dependency groupIdorg.apache.tapestry/groupId artifactIdtapestry-portlet/artifactId version4.1.2-SNAPSHOT/version scopetest/scope /dependency [EMAIL PROTECTED] wrote: my configuration is as follows: JDK 6.01 32bit JVM (I have also tested on a 64 bit with no luck) Tomcat 5.5.20 Debian Linux (2.6.15.28 kernel) Tapestry 4.1.2-SNAPSHOT with ognl 2.7 Hi Peter, 4.1.2-SNAPSHOT is a typo, I suppose? regards, Peter Jesse Kuhnert wrote: I'd downgrade then if I were you. Extensive profiling with yourkit hasn't shed any new light on whatever problems others are having. The OGNL classes indeed aren't being kept in the javassist pool from what I can tell. I have another yourkit snapshot binary to look at so we'll see what that says. I don't remember - have you filed any bug reports or reported any problems? Are you getting ognl exceptions during development as well as production? Do these errors sound like umpotential errors that should be looked at further? On 8/28/07, Peter Stavrinides [EMAIL PROTECTED] wrote: Hi Jessie Any progress on this? sorry to bug you, but I have to take a decision soon, I have two production machines that will need to upgrade or downgrade Tapestry. Best wishes, Peter Jon Oakes wrote: Hi Bryan, I am a relative newbie but I was wondering if you are running with caching enabled or disabled? It might make sense to keep things around when caching is disabled whereas I think it would clearly be a bug to keep things around with it disabled. Jon Oakes Jesse Kuhnert wrote: Hmmm...well, I don't think I like the sound of any of that. I'm just going to pretend this problem doesn't exist. (just kidding) I had thought I was doing something special with the ognl compilations that would cause its generated classes to not hang around afterwards in any pools. I'll take a look at things this weekend and am sure a fix will appear in between now and Monday - if there is a reasonable fix to be found. (in 4.1.3 snapshot form) On 8/24/07, Bryan Dotzour [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: I and another colleague of mine have been investigating what seems to be a memory leak in our Tapestry application for about a month since we upgraded to T4.1.2. I won't bore you with the saga of the last month, but I would like to present the data I've gathered and look to the list for a proposed solution. I was reading a recent thread in which Jesse said (08/09/2007): There is a map that grows as large as the system using it internally to javassist of various cached reflection info - but it doesn't leak in any way. This is precisely what I've found in profiling our application and it *appears* to be this map that is causing our applications to eventually run out of memory. The YourKit profiler shows me that, as time goes on, there is an instance of HiveMindClassPool that grows and grows as class instances are created. This class extends from javassist.ClassPool and is the map that Jesse is talking about in his quote above. And he's right, I wouldn't say that the class
Re: Memory consumption in T4.1.2 - Hard data
Correct me if I am wrong 4.1.2 is not released as yet, and not in the main repository, so how else would I get to it? [EMAIL PROTECTED] wrote: But that POM does use snapshots. You shouldn't need the repository http://people.apache.org/repo/m2-snapshot-repository at all. Probably, there are no very significant differences between the latest 4.1.2 snapshot and the release. But nevertheless, for a productive environment, I'd always go with a released version. -Original Message- From: Peter Stavrinides [mailto:[EMAIL PROTECTED] Sent: Wednesday, August 29, 2007 7:45 AM To: Tapestry users Subject: Re: Memory consumption in T4.1.2 - Hard data This is my POM: repositories repository releases updatePolicyalways/updatePolicy checksumPolicywarn/checksumPolicy /releases snapshots updatePolicyalways/updatePolicy checksumPolicyfail/checksumPolicy repository idapache.snapshots/id urlhttp://people.apache.org/repo/m2-snapshot-repository/url /repository /repositories dependency groupIdorg.apache.tapestry/groupId artifactIdtapestry-framework/artifactId version4.1.2-SNAPSHOT/version scopetest/scope /dependency dependency groupIdorg.apache.tapestry/groupId artifactIdtapestry-annotations/artifactId version4.1.2-SNAPSHOT/version scopetest/scope /dependency dependency groupIdorg.apache.tapestry/groupId artifactIdtapestry-contrib/artifactId version4.1.2-SNAPSHOT/version scopetest/scope /dependency dependency groupIdorg.apache.tapestry/groupId artifactIdtapestry-portlet/artifactId version4.1.2-SNAPSHOT/version scopetest/scope /dependency [EMAIL PROTECTED] wrote: my configuration is as follows: JDK 6.01 32bit JVM (I have also tested on a 64 bit with no luck) Tomcat 5.5.20 Debian Linux (2.6.15.28 kernel) Tapestry 4.1.2-SNAPSHOT with ognl 2.7 Hi Peter, 4.1.2-SNAPSHOT is a typo, I suppose? regards, Peter Jesse Kuhnert wrote: I'd downgrade then if I were you. Extensive profiling with yourkit hasn't shed any new light on whatever problems others are having. The OGNL classes indeed aren't being kept in the javassist pool from what I can tell. I have another yourkit snapshot binary to look at so we'll see what that says. I don't remember - have you filed any bug reports or reported any problems? Are you getting ognl exceptions during development as well as production? Do these errors sound like umpotential errors that should be looked at further? On 8/28/07, Peter Stavrinides [EMAIL PROTECTED] wrote: Hi Jessie Any progress on this? sorry to bug you, but I have to take a decision soon, I have two production machines that will need to upgrade or downgrade Tapestry. Best wishes, Peter Jon Oakes wrote: Hi Bryan, I am a relative newbie but I was wondering if you are running with caching enabled or disabled? It might make sense to keep things around when caching is disabled whereas I think it would clearly be a bug to keep things around with it disabled. Jon Oakes Jesse Kuhnert wrote: Hmmm...well, I don't think I like the sound of any of that. I'm just going to pretend this problem doesn't exist. (just kidding) I had thought I was doing something special with the ognl compilations that would cause its generated classes to not hang around afterwards in any pools. I'll take a look at things this weekend and am sure a fix will appear in between now and Monday - if there is a reasonable fix to be found. (in 4.1.3 snapshot form) On 8/24/07, Bryan Dotzour [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: I and another colleague of mine have been investigating what seems to be a memory leak in our Tapestry application for about a month since we upgraded to T4.1.2. I won't bore you with the saga of the last month, but I would like to present the data I've gathered and look to the list for a proposed solution. I was reading a recent thread in which Jesse said (08/09/2007): There is a map that grows as large as the system using it internally to javassist of various cached reflection info - but it doesn't leak in any way. This is precisely what I've found in profiling our application and it *appears* to be this map that is causing our applications to eventually run out
Re: Memory consumption in T4.1.2 - Hard data
Peter, though not officially final (I believe), http://news247.gr (tap-4.1.2) has been getting 40 req/day for an uptime of 26 days with memory set to no more than 128MB. On 8/28/07, Peter Stavrinides [EMAIL PROTECTED] wrote: Hi Jessie Any progress on this? sorry to bug you, but I have to take a decision soon, I have two production machines that will need to upgrade or downgrade Tapestry. Best wishes, Peter Jon Oakes wrote: Hi Bryan, I am a relative newbie but I was wondering if you are running with caching enabled or disabled? It might make sense to keep things around when caching is disabled whereas I think it would clearly be a bug to keep things around with it disabled. Jon Oakes Jesse Kuhnert wrote: Hmmm...well, I don't think I like the sound of any of that. I'm just going to pretend this problem doesn't exist. (just kidding) I had thought I was doing something special with the ognl compilations that would cause its generated classes to not hang around afterwards in any pools. I'll take a look at things this weekend and am sure a fix will appear in between now and Monday - if there is a reasonable fix to be found. (in 4.1.3 snapshot form) On 8/24/07, Bryan Dotzour [EMAIL PROTECTED] mailto: [EMAIL PROTECTED] wrote: I and another colleague of mine have been investigating what seems to be a memory leak in our Tapestry application for about a month since we upgraded to T4.1.2. I won't bore you with the saga of the last month, but I would like to present the data I've gathered and look to the list for a proposed solution. I was reading a recent thread in which Jesse said (08/09/2007): There is a map that grows as large as the system using it internally to javassist of various cached reflection info - but it doesn't leak in any way. This is precisely what I've found in profiling our application and it *appears* to be this map that is causing our applications to eventually run out of memory. The YourKit profiler shows me that, as time goes on, there is an instance of HiveMindClassPool that grows and grows as class instances are created. This class extends from javassist.ClassPool and is the map that Jesse is talking about in his quote above. And he's right, I wouldn't say that the class pool leaks either because it looks like it's designed to retain that memory until the class pool itself is no longer needed. Take this quote from the javassist.ClassPool javadocs: Memory consumption memo: ClassPool objects hold all the CtClasses that have been created so that the consistency among modified classes can be guaranteed. Thus if a large number of CtClasses are processed, the ClassPool will consume a huge amount of memory. To avoid this, a ClassPool object should be recreated, for example, every hundred classes processed. Note that getDefault() is a singleton factory. Otherwise, detach() in CtClass should be used to avoid huge memory consumption. This huge memory consumption by the ClassPool is what I was seeing. In particular it is the ClassPool that is held onto by OgnlRuntime. Inspecting this object in the profiler showed that it has a map containing about 45,000 classes. All of the keys into this map were things like: ASTTest_11494aca9af and ASTAnd_11494ace4fb and the values are instances of javassist.CtNewClass. Each entry in this map looks like it retains about 1,900 bytes, for a grand total of about 90 MB of memory used. These numbers came from my staging deployment where I had the profiler attached, using some reflection tricks I was able to look at a production site and found that it had about 240,000 items in that class pool.. approximately 450 MB of memory. So I guess the questions in my mind are: Why are there so many classes in the pool? Why does the number only ever go up? Do those classes really need to stay in the pool forever? - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- Andreas Andreou - [EMAIL PROTECTED] - http://andyhot.di.uoa.gr Tapestry / Tacos developer Open Source / JEE Consulting
Re: Memory consumption in T4.1.2 - Hard data
Andreas, give me a break! I am not trying to trash Tapestry if that's
what you are thinking. I am not the only one experiencing these problems.
Have a look at some of my postings (and others posts) and you will see
what I am talking about. I have been having these problems since
upgrading from 4.1.1 I never experienced these problems before with
uptime of months at a time.
Look at this log, id is an integer with the value 728:
21 Aug 2007 13:01:49,353 - INFO $Portfolio_89 - Setting the portfolio with the
id: 728
21 Aug 2007 13:01:50,479 - ERROR $Error_118 - Unable to parse OGNL expression
'id': Unable to parse OGNL expression 'id': Unable to parse OGNL expression
'id':
Exception [classpath:/org/apache/tapestry/form/TextArea.jwc, line 49, column 67]
at
org.apache.tapestry.binding.ExpressionBinding.resolveExpression(ExpressionBinding.java:145)
at
org.apache.tapestry.binding.ExpressionBinding.getObject(ExpressionBinding.java:125)
And then take a look at this:
21 Aug 2007 07:56:50,613 - ERROR
org.apache.tapestry.services.impl.HiveMindExpressionCompiler - Error generating
OGNL getter for expression exceptions with root [EMAIL PROTECTED]:Exception]
and body:
{ return (($Exception_119)$2).getExceptions();}
org.apache.hivemind.ApplicationRuntimeException: Unable to add method
java.lang.Object get(ognl.OgnlContext, java.lang.Object) to class
$ASTProperty_1146a471a52: [source error] no such class: $Exception_119
This happening after 3-4 days of normal operation? Suddenly exceptions
occur on random objects because OGNL can no longer compile them and this
is because Tapestry can no longer find instances of these classes, it
relates to JavaAssist dynamic class loading. Also, currently the server
is idle with no connections and JConsole shows 18000 classes loaded,
yesterday there were 7000 classes, this number never goes down only up,
so you tell me what's going on?
If Jessie can't explain it, then I have to go back a version we cant
afford for clients to see broken pages, my boss doesn't care about the
problem, he just wants a solution and its my job and reputation is at
stake.
Andreas Andreou wrote:
Peter,
though not officially final (I believe),
http://news247.gr (tap-4.1.2) has been getting 40 req/day
for an uptime of 26 days with memory set to no more than 128MB.
On 8/28/07, Peter Stavrinides [EMAIL PROTECTED] wrote:
Hi Jessie
Any progress on this? sorry to bug you, but I have to take a
decision soon, I have two production machines that will need to upgrade
or downgrade Tapestry.
Best wishes,
Peter
Jon Oakes wrote:
Hi Bryan,
I am a relative newbie but I was wondering if you are running with
caching enabled or disabled? It might make sense to keep things
around when caching is disabled whereas I think it would clearly be a
bug to keep things around with it disabled.
Jon Oakes
Jesse Kuhnert wrote:
Hmmm...well, I don't think I like the sound of any of that. I'm just
going to pretend this problem doesn't exist. (just kidding)
I had thought I was doing something special with the ognl compilations
that would cause its generated classes to not hang around afterwards
in any pools.
I'll take a look at things this weekend and am sure a fix will appear
in between now and Monday - if there is a reasonable fix to be found.
(in 4.1.3 snapshot form)
On 8/24/07, Bryan Dotzour [EMAIL PROTECTED] mailto:
[EMAIL PROTECTED] wrote:
I and another colleague of mine have been investigating what seems to
be
a memory leak in our Tapestry application for about a month since we
upgraded to T4.1.2. I won't bore you with the saga of the last month,
but I would like to present the data I've gathered and look to the
list
for a proposed solution. I was reading a recent thread in which Jesse
said (08/09/2007):
There is a map that grows as large as the system using it internally
to
javassist of various cached reflection info - but it doesn't leak in
any
way.
This is precisely what I've found in profiling our application and it
*appears* to be this map that is causing our applications to
eventually
run out of memory.
The YourKit profiler shows me that, as time goes on, there is an
instance of HiveMindClassPool that grows and grows as class instances
are created. This class extends from javassist.ClassPool and is the
map
that Jesse is talking about in his quote above. And he's right, I
wouldn't say that the class pool leaks either because it looks like
it's designed to retain that memory until the class pool itself is no
longer needed.
Take this quote from the javassist.ClassPool javadocs:
Memory consumption memo:
ClassPool objects hold all the CtClasses that have been created so
that
the consistency among modified classes can be guaranteed. Thus if a
large number of CtClasses are
Re: Memory consumption in T4.1.2 - Hard data
On 8/28/07, Peter Stavrinides [EMAIL PROTECTED] wrote:
Andreas, give me a break! I am not trying to trash Tapestry if that's
what you are thinking. I am not the only one experiencing these problems.
Hey - i never meant that!
It's obvious from the reports that something's going on + Jesse was on to
it..
I just thought i'd offer a counter-example + throw in an example
of a T4.1.2 public site ( i'm not aware of a lot)
Have a look at some of my postings (and others posts) and you will see
what I am talking about. I have been having these problems since
upgrading from 4.1.1 I never experienced these problems before with
uptime of months at a time.
Look at this log, id is an integer with the value 728:
21 Aug 2007 13:01:49,353 - INFO $Portfolio_89 - Setting the portfolio with
the id: 728
21 Aug 2007 13:01:50,479 - ERROR $Error_118 - Unable to parse OGNL
expression 'id': Unable to parse OGNL expression 'id': Unable to parse OGNL
expression 'id':
Exception [classpath:/org/apache/tapestry/form/TextArea.jwc, line 49,
column 67]
at org.apache.tapestry.binding.ExpressionBinding.resolveExpression
(ExpressionBinding.java:145)
at org.apache.tapestry.binding.ExpressionBinding.getObject(
ExpressionBinding.java:125)
And then take a look at this:
21 Aug 2007 07:56:50,613 - ERROR
org.apache.tapestry.services.impl.HiveMindExpressionCompiler - Error
generating OGNL getter for expression exceptions with root
[EMAIL PROTECTED]:Exception] and body:
{ return (($Exception_119)$2).getExceptions();}
org.apache.hivemind.ApplicationRuntimeException: Unable to add method
java.lang.Object get(ognl.OgnlContext, java.lang.Object) to class
$ASTProperty_1146a471a52: [source error] no such class: $Exception_119
This happening after 3-4 days of normal operation? Suddenly exceptions
occur on random objects because OGNL can no longer compile them and this
is because Tapestry can no longer find instances of these classes, it
relates to JavaAssist dynamic class loading. Also, currently the server
is idle with no connections and JConsole shows 18000 classes loaded,
yesterday there were 7000 classes, this number never goes down only up,
so you tell me what's going on?
If Jessie can't explain it, then I have to go back a version we cant
afford for clients to see broken pages, my boss doesn't care about the
problem, he just wants a solution and its my job and reputation is at
stake.
Andreas Andreou wrote:
Peter,
though not officially final (I believe),
http://news247.gr (tap-4.1.2) has been getting 40 req/day
for an uptime of 26 days with memory set to no more than 128MB.
On 8/28/07, Peter Stavrinides [EMAIL PROTECTED] wrote:
Hi Jessie
Any progress on this? sorry to bug you, but I have to take a
decision soon, I have two production machines that will need to upgrade
or downgrade Tapestry.
Best wishes,
Peter
Jon Oakes wrote:
Hi Bryan,
I am a relative newbie but I was wondering if you are running with
caching enabled or disabled? It might make sense to keep things
around when caching is disabled whereas I think it would clearly be a
bug to keep things around with it disabled.
Jon Oakes
Jesse Kuhnert wrote:
Hmmm...well, I don't think I like the sound of any of that. I'm
just
going to pretend this problem doesn't exist. (just kidding)
I had thought I was doing something special with the ognl
compilations
that would cause its generated classes to not hang around afterwards
in any pools.
I'll take a look at things this weekend and am sure a fix will appear
in between now and Monday - if there is a reasonable fix to be found.
(in 4.1.3 snapshot form)
On 8/24/07, Bryan Dotzour [EMAIL PROTECTED] mailto:
[EMAIL PROTECTED] wrote:
I and another colleague of mine have been investigating what seems
to
be
a memory leak in our Tapestry application for about a month since
we
upgraded to T4.1.2. I won't bore you with the saga of the last
month,
but I would like to present the data I've gathered and look to the
list
for a proposed solution. I was reading a recent thread in which
Jesse
said (08/09/2007):
There is a map that grows as large as the system using it
internally
to
javassist of various cached reflection info - but it doesn't leak in
any
way.
This is precisely what I've found in profiling our application and
it
*appears* to be this map that is causing our applications to
eventually
run out of memory.
The YourKit profiler shows me that, as time goes on, there is an
instance of HiveMindClassPool that grows and grows as class
instances
are created. This class extends from javassist.ClassPool and is the
map
that Jesse is talking about in his quote above. And he's right, I
wouldn't say that the class pool leaks either because it looks
like
it's designed to retain that memory until the
RE: Memory consumption in T4.1.2 - Hard data
So changing the page pool parameters didn't help then?
-Original Message-
From: Peter Stavrinides [mailto:[EMAIL PROTECTED]
Sent: Tuesday, August 28, 2007 10:34 AM
To: Tapestry users
Subject: Re: Memory consumption in T4.1.2 - Hard data
Andreas, give me a break! I am not trying to trash Tapestry
if that's what you are thinking. I am not the only one
experiencing these problems.
Have a look at some of my postings (and others posts) and you
will see what I am talking about. I have been having these
problems since upgrading from 4.1.1 I never experienced these
problems before with uptime of months at a time.
Look at this log, id is an integer with the value 728:
21 Aug 2007 13:01:49,353 - INFO $Portfolio_89 - Setting the
portfolio with the id: 728
21 Aug 2007 13:01:50,479 - ERROR $Error_118 - Unable to parse
OGNL expression 'id': Unable to parse OGNL expression 'id':
Unable to parse OGNL expression 'id':
Exception [classpath:/org/apache/tapestry/form/TextArea.jwc,
line 49, column 67]
at
org.apache.tapestry.binding.ExpressionBinding.resolveExpressio
n(ExpressionBinding.java:145)
at
org.apache.tapestry.binding.ExpressionBinding.getObject(Expres
sionBinding.java:125)
And then take a look at this:
21 Aug 2007 07:56:50,613 - ERROR
org.apache.tapestry.services.impl.HiveMindExpressionCompiler
- Error generating OGNL getter for expression exceptions with
root [EMAIL PROTECTED]:Exception] and body:
{ return (($Exception_119)$2).getExceptions();}
org.apache.hivemind.ApplicationRuntimeException: Unable to
add method java.lang.Object get(ognl.OgnlContext,
java.lang.Object) to class $ASTProperty_1146a471a52: [source
error] no such class: $Exception_119
This happening after 3-4 days of normal operation? Suddenly
exceptions occur on random objects because OGNL can no longer
compile them and this is because Tapestry can no longer find
instances of these classes, it relates to JavaAssist dynamic
class loading. Also, currently the server is idle with no
connections and JConsole shows 18000 classes loaded,
yesterday there were 7000 classes, this number never goes
down only up, so you tell me what's going on?
If Jessie can't explain it, then I have to go back a version
we cant afford for clients to see broken pages, my boss
doesn't care about the problem, he just wants a solution and
its my job and reputation is at stake.
Andreas Andreou wrote:
Peter,
though not officially final (I believe), http://news247.gr
(tap-4.1.2)
has been getting 40 req/day for an uptime of 26 days
with memory
set to no more than 128MB.
On 8/28/07, Peter Stavrinides [EMAIL PROTECTED] wrote:
Hi Jessie
Any progress on this? sorry to bug you, but I have to take a
decision soon, I have two production machines that will need to
upgrade or downgrade Tapestry.
Best wishes,
Peter
Jon Oakes wrote:
Hi Bryan,
I am a relative newbie but I was wondering if you are
running with
caching enabled or disabled? It might make sense to keep things
around when caching is disabled whereas I think it would
clearly be
a bug to keep things around with it disabled.
Jon Oakes
Jesse Kuhnert wrote:
Hmmm...well, I don't think I like the sound of any of
that. I'm just
going to pretend this problem doesn't exist. (just kidding)
I had thought I was doing something special with the ognl
compilations that would cause its generated classes to not hang
around afterwards in any pools.
I'll take a look at things this weekend and am sure a fix will
appear in between now and Monday - if there is a
reasonable fix to be found.
(in 4.1.3 snapshot form)
On 8/24/07, Bryan Dotzour [EMAIL PROTECTED] mailto:
[EMAIL PROTECTED] wrote:
I and another colleague of mine have been investigating
what seems
to
be
a memory leak in our Tapestry application for about a month
since we upgraded to T4.1.2. I won't bore you with the saga of
the last month, but I would like to present the data
I've gathered
and look to the
list
for a proposed solution. I was reading a recent thread
in which
Jesse said (08/09/2007):
There is a map that grows as large as the system using it
internally
to
javassist of various cached reflection info - but it
doesn't leak
in
any
way.
This is precisely what I've found in profiling our
application and
it
*appears* to be this map that is causing our applications to
eventually
run out of memory.
The YourKit profiler shows me that, as time goes on,
there is an
instance of HiveMindClassPool that grows and grows as class
instances are created. This class extends from
javassist.ClassPool and is the
map
Re: Memory consumption in T4.1.2 - Hard data
Now that's an OGNL compiler error. Have you tried with OGNL
2.7.1-SNAPSHOT(just use the latest). OGNL
2.7 that comes with Tap 4.1.2 by default is known to have bugs with the
compiled expressions. Please try with the latest OGNL; it might just do the
trick.
Kalle
On 8/28/07, Peter Stavrinides [EMAIL PROTECTED] wrote:
Andreas, give me a break! I am not trying to trash Tapestry if that's
what you are thinking. I am not the only one experiencing these problems.
Have a look at some of my postings (and others posts) and you will see
what I am talking about. I have been having these problems since
upgrading from 4.1.1 I never experienced these problems before with
uptime of months at a time.
Look at this log, id is an integer with the value 728:
21 Aug 2007 13:01:49,353 - INFO $Portfolio_89 - Setting the portfolio with
the id: 728
21 Aug 2007 13:01:50,479 - ERROR $Error_118 - Unable to parse OGNL
expression 'id': Unable to parse OGNL expression 'id': Unable to parse OGNL
expression 'id':
Exception [classpath:/org/apache/tapestry/form/TextArea.jwc, line 49,
column 67]
at org.apache.tapestry.binding.ExpressionBinding.resolveExpression
(ExpressionBinding.java:145)
at org.apache.tapestry.binding.ExpressionBinding.getObject(
ExpressionBinding.java:125)
And then take a look at this:
21 Aug 2007 07:56:50,613 - ERROR
org.apache.tapestry.services.impl.HiveMindExpressionCompiler - Error
generating OGNL getter for expression exceptions with root
[EMAIL PROTECTED]:Exception] and body:
{ return (($Exception_119)$2).getExceptions();}
org.apache.hivemind.ApplicationRuntimeException: Unable to add method
java.lang.Object get(ognl.OgnlContext, java.lang.Object) to class
$ASTProperty_1146a471a52: [source error] no such class: $Exception_119
This happening after 3-4 days of normal operation? Suddenly exceptions
occur on random objects because OGNL can no longer compile them and this
is because Tapestry can no longer find instances of these classes, it
relates to JavaAssist dynamic class loading. Also, currently the server
is idle with no connections and JConsole shows 18000 classes loaded,
yesterday there were 7000 classes, this number never goes down only up,
so you tell me what's going on?
If Jessie can't explain it, then I have to go back a version we cant
afford for clients to see broken pages, my boss doesn't care about the
problem, he just wants a solution and its my job and reputation is at
stake.
Andreas Andreou wrote:
Peter,
though not officially final (I believe),
http://news247.gr (tap-4.1.2) has been getting 40 req/day
for an uptime of 26 days with memory set to no more than 128MB.
On 8/28/07, Peter Stavrinides [EMAIL PROTECTED] wrote:
Hi Jessie
Any progress on this? sorry to bug you, but I have to take a
decision soon, I have two production machines that will need to upgrade
or downgrade Tapestry.
Best wishes,
Peter
Jon Oakes wrote:
Hi Bryan,
I am a relative newbie but I was wondering if you are running with
caching enabled or disabled? It might make sense to keep things
around when caching is disabled whereas I think it would clearly be a
bug to keep things around with it disabled.
Jon Oakes
Jesse Kuhnert wrote:
Hmmm...well, I don't think I like the sound of any of that. I'm
just
going to pretend this problem doesn't exist. (just kidding)
I had thought I was doing something special with the ognl
compilations
that would cause its generated classes to not hang around afterwards
in any pools.
I'll take a look at things this weekend and am sure a fix will appear
in between now and Monday - if there is a reasonable fix to be found.
(in 4.1.3 snapshot form)
On 8/24/07, Bryan Dotzour [EMAIL PROTECTED] mailto:
[EMAIL PROTECTED] wrote:
I and another colleague of mine have been investigating what seems
to
be
a memory leak in our Tapestry application for about a month since
we
upgraded to T4.1.2. I won't bore you with the saga of the last
month,
but I would like to present the data I've gathered and look to the
list
for a proposed solution. I was reading a recent thread in which
Jesse
said (08/09/2007):
There is a map that grows as large as the system using it
internally
to
javassist of various cached reflection info - but it doesn't leak in
any
way.
This is precisely what I've found in profiling our application and
it
*appears* to be this map that is causing our applications to
eventually
run out of memory.
The YourKit profiler shows me that, as time goes on, there is an
instance of HiveMindClassPool that grows and grows as class
instances
are created. This class extends from javassist.ClassPool and is the
map
that Jesse is talking about in his quote above. And he's right, I
wouldn't say that the class pool leaks either because it looks
like
it's
Re: Memory consumption in T4.1.2 - Hard data
No Marcus
[EMAIL PROTECTED] wrote:
So changing the page pool parameters didn't help then?
-Original Message-
From: Peter Stavrinides [mailto:[EMAIL PROTECTED]
Sent: Tuesday, August 28, 2007 10:34 AM
To: Tapestry users
Subject: Re: Memory consumption in T4.1.2 - Hard data
Andreas, give me a break! I am not trying to trash Tapestry
if that's what you are thinking. I am not the only one
experiencing these problems.
Have a look at some of my postings (and others posts) and you
will see what I am talking about. I have been having these
problems since upgrading from 4.1.1 I never experienced these
problems before with uptime of months at a time.
Look at this log, id is an integer with the value 728:
21 Aug 2007 13:01:49,353 - INFO $Portfolio_89 - Setting the
portfolio with the id: 728
21 Aug 2007 13:01:50,479 - ERROR $Error_118 - Unable to parse
OGNL expression 'id': Unable to parse OGNL expression 'id':
Unable to parse OGNL expression 'id':
Exception [classpath:/org/apache/tapestry/form/TextArea.jwc,
line 49, column 67]
at
org.apache.tapestry.binding.ExpressionBinding.resolveExpressio
n(ExpressionBinding.java:145)
at
org.apache.tapestry.binding.ExpressionBinding.getObject(Expres
sionBinding.java:125)
And then take a look at this:
21 Aug 2007 07:56:50,613 - ERROR
org.apache.tapestry.services.impl.HiveMindExpressionCompiler
- Error generating OGNL getter for expression exceptions with
root [EMAIL PROTECTED]:Exception] and body:
{ return (($Exception_119)$2).getExceptions();}
org.apache.hivemind.ApplicationRuntimeException: Unable to
add method java.lang.Object get(ognl.OgnlContext,
java.lang.Object) to class $ASTProperty_1146a471a52: [source
error] no such class: $Exception_119
This happening after 3-4 days of normal operation? Suddenly
exceptions occur on random objects because OGNL can no longer
compile them and this is because Tapestry can no longer find
instances of these classes, it relates to JavaAssist dynamic
class loading. Also, currently the server is idle with no
connections and JConsole shows 18000 classes loaded,
yesterday there were 7000 classes, this number never goes
down only up, so you tell me what's going on?
If Jessie can't explain it, then I have to go back a version
we cant afford for clients to see broken pages, my boss
doesn't care about the problem, he just wants a solution and
its my job and reputation is at stake.
Andreas Andreou wrote:
Peter,
though not officially final (I believe), http://news247.gr
(tap-4.1.2)
has been getting 40 req/day for an uptime of 26 days
with memory
set to no more than 128MB.
On 8/28/07, Peter Stavrinides [EMAIL PROTECTED] wrote:
Hi Jessie
Any progress on this? sorry to bug you, but I have to take a
decision soon, I have two production machines that will need to
upgrade or downgrade Tapestry.
Best wishes,
Peter
Jon Oakes wrote:
Hi Bryan,
I am a relative newbie but I was wondering if you are
running with
caching enabled or disabled? It might make sense to keep things
around when caching is disabled whereas I think it would
clearly be
a bug to keep things around with it disabled.
Jon Oakes
Jesse Kuhnert wrote:
Hmmm...well, I don't think I like the sound of any of
that. I'm just
going to pretend this problem doesn't exist. (just kidding)
I had thought I was doing something special with the ognl
compilations that would cause its generated classes to not hang
around afterwards in any pools.
I'll take a look at things this weekend and am sure a fix will
appear in between now and Monday - if there is a
reasonable fix to be found.
(in 4.1.3 snapshot form)
On 8/24/07, Bryan Dotzour [EMAIL PROTECTED] mailto:
[EMAIL PROTECTED] wrote:
I and another colleague of mine have been investigating
what seems
to
be
a memory leak in our Tapestry application for about a month
since we upgraded to T4.1.2. I won't bore you with the saga of
the last month, but I would like to present the data
I've gathered
and look to the
list
for a proposed solution. I was reading a recent thread
in which
Jesse said (08/09/2007):
There is a map that grows as large as the system using it
internally
to
javassist of various cached reflection info - but it
doesn't leak
in
any
way.
This is precisely what I've found in profiling our
application and
it
*appears* to be this map that is causing our applications to
eventually
run out of memory.
The YourKit profiler shows me
Re: Memory consumption in T4.1.2 - Hard data
Hi Jessie I'm gonna try update my version of ognl, if that won't work I will downgrade. I have not filed any bugs yet, but I sent a couple of emails to the list. I don't get any exceptions during development, but that is because its not running for 3 or 4 days. I strongly suggest you consider looking into it, because I use the standard libraries provided by Tapestry through Maven. my configuration is as follows: JDK 6.01 32bit JVM (I have also tested on a 64 bit with no luck) Tomcat 5.5.20 Debian Linux (2.6.15.28 kernel) Tapestry 4.1.2-SNAPSHOT with ognl 2.7 regards, Peter Jesse Kuhnert wrote: I'd downgrade then if I were you. Extensive profiling with yourkit hasn't shed any new light on whatever problems others are having. The OGNL classes indeed aren't being kept in the javassist pool from what I can tell. I have another yourkit snapshot binary to look at so we'll see what that says. I don't remember - have you filed any bug reports or reported any problems? Are you getting ognl exceptions during development as well as production? Do these errors sound like umpotential errors that should be looked at further? On 8/28/07, Peter Stavrinides [EMAIL PROTECTED] wrote: Hi Jessie Any progress on this? sorry to bug you, but I have to take a decision soon, I have two production machines that will need to upgrade or downgrade Tapestry. Best wishes, Peter Jon Oakes wrote: Hi Bryan, I am a relative newbie but I was wondering if you are running with caching enabled or disabled? It might make sense to keep things around when caching is disabled whereas I think it would clearly be a bug to keep things around with it disabled. Jon Oakes Jesse Kuhnert wrote: Hmmm...well, I don't think I like the sound of any of that. I'm just going to pretend this problem doesn't exist. (just kidding) I had thought I was doing something special with the ognl compilations that would cause its generated classes to not hang around afterwards in any pools. I'll take a look at things this weekend and am sure a fix will appear in between now and Monday - if there is a reasonable fix to be found. (in 4.1.3 snapshot form) On 8/24/07, Bryan Dotzour [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: I and another colleague of mine have been investigating what seems to be a memory leak in our Tapestry application for about a month since we upgraded to T4.1.2. I won't bore you with the saga of the last month, but I would like to present the data I've gathered and look to the list for a proposed solution. I was reading a recent thread in which Jesse said (08/09/2007): There is a map that grows as large as the system using it internally to javassist of various cached reflection info - but it doesn't leak in any way. This is precisely what I've found in profiling our application and it *appears* to be this map that is causing our applications to eventually run out of memory. The YourKit profiler shows me that, as time goes on, there is an instance of HiveMindClassPool that grows and grows as class instances are created. This class extends from javassist.ClassPool and is the map that Jesse is talking about in his quote above. And he's right, I wouldn't say that the class pool leaks either because it looks like it's designed to retain that memory until the class pool itself is no longer needed. Take this quote from the javassist.ClassPool javadocs: Memory consumption memo: ClassPool objects hold all the CtClasses that have been created so that the consistency among modified classes can be guaranteed. Thus if a large number of CtClasses are processed, the ClassPool will consume a huge amount of memory. To avoid this, a ClassPool object should be recreated, for example, every hundred classes processed. Note that getDefault() is a singleton factory. Otherwise, detach() in CtClass should be used to avoid huge memory consumption. This huge memory consumption by the ClassPool is what I was seeing. In particular it is the ClassPool that is held onto by OgnlRuntime. Inspecting this object in the profiler showed that it has a map containing about 45,000 classes. All of the keys into this map were things like: ASTTest_11494aca9af and ASTAnd_11494ace4fb and the values are instances of javassist.CtNewClass. Each entry in this map looks like it retains about 1,900 bytes, for a grand total of about 90 MB of memory used. These numbers came from my staging deployment where I had the profiler attached, using some reflection tricks I was able to look at a production site and found that it had about 240,000 items in that class pool.. approximately 450 MB of memory. So I guess the questions in my mind are: Why are there so many classes in the pool? Why does the number only ever go up? Do those classes really need to stay in the pool forever? - To unsubscribe, e-mail: [EMAIL PROTECTED]
Re: Memory consumption in T4.1.2 - Hard data
Umm yeah. I have looked in to it as I just said. The only thing left is for me to look at this youkit snapshot profile and play from there but I'm not capable of debugging issues that aren't reported properly. On 8/28/07, Peter Stavrinides [EMAIL PROTECTED] wrote: Hi Jessie I'm gonna try update my version of ognl, if that won't work I will downgrade. I have not filed any bugs yet, but I sent a couple of emails to the list. I don't get any exceptions during development, but that is because its not running for 3 or 4 days. I strongly suggest you consider looking into it, because I use the standard libraries provided by Tapestry through Maven. my configuration is as follows: JDK 6.01 32bit JVM (I have also tested on a 64 bit with no luck) Tomcat 5.5.20 Debian Linux (2.6.15.28 kernel) Tapestry 4.1.2-SNAPSHOT with ognl 2.7 regards, Peter Jesse Kuhnert wrote: I'd downgrade then if I were you. Extensive profiling with yourkit hasn't shed any new light on whatever problems others are having. The OGNL classes indeed aren't being kept in the javassist pool from what I can tell. I have another yourkit snapshot binary to look at so we'll see what that says. I don't remember - have you filed any bug reports or reported any problems? Are you getting ognl exceptions during development as well as production? Do these errors sound like umpotential errors that should be looked at further? On 8/28/07, Peter Stavrinides [EMAIL PROTECTED] wrote: Hi Jessie Any progress on this? sorry to bug you, but I have to take a decision soon, I have two production machines that will need to upgrade or downgrade Tapestry. Best wishes, Peter Jon Oakes wrote: Hi Bryan, I am a relative newbie but I was wondering if you are running with caching enabled or disabled? It might make sense to keep things around when caching is disabled whereas I think it would clearly be a bug to keep things around with it disabled. Jon Oakes Jesse Kuhnert wrote: Hmmm...well, I don't think I like the sound of any of that. I'm just going to pretend this problem doesn't exist. (just kidding) I had thought I was doing something special with the ognl compilations that would cause its generated classes to not hang around afterwards in any pools. I'll take a look at things this weekend and am sure a fix will appear in between now and Monday - if there is a reasonable fix to be found. (in 4.1.3 snapshot form) On 8/24/07, Bryan Dotzour [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: I and another colleague of mine have been investigating what seems to be a memory leak in our Tapestry application for about a month since we upgraded to T4.1.2. I won't bore you with the saga of the last month, but I would like to present the data I've gathered and look to the list for a proposed solution. I was reading a recent thread in which Jesse said (08/09/2007): There is a map that grows as large as the system using it internally to javassist of various cached reflection info - but it doesn't leak in any way. This is precisely what I've found in profiling our application and it *appears* to be this map that is causing our applications to eventually run out of memory. The YourKit profiler shows me that, as time goes on, there is an instance of HiveMindClassPool that grows and grows as class instances are created. This class extends from javassist.ClassPool and is the map that Jesse is talking about in his quote above. And he's right, I wouldn't say that the class pool leaks either because it looks like it's designed to retain that memory until the class pool itself is no longer needed. Take this quote from the javassist.ClassPool javadocs: Memory consumption memo: ClassPool objects hold all the CtClasses that have been created so that the consistency among modified classes can be guaranteed. Thus if a large number of CtClasses are processed, the ClassPool will consume a huge amount of memory. To avoid this, a ClassPool object should be recreated, for example, every hundred classes processed. Note that getDefault() is a singleton factory. Otherwise, detach() in CtClass should be used to avoid huge memory consumption. This huge memory consumption by the ClassPool is what I was seeing. In particular it is the ClassPool that is held onto by OgnlRuntime. Inspecting this object in the profiler showed that it has a map containing about 45,000 classes. All of the keys into this map were things like: ASTTest_11494aca9af and ASTAnd_11494ace4fb and the values are instances of javassist.CtNewClass. Each entry in this map looks like it retains about 1,900 bytes, for a grand total of about 90 MB of memory used. These numbers came from my staging deployment where I had the profiler attached, using some reflection tricks I was able to look at
RE: Memory consumption in T4.1.2 - Hard data
my configuration is as follows: JDK 6.01 32bit JVM (I have also tested on a 64 bit with no luck) Tomcat 5.5.20 Debian Linux (2.6.15.28 kernel) Tapestry 4.1.2-SNAPSHOT with ognl 2.7 Hi Peter, 4.1.2-SNAPSHOT is a typo, I suppose? regards, Peter Jesse Kuhnert wrote: I'd downgrade then if I were you. Extensive profiling with yourkit hasn't shed any new light on whatever problems others are having. The OGNL classes indeed aren't being kept in the javassist pool from what I can tell. I have another yourkit snapshot binary to look at so we'll see what that says. I don't remember - have you filed any bug reports or reported any problems? Are you getting ognl exceptions during development as well as production? Do these errors sound like umpotential errors that should be looked at further? On 8/28/07, Peter Stavrinides [EMAIL PROTECTED] wrote: Hi Jessie Any progress on this? sorry to bug you, but I have to take a decision soon, I have two production machines that will need to upgrade or downgrade Tapestry. Best wishes, Peter Jon Oakes wrote: Hi Bryan, I am a relative newbie but I was wondering if you are running with caching enabled or disabled? It might make sense to keep things around when caching is disabled whereas I think it would clearly be a bug to keep things around with it disabled. Jon Oakes Jesse Kuhnert wrote: Hmmm...well, I don't think I like the sound of any of that. I'm just going to pretend this problem doesn't exist. (just kidding) I had thought I was doing something special with the ognl compilations that would cause its generated classes to not hang around afterwards in any pools. I'll take a look at things this weekend and am sure a fix will appear in between now and Monday - if there is a reasonable fix to be found. (in 4.1.3 snapshot form) On 8/24/07, Bryan Dotzour [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: I and another colleague of mine have been investigating what seems to be a memory leak in our Tapestry application for about a month since we upgraded to T4.1.2. I won't bore you with the saga of the last month, but I would like to present the data I've gathered and look to the list for a proposed solution. I was reading a recent thread in which Jesse said (08/09/2007): There is a map that grows as large as the system using it internally to javassist of various cached reflection info - but it doesn't leak in any way. This is precisely what I've found in profiling our application and it *appears* to be this map that is causing our applications to eventually run out of memory. The YourKit profiler shows me that, as time goes on, there is an instance of HiveMindClassPool that grows and grows as class instances are created. This class extends from javassist.ClassPool and is the map that Jesse is talking about in his quote above. And he's right, I wouldn't say that the class pool leaks either because it looks like it's designed to retain that memory until the class pool itself is no longer needed. Take this quote from the javassist.ClassPool javadocs: Memory consumption memo: ClassPool objects hold all the CtClasses that have been created so that the consistency among modified classes can be guaranteed. Thus if a large number of CtClasses are processed, the ClassPool will consume a huge amount of memory. To avoid this, a ClassPool object should be recreated, for example, every hundred classes processed. Note that getDefault() is a singleton factory. Otherwise, detach() in CtClass should be used to avoid huge memory consumption. This huge memory consumption by the ClassPool is what I was seeing. In particular it is the ClassPool that is held onto by OgnlRuntime. Inspecting this object in the profiler showed that it has a map containing about 45,000 classes. All of the keys into this map were things like: ASTTest_11494aca9af and ASTAnd_11494ace4fb and the values are instances of javassist.CtNewClass. Each entry in this map looks like it retains about 1,900 bytes, for a grand total of about 90 MB of memory used. These numbers came from my staging deployment where I had the profiler attached, using some reflection tricks I was able to look at a production site and found that it had about 240,000 items in that class pool.. approximately 450 MB of memory. So I guess the questions in my mind are: Why are there so many classes in the pool? Why does the number only ever go up? Do those classes really need to stay in the pool forever? - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail:
Re: Memory consumption in T4.1.2 - Hard data
Yes [EMAIL PROTECTED] wrote: my configuration is as follows: JDK 6.01 32bit JVM (I have also tested on a 64 bit with no luck) Tomcat 5.5.20 Debian Linux (2.6.15.28 kernel) Tapestry 4.1.2-SNAPSHOT with ognl 2.7 Hi Peter, 4.1.2-SNAPSHOT is a typo, I suppose? regards, Peter Jesse Kuhnert wrote: I'd downgrade then if I were you. Extensive profiling with yourkit hasn't shed any new light on whatever problems others are having. The OGNL classes indeed aren't being kept in the javassist pool from what I can tell. I have another yourkit snapshot binary to look at so we'll see what that says. I don't remember - have you filed any bug reports or reported any problems? Are you getting ognl exceptions during development as well as production? Do these errors sound like umpotential errors that should be looked at further? On 8/28/07, Peter Stavrinides [EMAIL PROTECTED] wrote: Hi Jessie Any progress on this? sorry to bug you, but I have to take a decision soon, I have two production machines that will need to upgrade or downgrade Tapestry. Best wishes, Peter Jon Oakes wrote: Hi Bryan, I am a relative newbie but I was wondering if you are running with caching enabled or disabled? It might make sense to keep things around when caching is disabled whereas I think it would clearly be a bug to keep things around with it disabled. Jon Oakes Jesse Kuhnert wrote: Hmmm...well, I don't think I like the sound of any of that. I'm just going to pretend this problem doesn't exist. (just kidding) I had thought I was doing something special with the ognl compilations that would cause its generated classes to not hang around afterwards in any pools. I'll take a look at things this weekend and am sure a fix will appear in between now and Monday - if there is a reasonable fix to be found. (in 4.1.3 snapshot form) On 8/24/07, Bryan Dotzour [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: I and another colleague of mine have been investigating what seems to be a memory leak in our Tapestry application for about a month since we upgraded to T4.1.2. I won't bore you with the saga of the last month, but I would like to present the data I've gathered and look to the list for a proposed solution. I was reading a recent thread in which Jesse said (08/09/2007): There is a map that grows as large as the system using it internally to javassist of various cached reflection info - but it doesn't leak in any way. This is precisely what I've found in profiling our application and it *appears* to be this map that is causing our applications to eventually run out of memory. The YourKit profiler shows me that, as time goes on, there is an instance of HiveMindClassPool that grows and grows as class instances are created. This class extends from javassist.ClassPool and is the map that Jesse is talking about in his quote above. And he's right, I wouldn't say that the class pool leaks either because it looks like it's designed to retain that memory until the class pool itself is no longer needed. Take this quote from the javassist.ClassPool javadocs: Memory consumption memo: ClassPool objects hold all the CtClasses that have been created so that the consistency among modified classes can be guaranteed. Thus if a large number of CtClasses are processed, the ClassPool will consume a huge amount of memory. To avoid this, a ClassPool object should be recreated, for example, every hundred classes processed. Note that getDefault() is a singleton factory. Otherwise, detach() in CtClass should be used to avoid huge memory consumption. This huge memory consumption by the ClassPool is what I was seeing. In particular it is the ClassPool that is held onto by OgnlRuntime. Inspecting this object in the profiler showed that it has a map containing about 45,000 classes. All of the keys into this map were things like: ASTTest_11494aca9af and ASTAnd_11494ace4fb and the values are instances of javassist.CtNewClass. Each entry in this map looks like it retains about 1,900 bytes, for a grand total of about 90 MB of memory used. These numbers came from my staging deployment where I had the profiler attached, using some reflection tricks I was able to look at a production site and found that it had about 240,000 items in that class pool.. approximately 450 MB of memory. So I guess the questions in
Re: Memory consumption in T4.1.2 - Hard data
This is my POM: repositories repository releases updatePolicyalways/updatePolicy checksumPolicywarn/checksumPolicy /releases snapshots updatePolicyalways/updatePolicy checksumPolicyfail/checksumPolicy repository idapache.snapshots/id urlhttp://people.apache.org/repo/m2-snapshot-repository/url /repository /repositories dependency groupIdorg.apache.tapestry/groupId artifactIdtapestry-framework/artifactId version4.1.2-SNAPSHOT/version scopetest/scope /dependency dependency groupIdorg.apache.tapestry/groupId artifactIdtapestry-annotations/artifactId version4.1.2-SNAPSHOT/version scopetest/scope /dependency dependency groupIdorg.apache.tapestry/groupId artifactIdtapestry-contrib/artifactId version4.1.2-SNAPSHOT/version scopetest/scope /dependency dependency groupIdorg.apache.tapestry/groupId artifactIdtapestry-portlet/artifactId version4.1.2-SNAPSHOT/version scopetest/scope /dependency [EMAIL PROTECTED] wrote: my configuration is as follows: JDK 6.01 32bit JVM (I have also tested on a 64 bit with no luck) Tomcat 5.5.20 Debian Linux (2.6.15.28 kernel) Tapestry 4.1.2-SNAPSHOT with ognl 2.7 Hi Peter, 4.1.2-SNAPSHOT is a typo, I suppose? regards, Peter Jesse Kuhnert wrote: I'd downgrade then if I were you. Extensive profiling with yourkit hasn't shed any new light on whatever problems others are having. The OGNL classes indeed aren't being kept in the javassist pool from what I can tell. I have another yourkit snapshot binary to look at so we'll see what that says. I don't remember - have you filed any bug reports or reported any problems? Are you getting ognl exceptions during development as well as production? Do these errors sound like umpotential errors that should be looked at further? On 8/28/07, Peter Stavrinides [EMAIL PROTECTED] wrote: Hi Jessie Any progress on this? sorry to bug you, but I have to take a decision soon, I have two production machines that will need to upgrade or downgrade Tapestry. Best wishes, Peter Jon Oakes wrote: Hi Bryan, I am a relative newbie but I was wondering if you are running with caching enabled or disabled? It might make sense to keep things around when caching is disabled whereas I think it would clearly be a bug to keep things around with it disabled. Jon Oakes Jesse Kuhnert wrote: Hmmm...well, I don't think I like the sound of any of that. I'm just going to pretend this problem doesn't exist. (just kidding) I had thought I was doing something special with the ognl compilations that would cause its generated classes to not hang around afterwards in any pools. I'll take a look at things this weekend and am sure a fix will appear in between now and Monday - if there is a reasonable fix to be found. (in 4.1.3 snapshot form) On 8/24/07, Bryan Dotzour [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: I and another colleague of mine have been investigating what seems to be a memory leak in our Tapestry application for about a month since we upgraded to T4.1.2. I won't bore you with the saga of the last month, but I would like to present the data I've gathered and look to the list for a proposed solution. I was reading a recent thread in which Jesse said (08/09/2007): There is a map that grows as large as the system using it internally to javassist of various cached reflection info - but it doesn't leak in any way. This is precisely what I've found in profiling our application and it *appears* to be this map that is causing our applications to eventually run out of memory. The YourKit profiler shows me that, as time goes on, there is an instance of HiveMindClassPool that grows and grows as class instances are created. This class extends from javassist.ClassPool and is the map that Jesse is talking about in his quote above. And he's right, I wouldn't say that the class pool leaks either because it looks like it's designed to retain that memory until the class pool itself is no longer needed. Take this quote from the javassist.ClassPool javadocs: Memory consumption memo: ClassPool objects hold all the CtClasses that have been created so that the consistency among modified classes can be guaranteed. Thus if a large number of CtClasses are processed, the ClassPool will consume a huge amount of memory. To avoid this, a ClassPool object should be recreated, for example, every hundred classes processed. Note that getDefault() is a singleton factory. Otherwise, detach() in CtClass should be used to avoid huge memory consumption. This huge
Re: Memory consumption in T4.1.2 - Hard data
Hi Jessie Any progress on this? sorry to bug you, but I have to take a decision soon, I have two production machines that will need to upgrade or downgrade Tapestry. Best wishes, Peter Jon Oakes wrote: Hi Bryan, I am a relative newbie but I was wondering if you are running with caching enabled or disabled? It might make sense to keep things around when caching is disabled whereas I think it would clearly be a bug to keep things around with it disabled. Jon Oakes Jesse Kuhnert wrote: Hmmm...well, I don't think I like the sound of any of that. I'm just going to pretend this problem doesn't exist. (just kidding) I had thought I was doing something special with the ognl compilations that would cause its generated classes to not hang around afterwards in any pools. I'll take a look at things this weekend and am sure a fix will appear in between now and Monday - if there is a reasonable fix to be found. (in 4.1.3 snapshot form) On 8/24/07, Bryan Dotzour [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: I and another colleague of mine have been investigating what seems to be a memory leak in our Tapestry application for about a month since we upgraded to T4.1.2. I won't bore you with the saga of the last month, but I would like to present the data I've gathered and look to the list for a proposed solution. I was reading a recent thread in which Jesse said (08/09/2007): There is a map that grows as large as the system using it internally to javassist of various cached reflection info - but it doesn't leak in any way. This is precisely what I've found in profiling our application and it *appears* to be this map that is causing our applications to eventually run out of memory. The YourKit profiler shows me that, as time goes on, there is an instance of HiveMindClassPool that grows and grows as class instances are created. This class extends from javassist.ClassPool and is the map that Jesse is talking about in his quote above. And he's right, I wouldn't say that the class pool leaks either because it looks like it's designed to retain that memory until the class pool itself is no longer needed. Take this quote from the javassist.ClassPool javadocs: Memory consumption memo: ClassPool objects hold all the CtClasses that have been created so that the consistency among modified classes can be guaranteed. Thus if a large number of CtClasses are processed, the ClassPool will consume a huge amount of memory. To avoid this, a ClassPool object should be recreated, for example, every hundred classes processed. Note that getDefault() is a singleton factory. Otherwise, detach() in CtClass should be used to avoid huge memory consumption. This huge memory consumption by the ClassPool is what I was seeing. In particular it is the ClassPool that is held onto by OgnlRuntime. Inspecting this object in the profiler showed that it has a map containing about 45,000 classes. All of the keys into this map were things like: ASTTest_11494aca9af and ASTAnd_11494ace4fb and the values are instances of javassist.CtNewClass. Each entry in this map looks like it retains about 1,900 bytes, for a grand total of about 90 MB of memory used. These numbers came from my staging deployment where I had the profiler attached, using some reflection tricks I was able to look at a production site and found that it had about 240,000 items in that class pool.. approximately 450 MB of memory. So I guess the questions in my mind are: Why are there so many classes in the pool? Why does the number only ever go up? Do those classes really need to stay in the pool forever? - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Memory consumption in T4.1.2 - Hard data
Hmmm...well, I don't think I like the sound of any of that. I'm just going to pretend this problem doesn't exist. (just kidding) I had thought I was doing something special with the ognl compilations that would cause its generated classes to not hang around afterwards in any pools. I'll take a look at things this weekend and am sure a fix will appear in between now and Monday - if there is a reasonable fix to be found. (in 4.1.3 snapshot form) On 8/24/07, Bryan Dotzour [EMAIL PROTECTED] wrote: I and another colleague of mine have been investigating what seems to be a memory leak in our Tapestry application for about a month since we upgraded to T4.1.2. I won't bore you with the saga of the last month, but I would like to present the data I've gathered and look to the list for a proposed solution. I was reading a recent thread in which Jesse said (08/09/2007): There is a map that grows as large as the system using it internally to javassist of various cached reflection info - but it doesn't leak in any way. This is precisely what I've found in profiling our application and it *appears* to be this map that is causing our applications to eventually run out of memory. The YourKit profiler shows me that, as time goes on, there is an instance of HiveMindClassPool that grows and grows as class instances are created. This class extends from javassist.ClassPool and is the map that Jesse is talking about in his quote above. And he's right, I wouldn't say that the class pool leaks either because it looks like it's designed to retain that memory until the class pool itself is no longer needed. Take this quote from the javassist.ClassPool javadocs: Memory consumption memo: ClassPool objects hold all the CtClasses that have been created so that the consistency among modified classes can be guaranteed. Thus if a large number of CtClasses are processed, the ClassPool will consume a huge amount of memory. To avoid this, a ClassPool object should be recreated, for example, every hundred classes processed. Note that getDefault() is a singleton factory. Otherwise, detach() in CtClass should be used to avoid huge memory consumption. This huge memory consumption by the ClassPool is what I was seeing. In particular it is the ClassPool that is held onto by OgnlRuntime. Inspecting this object in the profiler showed that it has a map containing about 45,000 classes. All of the keys into this map were things like: ASTTest_11494aca9af and ASTAnd_11494ace4fb and the values are instances of javassist.CtNewClass. Each entry in this map looks like it retains about 1,900 bytes, for a grand total of about 90 MB of memory used. These numbers came from my staging deployment where I had the profiler attached, using some reflection tricks I was able to look at a production site and found that it had about 240,000 items in that class pool.. approximately 450 MB of memory. So I guess the questions in my mind are: Why are there so many classes in the pool? Why does the number only ever go up? Do those classes really need to stay in the pool forever? -- Jesse Kuhnert Tapestry/Dojo team member/developer Open source based consulting work centered around dojo/tapestry/tacos/hivemind. http://blog.opencomponentry.com - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
