John Embretsen wrote:
Deepa Remesh wrote:
Thanks John for uploading your repro. I have not been able to look at
it closely as jira seems to be having problems again. I was wondering
if you have the results from your DOTS test. Please post if you have
any updates.
Actually, the DOTS test failed, i.e. the network server hit an
OutOfMemoryError (PermGenSpace) after about 56 hours (not much longer
than without the patch), but I suspect that something is not right...
I get strange results from running my repro with the same jars, and
*sometimes* with freshly built jars even with your patch included. This
is not what I was seeing before the weekend. I am doing some detective
work to figure out what could be wrong with my environment (scripts,
classpaths, JVM settings, build process, patch, etc.). When I figure it
out, I will re-run the tests and post results.
I think I figured out what was causing the unpredictable results when running
my repro:
I was (most of the time, both for DOTS and my repro) running with the
following flags on the server JVM:
-Xms128m -Xmx128m -XX:PermSize=64m -XX:MaxPermSize=64m
These flags are setting both initial and max size of both heap and PermGen
space, with initial size equal to max size. This was to ease debugging (I
think it is easier to interpret memory usage graphs when I know the
capacities of the different parts of the heap). Besides, people recommend
setting initial and max heap size equal to each other for performance
reasons, so I did the same for the permanent generation.
Anyway, I found out that *without* the flag "-XX:PermSize=64m" the test/repro
"passed", while *with* the flag set, the server usually crashed at
approximately 11000 statements. Here is my theory, based on observation of
jstat output:
With my repro, derby generates classes faster than the VM is able to garbage
collect in one go. According to PermGen usage logs (jstat), roughly 500k
worth of bytecode is injected into the permanent generation per second (I
guess mostly due to class generation since most statements sent from the
client are unique).
So, when the initial PermGen size is smaller, i.e. not close to, the max
size, the VM can increase PermGen capacity while garbage collecting old
classes (or something like that). Perhaps there is some small delay before
classes can be unloaded/GCed, I don't know.
I have successfully and predicably run my repro again, now without the
"-XX:PermSize" flag set. Without patch5 for DERBY-210, the server hits an
OutOfMemoryError after roughly 11000 statements. With patch5, the server is
able to gc all parts of the heap, including the PermGen space (I have tested
100000 statements so far, after I discovered this issue).
I have started a new DOTS test run, this time without setting the initial
PermGen size. I am crossing my fingers and hoping this will improve the
results of the DOTS test.
There could of course be other issues with my environment, so I still
recommend that others try some of these tests/repros. Use whatever JVM
settings you feel like, but I recommend not setting initial PermSize equal
(or close) to max PermSize.
--
John
