Re: Instance becomes unresponsive after 12 hours

[Chuck Hill]

Hi Maik,

The snapshots are purged from the cache when an EC is disposed if no other EO 
in another EC is still referencing them.  That is, disposing will decrement the 
reference count for the snapshot of the EOs in the EC.  That should happen 
automatically when the page is garbage collecting, assuming of course there are 
no other references to an EC in that EC.  Dispose is commonly called to speed 
up this process.  But to do that, you need to know when you are done with the 
EC which can be hard in a page.

The errors you got from calling dispose may have been due to references on the 
page or elsewhere to EOs in the EC.   Calling methods on them after their EC 
was disposed would result in NPE or other exceptions.  Alternatives would be to 
call refaultAllObjects() or nulling the EC variable.

This is interesting: “With the number of objects growing we observed quadratic 
runtime growth when retrieving objects form the snapshot cache”.   That 
suggests that with large numbers of objects inefficiencies in the hash function 
(too many collisions) come into play, or there are algorithmic deficiencies in 
the dictionary used to store the snapshots.  Neither of which would be easy to 
fix.

Chuck


From: 
mailto:webobjects-dev-bounces+chill=gevityinc@lists.apple.com>>
 on behalf of Musall Maik mailto:m...@selbstdenker.ag>>
Date: Wednesday, January 27, 2016 at 5:38 AM
To: Apple WebObjects Developer List 
mailto:webobjects-dev@lists.apple.com>>
Subject: Re: Instance becomes unresponsive after 12 hours

Hi Dennis,

thanks for your suggestions.

Currently, I don't dispose of editing contexts explicitly. I had EOF errors a 
few years ago when I used to have dispose() calls after the final 
saveChanges(), so I stopped doing dispose(). (I don't recall the details.) I do 
however create editing contexts frequently and do only limited stuff in each 
one. I don't use the session editing context.

So in theory, any editing context should become unreachable once the component 
is no longer in the page cache, and then the garbage collector should take care 
of it. Are you suggesting something different?

And you also suggest that when I dispose() of editing contexts, the 
corresponding snapshots are purged from the snapshot cache? That would be new 
to me. Currently I don't know of a method to do that other than 
invalidateAllObjects() or invalidateObjectsWithGlobalIDs().

Maik


Am 27.01.2016 um 10:39 schrieb Dennis Bliefernicht 
mailto:dennis.blieferni...@xyrality.com>>:

Hi,

On 26.01.2016, at 14:28, Musall Maik 
mailto:m...@selbstdenker.ag>> wrote:

Anyway, heap sizes of up to 60 GByte are apparently not so common, at least not 
with WebObjects. I found a helpful article [1] about very large JVM heaps, 
including the hint to the Zing Azul JVM, which features a stopless garbage 
collector [3], but costs $8000 per server and year. In the end, we may actually 
be trying that if other options fail, but first I need to understand more about 
what's in memory. I'm currently approaching that question via JProfiler, 
especially to find out if that's all snapshot cache (how would I limit that in 
size?) or some other homemade memory leak.

Just a shot in the dark (especially the regular timing makes this weird), but 
we have had some issues with the snapshot cache accumulating a lot of data when 
some editing contexts weren't disposed early on (and the referred objects thus 
were carried on until session timeouts may finally remove them). With the 
number of objects growing we observed (if I remember correctly) quadratic 
runtime growth when retrieving objects form the snapshot cache. As we handle a 
large amount of small objects, this escalated rather suddenly when the cache 
exceeded a certain size. In this case JProfiler should show you larger amounts 
of time spent retrieving objects from the snapshot cache in the Hotspots list. 
In our case we were able to dispose editing contexts as soon as possible, 
flushing the objects out of the cache and keeping retrieval time speedy. 
Restarting instances would in this case also keep the accumulated cruft down, 
this scenario would probably only match though, if a significant amount of 
objects come into the system shortly before or at the waiting times.

Greetings
Dennis

--

[cid:B4A9CB32-6254-49B0-9576-C3C5EE9346A5@datacenter]



-
Dennis Bliefernicht • Backend Development
T +49 40 357 3001 62
dennis.blieferni...@xyrality.com<mailto:dennis.blieferni...@xyrality.com>

XYRALITY GmbH • Friedensallee 290 • 22763 Hamburg
www.xyrality.com<http://www.xyrality.com/>
Registergericht: Hamburg HRB 115332
Geschäftsführer: Sven Ossenbrüggen
-


 ___
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list 

Re: Instance becomes unresponsive after 12 hours

[Musall Maik]

Hi Dennis,

thanks for your suggestions.

Currently, I don't dispose of editing contexts explicitly. I had EOF errors a 
few years ago when I used to have dispose() calls after the final 
saveChanges(), so I stopped doing dispose(). (I don't recall the details.) I do 
however create editing contexts frequently and do only limited stuff in each 
one. I don't use the session editing context.

So in theory, any editing context should become unreachable once the component 
is no longer in the page cache, and then the garbage collector should take care 
of it. Are you suggesting something different?

And you also suggest that when I dispose() of editing contexts, the 
corresponding snapshots are purged from the snapshot cache? That would be new 
to me. Currently I don't know of a method to do that other than 
invalidateAllObjects() or invalidateObjectsWithGlobalIDs().

Maik


> Am 27.01.2016 um 10:39 schrieb Dennis Bliefernicht 
> :
> 
> Hi,
> 
>> On 26.01.2016, at 14:28, Musall Maik > > wrote:
>> 
>> Anyway, heap sizes of up to 60 GByte are apparently not so common, at least 
>> not with WebObjects. I found a helpful article [1] about very large JVM 
>> heaps, including the hint to the Zing Azul JVM, which features a stopless 
>> garbage collector [3], but costs $8000 per server and year. In the end, we 
>> may actually be trying that if other options fail, but first I need to 
>> understand more about what's in memory. I'm currently approaching that 
>> question via JProfiler, especially to find out if that's all snapshot cache 
>> (how would I limit that in size?) or some other homemade memory leak.
> 
> Just a shot in the dark (especially the regular timing makes this weird), but 
> we have had some issues with the snapshot cache accumulating a lot of data 
> when some editing contexts weren't disposed early on (and the referred 
> objects thus were carried on until session timeouts may finally remove them). 
> With the number of objects growing we observed (if I remember correctly) 
> quadratic runtime growth when retrieving objects form the snapshot cache. As 
> we handle a large amount of small objects, this escalated rather suddenly 
> when the cache exceeded a certain size. In this case JProfiler should show 
> you larger amounts of time spent retrieving objects from the snapshot cache 
> in the Hotspots list. In our case we were able to dispose editing contexts as 
> soon as possible, flushing the objects out of the cache and keeping retrieval 
> time speedy. Restarting instances would in this case also keep the 
> accumulated cruft down, this scenario would probably only match though, if a 
> significant amount of objects come into the system shortly before or at the 
> waiting times.
> 
> Greetings
> Dennis
> 
> --
> 
> 
> 
> 
> 
> -
> Dennis Bliefernicht • Backend Development
> T +49 40 357 3001 62
> dennis.blieferni...@xyrality.com 
> 
> XYRALITY GmbH • Friedensallee 290 • 22763 Hamburg
> www.xyrality.com 
> Registergericht: Hamburg HRB 115332
> Geschäftsführer: Sven Ossenbrüggen
> -
> 

 ___
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list  (Webobjects-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: Instance becomes unresponsive after 12 hours

[Musall Maik]

Hi Chuck,

of course it could be something other than garbage collection, but at this 
point I'm pretty sure it's something related to memory, either GC, or snapshot 
cache size, or something related.

I checked for all sorts of things that can occur at 5pm and didn't find any. In 
fact the problematic behaviour can also start a few minutes before or after 
that sometimes, and I don't see any meaningful spikes in resource consumption 
of any kind at that time. Backup is at night, I have monitoring for expensive 
processing jobs and also would see them in the application log. Database is 
exclusive for this application, and EOF transaction times are fast.

The one thing that I always see when this happens is a large heap, and it never 
happens as long as the heap is small. I spent the last 24 hours learning about 
garbage collectors, and there's a lot to test and try there, but ultimately 
there are unavoidable Stop-The-World events with all available options in the 
GC implementations we have in the Oracle JVM, and the only thing you can tune 
is how often they occur vs. how long they take, and both depend on either the 
live object set size or the total heap size.

Maik



> Am 26.01.2016 um 19:44 schrieb Chuck Hill :
> 
> Hi Maik,
> 
> I will guess that the answer to the following is No as you said if you 
> restart the apps at 3PM the problem does not a happen.
> 
> Could it be something other than garbage collection?  I’ve seen a similar 
> problem when someone installed backup software that ran during business 
> hours.  Could it be something unrelated to  your apps that starts at 5PM?  Or 
> are some users starting very intensive processing at the end of their day and 
> impacting other users?  Could it be on the database side?  A delay there 
> would leave the app stalled with very little CPU or IO activity.  That is the 
> only thing that occurred to me other than the GC you are investigating.
> 
> Chuck
> 
> 
> From:  <mailto:webobjects-dev-bounces+chill=gevityinc@lists.apple.com>> on 
> behalf of Musall Maik mailto:m...@selbstdenker.ag>>
> Date: Tuesday, January 26, 2016 at 5:28 AM
> To: Apple WebObjects Developer List  <mailto:webobjects-dev@lists.apple.com>>
> Subject: Re: Instance becomes unresponsive after 12 hours
> 
> Hi all,
> 
> a few of you also replied by private mail, thanks. Supplying the -Xloggc 
> option worked, although I don't understand where the info went before, and 
> the log rotation option doesn't rotate anything but just truncates the log 
> file on every JVM restart, which sucks. But ok, I have some data.
> 
> Below you find the log of one session. Users reported bad response times that 
> correspond with the Full GC runs, but also very bad response times from 17:00 
> onward where there is often no GC activity logged at the times that users 
> experienced waiting time. So I'm not entirely sure if this is all caused by 
> GC.
> 
> Anyway, heap sizes of up to 60 GByte are apparently not so common, at least 
> not with WebObjects. I found a helpful article [1] about very large JVM 
> heaps, including the hint to the Zing Azul JVM, which features a stopless 
> garbage collector [3], but costs $8000 per server and year. In the end, we 
> may actually be trying that if other options fail, but first I need to 
> understand more about what's in memory. I'm currently approaching that 
> question via JProfiler, especially to find out if that's all snapshot cache 
> (how would I limit that in size?) or some other homemade memory leak.
> 
> Does anyone have experience with the combination of Azul and WO?
> 
> I also found one strange event today where the application appeared stuck, 
> but continued after a minute or two, while the gc log file was truncated at 
> that time, without the JVM restarting. I can't see how that's supposed to 
> happen. Here's how that looked in my shell running "tail -F gc.log":
> 
> 
> [PSYoungGen: 15256082K->3431947K(16962048K)] 25580475K->13801928K(27895808K), 
> 1.1505415 secs] [Times: user=6.00 sys=0.03, real=1.15 secs]
> 2016-01-26T12:11:38.640+0100: 26136.407: [GC (Allocation Failure)
> Desired survivor size 4679794688 bytes, new threshold 15 (max 15)
> [PSYoungGen: 15791627K->3414349K(16958976K)] 26161608K->13935770K(27892736K), 
> 1.0761646 secs] [Times: user=6.10 sys=0.17, real=1.08 secs]
> 2016-01-26T12:16:10.835+0100: 26408.603: [GC (Allocation Failure)
> Desired survivor size 4657250304 bytes, new threshold 15 (max 15)
> [PSYoungGen: 15774029K->3449938K(17397760K)] 26295450K->14128338K(28331520K), 
> 1.3285101 secs] [Times: user=6.24 sys=1.39, real=1.33 secs]
> 2016-01-26T12:16:12.164+0100: 26409.931: [Full GC (Ergonomics) 

Re: Instance becomes unresponsive after 12 hours

[Dennis Bliefernicht]

Hi,

> On 26.01.2016, at 14:28, Musall Maik  wrote:
> 
> Anyway, heap sizes of up to 60 GByte are apparently not so common, at least 
> not with WebObjects. I found a helpful article [1] about very large JVM 
> heaps, including the hint to the Zing Azul JVM, which features a stopless 
> garbage collector [3], but costs $8000 per server and year. In the end, we 
> may actually be trying that if other options fail, but first I need to 
> understand more about what's in memory. I'm currently approaching that 
> question via JProfiler, especially to find out if that's all snapshot cache 
> (how would I limit that in size?) or some other homemade memory leak.

Just a shot in the dark (especially the regular timing makes this weird), but 
we have had some issues with the snapshot cache accumulating a lot of data when 
some editing contexts weren't disposed early on (and the referred objects thus 
were carried on until session timeouts may finally remove them). With the 
number of objects growing we observed (if I remember correctly) quadratic 
runtime growth when retrieving objects form the snapshot cache. As we handle a 
large amount of small objects, this escalated rather suddenly when the cache 
exceeded a certain size. In this case JProfiler should show you larger amounts 
of time spent retrieving objects from the snapshot cache in the Hotspots list. 
In our case we were able to dispose editing contexts as soon as possible, 
flushing the objects out of the cache and keeping retrieval time speedy. 
Restarting instances would in this case also keep the accumulated cruft down, 
this scenario would probably only match though, if a significant amount of 
objects come into the system shortly before or at the waiting times.

Greetings
Dennis

--





-
Dennis Bliefernicht • Backend Development
T +49 40 357 3001 62
dennis.blieferni...@xyrality.com

XYRALITY GmbH • Friedensallee 290 • 22763 Hamburg
www.xyrality.com 
Registergericht: Hamburg HRB 115332
Geschäftsführer: Sven Ossenbrüggen
-

 ___
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list  (Webobjects-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: Instance becomes unresponsive after 12 hours

[Chuck Hill]

Hi Maik,

I will guess that the answer to the following is No as you said if you restart 
the apps at 3PM the problem does not a happen.

Could it be something other than garbage collection?  I’ve seen a similar 
problem when someone installed backup software that ran during business hours.  
Could it be something unrelated to  your apps that starts at 5PM?  Or are some 
users starting very intensive processing at the end of their day and impacting 
other users?  Could it be on the database side?  A delay there would leave the 
app stalled with very little CPU or IO activity.  That is the only thing that 
occurred to me other than the GC you are investigating.

Chuck


From: 
mailto:webobjects-dev-bounces+chill=gevityinc@lists.apple.com>>
 on behalf of Musall Maik mailto:m...@selbstdenker.ag>>
Date: Tuesday, January 26, 2016 at 5:28 AM
To: Apple WebObjects Developer List 
mailto:webobjects-dev@lists.apple.com>>
Subject: Re: Instance becomes unresponsive after 12 hours

Hi all,

a few of you also replied by private mail, thanks. Supplying the -Xloggc option 
worked, although I don't understand where the info went before, and the log 
rotation option doesn't rotate anything but just truncates the log file on 
every JVM restart, which sucks. But ok, I have some data.

Below you find the log of one session. Users reported bad response times that 
correspond with the Full GC runs, but also very bad response times from 17:00 
onward where there is often no GC activity logged at the times that users 
experienced waiting time. So I'm not entirely sure if this is all caused by GC.

Anyway, heap sizes of up to 60 GByte are apparently not so common, at least not 
with WebObjects. I found a helpful article [1] about very large JVM heaps, 
including the hint to the Zing Azul JVM, which features a stopless garbage 
collector [3], but costs $8000 per server and year. In the end, we may actually 
be trying that if other options fail, but first I need to understand more about 
what's in memory. I'm currently approaching that question via JProfiler, 
especially to find out if that's all snapshot cache (how would I limit that in 
size?) or some other homemade memory leak.

Does anyone have experience with the combination of Azul and WO?

I also found one strange event today where the application appeared stuck, but 
continued after a minute or two, while the gc log file was truncated at that 
time, without the JVM restarting. I can't see how that's supposed to happen. 
Here's how that looked in my shell running "tail -F gc.log":


[PSYoungGen: 15256082K->3431947K(16962048K)] 25580475K->13801928K(27895808K), 
1.1505415 secs] [Times: user=6.00 sys=0.03, real=1.15 secs]
2016-01-26T12:11:38.640+0100: 26136.407: [GC (Allocation Failure)
Desired survivor size 4679794688 bytes, new threshold 15 (max 15)
[PSYoungGen: 15791627K->3414349K(16958976K)] 26161608K->13935770K(27892736K), 
1.0761646 secs] [Times: user=6.10 sys=0.17, real=1.08 secs]
2016-01-26T12:16:10.835+0100: 26408.603: [GC (Allocation Failure)
Desired survivor size 4657250304 bytes, new threshold 15 (max 15)
[PSYoungGen: 15774029K->3449938K(17397760K)] 26295450K->14128338K(28331520K), 
1.3285101 secs] [Times: user=6.24 sys=1.39, real=1.33 secs]
2016-01-26T12:16:12.164+0100: 26409.931: [Full GC (Ergonomics) [PSYoungGen: 
3449938K->2355719K(17397760K)] [ParOldGen: 10678400K->10933508K(13649408K)] 
14128338K->13289227K(31047168K), [Metaspace: 84345K->84345K(94208K)], 
12.1170523 secs] [Times: user=67.98 sys=1.03, real=12.11 secs]
2016-01-26T12:21:18.990+0100: 26716.757: [GC (Allocation Failure)
Desired survivor size 4644667392 bytes, new threshold 15 (max 15)
[PSYoungGen: 15218183K->2398851K(17410560K)] 26151691K->13458637K(31059968K), 
1.3389149 secs] [Times: user=8.72 sys=0.17, real=1.34 secs]
tail: gc.log: file truncated
Java HotSpot(TM) 64-Bit Server VM (25.66-b17) for linux-amd64 JRE 
(1.8.0_66-b17), built on Oct  6 2015 17:28:34 by "java_re" with gcc 4.3.0 
20080428 (Red Hat 4.3.0-8)
Memory: 4k page, physical 99008820k(13655516k free), swap 67108860k(66990156k 
free)
CommandLine flags: -XX:InitialHeapSize=1584141120 -XX:+ManagementServer 
-XX:MaxHeapSize=90194313216 -XX:+PrintGC -XX:+PrintGCDateStamps 
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution 
-XX:-UseGCLogFileRotation -XX:+UseParallelGC
2016-01-26T12:33:36.029+0100: 27453.796: [GC (Allocation Failure)
Desired survivor size 4588568576 bytes, new threshold 15 (max 15)
[PSYoungGen: 15261315K->2538940K(17720832K)] 26321101K->13670347K(31370240K), 
0.7936427 secs] [Times: user=4.62 sys=0.03, real=0.80 secs]
2016-01-26T12:39:29.156+0100: 27806.923: [GC (Allocation Failure)


And here's the above-mentioned full gc log of 4 hours during the most critical 
usage time where users experienced waits. Notable wait times according to 
user's notes occurre

Re: Instance becomes unresponsive after 12 hours

[Musall Maik]

Hi all,

a few of you also replied by private mail, thanks. Supplying the -Xloggc option 
worked, although I don't understand where the info went before, and the log 
rotation option doesn't rotate anything but just truncates the log file on 
every JVM restart, which sucks. But ok, I have some data.

Below you find the log of one session. Users reported bad response times that 
correspond with the Full GC runs, but also very bad response times from 17:00 
onward where there is often no GC activity logged at the times that users 
experienced waiting time. So I'm not entirely sure if this is all caused by GC.

Anyway, heap sizes of up to 60 GByte are apparently not so common, at least not 
with WebObjects. I found a helpful article [1] about very large JVM heaps, 
including the hint to the Zing Azul JVM, which features a stopless garbage 
collector [3], but costs $8000 per server and year. In the end, we may actually 
be trying that if other options fail, but first I need to understand more about 
what's in memory. I'm currently approaching that question via JProfiler, 
especially to find out if that's all snapshot cache (how would I limit that in 
size?) or some other homemade memory leak.

Does anyone have experience with the combination of Azul and WO?

I also found one strange event today where the application appeared stuck, but 
continued after a minute or two, while the gc log file was truncated at that 
time, without the JVM restarting. I can't see how that's supposed to happen. 
Here's how that looked in my shell running "tail -F gc.log":


[PSYoungGen: 15256082K->3431947K(16962048K)] 25580475K->13801928K(27895808K), 
1.1505415 secs] [Times: user=6.00 sys=0.03, real=1.15 secs]
2016-01-26T12:11:38.640+0100: 26136.407: [GC (Allocation Failure)
Desired survivor size 4679794688 bytes, new threshold 15 (max 15)
[PSYoungGen: 15791627K->3414349K(16958976K)] 26161608K->13935770K(27892736K), 
1.0761646 secs] [Times: user=6.10 sys=0.17, real=1.08 secs]
2016-01-26T12:16:10.835+0100: 26408.603: [GC (Allocation Failure)
Desired survivor size 4657250304 bytes, new threshold 15 (max 15)
[PSYoungGen: 15774029K->3449938K(17397760K)] 26295450K->14128338K(28331520K), 
1.3285101 secs] [Times: user=6.24 sys=1.39, real=1.33 secs]
2016-01-26T12:16:12.164+0100: 26409.931: [Full GC (Ergonomics) [PSYoungGen: 
3449938K->2355719K(17397760K)] [ParOldGen: 10678400K->10933508K(13649408K)] 
14128338K->13289227K(31047168K), [Metaspace: 84345K->84345K(94208K)], 
12.1170523 secs] [Times: user=67.98 sys=1.03, real=12.11 secs]
2016-01-26T12:21:18.990+0100: 26716.757: [GC (Allocation Failure)
Desired survivor size 4644667392 bytes, new threshold 15 (max 15)
[PSYoungGen: 15218183K->2398851K(17410560K)] 26151691K->13458637K(31059968K), 
1.3389149 secs] [Times: user=8.72 sys=0.17, real=1.34 secs]
tail: gc.log: file truncated
Java HotSpot(TM) 64-Bit Server VM (25.66-b17) for linux-amd64 JRE 
(1.8.0_66-b17), built on Oct  6 2015 17:28:34 by "java_re" with gcc 4.3.0 
20080428 (Red Hat 4.3.0-8)
Memory: 4k page, physical 99008820k(13655516k free), swap 67108860k(66990156k 
free)
CommandLine flags: -XX:InitialHeapSize=1584141120 -XX:+ManagementServer 
-XX:MaxHeapSize=90194313216 -XX:+PrintGC -XX:+PrintGCDateStamps 
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution 
-XX:-UseGCLogFileRotation -XX:+UseParallelGC
2016-01-26T12:33:36.029+0100: 27453.796: [GC (Allocation Failure)
Desired survivor size 4588568576 bytes, new threshold 15 (max 15)
[PSYoungGen: 15261315K->2538940K(17720832K)] 26321101K->13670347K(31370240K), 
0.7936427 secs] [Times: user=4.62 sys=0.03, real=0.80 secs]
2016-01-26T12:39:29.156+0100: 27806.923: [GC (Allocation Failure)


And here's the above-mentioned full gc log of 4 hours during the most critical 
usage time where users experienced waits. Notable wait times according to 
user's notes occurred at least at the following times: 15:07, 16:05, 16:57, 
17:00-17:35, 17:59.


2016-01-25T15:06:54.344+0100: 36634.642: [GC (Allocation Failure) [PSYoungGen: 
18701268K->2058545K(23079424K)] 33613120K->17079754K(41686016K), 0.8635088 
secs] [Times: user=4.22 sys=0.50, real=0.86 secs] 
2016-01-25T15:15:48.438+0100: 37168.736: [GC (Allocation Failure) [PSYoungGen: 
19683633K->2231668K(23157248K)] 34704842K->17370085K(41763840K), 0.9898285 
secs] [Times: user=4.98 sys=0.05, real=0.99 secs] 
2016-01-25T15:24:54.891+0100: 37715.189: [GC (Allocation Failure) [PSYoungGen: 
19856756K->2254238K(23623168K)] 34995173K->17626274K(42229760K), 1.0030974 
secs] [Times: user=4.91 sys=0.30, real=1.00 secs] 
2016-01-25T15:33:58.784+0100: 38259.082: [GC (Allocation Failure) [PSYoungGen: 
20552606K->2502793K(23740416K)] 35924642K->17914660K(42347008K), 1.0203825 
secs] [Times: user=5.32 sys=0.24, real=1.02 secs] 
2016-01-25T15:43:12.959+0100: 38813.256: [GC (Allocation Failure) [PSYoungGen: 
20801161K->2677482K(23608320K)] 36213028K->18138909K(42214912K), 1.231 
secs] [Times: user=5.53 sys=0.22, real=1.23 secs] 
2016-01-25T15:51

Re: Instance becomes unresponsive after 12 hours

[Andrus Adamchik]

Here is my favorite:

jstat -gc  1 3000

Attaches to a local Java process and prints all minor and major GC's. Usually 
more detailed than VisualVM.

Andrus

---
Andrus Adamchik
@andrus_a | @ApacheCayenne


> On Jan 24, 2016, at 10:19 PM, Maik Musall  wrote:
> 
> Hi,
> 
> yes, I tried, but the documented log output would never appear anywhere in my 
> logs. These are the options I'm using:
> 
> -verbose:gc
> -XX:+PrintGCDetails
> -XX:+PrintGCDateStamps
> 
> Despite these settings, I can't see ANYTHING in the logs about GC. I don't 
> know what I'm doing wrong there?
> 
> Maik
> 
>> Am 24.01.2016 um 13:27 schrieb Bogdan Zlatanov :
>> 
>> Hi,
>> 
>> Have you tried tracing the garbage collection -> 
>> https://docs.oracle.com/cd/E19159-01/819-3681/abeig/index.html?
>> 
>> Cheers,
>> Bogdan
>> 
>>> On 24/01/2016 13:01, Musall Maik wrote:
>>> Hi,
>>> 
>>> I have an application that often becomes laggy and shows slow responses 
>>> almost every day around 5 pm. The instance has daily scheduled restarts at 
>>> 5 am, so it occurs after 12 hours of use. When I restart it preemptively at 
>>> 3 pm, the problem does not occur. When I don't, users suffer bad response 
>>> times on many requests, but not all.
>>> 
>>> Users are computing heavy statistics in this application, instance size 
>>> (mostly snapshot cache) at that time is usually about 50 GByte, while Xmx 
>>> is 84 GByte, and XX:MaxPermSize is 1024m. The biggest instance size I've 
>>> seen so far in production is 61 GByte, so 50 seems not like a limit where 
>>> something weird happens.
>>> 
>>> CPU, memory, filesystem, database and network all show nothing special 
>>> before and after 5 pm. I monitored the instance with jvisualvm to no avail. 
>>> I've been searching for the reason for quite some time now, so I'd 
>>> appreciate any suggestion you might have what else to check.
>>> 
>>> Thanks
>>> Maik



 ___
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list  (Webobjects-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: Instance becomes unresponsive after 12 hours

[Bogdan Zlatanov]

Hi again,

Try redirecting all the GC output in it's own file with /the -Xloggc: 
/path/to/gc/log /option.


Cheers,
Bogdan

On 24/01/2016 20:19, Maik Musall wrote:

Hi,

yes, I tried, but the documented log output would never appear anywhere in my 
logs. These are the options I'm using:

-verbose:gc
-XX:+PrintGCDetails
-XX:+PrintGCDateStamps

Despite these settings, I can't see ANYTHING in the logs about GC. I don't know 
what I'm doing wrong there?

Maik


Am 24.01.2016 um 13:27 schrieb Bogdan Zlatanov :

Hi,

Have you tried tracing the garbage collection -> 
https://docs.oracle.com/cd/E19159-01/819-3681/abeig/index.html?

Cheers,
Bogdan


On 24/01/2016 13:01, Musall Maik wrote:
Hi,

I have an application that often becomes laggy and shows slow responses almost 
every day around 5 pm. The instance has daily scheduled restarts at 5 am, so it 
occurs after 12 hours of use. When I restart it preemptively at 3 pm, the 
problem does not occur. When I don't, users suffer bad response times on many 
requests, but not all.

Users are computing heavy statistics in this application, instance size (mostly 
snapshot cache) at that time is usually about 50 GByte, while Xmx is 84 GByte, 
and XX:MaxPermSize is 1024m. The biggest instance size I've seen so far in 
production is 61 GByte, so 50 seems not like a limit where something weird 
happens.

CPU, memory, filesystem, database and network all show nothing special before 
and after 5 pm. I monitored the instance with jvisualvm to no avail. I've been 
searching for the reason for quite some time now, so I'd appreciate any 
suggestion you might have what else to check.

Thanks
Maik


___
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list  (Webobjects-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/webobjects-dev/bogdan.zlatanov%40gmail.com

This email sent to bogdan.zlata...@gmail.com

___
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list  (Webobjects-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/webobjects-dev/maik%40selbstdenker.ag

This email sent to m...@selbstdenker.ag

  ___
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list  (Webobjects-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/webobjects-dev/bogdan.zlatanov%40gmail.com

This email sent to bogdan.zlata...@gmail.com


 ___
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list  (Webobjects-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: Instance becomes unresponsive after 12 hours

[Maik Musall]

Hi,

yes, I tried, but the documented log output would never appear anywhere in my 
logs. These are the options I'm using:

-verbose:gc
-XX:+PrintGCDetails
-XX:+PrintGCDateStamps

Despite these settings, I can't see ANYTHING in the logs about GC. I don't know 
what I'm doing wrong there?

Maik

> Am 24.01.2016 um 13:27 schrieb Bogdan Zlatanov :
> 
> Hi,
> 
> Have you tried tracing the garbage collection -> 
> https://docs.oracle.com/cd/E19159-01/819-3681/abeig/index.html?
> 
> Cheers,
> Bogdan
> 
>> On 24/01/2016 13:01, Musall Maik wrote:
>> Hi,
>> 
>> I have an application that often becomes laggy and shows slow responses 
>> almost every day around 5 pm. The instance has daily scheduled restarts at 5 
>> am, so it occurs after 12 hours of use. When I restart it preemptively at 3 
>> pm, the problem does not occur. When I don't, users suffer bad response 
>> times on many requests, but not all.
>> 
>> Users are computing heavy statistics in this application, instance size 
>> (mostly snapshot cache) at that time is usually about 50 GByte, while Xmx is 
>> 84 GByte, and XX:MaxPermSize is 1024m. The biggest instance size I've seen 
>> so far in production is 61 GByte, so 50 seems not like a limit where 
>> something weird happens.
>> 
>> CPU, memory, filesystem, database and network all show nothing special 
>> before and after 5 pm. I monitored the instance with jvisualvm to no avail. 
>> I've been searching for the reason for quite some time now, so I'd 
>> appreciate any suggestion you might have what else to check.
>> 
>> Thanks
>> Maik
>> 
>> 
>> ___
>> Do not post admin requests to the list. They will be ignored.
>> Webobjects-dev mailing list  (Webobjects-dev@lists.apple.com)
>> Help/Unsubscribe/Update your Subscription:
>> https://lists.apple.com/mailman/options/webobjects-dev/bogdan.zlatanov%40gmail.com
>> 
>> This email sent to bogdan.zlata...@gmail.com
> 
> ___
> Do not post admin requests to the list. They will be ignored.
> Webobjects-dev mailing list  (Webobjects-dev@lists.apple.com)
> Help/Unsubscribe/Update your Subscription:
> https://lists.apple.com/mailman/options/webobjects-dev/maik%40selbstdenker.ag
> 
> This email sent to m...@selbstdenker.ag

 ___
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list  (Webobjects-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: Instance becomes unresponsive after 12 hours

[Bogdan Zlatanov]

Hi,

Have you tried tracing the garbage collection -> 
https://docs.oracle.com/cd/E19159-01/819-3681/abeig/index.html?


Cheers,
Bogdan

On 24/01/2016 13:01, Musall Maik wrote:

Hi,

I have an application that often becomes laggy and shows slow responses almost 
every day around 5 pm. The instance has daily scheduled restarts at 5 am, so it 
occurs after 12 hours of use. When I restart it preemptively at 3 pm, the 
problem does not occur. When I don't, users suffer bad response times on many 
requests, but not all.

Users are computing heavy statistics in this application, instance size (mostly 
snapshot cache) at that time is usually about 50 GByte, while Xmx is 84 GByte, 
and XX:MaxPermSize is 1024m. The biggest instance size I've seen so far in 
production is 61 GByte, so 50 seems not like a limit where something weird 
happens.

CPU, memory, filesystem, database and network all show nothing special before 
and after 5 pm. I monitored the instance with jvisualvm to no avail. I've been 
searching for the reason for quite some time now, so I'd appreciate any 
suggestion you might have what else to check.

Thanks
Maik


  ___
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list  (Webobjects-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/webobjects-dev/bogdan.zlatanov%40gmail.com

This email sent to bogdan.zlata...@gmail.com


___
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list  (Webobjects-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Instance becomes unresponsive after 12 hours

[Musall Maik]

Hi,

I have an application that often becomes laggy and shows slow responses almost 
every day around 5 pm. The instance has daily scheduled restarts at 5 am, so it 
occurs after 12 hours of use. When I restart it preemptively at 3 pm, the 
problem does not occur. When I don't, users suffer bad response times on many 
requests, but not all.

Users are computing heavy statistics in this application, instance size (mostly 
snapshot cache) at that time is usually about 50 GByte, while Xmx is 84 GByte, 
and XX:MaxPermSize is 1024m. The biggest instance size I've seen so far in 
production is 61 GByte, so 50 seems not like a limit where something weird 
happens.

CPU, memory, filesystem, database and network all show nothing special before 
and after 5 pm. I monitored the instance with jvisualvm to no avail. I've been 
searching for the reason for quite some time now, so I'd appreciate any 
suggestion you might have what else to check.

Thanks
Maik


 ___
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list  (Webobjects-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com