Re: JVM detection of thread at safepoint

2017-12-05 Thread Gil Tene
Alex, capturing the stack trace as the thread actually reaches a taken safepoint (via SafepointSynchronize::block) is definitely a cool trick. As a "where was I when I finally reached a safepoint after a long TTSP" tracer, it may well help in investigating the problematic code paths if you are

Re: JVM detection of thread at safepoint

2017-12-05 Thread Alex Bagehot
On Tue, Dec 5, 2017 at 11:22 AM, Mark Price wrote: > Hi Kirk, > thanks for the response. > > I've run the same workload with -XX:+KeepSafepointsInCountedLoops, and > the problem persists, so I'm fairly certain it's not that (assuming the JVM > is respecting the flags...). The application is GC-fr

Re: JVM detection of thread at safepoint

2017-12-05 Thread Mark Price
Great, plenty of things to try :) Thanks for your input everyone. On Tuesday, 5 December 2017 12:39:22 UTC, Gil Tene wrote: > > > > Sent from my iPad > > On Dec 5, 2017, at 1:26 PM, Mark Price > wrote: > > >> That (each process having it's own copy) is surprising to me. Unless the >> mapping

Re: JVM detection of thread at safepoint

2017-12-05 Thread Gil Tene
Sent from my iPad On Dec 5, 2017, at 1:26 PM, Mark Price mailto:m...@aitusoftware.com>> wrote: That (each process having it's own copy) is surprising to me. Unless the mapping is such that private copies are required, I'd expect the processes to share the page cache entries. I can't recrea

Re: JVM detection of thread at safepoint

2017-12-05 Thread Gil Tene
On Tuesday, December 5, 2017 at 1:26:23 PM UTC+1, Mark Price wrote: > That (each process having it's own copy) is surprising to me. Unless the > mapping is such that private copies are required, I'd expect the processes to > share the page cache entries. > > > I can't recreate this effect local

Re: JVM detection of thread at safepoint

2017-12-05 Thread Mark Price
> > > That (each process having it's own copy) is surprising to me. Unless the > mapping is such that private copies are required, I'd expect the processes > to share the page cache entries. > I can't recreate this effect locally using FileChannel.map(); the library in use in the application u

Re: JVM detection of thread at safepoint

2017-12-05 Thread Gil Tene
On Tuesday, December 5, 2017 at 12:31:52 PM UTC+1, Mark Price wrote: > > Hi Gil, > thanks for the response. > > I'm fairly sure that interaction with the page-cache is one of the > problems. When this is occurring, the free-mem is already hovering around > vm.min_free_kbytes, and the mapped fil

Re: JVM detection of thread at safepoint

2017-12-05 Thread Gil Tene
You can use it to track down your problem. You can download a Zing trial and play with it. It won't tell you anything about the various HotSpot specific TTSP paths (since most of those don't exist in Zing), but since you suspect mapped i/o based TTSPs, Zing should run into those just as much, a

Re: JVM detection of thread at safepoint

2017-12-05 Thread Mark Price
> > > In Zing, we have a built-in TTSP profiler for exactly this reason. > I remember it fondly :) -- You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group. To unsubscribe from this group and stop receiving emails from it, send an email to

Re: JVM detection of thread at safepoint

2017-12-05 Thread Mark Price
Hi Gil, thanks for the response. I'm fairly sure that interaction with the page-cache is one of the problems. When this is occurring, the free-mem is already hovering around vm.min_free_kbytes, and the mapped files are a significant fraction of system memory. From what I can see, each process t

Re: JVM detection of thread at safepoint

2017-12-05 Thread Gil Tene
One of the challenges in using GC logs to find out if you have TTSP issues is that GC events are extremely infrequent, so not having seen a high TTSP in the log doesn't mean you won't hit one soon... It's literally like deducing the likelihood of hitting something bad from 1000 data points coll

Re: JVM detection of thread at safepoint

2017-12-05 Thread Mark Price
Hi Kirk, thanks for the response. I've run the same workload with -XX:+KeepSafepointsInCountedLoops, and the problem persists, so I'm fairly certain it's not that (assuming the JVM is respecting the flags...). The application is GC-free, so the safepoints are only going to be coming from (non-exha

Re: JVM detection of thread at safepoint

2017-12-05 Thread Kirk Pepperdine
Hi, > On Dec 5, 2017, at 11:53 AM, Gil Tene wrote: > > Page faults in mapped file i/o and counted loops are certainly two common > causes of long TTSP. But there are many other paths that *could* cause it as > well in HotSpot. Without catching it and looking at the stack trace, it's > hard t

Re: JVM detection of thread at safepoint

2017-12-05 Thread Gil Tene
Page faults in mapped file i/o and counted loops are certainly two common causes of long TTSP. But there are many other paths that *could* cause it as well in HotSpot. Without catching it and looking at the stack trace, it's hard to know which ones to blame. Once you knock out one cause, you'll

Re: JVM detection of thread at safepoint

2017-12-05 Thread Mark Price
Hi Aleksey, thanks for the response. The I/O is definitely one problem, but I was trying to figure out whether it was contributing to the long TTSP times, or whether I might have some code that was misbehaving (e.g. NonCountedLoops). Your response aligns with my guesswork, so hopefully I just ha

Re: JVM detection of thread at safepoint

2017-12-05 Thread Aleksey Shipilev
On 12/05/2017 09:26 AM, Mark Price wrote: > I'm investigating some long time-to-safepoint pauses in oracle/openjdk. The > application in question > is also suffering from some fairly nasty I/O problems where latency-sensitive > threads are being > descheduled in uninterruptible sleep state due to

JVM detection of thread at safepoint

2017-12-05 Thread Mark Price
Hi Folks, I'm investigating some long time-to-safepoint pauses in oracle/openjdk. The application in question is also suffering from some fairly nasty I/O problems where latency-sensitive threads are being descheduled in uninterruptible sleep state due to needing a file-system lock. My question