Re: Java performance under a second level VM
Things may be more clear when you see ttime for the guest on 1st level. I suspect you see the missing cycles there. It's not 2nd level CP doing the hard work but 1st level. Rob -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Java performance under a second level VM
The JIT for example uses polling to sample the state. It sets a timer to pop in a few ms and goes to sleep. But since that state change and timer interrupt reflection isn't in hardware the cost appears excessive. If you would be computing pi second level in Linux it may still be pretty decent. And because the optimizer uses elapsed time rather than cpu time it keeps recompiling classes and kind of digs is own grave. You might try without the optimizer on 2nd level Linux. This is probably too specialist topic for general consumption. We can have the discussion offline next week when I'm online again :-) On Apr 17, 2015 8:36 AM, Pavelka, Tomas tomas.pave...@ca.com wrote: Hi Rob, thanks for the reply. The performance of Linux under a second level VM always seemed unpredictable to me but after reading your response I started seeing a pattern: whenever I had a poorly performing application on Linux on a second level VM, there was always some kind of networking involved which in turn involved waiting on open sockets. I just want to ensure I am understanding correctly what your are saying: Is it true that waits under software SIE burn CPU where the same waits under hardware SIE do not? That would explain a lot of the performance problems I have seen but I am sure there are caveats. Thanks, Tomas -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/ -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Java performance under a second level VM
Hi Rob, thanks for the reply. The performance of Linux under a second level VM always seemed unpredictable to me but after reading your response I started seeing a pattern: whenever I had a poorly performing application on Linux on a second level VM, there was always some kind of networking involved which in turn involved waiting on open sockets. I just want to ensure I am understanding correctly what your are saying: Is it true that waits under software SIE burn CPU where the same waits under hardware SIE do not? That would explain a lot of the performance problems I have seen but I am sure there are caveats. Thanks, Tomas -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Java performance under a second level VM
And because the optimizer uses elapsed time rather than cpu time it keeps recompiling classes and kind of digs is own grave. You might try without the optimizer on 2nd level Linux. I wrote a simple benchmark that just prints a number of lines on the console. Java uses about 180 times more CPU on the second level when compared to first level. It uses about 40 times more CPU than Perl doing the same thing on second level. When I turn the JIT off the CPU time on the second level is slightly less than with the JIT on, but the wallclock time is cut by half. This is probably too specialist topic for general consumption. I find this pretty fascinating, not sure about the others. If no one else replies let's move this offline. Thanks, Tomas -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Java performance under a second level VM
On Apr 17, 2015 9:40 AM, Pavelka, Tomas tomas.pave...@ca.com wrote: This is probably too specialist topic for general consumption. I find this pretty fascinating, not sure about the others. If no one else replies let's move this offline. Yes me too. But i have that with most exotic performance challenges :-) -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Java performance under a second level VM
On Friday, 04/17/2015 at 02:37 EDT, Pavelka, Tomas tomas.pave...@ca.com wrote: I just want to ensure I am understanding correctly what your are saying: Is it true that waits under software SIE burn CPU where the same waits under hardware SIE do not? That would explain a lot of the performance problems I have seen but I am sure there are caveats. It depends on how the wait is done. If you set a timer to pop when you want to wake up, there is a lot of variation between 1st and 2nd level, depending on how far in the future the timer goes off. The derivative and worst case is to keep waking up every few milliseconds to see if it's time to wake up. Imagine hitting the snooze button on your alarm clock every 7 minutes just to see if it's time to get up. Right. You'd be cranky, too. When the 2nd level guest issues the SIE instruction, the 1st level system converts the 2nd level SIE into 1st level SIE. I call this the SIE Pancake Effect. This includes a time slice of a time slice and just a few pages of 3rd level guest resident memory. Or CP simulates an instruction. But the more often do you those things, the more overhead is incurred. And if all you're doing is going back to sleep, then it's a lot of CPU burn for no benefit. The miracle is the 3rd level guests run as well as they do. It's the speed of the CPU that save us and is what has allowed 2nd level systems to operate in production. This polling issue helps illustrate why future performance improvements must come primarily from improvements in software - we're nearing the end of the acceleration lane on the current processor technology. Alan Altmark Senior Managing z/VM and Linux Consultant Lab Services System z Delivery Practice IBM Systems Technology Group ibm.com/systems/services/labservices office: 607.429.3323 mobile; 607.321.7556 alan_altm...@us.ibm.com IBM Endicott -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Java performance under a second level VM
Yes. Hardware virtualization support (aka SIE) is only two levels: LPAR and z/VM. Linux running its processes is one too many so must be done in software. That creates overhead. In your case is likely the polling of the JVM that does it, which means is not really related to workload but just burns cycles while idle. CMS does still fine that way but Linux not. The virtual time is while CP has the guest dispatched under SIE. When the user code in SIE is slow due to exceptions that CP must handle the clock is still ticking. Just like when the guest suffers from heavy cache miss. The IBM support assistant (or what it's called now) did pretty nice profile the IBM JVM. There are also lots of GC analyzers if that's where you're looking. Rob On Apr 16, 2015 3:58 PM, Pavelka, Tomas tomas.pave...@ca.com wrote: I have a Java application running a REST API built on top of the Dropwizard stack (www.dropwizard.iohttp://www.dropwizard.io). I have been running it on a first level z/VM and it was always well behaved consuming less than 1% of a CPU on an EC12 during normal operation. I needed to do some testing which I need to run on a second level z/VM. I have experienced performance that pretty much rendered the application unusable. To get a better understanding of the difference in performance I tried to measure the initialization time of the application. This I have done with CP INDICATE USER EXPANDED, measuring Ttime and Vtime, subtracting the values at the beginning from values at the end of the measured interval. For the first level z/VM the initialization took 5 seconds for both Ttime and Vtime. For the second level system Ttime was 0:41:12 and Vtime was 0:41:11, i.e. the initialization on the second level took a little less than 500 times as much as on the first level. We have another group testing software on that system written in Perl and they have not complained about performance (unfortunately that is the closest I have to performance data for the Perl app). The Linux machines on the second level do not feel sluggish (when doing things like text editing directory browsing and copying files, all under an SSH session). This makes me believe that the problem may be Java specific. I also tried running other Java applications, the response was pretty bad especially if the applications did console or file IO (but I don't have any measurements). I tried this under IBM JRE 7.1 and on 8.0 without noticeable change in performance. I have a few questions: 1) Can the total and virtual time on a second level system be relied upon? Or can the time be skewed? Unfortunately I don't have the performance data from the first level VM on which the second level runs. 2) Could anyone recommend a Java profiler for s390x? 3) Has anyone experienced something similar? Thanks, Tomas Tomas Pavelka CA Technologies Sr Software Engineer CA CZ, s.r.o V Parku 12, 148 00 Praha Czech Republic Office: +25996 | tomas.pave...@ca.com Id. Císlo 25694073, z obchodního rejstříku, vedeného Městským soudem v Praze, oddíl C, vložka 61808 / Id. No. 25694073, registered in the Commercial Register maintained by the Municipal Court in Praque, Section C, File 61808 -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/ -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
