Re: Java performance under a second level VM

[Pavelka, Tomas]

Hi Rob, 
thanks for the reply. The performance of Linux under a second level VM always 
seemed unpredictable to me but after reading your response I started seeing a 
pattern: whenever I had a poorly performing application on Linux on a second 
level VM, there was always some kind of networking involved which in turn 
involved waiting on open sockets.
I just want to ensure I am understanding correctly what your are saying: Is it 
true that waits under software SIE burn CPU where the same waits under hardware 
SIE do not? That would explain a lot of the performance problems I have seen 
but I am sure there are caveats.

Thanks,
Tomas

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/

Re: Java performance under a second level VM

[Rob van der Heij]

Yes. Hardware virtualization  support (aka SIE) is only two levels: LPAR
and z/VM. Linux running its processes is one too many so must be done in
software. That creates overhead. In your case is likely the polling of the
JVM that does it, which means is not really related to workload but just
burns cycles while idle. CMS does still fine that way but Linux not.
The virtual time is while CP has the guest dispatched under SIE. When the
user code in SIE is slow due to exceptions that CP must handle the clock is
still ticking. Just like when the guest suffers from heavy cache miss.
The IBM support assistant  (or what it's called now) did pretty nice
profile the IBM JVM. There are also lots of GC  analyzers if that's where
you're looking.
Rob
On Apr 16, 2015 3:58 PM, "Pavelka, Tomas"  wrote:

> I have a Java application running a REST API built on top of the
> Dropwizard stack (www.dropwizard.io). I have
> been running it on a first level z/VM and it was always well behaved
> consuming less than 1% of a CPU on an EC12 during normal operation. I
> needed to do some testing which I need to run on a second level z/VM. I
> have experienced performance that pretty much rendered the application
> unusable. To get a better understanding of the difference in performance I
> tried to measure the initialization time of the application. This I have
> done with CP INDICATE USER EXPANDED, measuring Ttime and Vtime, subtracting
> the values at the beginning from values at the end of the measured interval.
> For the first level z/VM the initialization took 5 seconds for both Ttime
> and Vtime. For the second level system Ttime was  0:41:12 and Vtime was
> 0:41:11, i.e. the initialization on the second level took a little less
> than 500 times as much as on the first level.
>
> We have another group testing software on that system written in Perl and
> they have not complained about performance (unfortunately that is the
> closest I have to performance data for the Perl app). The Linux machines on
> the second level do not feel sluggish (when doing things like text editing
> directory browsing and copying files, all under an SSH session). This makes
> me believe that the problem may be Java specific. I also tried running
> other Java applications, the response was pretty bad especially if the
> applications did console or file IO (but I don't have any measurements). I
> tried this under IBM JRE 7.1 and on 8.0 without noticeable change in
> performance.
>
> I have a few questions:
>
> 1) Can the total and virtual time on a second level system be relied upon?
> Or can the time be skewed? Unfortunately I don't have the performance data
> from the first level VM on which the second level runs.
> 2) Could anyone recommend a Java profiler for s390x?
> 3) Has anyone experienced something similar?
>
> Thanks,
> Tomas
>
>
> Tomas Pavelka
> CA Technologies
> Sr Software Engineer
>
> CA CZ, s.r.o
> V Parku 12,
> 148 00 Praha
> Czech Republic
>
> Office: +25996 | tomas.pave...@ca.com
>
> Id. Císlo 25694073, z obchodního rejstříku, vedeného Městským soudem v
> Praze, oddíl C, vložka 61808 / Id. No. 25694073, registered in the
> Commercial Register maintained by the Municipal Court in Praque, Section C,
> File 61808
>
> --
> For LINUX-390 subscribe / signoff / archive access instructions,
> send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or
> visit
> http://www.marist.edu/htbin/wlvindex?LINUX-390
> --
> For more information on Linux on System z, visit
> http://wiki.linuxvm.org/
>

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/

Java performance under a second level VM

[Pavelka, Tomas]

I have a Java application running a REST API built on top of the Dropwizard 
stack (www.dropwizard.io). I have been running it on 
a first level z/VM and it was always well behaved consuming less than 1% of a 
CPU on an EC12 during normal operation. I needed to do some testing which I 
need to run on a second level z/VM. I have experienced performance that pretty 
much rendered the application unusable. To get a better understanding of the 
difference in performance I tried to measure the initialization time of the 
application. This I have done with CP INDICATE USER EXPANDED, measuring Ttime 
and Vtime, subtracting the values at the beginning from values at the end of 
the measured interval.
For the first level z/VM the initialization took 5 seconds for both Ttime and 
Vtime. For the second level system Ttime was  0:41:12 and Vtime was 0:41:11, 
i.e. the initialization on the second level took a little less than 500 times 
as much as on the first level.

We have another group testing software on that system written in Perl and they 
have not complained about performance (unfortunately that is the closest I have 
to performance data for the Perl app). The Linux machines on the second level 
do not feel sluggish (when doing things like text editing directory browsing 
and copying files, all under an SSH session). This makes me believe that the 
problem may be Java specific. I also tried running other Java applications, the 
response was pretty bad especially if the applications did console or file IO 
(but I don't have any measurements). I tried this under IBM JRE 7.1 and on 8.0 
without noticeable change in performance.

I have a few questions:

1) Can the total and virtual time on a second level system be relied upon? Or 
can the time be skewed? Unfortunately I don't have the performance data from 
the first level VM on which the second level runs.
2) Could anyone recommend a Java profiler for s390x?
3) Has anyone experienced something similar?

Thanks,
Tomas


Tomas Pavelka
CA Technologies
Sr Software Engineer

CA CZ, s.r.o 
V Parku 12, 
148 00 Praha 
Czech Republic

Office: +25996 | tomas.pave...@ca.com

Id. Císlo 25694073, z obchodního rejstříku, vedeného Městským soudem v Praze, 
oddíl C, vložka 61808 / Id. No. 25694073, registered in the Commercial Register 
maintained by the Municipal Court in Praque, Section C, File 61808

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/