Re: [Qemu-devel] Reporting Heisenbugs in qemu
On Wed, May 8, 2013 at 10:18 AM, Paolo Bonzini pbonz...@redhat.com wrote: Paolo Bonzini pbonz...@redhat.com writes: I guess that's the register windows. There's only so much you can do to optimize them, and heavily recursive workloads (like Perl, or the RTL half of GCC) pay a hefty price. Two qemu targets stand out for slowness, sparc (32 and 64) and mips (64, don't know about 32). x86 (32 and 64), arm, and ppc run with a slowdown of 30 for my bogus benchmark of GMP configure+make. With FreeBSD x86_64 I see a slowdown of just 13. (My reference system runs FreeBSD, so running FreeBSD under qemu is only far.) My claimed slowdown factors are affected by kernel, libraries, and unfortunately very much by gcc speed, which vary with target. If the sparc emulation speed is due to register windows, then why does mips seem just as slow? No idea. :) Of all the architectures you listed, MIPS is really the one that I have no inkling of... If register windows shortage is a problem, it should be easy to pretend to have lots of them, right? That can help to avoid trapping to the kernel. But you still have to spill the whole window to the stack on every call to a non-leaf function, which can be expensive even if you do it in the translated code. The kernel wouldn't be able to spill or fill more than 8 windows anyway. But maybe we could implement window cleaning in QEMU, if nobody minds not seeing any CLEANWIN traps. I don't think real HW can clean the windows. Register window access in QEMU could be improved slightly, for example it may be possible to handle wrapping of the registers with sparse layout and page mapping tricks. Also the translator might be able to keep track of current CWP and generate direct accesses instead of going through regwptr. Paolo
Re: [Qemu-devel] Reporting Heisenbugs in qemu
Il 08/05/2013 00:57, Aurelien Jarno ha scritto:
On Tue, May 07, 2013 at 11:29:20PM +0200, Artyom Tarasenko wrote:
On Tue, May 7, 2013 at 1:38 PM, Torbjorn Granlund t...@gmplib.org wrote:
The 2nd table of http://gmplib.org/devel/testsystems.html shows all
emulated systems I am using, most of which are qemu-based.
Do I read it correct that qemu-system-ppc64 with the slowdown factor
of 33 is ~3 times faster than qemu-system-sparc64 with the slowdown
factor of 96 ?
Do they both use Debian Wheezy guest? You have a remark that ppc64 has
problems with its clock. Was it taken into account when the slowdown
factors were calculated?
Clock or not, it should be noted that qemu-system-sparc64 is undoubtedly
slower (at least 5 to 10 times) than qemu-system-{arm,ppc,mips,...} on
some type of load like perl scripts.
I guess that's the register windows. There's only so much you can do to
optimize them, and heavily recursive workloads (like Perl, or the RTL
half of GCC) pay a hefty price.
Paolo
Re: [Qemu-devel] Reporting Heisenbugs in qemu
Paolo Bonzini pbonz...@redhat.com writes: I guess that's the register windows. There's only so much you can do to optimize them, and heavily recursive workloads (like Perl, or the RTL half of GCC) pay a hefty price. Two qemu targets stand out for slowness, sparc (32 and 64) and mips (64, don't know about 32). x86 (32 and 64), arm, and ppc run with a slowdown of 30 for my bogus benchmark of GMP configure+make. With FreeBSD x86_64 I see a slowdown of just 13. (My reference system runs FreeBSD, so running FreeBSD under qemu is only far.) My claimed slowdown factors are affected by kernel, libraries, and unfortunately very much by gcc speed, which vary with target. If the sparc emulation speed is due to register windows, then why does mips seem just as slow? If register windows shortage is a problem, it should be easy to pretend to have lots of them, right? -- Torbjörn
Re: [Qemu-devel] Reporting Heisenbugs in qemu
Paolo Bonzini pbonz...@redhat.com writes: I guess that's the register windows. There's only so much you can do to optimize them, and heavily recursive workloads (like Perl, or the RTL half of GCC) pay a hefty price. Two qemu targets stand out for slowness, sparc (32 and 64) and mips (64, don't know about 32). x86 (32 and 64), arm, and ppc run with a slowdown of 30 for my bogus benchmark of GMP configure+make. With FreeBSD x86_64 I see a slowdown of just 13. (My reference system runs FreeBSD, so running FreeBSD under qemu is only far.) My claimed slowdown factors are affected by kernel, libraries, and unfortunately very much by gcc speed, which vary with target. If the sparc emulation speed is due to register windows, then why does mips seem just as slow? No idea. :) Of all the architectures you listed, MIPS is really the one that I have no inkling of... If register windows shortage is a problem, it should be easy to pretend to have lots of them, right? That can help to avoid trapping to the kernel. But you still have to spill the whole window to the stack on every call to a non-leaf function, which can be expensive even if you do it in the translated code. Paolo
Re: [Qemu-devel] Reporting Heisenbugs in qemu
On Wed, May 08, 2013 at 11:45:45AM +0200, Torbjorn Granlund wrote: Paolo Bonzini pbonz...@redhat.com writes: I guess that's the register windows. There's only so much you can do to optimize them, and heavily recursive workloads (like Perl, or the RTL half of GCC) pay a hefty price. Two qemu targets stand out for slowness, sparc (32 and 64) and mips (64, don't know about 32). x86 (32 and 64), arm, and ppc run with a slowdown of 30 for my bogus benchmark of GMP configure+make. With FreeBSD x86_64 I see a slowdown of just 13. (My reference system runs FreeBSD, so running FreeBSD under qemu is only far.) My claimed slowdown factors are affected by kernel, libraries, and unfortunately very much by gcc speed, which vary with target. If the sparc emulation speed is due to register windows, then why does mips seem just as slow? I guess it's related to your benchmark, as it is not something really observed (though not scientifically) on other loads. How much memory do you need for it, couldn't it be that building GMP needs more than 256MB of RAM at some point and that the system starts swapping? -- Aurelien Jarno GPG: 1024D/F1BCDB73 aurel...@aurel32.net http://www.aurel32.net
Re: [Qemu-devel] Reporting Heisenbugs in qemu
On 05/08/2013 04:45:45 AM, Torbjorn Granlund wrote: Paolo Bonzini pbonz...@redhat.com writes: I guess that's the register windows. There's only so much you can do to optimize them, and heavily recursive workloads (like Perl, or the RTL half of GCC) pay a hefty price. Two qemu targets stand out for slowness, sparc (32 and 64) and mips (64, don't know about 32). x86 (32 and 64), arm, and ppc run with a slowdown of 30 for my bogus benchmark of GMP configure+make. With FreeBSD x86_64 I see a slowdown of just 13. (My reference system runs FreeBSD, so running FreeBSD under qemu is only far.) My claimed slowdown factors are affected by kernel, libraries, and unfortunately very much by gcc speed, which vary with target. If the sparc emulation speed is due to register windows, then why does mips seem just as slow? If register windows shortage is a problem, it should be easy to pretend to have lots of them, right? sh4 is pretty slow too. Unfortunately: http://landley.net/aboriginal/bin/system-image-sh4.tar.bz2 Only has 64 megs of memory in the emulated board. (Enough to build hello world, not enough to build most packages.) I have a vague todo item to add a command line thing to qemu to plug a physical memory address range into an aribtrary address and then tell linux discontigmem add memory range HERE on the command line. That way I wouldn't have to hack up each board emulation to get more memory...) Rob
Re: [Qemu-devel] Reporting Heisenbugs in qemu
On 07/05/13 12:38, Torbjorn Granlund wrote: Hi Torbjorn, I am trying to use qemu to 1. cover more of the assembly code in GMP 2. check configuration logic of GMP but I am not as successful as I would like to be. The 2nd table of http://gmplib.org/devel/testsystems.html shows all emulated systems I am using, most of which are qemu-based. Wow - that's a really impressive setup :) Unfortunately, several of the qemu-based systems experience intermittent but common segfaults: 1. Linux mips64eb 2.6.32-5-5kc-malta #1 Sun Sep 23 12:29:36 UTC 2012 mips64 GNU/Linux 2. Linux mips64el 2.6.32-5-5kc-malta #1 Fri Feb 15 21:38:11 UTC 2013 mips64 GNU/Linux 3. Linux kick.gmplib.org 2.6.18-6-sparc32 #1 Sat Dec 27 09:13:12 UTC 2008 sparc GNU/Linux An example of a failure is: gmp/tests/cxx/t-ops2.cc: In function 'void checkz()': gmp/tests/cxx/t-ops2.cc:86: internal compiler error: Segmentation fault Please submit a full bug report, with preprocessed source if appropriate. SeeURL:http://gcc.gnu.org/bugs.html for instructions. For Debian GNU/Linux specific bug reporting instructions, seeURL:file:///usr/share/doc/gcc-4.1/README.Bugs. The bug is not reproducible, so it is likely a hardware or OS problem. (This was from the sparc32 system.) rootrem.c: In function 'mpn_rootrem_internal': rootrem.c:120:1: internal compiler error: Segmentation fault Please submit a full bug report, with preprocessed source if appropriate. Seefile:///usr/share/doc/gcc-4.6/README.Bugs for instructions. The bug is not reproducible, so it is likely a hardware or OS problem. (From the mips64eb system.) I am aware of that these systems don't exactly use the kernel-of-the-week. Newer kernels I have tried cause non-boot. (I don't think I've tried any newer sparc kernel, as building that would require a stable sparc system...) I realise that linux might have been debugged until it works on real hardware, but that qemu might trigger untested linux execution paths. Yesterday, I disabled GMP testing on these qemu systems, as I got tired of the many false alarms, and since GMP looked bad. Is there any hope that these qemu systems will become stable? Or aren't these problems qemu's fault? I actually spent quite a bit of time a couple of years ago trying to troubleshoot a similar intermittent buildfarm compiler blowout in a Xen x86 guest, and in the end it turned out to be simply a lack of memory (both physical and virtual) assigned to the VM. It just so happened that the code in question was tickling a gcc bug which caused excess memory usage during compilation, but only when compiling with a certain combination of flags. This may not necessarily be the problem here, but it's certainly worth a little experimentation first before looking deeper into QEMU. HTH, Mark.
Re: [Qemu-devel] Reporting Heisenbugs in qemu
On Tue, May 07, 2013 at 01:38:57PM +0200, Torbjorn Granlund wrote: I am trying to use qemu to 1. cover more of the assembly code in GMP 2. check configuration logic of GMP but I am not as successful as I would like to be. The 2nd table of http://gmplib.org/devel/testsystems.html shows all emulated systems I am using, most of which are qemu-based. Unfortunately, several of the qemu-based systems experience intermittent but common segfaults: 1. Linux mips64eb 2.6.32-5-5kc-malta #1 Sun Sep 23 12:29:36 UTC 2012 mips64 GNU/Linux 2. Linux mips64el 2.6.32-5-5kc-malta #1 Fri Feb 15 21:38:11 UTC 2013 mips64 GNU/Linux 3. Linux kick.gmplib.org 2.6.18-6-sparc32 #1 Sat Dec 27 09:13:12 UTC 2008 sparc GNU/Linux An example of a failure is: gmp/tests/cxx/t-ops2.cc: In function 'void checkz()': gmp/tests/cxx/t-ops2.cc:86: internal compiler error: Segmentation fault Please submit a full bug report, with preprocessed source if appropriate. See URL:http://gcc.gnu.org/bugs.html for instructions. For Debian GNU/Linux specific bug reporting instructions, see URL:file:///usr/share/doc/gcc-4.1/README.Bugs. The bug is not reproducible, so it is likely a hardware or OS problem. (This was from the sparc32 system.) rootrem.c: In function 'mpn_rootrem_internal': rootrem.c:120:1: internal compiler error: Segmentation fault Please submit a full bug report, with preprocessed source if appropriate. See file:///usr/share/doc/gcc-4.6/README.Bugs for instructions. The bug is not reproducible, so it is likely a hardware or OS problem. (From the mips64eb system.) I am aware of that these systems don't exactly use the kernel-of-the-week. Newer kernels I have tried cause non-boot. (I don't think I've tried any newer sparc kernel, as building that would require a stable sparc system...) I realise that linux might have been debugged until it works on real hardware, but that qemu might trigger untested linux execution paths. Yesterday, I disabled GMP testing on these qemu systems, as I got tired of the many false alarms, and since GMP looked bad. Is there any hope that these qemu systems will become stable? Or aren't these problems qemu's fault? Which version of QEMU are you using? This looks like a bug that has appeared in version 1.3.0 and fixed in version 1.4.1. The corresponding commit in master is 52ae646d4a3ebdcdcc973492c6a56f2c49b6578f -- Aurelien Jarno GPG: 1024D/F1BCDB73 aurel...@aurel32.net http://www.aurel32.net
Re: [Qemu-devel] Reporting Heisenbugs in qemu
Aurelien Jarno aurel...@aurel32.net writes:
Which version of QEMU are you using? This looks like a bug that has
appeared in version 1.3.0 and fixed in version 1.4.1. The corresponding
commit in master is 52ae646d4a3ebdcdcc973492c6a56f2c49b6578f
I cannot tell exactly, since I didn't keep sources to each qemu compile.
The compile for mips64e{l,b} is from 2013-03-06, the sources are not
much older.
Older qemu versions determinisitcally crashed for GMP's make check
target for both mips64 targets. (I cannot recall exactly how qemu
crashed, I think it might have been an abort trap.)
The compile for sparc32 is from 2013-03-19.
I now updated these systems to a qemu build, and will re-enable
testing. (Results tomorrow.)
--
Torbjörn
Re: [Qemu-devel] Reporting Heisenbugs in qemu
On Tue, May 7, 2013 at 1:38 PM, Torbjorn Granlund t...@gmplib.org wrote: The 2nd table of http://gmplib.org/devel/testsystems.html shows all emulated systems I am using, most of which are qemu-based. Do I read it correct that qemu-system-ppc64 with the slowdown factor of 33 is ~3 times faster than qemu-system-sparc64 with the slowdown factor of 96 ? Do they both use Debian Wheezy guest? You have a remark that ppc64 has problems with its clock. Was it taken into account when the slowdown factors were calculated? Artyom -- Regards, Artyom Tarasenko linux/sparc and solaris/sparc under qemu blog: http://tyom.blogspot.com/search/label/qemu
Re: [Qemu-devel] Reporting Heisenbugs in qemu
Artyom Tarasenko atar4q...@gmail.com writes: Do I read it correct that qemu-system-ppc64 with the slowdown factor of 33 is ~3 times faster than qemu-system-sparc64 with the slowdown factor of 96 ? You read it correctly. But please see the caveat at the table end. Do they both use Debian Wheezy guest? You have a remark that ppc64 has problems with its clock. Was it taken into account when the slowdown factors were calculated? The time of a job is computed as the difference between a time stamp made by the test system and the report time. The first time is affected by ppc64's slow clock, the latter is correct. A cron job sets the clock every 30 minutes, so the ppc time might be *overestimated* 30 minutes. -- Torbjörn
Re: [Qemu-devel] Reporting Heisenbugs in qemu
On Tue, May 7, 2013 at 11:43 PM, Torbjorn Granlund t...@gmplib.org wrote: Artyom Tarasenko atar4q...@gmail.com writes: Do I read it correct that qemu-system-ppc64 with the slowdown factor of 33 is ~3 times faster than qemu-system-sparc64 with the slowdown factor of 96 ? You read it correctly. But please see the caveat at the table end. Do they both use Debian Wheezy guest? You have a remark that ppc64 has problems with its clock. Was it taken into account when the slowdown factors were calculated? The time of a job is computed as the difference between a time stamp made by the test system and the report time. The first time is affected by ppc64's slow clock, the latter is correct. A cron job sets the clock every 30 minutes, so the ppc time might be *overestimated* 30 minutes. Meaning it's even faster than estimated? That's fast. I don't see in the table, how long does the build take? -- Regards, Artyom Tarasenko linux/sparc and solaris/sparc under qemu blog: http://tyom.blogspot.com/search/label/qemu
Re: [Qemu-devel] Reporting Heisenbugs in qemu
On Tue, May 07, 2013 at 11:29:20PM +0200, Artyom Tarasenko wrote:
On Tue, May 7, 2013 at 1:38 PM, Torbjorn Granlund t...@gmplib.org wrote:
The 2nd table of http://gmplib.org/devel/testsystems.html shows all
emulated systems I am using, most of which are qemu-based.
Do I read it correct that qemu-system-ppc64 with the slowdown factor
of 33 is ~3 times faster than qemu-system-sparc64 with the slowdown
factor of 96 ?
Do they both use Debian Wheezy guest? You have a remark that ppc64 has
problems with its clock. Was it taken into account when the slowdown
factors were calculated?
Clock or not, it should be noted that qemu-system-sparc64 is undoubtedly
slower (at least 5 to 10 times) than qemu-system-{arm,ppc,mips,...} on
some type of load like perl scripts.
--
Aurelien Jarno GPG: 1024D/F1BCDB73
aurel...@aurel32.net http://www.aurel32.net
Re: [Qemu-devel] Reporting Heisenbugs in qemu
Artyom Tarasenko atar4q...@gmail.com writes: Meaning it's even faster than estimated? That's fast. I don't see in the table, how long does the build take? About one hour. That's for: configure + make + make check TESTS= (The host is a 4X 3.6 GHz Bulldozer Zambezi.) -- Torbjörn
