Hi, Since qemu 5.0 I found that Ubuntu Test environments crashed often. After a while I found that all affected tests run qemu TCG and they get into OOM conditions that kills the qemu process.
Steps to reproduce: Run TCG on a guest image until boot settles: $ wget http://cloud-images.ubuntu.com/daily/server/groovy/20200714/groovy-server-cloudimg-amd64.img $ qemu-system-x86_64 -name guest=groovy-testguest,debug-threads=on -machine pc-q35-focal,accel=tcg,usb=off,dump-guest-core=off -cpu qemu64 -m 512 -overcommit mem-lock=off -smp 1,sockets=1,cores=1,threads=1 -hda groovy-server-cloudimg-amd64.img -nographic -serial file:/tmp/serial.debug I usually wait until I see no faults anymore in pidstat (indicates bootup is complete). Also at that point the worker threads vanish or at least reduce significantly. Then I checked RSS sizes: $ pidstat -p $(pgrep -f 'name guest=groovy-testguest') -T ALL -rt 5 To know the expected deviations I ran it a few times with old/new version qemu 4.2 qemu 5.0 VSZ RSS VSZ RSS 1735828 642172 2322668 1641532 1424596 602368 2374068 1628788 1570060 611372 2789648 1676748 1556696 611240 2981112 1658196 1388844 649696 2443716 1636896 1597788 644584 2989336 1635516 That is ~+160% I was wondering if that might be the new toolchain or due to features instead of TCG (even though all non TCG tests showed no issue). I ran the same with -enable-kvm which shows no difference worth to report: accel=kvm Old qemu: accel=kvm New qemu: VSZ RSS VSZ RSS 1844232 489224 1195880 447696 1068784 448324 1330036 484464 1583020 448708 1380408 468588 1247244 493980 1244148 493188 1702912 483444 1247672 454444 1287980 448480 1983548 501184 So it seems to come down to "4.2 TCG vs 5.0 TCG" Therefore I have spun up a 4.2 and a 5.0 qemu with TCG showing this ~+160% increased memory consumption. Using smem I then check where the consumption was per mapping: # smem --reverse --mappings --abbreviate --processfilter=qemu-system-x86_64 | head -n 10 qemu 4.2 qemu 5.0 Map AVGPSS PSS AVGPSS PSS <anonymous> 289.5M 579.0M 811.5M 1.6G qemu-system-x86_64 9.1M 9.1M 9.2M 9.2M [heap] 2.8M 5.6M 3.4M 6.8M /usr/bin/python3.8 1.8M 1.8M 1.8M 1.8M /libepoxy.so.0. 448.0K 448.0K 448.0K 448.0K /libcrypto.so.1 296.0K 296.0K 275.0K 275.0K /libgnutls.so.3 234.0K 234.0K 230.0K 230.0K /libasound.so.2 208.0K 208.0K 208.0K 208.0K /libssl.so.1.1 180.0K 180.0K 92.0K 184.0K So all the increase is in anon memory of qemu. Since it is TCG I ran `info jit` in the Monitor qemu 4.2: (qemu) info jit Translation buffer state: gen code size 99.781.715/134.212.563 TB count 183622 TB avg target size 18 max=1992 bytes TB avg host size 303 bytes (expansion ratio: 16.4) cross page TB count 797 (0%) direct jump count 127941 (69%) (2 jumps=91451 49%) TB hash buckets 98697/131072 (75.30% head buckets used) TB hash occupancy 34.04% avg chain occ. Histogram: [0,10)%|▆ █ ▅▁▃▁▁|[90,100]% TB hash avg chain 1.020 buckets. Histogram: 1|█▁▁|3 Statistics: TB flush count 14 TB invalidate count 92226 TLB full flushes 1 TLB partial flushes 175405 TLB elided flushes 233747 [TCG profiler not compiled] qemu 5.0: (qemu) info jit Translation buffer state: gen code size 259.896.403/1.073.736.659 TB count 456365 TB avg target size 20 max=1992 bytes TB avg host size 328 bytes (expansion ratio: 16.1) cross page TB count 2020 (0%) direct jump count 309815 (67%) (2 jumps=227122 49%) TB hash buckets 216220/262144 (82.48% head buckets used) TB hash occupancy 41.36% avg chain occ. Histogram: [0,10)%|▅ █ ▇▁▄▁▂|[90,100]% TB hash avg chain 1.039 buckets. Histogram: 1|█▁▁|3 Statistics: TB flush count 1 TB invalidate count 463653 TLB full flushes 0 TLB partial flushes 178464 TLB elided flushes 242382 [TCG profiler not compiled] Well I see the numbers increase, but this isn't my home turf anymore. The one related tunabel I know is -tb-size I ran both versions with -tb-size 150 And the result gave me two similarly working qemu processes. RSS qemu 4.2: 628072 635528 qemu 5.0: 655628 634952 Seems to be "good again" with that tunable set. It seems the TB default sizing, self size reduction or something like it cache has changed. On a system with ~1.5G for example (which matches our testbeds) I'd expect it to back down a bit before being OOM Killed consuming almost 100% of the memory. My next step is to build qemu from source without an Ubuntu downstream delta. That should help to further track it down and also provide some results of the soon to be released 5.1. That will probably take until tomorrow, I'll report here again then. I searched the mailing list and the web for this behavior, but either I use the wrong keywords or it wasn't reported/discussed yet. Nor does [1] list anything that sounds related But if this already rings a bell for someone please let me know. Thanks in advance! [1]: https://wiki.qemu.org/ChangeLog/5.0#TCG -- Christian Ehrhardt Staff Engineer, Ubuntu Server Canonical Ltd
