On 2/20/2026 1:00 PM, JAEHOON KIM wrote:
On 2/19/2026 4:27 PM, Stefan Hajnoczi wrote:
Hi Jaehoon,
Following the call earlier this week I ran a single fio job to get a
clearer picture of:
1. The QEMU 10.0.0 regression that prompted you to optimize AioContext
polling.
2. How the poll-weight parameter affects IOPS.
run rw bs numjobs iothreads iops diff
v9.2.0 randread 8k 1 1 174944 3.6%
v10.0.0 randread 8k 1 1 174285 3.2%
baseline randread 8k 1 1 168908 0.0%
w2 randread 8k 1 1 163718 -3.1%
w3 randread 8k 1 1 165805 -1.8%
w4 randread 8k 1 1 167388 -0.9%
This time I only ran randread bs=8k iodepth=8 numjobs=1 with a single
IOThread.
Observations:
- There might be an IOPS regression between v10.0.0 and the baseline
(9ad7f544c696) that your patches apply on top of. This is different
from the CPU utilization regression that you found in v9.2.0 ->
v10.0.0. I will bisect it.
- poll-weight=3 and 4 improve IOPS to a level that is acceptable. CPU
utilization looks like this:
run %usr %nice %sys %iowait %steal %irq
%soft %guest %gnice %idle
baseline 49.37 0.00 31.10 0.00 0.00 11.61
0.04 0.00 0.00 7.89
w2 46.24 0.00 32.61 0.00 0.00 11.84
0.10 0.00 0.00 9.21
w3 48.04 0.00 32.17 0.00 0.00 11.98
0.08 0.00 0.00 7.73
w4 48.56 0.00 31.23 0.00 0.00 11.48
0.03 0.00 0.00 8.69
poll-weight=2 is the winner at CPU utilization. I'm not sure if
poll-weight=3 will produce an acceptable CPU utilization improvement for
you. Do you have data or want to re-run to measure poll-weight=3?
Stefan
Thank you very much for sharing the detailed measurement results.
I truly appreciate the effort.
Regarding w=3, I will discuss with our performance team to see
if the CPU consumption levels are acceptable within our internal
test environment. I will get back to you with more definitive data
as soon as possible.
Thanks again for your thorough analysis.
Regards,
Jaehoon.
Hello Stefan,
Thank you for your patience. I would like to share the observed changes
in throughput
and CPU consumption in our performance test environment as follows.
We observed that when using W=3, performance returns to a level
comparable to
QEMU v9.1, while W=2 results in slightly lower CPU consumption.
Our initial preference is to use W=2 by default.
However, we fully understand your concerns regarding the potential
performance drop.
Should the patch be accepted, we would provide guidance on using the
weighted value as
a configurable option.
For reference, we consider values between -2 and 2 as noise in our analysis.
I look forward to your feedback.
The table below shows a comparison between:
* - Host:* RHEL10.1-GA+qemu-10.0.0-14.el10_1, *Guest:* RHEL 9.6 GA vs.
* - Host:* RHEL10.1-GA+qemu-10.0.0-14.el10_1 (w=2, w=3), *Guest:* RHEL
9.6 GA
for FIO FCP and FICON with 1 iothread and 8 iothreads.
The values shown are the averages for numjobs 1, 4, and 8.
FIO FCP -1 iothread
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
| | Seq. Read | Seq. Read | Seq. Write | Seq.
Write | Rand. Read | Rand. Read | Rand. Write| Rand. Write|
| | w2 (%) | w3 (%) | w2 (%) | w3 (%)
| w2 (%) | w3 (%) | w2 (%) | w3 (%) |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
| Throughput avg | -3.00 | -2.33 | 0.00 | -0.33
| -3.00 | -3.33 | 1.33 | -0.33 |
| CPU consumption avg | -5.67 | -4.33 | -6.33 | -5.33
| -7.33 | -5.33 | -10.33 | -8.67 |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
FIO FCP -8 iothread
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
| | Seq. Read | Seq. Read | Seq. Write | Seq.
Write | Rand. Read | Rand. Read | Rand. Write| Rand. Write|
| | w2 (%) | w3 (%) | w2 (%) | w3 (%)
| w2 (%) | w3 (%) | w2 (%) | w3 (%) |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
| Throughput avg | -5.00 | -4.00 | -3.67 | -3.33
| -4.67 | -4.00 | -0.33 | -0.67 |
| CPU consumption avg | -13.00 | -10.67 | -16.00 | -14.33
| -14.00 | -9.33 | -13.67 | -11.00 |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
FIO FICON -1 iothread
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
| | Seq. Read | Seq. Read | Seq. Write | Seq.
Write | Rand. Read | Rand. Read | Rand. Write| Rand. Write|
| | w2 (%) | w3 (%) | w2 (%) | w3 (%)
| w2 (%) | w3 (%) | w2 (%) | w3 (%) |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
| Throughput avg | -0.67 | -0.67 | -6.33 | -6.67
| -0.67 | 0.00 | 1.33 | 1.33 |
| CPU consumption avg | -7.67 | -7.33 | -13.67 | -13.00
| -9.00 | -8.33 | -5.00 | -4.33 |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
FIO FICON -8 iothread
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
| | Seq. Read | Seq. Read | Seq. Write | Seq.
Write | Rand. Read | Rand. Read | Rand. Write| Rand. Write|
| | w2 (%) | w3 (%) | w2 (%) | w3 (%)
| w2 (%) | w3 (%) | w2 (%) | w3 (%) |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
| Throughput avg | -3.00 | -2.67 | -7.33 | -7.00
| -0.67 | -1.00 | 0.67 | 0.67 |
| CPU consumption avg | -16.33 | -14.33 | -25.33 | -27.00
| -8.67 | -7.00 | -6.67 | -5.00 |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
The table below shows a comparison between:
* - Host:* RHEL 10.0 GA + qemu-9.1.0-15.el10, *Guest:* RHEL 9.6 GA vs.
* - Host:* RHEL 10.1 GA + qemu-10.0.0-14.el10_1 (w=2, w=3), *Guest:*
RHEL 9.6 GA.
FIO FCP -1 iothread
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
| | Seq. Read | Seq. Read | Seq. Write | Seq.
Write | Rand. Read | Rand. Read | Rand. Write| Rand. Write|
| | w2 (%) | w3 (%) | w2 (%) | w3 (%)
| w2 (%) | w3 (%) | w2 (%) | w3 (%) |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
| Throughput avg | -0.67 | 0.00 | -0.33 | -1.00
| -1.00 | -0.67 | 3.33 | 2.00 |
| CPU consumption avg | 0.67 | 2.00 | 1.67 | 3.00
| -3.33 | -0.33 | -2.33 | 0.00 |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
FIO FCP -8 iothread
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
| | Seq. Read | Seq. Read | Seq. Write | Seq.
Write | Rand. Read | Rand. Read | Rand. Write| Rand. Write|
| | w2 (%) | w3 (%) | w2 (%) | w3 (%)
| w2 (%) | w3 (%) | w2 (%) | w3 (%) |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
| Throughput avg | -2.00 | -1.33 | -1.00 | 0.00
| -0.33 | 1.00 | 1.00 | 0.67 |
| CPU consumption avg | -3.00 | -1.00 | -2.00 | 0.00
| -10.33 | -5.67 | -6.67 | -3.33 |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
FIO FICON -1 iothread
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
| | Seq. Read | Seq. Read | Seq. Write | Seq.
Write | Rand. Read | Rand. Read | Rand. Write| Rand. Write|
| | w2 (%) | w3 (%) | w2 (%) | w3 (%)
| w2 (%) | w3 (%) | w2 (%) | w3 (%) |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
| Throughput avg | -1.67 | -1.67 | -0.33 | 0.00
| -1.33 | -1.00 | -1.67 | -2.33 |
| CPU consumption avg | -0.33 | 1.00 | 1.00 | 2.00
| -2.33 | -1.33 | -1.33 | -0.33 |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
FIO FICON -8 iothread
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
| | Seq. Read | Seq. Read | Seq. Write | Seq.
Write | Rand. Read | Rand. Read | Rand. Write| Rand. Write|
| | w2 (%) | w3 (%) | w2 (%) | w3 (%)
| w2 (%) | w3 (%) | w2 (%) | w3 (%) |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
| Throughput avg | -1.00 | -0.67 | -0.33 | 0.33
| 0.67 | 0.33 | 0.33 | 0.33 |
| CPU consumption avg | -1.33 | 1.00 | 4.33 | 2.33
| -2.00 | 0.00 | -3.00 | -1.33 |
+---------------------+------------+------------+------------+------------+------------+------------+------------+------------+
Regards,
Jaehoon.
