[Bug 1880943] Re: [focal] disk I/O performance regression

2020-07-03 Thread Kai-Heng Feng
If ext4lazyinit isn't in used, do we still see the performance regression? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1880943 Title: [focal] disk I/O performance regression To manage

[Bug 1880943] Re: [focal] disk I/O performance regression

2020-05-29 Thread Frode Nordahl
Playing around with the now available IO schedulers does not appear to help much and looking at iotop while the test runs does show ext4lazyinit consuming much of the IO most of the time while the test runs. I wonder if this is a more ominous change of behavior after all Total DISK READ:

[Bug 1880943] Re: [focal] disk I/O performance regression

2020-05-29 Thread Frode Nordahl
We have been using the ext4lazyinit for quite some time, so I guess this must be a combination of multiple things. I see that the 5.4 kernel brings a change of io scheduler to mq- deadline. Could the combination of ext4lazyinit+mq-deadline+rotational drives be a problem? -- You received this

[Bug 1880943] Re: [focal] disk I/O performance regression

2020-05-29 Thread Frode Nordahl
For any future travelers this issue was caused by the much debated ext4lazyinit "feature". # iotop Total DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s Current DISK READ: 0.00 B/s | Current DISK WRITE: 7.38 M/s TID PRIO USER DISK READ DISK WRITE SWAPIN

[Bug 1880943] Re: [focal] disk I/O performance regression

2020-05-29 Thread Frode Nordahl
That is an excellent idea. Before embarking on that endeavor I did a last control test which involved deploying Bionic and then installing the Focal 5.4 kernel packages, and lo and behold the system is still performant. Redeploying Focal (with the Focal kernel obviously) makes it non- performant

[Bug 1880943] Re: [focal] disk I/O performance regression

2020-05-28 Thread Kai-Heng Feng
Would it be possible for you to do a kernel bisection? First, find the last -rc kernel works and the first -rc kernel doesn’t work from http://kernel.ubuntu.com/~kernel-ppa/mainline/ Then, $ sudo apt build-dep linux $ git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git $

[Bug 1880943] Re: [focal] disk I/O performance regression

2020-05-28 Thread Frode Nordahl
Thank you for the suggestion to test mainline. Regarding the kernel parameters: The test is run on the host and not in a virtual machine, for completeness I have removed the iommu and vf related kernel parameters and they do not affect the outcome of the test. Unfortunately using the mainline

[Bug 1880943] Re: [focal] disk I/O performance regression

2020-05-27 Thread Kai-Heng Feng
Note that I didn't add any kernel parameter, so my first guess would be "iommu=pt" doesn't get passthrough right. ** Changed in: linux (Ubuntu) Status: Confirmed => Incomplete -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

[Bug 1880943] Re: [focal] disk I/O performance regression

2020-05-27 Thread Kai-Heng Feng
Tried 4.15.0-102, 5.4.0-33 and mailing kernel all three show similar performance (~2100MB/s). Please test latest mainline kernel: https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.7-rc7/ And decide what's the next step based on the performance of mainline kernel. -- You received this bug

[Bug 1880943] Re: [focal] disk I/O performance regression

2020-05-27 Thread Frode Nordahl
Note that to rule out any hardware configuration/malfunction issue between the two hosts I have re-run the tests on the exact same machine used for the non-performant Focal tests with good performance for Bionic. -- You received this bug notification because you are a member of Ubuntu Bugs,

[Bug 1880943] Re: [focal] disk I/O performance regression

2020-05-27 Thread Ryan Beisner
** Tags added: uosci -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1880943 Title: [focal] disk I/O performance regression To manage notifications about this bug go to: