** Description changed:
[Impact]
Commit f382fb0bcef4 ("block: remove legacy IO schedulers") introduced a
behavior change in the blkio throttle cgroup subsystem: IO statistics
are not reported anymore unless a throttling rule is explicitly defined,
because the current code only counts bios that are actually throttled.
- This behavior change is breaking the user-space (see original bug report
- below).
+ This behavior change is potentially breaking some user-space
+ applications that are relying on the old behavior (see original bug
+ report below).
[Test case]
- - mount cgroup v1
- - create a blkio cgroup
- - move a task into the blkio cgroup
- - perform some I/O (i.e., dd)
- - read the IO stats for the cgroup (blkio.throttle.io_serviced and
blkio.throttle.io_service_bytes in cgroupfs)
- - IO stats are all 0, unless a throttle rule is defined
+ - mount cgroup v1
+ - create a blkio cgroup
+ - move a task into the blkio cgroup
+ - perform some I/O (i.e., dd)
+ - read the IO stats for the cgroup (blkio.throttle.io_serviced and
blkio.throttle.io_service_bytes in cgroupfs)
+ - IO stats are all 0, unless a throttle rule is defined
Previous behavior (kernel 5.15) was showing I/O statistics even without
throttling rules defined.
[Fix]
Apply / backport this fix:
https://lore.kernel.org/lkml/[email protected]/t/
[Regression potential]
The fix is affecting the block IO cgroup subsystem, we may see potential
regressions in this particular cgroup subsystem with this fix applied.
[Original bug report]
Hi,
I'm still investigating but, am a bit stuck. Here's what I've found so
far.
Today I've upgraded some nodes in AWS EC2 from the previous v5.15 linux-
aws package to the recently pusblished v5.19 package and rebooted. It
seems that even when there's disk activity, the files:
/sys/fs/cgroup/blkio/blkio.throttle.io_serviced
/sys/fs/cgroup/blkio/blkio.throttle.io_service_bytes
Are only ever populated with 0's. Prior on v5.15 these would reflect the
actual disk usage. No other system configuration changes were applied
just the kernel upgrade and reboot. I've also verified that simply
rebooting a v5.15 where this does work doesn't break the reporting.
These EC2 instances are running with cgroups v1 due to other
compatability issues and I suspect that might be the issue. So far, I
cannot find any differences. mtab shows the same v1 mount setup, the
kernel options match betwen v5.15 and v5.19.
I'm more than happy to fetch whatever info would help out here. I'd love
to get 5.19 working for us but, we really need the data from these
files.
Info:
Prior version that works: Linux ip-10-128-168-154 5.15.0-1031-aws #35-Ubuntu
SMP Fri Feb 10 02:07:18 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Upgraded version that's broken: Linux ip-10-128-166-219 5.19.0-1022-aws
#23~22.04.1-Ubuntu SMP Fri Mar 17 15:38:24 UTC 2023 x86_64 x86_64 x86_64
GNU/Linux
EC2 instances built off of the published 22.04 LTS AMI in us-east-1.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/2016186
Title:
5.19 not reporting cgroups v1 blkio.throttle.io_serviced
Status in linux package in Ubuntu:
Incomplete
Status in linux source package in Kinetic:
Incomplete
Status in linux source package in Lunar:
Incomplete
Status in linux source package in Mantic:
Incomplete
Bug description:
[Impact]
Commit f382fb0bcef4 ("block: remove legacy IO schedulers") introduced
a behavior change in the blkio throttle cgroup subsystem: IO
statistics are not reported anymore unless a throttling rule is
explicitly defined, because the current code only counts bios that are
actually throttled.
This behavior change is potentially breaking some user-space
applications that are relying on the old behavior (see original bug
report below).
[Test case]
- mount cgroup v1
- create a blkio cgroup
- move a task into the blkio cgroup
- perform some I/O (i.e., dd)
- read the IO stats for the cgroup (blkio.throttle.io_serviced and
blkio.throttle.io_service_bytes in cgroupfs)
- IO stats are all 0, unless a throttle rule is defined
Previous behavior (kernel 5.15) was showing I/O statistics even
without throttling rules defined.
[Fix]
Apply / backport this fix:
https://lore.kernel.org/lkml/[email protected]/t/
[Regression potential]
The fix is affecting the block IO cgroup subsystem, we may see
potential regressions in this particular cgroup subsystem with this
fix applied.
[Original bug report]
Hi,
I'm still investigating but, am a bit stuck. Here's what I've found so
far.
Today I've upgraded some nodes in AWS EC2 from the previous v5.15
linux-aws package to the recently pusblished v5.19 package and
rebooted. It seems that even when there's disk activity, the files:
/sys/fs/cgroup/blkio/blkio.throttle.io_serviced
/sys/fs/cgroup/blkio/blkio.throttle.io_service_bytes
Are only ever populated with 0's. Prior on v5.15 these would reflect
the actual disk usage. No other system configuration changes were
applied just the kernel upgrade and reboot. I've also verified that
simply rebooting a v5.15 where this does work doesn't break the
reporting. These EC2 instances are running with cgroups v1 due to
other compatability issues and I suspect that might be the issue. So
far, I cannot find any differences. mtab shows the same v1 mount
setup, the kernel options match betwen v5.15 and v5.19.
I'm more than happy to fetch whatever info would help out here. I'd
love to get 5.19 working for us but, we really need the data from
these files.
Info:
Prior version that works: Linux ip-10-128-168-154 5.15.0-1031-aws #35-Ubuntu
SMP Fri Feb 10 02:07:18 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Upgraded version that's broken: Linux ip-10-128-166-219 5.19.0-1022-aws
#23~22.04.1-Ubuntu SMP Fri Mar 17 15:38:24 UTC 2023 x86_64 x86_64 x86_64
GNU/Linux
EC2 instances built off of the published 22.04 LTS AMI in us-east-1.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2016186/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : [email protected]
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
