[Kernel-packages] [Bug 2021571] Re: cat /sys/fs/cgroup/blkio/blkio.time_recursive took 1second to complete
It examines the blkio device tree and sums data as far as i know. Removed devices do not get deleted from the blkio statistics tree and are just marked as offline somehow. In our case racadm lclog created and destroyed usb devices to pull data and the blkio tree reached over 65000 leafs. It takes time to process a tree that large. WE believe our issue was caused by accumulation over time of the usb device leafs from racadm lclog. WE had it running in a cron which has now been stopped. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-meta in Ubuntu. https://bugs.launchpad.net/bugs/2021571 Title: cat /sys/fs/cgroup/blkio/blkio.time_recursive took 1second to complete Status in linux-meta package in Ubuntu: Confirmed Bug description: Issue: ```console # time cat /sys/fs/cgroup/blkio/blkio.time_recursive 8:16 354721435 real0m1.297s user0m0.000s sys 0m1.297s ``` As could be seen from above result, cat blkio took about 1 second to complete which is much longer than a normal case. Kernel Version: Linux compute08 4.15.0-72-generic #81-Ubuntu SMP Tue Nov 26 12:20:02 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux Would be appreciated to know any operations to identify the issue, Thank you very much for the help. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-meta/+bug/2021571/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2021571] Re: cat /sys/fs/cgroup/blkio/blkio.time_recursive took 1second to complete
the hardware does not seem to be relevant. Do you have anything calling "racadm lclog" on fairly regular basis? it creates and destroys a usb device over and over. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-meta in Ubuntu. https://bugs.launchpad.net/bugs/2021571 Title: cat /sys/fs/cgroup/blkio/blkio.time_recursive took 1second to complete Status in linux-meta package in Ubuntu: Confirmed Bug description: Issue: ```console # time cat /sys/fs/cgroup/blkio/blkio.time_recursive 8:16 354721435 real0m1.297s user0m0.000s sys 0m1.297s ``` As could be seen from above result, cat blkio took about 1 second to complete which is much longer than a normal case. Kernel Version: Linux compute08 4.15.0-72-generic #81-Ubuntu SMP Tue Nov 26 12:20:02 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux Would be appreciated to know any operations to identify the issue, Thank you very much for the help. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-meta/+bug/2021571/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2021571] Re: cat /sys/fs/cgroup/blkio/blkio.time_recursive took 1second to complete
Do you have a lot of blkio devices coming and going? For example iscsi/usb/etc. We currently believe it's tree bloat due to added/removed devices. We are still digging into the issue. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-meta in Ubuntu. https://bugs.launchpad.net/bugs/2021571 Title: cat /sys/fs/cgroup/blkio/blkio.time_recursive took 1second to complete Status in linux-meta package in Ubuntu: Confirmed Bug description: Issue: ```console # time cat /sys/fs/cgroup/blkio/blkio.time_recursive 8:16 354721435 real0m1.297s user0m0.000s sys 0m1.297s ``` As could be seen from above result, cat blkio took about 1 second to complete which is much longer than a normal case. Kernel Version: Linux compute08 4.15.0-72-generic #81-Ubuntu SMP Tue Nov 26 12:20:02 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux Would be appreciated to know any operations to identify the issue, Thank you very much for the help. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-meta/+bug/2021571/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2021571] Re: cat /sys/fs/cgroup/blkio/blkio.time_recursive took 1second to complete
yes. just confirms it takes a while. ** Attachment added: "flame2.html" https://bugs.launchpad.net/ubuntu/+source/linux-meta/+bug/2021571/+attachment/5700645/+files/flame2.html -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-meta in Ubuntu. https://bugs.launchpad.net/bugs/2021571 Title: cat /sys/fs/cgroup/blkio/blkio.time_recursive took 1second to complete Status in linux-meta package in Ubuntu: Confirmed Bug description: Issue: ```console # time cat /sys/fs/cgroup/blkio/blkio.time_recursive 8:16 354721435 real0m1.297s user0m0.000s sys 0m1.297s ``` As could be seen from above result, cat blkio took about 1 second to complete which is much longer than a normal case. Kernel Version: Linux compute08 4.15.0-72-generic #81-Ubuntu SMP Tue Nov 26 12:20:02 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux Would be appreciated to know any operations to identify the issue, Thank you very much for the help. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-meta/+bug/2021571/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 2021571] Re: cat /sys/fs/cgroup/blkio/blkio.time_recursive took 1second to complete
Was the cause of this ever determined? I am seeing something similar on some of my systems. Times are usually around 1s, sometimes slightly higher. # uname -r 4.15.0-193-generic # time cat /sys/fs/cgroup/blkio/blkio.time_recursive 8:144 69349171178307 8:128 4718621058 8:112 4604996397 8:96 4625378704 8:80 4873085108 8:64 3777190161 8:48 4749702443 8:32 8346248400 8:16 9260262419 8:0 154939957376379 real0m1.03s user0m0.00s sys 0m1.03s -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-meta in Ubuntu. https://bugs.launchpad.net/bugs/2021571 Title: cat /sys/fs/cgroup/blkio/blkio.time_recursive took 1second to complete Status in linux-meta package in Ubuntu: Confirmed Bug description: Issue: ```console # time cat /sys/fs/cgroup/blkio/blkio.time_recursive 8:16 354721435 real0m1.297s user0m0.000s sys 0m1.297s ``` As could be seen from above result, cat blkio took about 1 second to complete which is much longer than a normal case. Kernel Version: Linux compute08 4.15.0-72-generic #81-Ubuntu SMP Tue Nov 26 12:20:02 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux Would be appreciated to know any operations to identify the issue, Thank you very much for the help. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-meta/+bug/2021571/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
