Public bug reported:
== Comment: #0 - Anton Blanchard <ant...@au1.ibm.com> - 2015-07-16 22:22:09 ==
We are seeing large amounts of jitter caused by od_dbs_timer(). We should slow
down the rate of updates and or turn this into a timer. Having a workqueue
execute so often is very noticeable.
# echo 1 >
/sys/kernel/debug/tracing/events/workqueue/workqueue_execute_start/enable
(wait a while)
# cat /sys/kernel/debug/trace
<...>-67605 [040] .... 849622.393576: workqueue_execute_start: work
struct c0000007fba1ba20: function od_dbs_timer
<...>-67605 [040] .... 849622.403574: workqueue_execute_start: work
struct c0000007fba1ba20: function od_dbs_timer
<...>-116685 [048] .... 849622.403575: workqueue_execute_start: work
struct c0000007fbc1ba20: function od_dbs_timer
<...>-116685 [048] .... 849622.413574: workqueue_execute_start: work
struct c0000007fbc1ba20: function od_dbs_timer
<...>-67605 [040] .... 849622.413575: workqueue_execute_start: work
struct c0000007fba1ba20: function od_dbs_timer
<...>-67605 [040] .... 849622.423575: workqueue_execute_start: work
struct c0000007fba1ba20: function od_dbs_timer
<...>-116685 [048] .... 849622.433574: workqueue_execute_start: work
struct c0000007fbc1ba20: function od_dbs_timer
<...>-67605 [040] .... 849622.433574: workqueue_execute_start: work
struct c0000007fba1ba20: function od_dbs_timer
<...>-116685 [048] .... 849622.443573: workqueue_execute_start: work
struct c0000007fbc1ba20: function od_dbs_timer
== Comment: #1 - Shilpasri G. Bhat <shigb...@in.ibm.com> - 2015-07-22 19:42:38
==
Hi Anton,
We can set the governor's tunable 'sampling_down_factor' to decrease the
rate of updates. When this tunable is set to a value greater than 1, the
sampling period of the governor is increased during the peak load to
sampling_period times sampling_down_factor. This will reduce the jitter
caused by od_dbs_timer() when the cpu is busy.
I am currently running benchmarks to find out the optimal value for this
tunable and will post them soon.
Thanks and Regards,
Shilpa
== Comment: #2 - Anton Blanchard <ant...@au1.ibm.com> - 2015-07-31 03:44:49 ==
FYI We are also seeing high levels of CPU consumed by this on a LAMP workload:
2.54% kworker/0:0 [kernel.kallsyms] [k] osq_lock
|
---osq_lock
|
|--99.83%-- mutex_optimistic_spin
| __mutex_lock_slowpath
| mutex_lock
| |
| |--80.08%-- od_dbs_timer
2.5% of total CPU time spent in the od_dbs_timer mutex.
== Comment: #3 - Anton Blanchard <ant...@au1.ibm.com> - 2015-07-31 06:00:45 ==
Hitting this on a customer setup, raising priority
== Comment: #4 - Shilpasri G. Bhat <shigb...@in.ibm.com> - 2015-08-03 06:47:40
==
I used `perf top` and `perf record` to observe the overhead caused by
'osq_lock'.
Both with ebizzy and SPECPower's ssjb workload I am able to see an overhead of
0.03% caused by 'osq_lock' with default governor settings.
With sampling_down_factor=100, (1second) I am able to see 0.00% of overhead by
'osq_lock'.
So this might not be a good data point to showcase, but by reducing the
od_dbs_timer interrupts we are guaranteed to decrease the overhead
caused by 'osq_lock'.
== Comment: #5 - VAIDYANATHAN SRINIVASAN <svaid...@in.ibm.com> - 2015-08-03
09:09:09 ==
Hi Anton,
Thanks for opening the bz to track and fix this issue. Shilpa is trying
different workarounds. Here is our plan:
(1) Use sampling_down_factor and other tunables in current Ubuntu
releases to workaround the issue or minimise the impact.
(2) Redesign cpufreq subsystem on powerpc similar to intel pstate driver
so that we can program timers and cancel them dynamically based on
different utilization points. Target Ubuntu 16.04 and then backport to
14.04.x and other distros.
(3) Enhance design for (2) buy estimating core level utilization without
running timers in each thread and then decide the target PState
(4) Explore hardware assist so that we can avoid per-core estimation in
software but still be able to set per-core PState. We need to take an
interrupt or work-queue only to change PState and not really for
estimation of load. Hence steady state load will experience zero jitter
from cpufreq.
--Vaidy
== Comment: #7 - Shilpasri G. Bhat <shigb...@in.ibm.com> - 2015-08-04 07:49:23
==
Workaround using ondemand tunable 'sampling_down_factor':
File: /etc/init.d/ondemand (shell script which sets the governor after boot)
if [ "$GOVERNOR" = "ondemand" ]; then
echo 100 >
/sys/devices/system/cpu/cpufreq/ondemand/sampling_down_factor
fi
Setting the value of 100 to sampling_down_factor will increase the
sampling period of ondemand governor to one second when the cpu is busy.
** Affects: ubuntu
Importance: Undecided
Status: New
** Tags: architecture-ppc64le bugnameltc-127761 severity-high
targetmilestone-inin1504
** Tags added: architecture-ppc64le bugnameltc-127761 severity-high
targetmilestone-inin1504
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1483586
Title:
On demand cpufreq govneror causes large amounts of jitter
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+bug/1483586/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
