[Kernel-packages] [Bug 1579278] Re: Keep powersave CPU frequency scaling governor for CPUs that support intel_pstate
The last outstanding open task for sysvinit/xenial is also invalid, to my understanding. Both /etc/init.d/ondemand in <= xenial and /lib/systemd/set-cpufreq in >= bionic implement the intended default policy of interactive/ondemand/powersave, not performance. ** Changed in: sysvinit (Ubuntu Xenial) Status: Triaged => Invalid -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1579278 Title: Keep powersave CPU frequency scaling governor for CPUs that support intel_pstate Status in linux package in Ubuntu: Invalid Status in systemd package in Ubuntu: Fix Released Status in sysvinit package in Ubuntu: Invalid Status in linux source package in Xenial: Invalid Status in systemd source package in Xenial: Invalid Status in sysvinit source package in Xenial: Invalid Bug description: Hi, With the new Ubuntu archive servers, we saw constantly high load and after some tinkering, we found that it was mostly CPUs being woken up to see if they should enter idle states. Changing the CPU frequency scaling governor to "performance" saw a considerable drop. Perf report using the following commands: | perf record -g -a sleep 10 | perf report | Samples: 287K of event 'cycles:pp', Event count (approx.): 124776998906 | Children Self Command Shared Object Symbol | + 55.24% 0.20% swapper [kernel.kallsyms] [k] cpu_startup_entry | + 53.51% 0.00% swapper [kernel.kallsyms] [k] start_secondary | + 53.02% 0.08% swapper [kernel.kallsyms] [k] call_cpuidle | + 52.94% 0.02% swapper [kernel.kallsyms] [k] cpuidle_enter | + 31.81% 0.67% swapper [kernel.kallsyms] [k] cpuidle_enter_state | + 29.59% 0.12% swapper [kernel.kallsyms] [k] acpi_idle_enter | + 29.45% 0.05% swapper [kernel.kallsyms] [k] acpi_idle_do_entry | + 29.43%29.43% swapper [kernel.kallsyms] [k] acpi_processor_ffh_cstate_enter | + 20.51% 0.04% swapper [kernel.kallsyms] [k] ret_from_intr | + 20.47% 0.12% swapper [kernel.kallsyms] [k] do_IRQ | + 19.30% 0.07% swapper [kernel.kallsyms] [k] irq_exit | + 19.18% 0.07% apache2 [kernel.kallsyms] [k] entry_SYSCALL_64_fastpath | + 18.80% 0.17% swapper [kernel.kallsyms] [k] __do_softirq | + 16.45% 0.11% swapper [kernel.kallsyms] [k] net_rx_action | + 16.25% 0.43% swapper [kernel.kallsyms] [k] be_poll | + 14.74% 0.21% swapper [kernel.kallsyms] [k] be_process_rx | + 13.61% 0.07% swapper [kernel.kallsyms] [k] napi_gro_frags | + 12.58% 0.04% swapper [kernel.kallsyms] [k] netif_receive_skb_internal | + 12.48% 0.03% swapper [kernel.kallsyms] [k] __netif_receive_skb | + 12.42% 0.24% swapper [kernel.kallsyms] [k] __netif_receive_skb_core | + 12.41% 0.00% apache2 [unknown] [k] 0x7f27983b5028 | + 12.41% 0.00% apache2 [unknown] [k] 0x7f2798369028 | + 11.49% 0.16% swapper [kernel.kallsyms] [k] ip_rcv | + 11.29% 0.09% swapper [kernel.kallsyms] [k] ip_rcv_finish | + 10.77% 0.05% swapper [kernel.kallsyms] [k] ip_local_deliver | + 10.70% 0.06% swapper [kernel.kallsyms] [k] ip_local_deliver_finish | + 10.55% 0.22% swapper [kernel.kallsyms] [k] tcp_v4_rcv | + 10.10% 0.00% apache2 [unknown] [k] | + 10.01% 0.04% swapper [kernel.kallsyms] [k] tcp_v4_do_rcv Expanding in a few of those, you'll see: | - 55.24% 0.20% swapper [kernel.kallsyms] [k] cpu_startup_entry |- 55.04% cpu_startup_entry | - 52.98% call_cpuidle | + 52.93% cpuidle_enter | + 0.00% ret_from_intr |0.00% cpuidle_enter_state |0.00% irq_entries_start | + 1.14% cpuidle_select | + 0.47% schedule_preempt_disabled | 0.10% rcu_idle_enter | 0.09% rcu_idle_exit | + 0.05% ret_from_intr | + 0.05% tick_nohz_idle_enter | + 0.04% arch_cpu_idle_enter | 0.02% cpuidle_enter | 0.02% tick_check_broadcast_expired | + 0.01% cpuidle_reflect | 0.01% menu_reflect | 0.01% atomic_notifier_call_chain | 0.01% local_touch_nmi | 0.01% cpuidle_not_available | 0.01% menu_select | 0.01% cpuidle_get_
[Kernel-packages] [Bug 1579278] Re: Keep powersave CPU frequency scaling governor for CPUs that support intel_pstate
> Setting the governor to performance 'fixes' it. > I don't know if this is *this* issue or another one, so happy to open a new > issue if you prefer. we will be changing the built-in 'ondemand' systemd service (set-cpufreq script, technically) to allow end-user configuration instead of using only its hardcoded values. So if you want to use the 'performance' governor you will be able to configure it to do so. See bug 1806012 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1579278 Title: Keep powersave CPU frequency scaling governor for CPUs that support intel_pstate Status in linux package in Ubuntu: Invalid Status in systemd package in Ubuntu: Fix Released Status in sysvinit package in Ubuntu: Invalid Status in linux source package in Xenial: Invalid Status in systemd source package in Xenial: Invalid Status in sysvinit source package in Xenial: Invalid Bug description: Hi, With the new Ubuntu archive servers, we saw constantly high load and after some tinkering, we found that it was mostly CPUs being woken up to see if they should enter idle states. Changing the CPU frequency scaling governor to "performance" saw a considerable drop. Perf report using the following commands: | perf record -g -a sleep 10 | perf report | Samples: 287K of event 'cycles:pp', Event count (approx.): 124776998906 | Children Self Command Shared Object Symbol | + 55.24% 0.20% swapper [kernel.kallsyms] [k] cpu_startup_entry | + 53.51% 0.00% swapper [kernel.kallsyms] [k] start_secondary | + 53.02% 0.08% swapper [kernel.kallsyms] [k] call_cpuidle | + 52.94% 0.02% swapper [kernel.kallsyms] [k] cpuidle_enter | + 31.81% 0.67% swapper [kernel.kallsyms] [k] cpuidle_enter_state | + 29.59% 0.12% swapper [kernel.kallsyms] [k] acpi_idle_enter | + 29.45% 0.05% swapper [kernel.kallsyms] [k] acpi_idle_do_entry | + 29.43%29.43% swapper [kernel.kallsyms] [k] acpi_processor_ffh_cstate_enter | + 20.51% 0.04% swapper [kernel.kallsyms] [k] ret_from_intr | + 20.47% 0.12% swapper [kernel.kallsyms] [k] do_IRQ | + 19.30% 0.07% swapper [kernel.kallsyms] [k] irq_exit | + 19.18% 0.07% apache2 [kernel.kallsyms] [k] entry_SYSCALL_64_fastpath | + 18.80% 0.17% swapper [kernel.kallsyms] [k] __do_softirq | + 16.45% 0.11% swapper [kernel.kallsyms] [k] net_rx_action | + 16.25% 0.43% swapper [kernel.kallsyms] [k] be_poll | + 14.74% 0.21% swapper [kernel.kallsyms] [k] be_process_rx | + 13.61% 0.07% swapper [kernel.kallsyms] [k] napi_gro_frags | + 12.58% 0.04% swapper [kernel.kallsyms] [k] netif_receive_skb_internal | + 12.48% 0.03% swapper [kernel.kallsyms] [k] __netif_receive_skb | + 12.42% 0.24% swapper [kernel.kallsyms] [k] __netif_receive_skb_core | + 12.41% 0.00% apache2 [unknown] [k] 0x7f27983b5028 | + 12.41% 0.00% apache2 [unknown] [k] 0x7f2798369028 | + 11.49% 0.16% swapper [kernel.kallsyms] [k] ip_rcv | + 11.29% 0.09% swapper [kernel.kallsyms] [k] ip_rcv_finish | + 10.77% 0.05% swapper [kernel.kallsyms] [k] ip_local_deliver | + 10.70% 0.06% swapper [kernel.kallsyms] [k] ip_local_deliver_finish | + 10.55% 0.22% swapper [kernel.kallsyms] [k] tcp_v4_rcv | + 10.10% 0.00% apache2 [unknown] [k] | + 10.01% 0.04% swapper [kernel.kallsyms] [k] tcp_v4_do_rcv Expanding in a few of those, you'll see: | - 55.24% 0.20% swapper [kernel.kallsyms] [k] cpu_startup_entry |- 55.04% cpu_startup_entry | - 52.98% call_cpuidle | + 52.93% cpuidle_enter | + 0.00% ret_from_intr |0.00% cpuidle_enter_state |0.00% irq_entries_start | + 1.14% cpuidle_select | + 0.47% schedule_preempt_disabled | 0.10% rcu_idle_enter | 0.09% rcu_idle_exit | + 0.05% ret_from_intr | + 0.05% tick_nohz_idle_enter | + 0.04% arch_cpu_idle_enter | 0.02% cpuidle_enter | 0.02% tick_check_broadcast_expired | + 0.01% cpuidle_reflect | 0.01% menu_reflect | 0.01% atomic_notifier_call_chain | 0.01% local_touch_nmi |
[Kernel-packages] [Bug 1579278] Re: Keep powersave CPU frequency scaling governor for CPUs that support intel_pstate
This bug was fixed in the package systemd - 230-3git1 --- systemd (230-3git1) yakkety; urgency=medium Upload current Debian packaging git to fix tests. [ Martin Pitt ] * tmp.mount: Add nosuid and nodev mount options. This restores compatibility with the original SysV int RAMTMP defaults. (Closes: #826377) * debian/tests/upstream: Some tests fail on platforms without QEMU at the moment due to upstream PR#3587; blacklist these for now if QEMU is not available. * debian/rules: Don't run the "anything links against /usr" check for upstream tests, as those run on Ubuntu 16.04 LTS which does not yet have libidn moved to /lib. * debian/tests/upstream: Clean up old journals before running a test, to avoid printing a wrong one on failure. * debian/tests/upstream: Do not run the QEMU tests on i386. Nested QEMU on i386 causes testbed hangs on Ubuntu's cloud infrastructure, which is the only place where these actually run. [ Laurent Bigonville ] * Build with IDN support. (Closes: #814528) -- Martin Pitt Thu, 23 Jun 2016 10:51:14 +0200 ** Changed in: systemd (Ubuntu) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1579278 Title: Keep powersave CPU frequency scaling governor for CPUs that support intel_pstate Status in linux package in Ubuntu: Invalid Status in systemd package in Ubuntu: Fix Released Status in sysvinit package in Ubuntu: Invalid Status in linux source package in Xenial: Invalid Status in systemd source package in Xenial: Invalid Status in sysvinit source package in Xenial: Triaged Bug description: Hi, With the new Ubuntu archive servers, we saw constantly high load and after some tinkering, we found that it was mostly CPUs being woken up to see if they should enter idle states. Changing the CPU frequency scaling governor to "performance" saw a considerable drop. Perf report using the following commands: | perf record -g -a sleep 10 | perf report | Samples: 287K of event 'cycles:pp', Event count (approx.): 124776998906 | Children Self Command Shared Object Symbol | + 55.24% 0.20% swapper [kernel.kallsyms] [k] cpu_startup_entry | + 53.51% 0.00% swapper [kernel.kallsyms] [k] start_secondary | + 53.02% 0.08% swapper [kernel.kallsyms] [k] call_cpuidle | + 52.94% 0.02% swapper [kernel.kallsyms] [k] cpuidle_enter | + 31.81% 0.67% swapper [kernel.kallsyms] [k] cpuidle_enter_state | + 29.59% 0.12% swapper [kernel.kallsyms] [k] acpi_idle_enter | + 29.45% 0.05% swapper [kernel.kallsyms] [k] acpi_idle_do_entry | + 29.43%29.43% swapper [kernel.kallsyms] [k] acpi_processor_ffh_cstate_enter | + 20.51% 0.04% swapper [kernel.kallsyms] [k] ret_from_intr | + 20.47% 0.12% swapper [kernel.kallsyms] [k] do_IRQ | + 19.30% 0.07% swapper [kernel.kallsyms] [k] irq_exit | + 19.18% 0.07% apache2 [kernel.kallsyms] [k] entry_SYSCALL_64_fastpath | + 18.80% 0.17% swapper [kernel.kallsyms] [k] __do_softirq | + 16.45% 0.11% swapper [kernel.kallsyms] [k] net_rx_action | + 16.25% 0.43% swapper [kernel.kallsyms] [k] be_poll | + 14.74% 0.21% swapper [kernel.kallsyms] [k] be_process_rx | + 13.61% 0.07% swapper [kernel.kallsyms] [k] napi_gro_frags | + 12.58% 0.04% swapper [kernel.kallsyms] [k] netif_receive_skb_internal | + 12.48% 0.03% swapper [kernel.kallsyms] [k] __netif_receive_skb | + 12.42% 0.24% swapper [kernel.kallsyms] [k] __netif_receive_skb_core | + 12.41% 0.00% apache2 [unknown] [k] 0x7f27983b5028 | + 12.41% 0.00% apache2 [unknown] [k] 0x7f2798369028 | + 11.49% 0.16% swapper [kernel.kallsyms] [k] ip_rcv | + 11.29% 0.09% swapper [kernel.kallsyms] [k] ip_rcv_finish | + 10.77% 0.05% swapper [kernel.kallsyms] [k] ip_local_deliver | + 10.70% 0.06% swapper [kernel.kallsyms] [k] ip_local_deliver_finish | + 10.55% 0.22% swapper [kernel.kallsyms] [k] tcp_v4_rcv | + 10.10% 0.00% apache2 [unknown] [k] | + 10.01% 0.04% swapper [kernel.kallsyms] [k] tcp_v4_do_rcv Expanding in a few of those, you'll see: | - 55.24% 0.20% swappe
[Kernel-packages] [Bug 1579278] Re: Keep powersave CPU frequency scaling governor for CPUs that support intel_pstate
http://anonscm.debian.org/cgit/pkg- systemd/systemd.git/commit/?id=2558ca88 ** Changed in: systemd (Ubuntu) Status: In Progress => Fix Committed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1579278 Title: Keep powersave CPU frequency scaling governor for CPUs that support intel_pstate Status in linux package in Ubuntu: Invalid Status in systemd package in Ubuntu: Fix Committed Status in sysvinit package in Ubuntu: Invalid Status in linux source package in Xenial: Invalid Status in systemd source package in Xenial: Invalid Status in sysvinit source package in Xenial: Triaged Bug description: Hi, With the new Ubuntu archive servers, we saw constantly high load and after some tinkering, we found that it was mostly CPUs being woken up to see if they should enter idle states. Changing the CPU frequency scaling governor to "performance" saw a considerable drop. Perf report using the following commands: | perf record -g -a sleep 10 | perf report | Samples: 287K of event 'cycles:pp', Event count (approx.): 124776998906 | Children Self Command Shared Object Symbol | + 55.24% 0.20% swapper [kernel.kallsyms] [k] cpu_startup_entry | + 53.51% 0.00% swapper [kernel.kallsyms] [k] start_secondary | + 53.02% 0.08% swapper [kernel.kallsyms] [k] call_cpuidle | + 52.94% 0.02% swapper [kernel.kallsyms] [k] cpuidle_enter | + 31.81% 0.67% swapper [kernel.kallsyms] [k] cpuidle_enter_state | + 29.59% 0.12% swapper [kernel.kallsyms] [k] acpi_idle_enter | + 29.45% 0.05% swapper [kernel.kallsyms] [k] acpi_idle_do_entry | + 29.43%29.43% swapper [kernel.kallsyms] [k] acpi_processor_ffh_cstate_enter | + 20.51% 0.04% swapper [kernel.kallsyms] [k] ret_from_intr | + 20.47% 0.12% swapper [kernel.kallsyms] [k] do_IRQ | + 19.30% 0.07% swapper [kernel.kallsyms] [k] irq_exit | + 19.18% 0.07% apache2 [kernel.kallsyms] [k] entry_SYSCALL_64_fastpath | + 18.80% 0.17% swapper [kernel.kallsyms] [k] __do_softirq | + 16.45% 0.11% swapper [kernel.kallsyms] [k] net_rx_action | + 16.25% 0.43% swapper [kernel.kallsyms] [k] be_poll | + 14.74% 0.21% swapper [kernel.kallsyms] [k] be_process_rx | + 13.61% 0.07% swapper [kernel.kallsyms] [k] napi_gro_frags | + 12.58% 0.04% swapper [kernel.kallsyms] [k] netif_receive_skb_internal | + 12.48% 0.03% swapper [kernel.kallsyms] [k] __netif_receive_skb | + 12.42% 0.24% swapper [kernel.kallsyms] [k] __netif_receive_skb_core | + 12.41% 0.00% apache2 [unknown] [k] 0x7f27983b5028 | + 12.41% 0.00% apache2 [unknown] [k] 0x7f2798369028 | + 11.49% 0.16% swapper [kernel.kallsyms] [k] ip_rcv | + 11.29% 0.09% swapper [kernel.kallsyms] [k] ip_rcv_finish | + 10.77% 0.05% swapper [kernel.kallsyms] [k] ip_local_deliver | + 10.70% 0.06% swapper [kernel.kallsyms] [k] ip_local_deliver_finish | + 10.55% 0.22% swapper [kernel.kallsyms] [k] tcp_v4_rcv | + 10.10% 0.00% apache2 [unknown] [k] | + 10.01% 0.04% swapper [kernel.kallsyms] [k] tcp_v4_do_rcv Expanding in a few of those, you'll see: | - 55.24% 0.20% swapper [kernel.kallsyms] [k] cpu_startup_entry |- 55.04% cpu_startup_entry | - 52.98% call_cpuidle | + 52.93% cpuidle_enter | + 0.00% ret_from_intr |0.00% cpuidle_enter_state |0.00% irq_entries_start | + 1.14% cpuidle_select | + 0.47% schedule_preempt_disabled | 0.10% rcu_idle_enter | 0.09% rcu_idle_exit | + 0.05% ret_from_intr | + 0.05% tick_nohz_idle_enter | + 0.04% arch_cpu_idle_enter | 0.02% cpuidle_enter | 0.02% tick_check_broadcast_expired | + 0.01% cpuidle_reflect | 0.01% menu_reflect | 0.01% atomic_notifier_call_chain | 0.01% local_touch_nmi | 0.01% cpuidle_not_available | 0.01% menu_select | 0.01% cpuidle_get_cpu_driver | + 0.01% tick_nohz_idle_exit | + 0.01% sched_ttwu_pending | 0.00% set_cpu_sd_state_idle | 0.00% native_irq_return_iret | 0.0
[Kernel-packages] [Bug 1579278] Re: Keep powersave CPU frequency scaling governor for CPUs that support intel_pstate
@Haw Loeung: It would be good to understand in more detail your findings. As far as I know (and I am not an expert in this area), great care has been taken to avoid things like wakeups just to decide to go idle. I did a test on my computer (Ubuntu 16.04 server, no GUI), but to avoid extra overheads, I used the /proc/timer_stats method. I also did the test over 300 seconds, so as to dilute influence from the command itself. Kernel: 4.8.0-040800rc5-generic intel_pstate, "idle" system: powersave: 14722 events, 49.073 events / second performance: 14964 events, 49.879 events / second intel_pstate: extra "idle" system: powersave: 6381 total events, 21.269 events / second performance: 6788 events, 22.626 events / second Which kernel are you using? There was a flurry of high wakeups activity in June, but I thought it all got resolved. Note that there was a slight increase in wakeups with the recent change to a utilization based algorithm, but that was good thing because it solved a very long standing issue whereby the CPU frequency might not increase under high load for periodic workflows that just so happened to be idle on jiffy boundaries. You mention "on some workloads". Could you be more specific? i.e. so that I can attempt to recreate your scenario on my computer. My script: $ cat ./set_cpu_timer_stats_2 #! /bin/bash # Take a 300 second time sample # A long time is needed to amortize/dilute the effect of the command itself. # Reset the counters echo "0" > /proc/timer_stats echo "1" > /proc/timer_stats # Now let the sample time elapse sleep 300 # Now observe the timer counters cat /proc/timer_stats # And turn off timer stats. echo "0" > /proc/timer_stats What does "extra idle" mean?: $ cat set_cpu_turn_off_services #! /bin/bash # Turn off some services to try to get "idle" to be more "idle" sudo systemctl stop mysql.service sudo systemctl stop apache2.service sudo systemctl stop nmbd.service sudo systemctl stop smbd.service sudo systemctl stop cron.service sudo systemctl stop winbind.service sudo systemctl stop apt-daily.timer Recent wakeups thread: http://marc.info/?t=14661775703&r=1&w=2 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1579278 Title: Keep powersave CPU frequency scaling governor for CPUs that support intel_pstate Status in linux package in Ubuntu: Invalid Status in systemd package in Ubuntu: Fix Committed Status in sysvinit package in Ubuntu: Invalid Status in linux source package in Xenial: Invalid Status in systemd source package in Xenial: Invalid Status in sysvinit source package in Xenial: Triaged Bug description: Hi, With the new Ubuntu archive servers, we saw constantly high load and after some tinkering, we found that it was mostly CPUs being woken up to see if they should enter idle states. Changing the CPU frequency scaling governor to "performance" saw a considerable drop. Perf report using the following commands: | perf record -g -a sleep 10 | perf report | Samples: 287K of event 'cycles:pp', Event count (approx.): 124776998906 | Children Self Command Shared Object Symbol | + 55.24% 0.20% swapper [kernel.kallsyms] [k] cpu_startup_entry | + 53.51% 0.00% swapper [kernel.kallsyms] [k] start_secondary | + 53.02% 0.08% swapper [kernel.kallsyms] [k] call_cpuidle | + 52.94% 0.02% swapper [kernel.kallsyms] [k] cpuidle_enter | + 31.81% 0.67% swapper [kernel.kallsyms] [k] cpuidle_enter_state | + 29.59% 0.12% swapper [kernel.kallsyms] [k] acpi_idle_enter | + 29.45% 0.05% swapper [kernel.kallsyms] [k] acpi_idle_do_entry | + 29.43%29.43% swapper [kernel.kallsyms] [k] acpi_processor_ffh_cstate_enter | + 20.51% 0.04% swapper [kernel.kallsyms] [k] ret_from_intr | + 20.47% 0.12% swapper [kernel.kallsyms] [k] do_IRQ | + 19.30% 0.07% swapper [kernel.kallsyms] [k] irq_exit | + 19.18% 0.07% apache2 [kernel.kallsyms] [k] entry_SYSCALL_64_fastpath | + 18.80% 0.17% swapper [kernel.kallsyms] [k] __do_softirq | + 16.45% 0.11% swapper [kernel.kallsyms] [k] net_rx_action | + 16.25% 0.43% swapper [kernel.kallsyms] [k] be_poll | + 14.74% 0.21% swapper [kernel.kallsyms] [k] be_process_rx | + 13.61% 0.07% swapper [kernel.kallsyms] [k] napi_gro_frags | + 12.58% 0.04% swapper [kernel.kallsyms] [k] netif_receive_skb_internal | + 12.48% 0.03% swapper [kernel.kallsyms] [k] __netif_receive_skb | + 12.42% 0.24% swapper
[Kernel-packages] [Bug 1579278] Re: Keep powersave CPU frequency scaling governor for CPUs that support intel_pstate
@Haw Loeung: In addition to what I wrote earlier: > With the new Ubuntu archive servers, we saw constantly high load > and after some tinkering, we found that it was mostly CPUs > being woken up to see if they should enter idle states. > Changing the CPU frequency scaling governor to "performance" saw a > considerable drop. What do you mean by "high load"? And when you say "saw a considerable drop", does that mean in wakeups per second or load? Note that you should observe a significant difference in load average on a server between powersave and performance mode, and that actually indicates things are working as they should be. For the SpecPower simulator test I posted above, I'll add some more data for the 0.5X and X lines: 0.5X, where Performance used 31.7% more package power: Powersave: Busy%: 12.58% (load average = 1.01) Bzy MHz: 1651 Performance: Busy%: 5.04% (load average = 0.40) Bzy MHz: 3686 X, where Performance used 42.1% more package power: Powersave: Busy%: 23.66% (load average = 1.89) Bzy MHz: 1798 Performance: Busy%: 10.56% (load average = 0.84) Bzy MHz: 3681 Isn't energy consumption what really matters, as long as performance doesn't suffer too much? What I would like to see for your servers is the results from: sudo turbostat -J -S --debug sleep 300 For the intel_pstate CPU frequency scaling driver and the powersave and performance scaling governors with your work flow. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1579278 Title: Keep powersave CPU frequency scaling governor for CPUs that support intel_pstate Status in linux package in Ubuntu: Invalid Status in systemd package in Ubuntu: Fix Committed Status in sysvinit package in Ubuntu: Invalid Status in linux source package in Xenial: Invalid Status in systemd source package in Xenial: Invalid Status in sysvinit source package in Xenial: Triaged Bug description: Hi, With the new Ubuntu archive servers, we saw constantly high load and after some tinkering, we found that it was mostly CPUs being woken up to see if they should enter idle states. Changing the CPU frequency scaling governor to "performance" saw a considerable drop. Perf report using the following commands: | perf record -g -a sleep 10 | perf report | Samples: 287K of event 'cycles:pp', Event count (approx.): 124776998906 | Children Self Command Shared Object Symbol | + 55.24% 0.20% swapper [kernel.kallsyms] [k] cpu_startup_entry | + 53.51% 0.00% swapper [kernel.kallsyms] [k] start_secondary | + 53.02% 0.08% swapper [kernel.kallsyms] [k] call_cpuidle | + 52.94% 0.02% swapper [kernel.kallsyms] [k] cpuidle_enter | + 31.81% 0.67% swapper [kernel.kallsyms] [k] cpuidle_enter_state | + 29.59% 0.12% swapper [kernel.kallsyms] [k] acpi_idle_enter | + 29.45% 0.05% swapper [kernel.kallsyms] [k] acpi_idle_do_entry | + 29.43%29.43% swapper [kernel.kallsyms] [k] acpi_processor_ffh_cstate_enter | + 20.51% 0.04% swapper [kernel.kallsyms] [k] ret_from_intr | + 20.47% 0.12% swapper [kernel.kallsyms] [k] do_IRQ | + 19.30% 0.07% swapper [kernel.kallsyms] [k] irq_exit | + 19.18% 0.07% apache2 [kernel.kallsyms] [k] entry_SYSCALL_64_fastpath | + 18.80% 0.17% swapper [kernel.kallsyms] [k] __do_softirq | + 16.45% 0.11% swapper [kernel.kallsyms] [k] net_rx_action | + 16.25% 0.43% swapper [kernel.kallsyms] [k] be_poll | + 14.74% 0.21% swapper [kernel.kallsyms] [k] be_process_rx | + 13.61% 0.07% swapper [kernel.kallsyms] [k] napi_gro_frags | + 12.58% 0.04% swapper [kernel.kallsyms] [k] netif_receive_skb_internal | + 12.48% 0.03% swapper [kernel.kallsyms] [k] __netif_receive_skb | + 12.42% 0.24% swapper [kernel.kallsyms] [k] __netif_receive_skb_core | + 12.41% 0.00% apache2 [unknown] [k] 0x7f27983b5028 | + 12.41% 0.00% apache2 [unknown] [k] 0x7f2798369028 | + 11.49% 0.16% swapper [kernel.kallsyms] [k] ip_rcv | + 11.29% 0.09% swapper [kernel.kallsyms] [k] ip_rcv_finish | + 10.77% 0.05% swapper [kernel.kallsyms] [k] ip_local_deliver | + 10.70% 0.06% swapper [kernel.kallsyms] [k] ip_local_deliver_finish | + 10.55% 0.22% swapper [kernel.kallsyms] [k] tcp_v4_rcv | + 10.10% 0.0
[Kernel-packages] [Bug 1579278] Re: Keep powersave CPU frequency scaling governor for CPUs that support intel_pstate
This bug was fixed in the package systemd - 231-6git1 --- systemd (231-6git1) yakkety; urgency=medium Upload current Debian packaging git. [ Michael Biebl ] * fsckd: Do not exit on idle timeout if there are still clients connected systemd-fsckd's event loop terminates if nothing happens for 30 seconds. Exiting prematurely while fsck is still running but simply too slow to send us progress updates would close the socket and fsck would receive SIGPIPE when it writes to the socket. If this happens, the fsck process is aborted and the file system check is not completed. (Closes: #788050) (LP: #1547844) [ Martin Pitt ] * 73-usb-net-by-mac.rules: Split kernel command line import line. Reportedly this makes the rule actually work on some platforms. Thanks Alp Toker! (LP: #1593379) * debian/tests/boot-smoke: Only run 5 iterations * systemd.postinst: Drop obsolete setcap call for systemd-detect-virt. Drop corresponding libcap2-bin dependency. * debian/tests/systemd-fsckd: Robustify check for "unit was running" (LP: #1624406) * debian/extra/set-cpufreq: Use powersave with intel_pstate. This is what we did on xenial, and apparently powersave is still actually better than performance. Thanks to Doug Smythies for the measurements! (LP: #1579278) * Ubuntu: Move ondemand.service from static to runtime enablement. This makes it easier to keep performance, by disabling ondemand.service. Side issue in LP: #1579278 -- Martin Pitt Mon, 19 Sep 2016 22:37:51 +0200 ** Changed in: systemd (Ubuntu) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1579278 Title: Keep powersave CPU frequency scaling governor for CPUs that support intel_pstate Status in linux package in Ubuntu: Invalid Status in systemd package in Ubuntu: Fix Released Status in sysvinit package in Ubuntu: Invalid Status in linux source package in Xenial: Invalid Status in systemd source package in Xenial: Invalid Status in sysvinit source package in Xenial: Triaged Bug description: Hi, With the new Ubuntu archive servers, we saw constantly high load and after some tinkering, we found that it was mostly CPUs being woken up to see if they should enter idle states. Changing the CPU frequency scaling governor to "performance" saw a considerable drop. Perf report using the following commands: | perf record -g -a sleep 10 | perf report | Samples: 287K of event 'cycles:pp', Event count (approx.): 124776998906 | Children Self Command Shared Object Symbol | + 55.24% 0.20% swapper [kernel.kallsyms] [k] cpu_startup_entry | + 53.51% 0.00% swapper [kernel.kallsyms] [k] start_secondary | + 53.02% 0.08% swapper [kernel.kallsyms] [k] call_cpuidle | + 52.94% 0.02% swapper [kernel.kallsyms] [k] cpuidle_enter | + 31.81% 0.67% swapper [kernel.kallsyms] [k] cpuidle_enter_state | + 29.59% 0.12% swapper [kernel.kallsyms] [k] acpi_idle_enter | + 29.45% 0.05% swapper [kernel.kallsyms] [k] acpi_idle_do_entry | + 29.43%29.43% swapper [kernel.kallsyms] [k] acpi_processor_ffh_cstate_enter | + 20.51% 0.04% swapper [kernel.kallsyms] [k] ret_from_intr | + 20.47% 0.12% swapper [kernel.kallsyms] [k] do_IRQ | + 19.30% 0.07% swapper [kernel.kallsyms] [k] irq_exit | + 19.18% 0.07% apache2 [kernel.kallsyms] [k] entry_SYSCALL_64_fastpath | + 18.80% 0.17% swapper [kernel.kallsyms] [k] __do_softirq | + 16.45% 0.11% swapper [kernel.kallsyms] [k] net_rx_action | + 16.25% 0.43% swapper [kernel.kallsyms] [k] be_poll | + 14.74% 0.21% swapper [kernel.kallsyms] [k] be_process_rx | + 13.61% 0.07% swapper [kernel.kallsyms] [k] napi_gro_frags | + 12.58% 0.04% swapper [kernel.kallsyms] [k] netif_receive_skb_internal | + 12.48% 0.03% swapper [kernel.kallsyms] [k] __netif_receive_skb | + 12.42% 0.24% swapper [kernel.kallsyms] [k] __netif_receive_skb_core | + 12.41% 0.00% apache2 [unknown] [k] 0x7f27983b5028 | + 12.41% 0.00% apache2 [unknown] [k] 0x7f2798369028 | + 11.49% 0.16% swapper [kernel.kallsyms] [k] ip_rcv | + 11.29% 0.09% swapper [kernel.kallsyms] [k] ip_rcv_finish | + 10.77% 0.05% swapper [kernel.kallsyms] [k]
[Kernel-packages] [Bug 1579278] Re: Keep powersave CPU frequency scaling governor for CPUs that support intel_pstate
What has been done here is incorrect. The old "ondemand" script was modified so that if the intel_pstate driver was being used, and therefore "ondemand" did not exist, it would fall through to setting the "powersave" governor (refer to bug #1314643). Note that "powersave" with the intel_pstate CPU frequency scaling driver is NOT the same as "powersave" with the acpi-cpufreq CPU frequency scaling driver. The preferred governor with the intel_pstate driver is powersave. I'm saying that this: # more modern processors that support intel_pstate behave worse with # ondemand/powersave and should use performance if [ `cat /sys/devices/system/cpu/cpu$FIRSTCPU/cpufreq/scaling_driver` = intel_pstate ]; then echo 'CPU supports intel_pstate, keeping "performance"' exit 0 fi is incorrect. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1579278 Title: Keep powersave CPU frequency scaling governor for CPUs that support intel_pstate Status in linux package in Ubuntu: Invalid Status in systemd package in Ubuntu: Fix Released Status in sysvinit package in Ubuntu: Invalid Status in linux source package in Xenial: Invalid Status in systemd source package in Xenial: Invalid Status in sysvinit source package in Xenial: Triaged Bug description: Hi, With the new Ubuntu archive servers, we saw constantly high load and after some tinkering, we found that it was mostly CPUs being woken up to see if they should enter idle states. Changing the CPU frequency scaling governor to "performance" saw a considerable drop. Perf report using the following commands: | perf record -g -a sleep 10 | perf report | Samples: 287K of event 'cycles:pp', Event count (approx.): 124776998906 | Children Self Command Shared Object Symbol | + 55.24% 0.20% swapper [kernel.kallsyms] [k] cpu_startup_entry | + 53.51% 0.00% swapper [kernel.kallsyms] [k] start_secondary | + 53.02% 0.08% swapper [kernel.kallsyms] [k] call_cpuidle | + 52.94% 0.02% swapper [kernel.kallsyms] [k] cpuidle_enter | + 31.81% 0.67% swapper [kernel.kallsyms] [k] cpuidle_enter_state | + 29.59% 0.12% swapper [kernel.kallsyms] [k] acpi_idle_enter | + 29.45% 0.05% swapper [kernel.kallsyms] [k] acpi_idle_do_entry | + 29.43%29.43% swapper [kernel.kallsyms] [k] acpi_processor_ffh_cstate_enter | + 20.51% 0.04% swapper [kernel.kallsyms] [k] ret_from_intr | + 20.47% 0.12% swapper [kernel.kallsyms] [k] do_IRQ | + 19.30% 0.07% swapper [kernel.kallsyms] [k] irq_exit | + 19.18% 0.07% apache2 [kernel.kallsyms] [k] entry_SYSCALL_64_fastpath | + 18.80% 0.17% swapper [kernel.kallsyms] [k] __do_softirq | + 16.45% 0.11% swapper [kernel.kallsyms] [k] net_rx_action | + 16.25% 0.43% swapper [kernel.kallsyms] [k] be_poll | + 14.74% 0.21% swapper [kernel.kallsyms] [k] be_process_rx | + 13.61% 0.07% swapper [kernel.kallsyms] [k] napi_gro_frags | + 12.58% 0.04% swapper [kernel.kallsyms] [k] netif_receive_skb_internal | + 12.48% 0.03% swapper [kernel.kallsyms] [k] __netif_receive_skb | + 12.42% 0.24% swapper [kernel.kallsyms] [k] __netif_receive_skb_core | + 12.41% 0.00% apache2 [unknown] [k] 0x7f27983b5028 | + 12.41% 0.00% apache2 [unknown] [k] 0x7f2798369028 | + 11.49% 0.16% swapper [kernel.kallsyms] [k] ip_rcv | + 11.29% 0.09% swapper [kernel.kallsyms] [k] ip_rcv_finish | + 10.77% 0.05% swapper [kernel.kallsyms] [k] ip_local_deliver | + 10.70% 0.06% swapper [kernel.kallsyms] [k] ip_local_deliver_finish | + 10.55% 0.22% swapper [kernel.kallsyms] [k] tcp_v4_rcv | + 10.10% 0.00% apache2 [unknown] [k] | + 10.01% 0.04% swapper [kernel.kallsyms] [k] tcp_v4_do_rcv Expanding in a few of those, you'll see: | - 55.24% 0.20% swapper [kernel.kallsyms] [k] cpu_startup_entry |- 55.04% cpu_startup_entry | - 52.98% call_cpuidle | + 52.93% cpuidle_enter | + 0.00% ret_from_intr |0.00% cpuidle_enter_state |0.00% irq_entries_start | + 1.14% cpuidle_select | + 0.47% schedule_preempt_disabled | 0.10% rcu_idle_enter
[Kernel-packages] [Bug 1579278] Re: Keep powersave CPU frequency scaling governor for CPUs that support intel_pstate
> The preferred governor with the intel_pstate driver is powersave. Do you have some references/proof for that? This is contrary to what kernel developers say, see comment 1. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1579278 Title: Keep powersave CPU frequency scaling governor for CPUs that support intel_pstate Status in linux package in Ubuntu: Invalid Status in systemd package in Ubuntu: Fix Released Status in sysvinit package in Ubuntu: Invalid Status in linux source package in Xenial: Invalid Status in systemd source package in Xenial: Invalid Status in sysvinit source package in Xenial: Triaged Bug description: Hi, With the new Ubuntu archive servers, we saw constantly high load and after some tinkering, we found that it was mostly CPUs being woken up to see if they should enter idle states. Changing the CPU frequency scaling governor to "performance" saw a considerable drop. Perf report using the following commands: | perf record -g -a sleep 10 | perf report | Samples: 287K of event 'cycles:pp', Event count (approx.): 124776998906 | Children Self Command Shared Object Symbol | + 55.24% 0.20% swapper [kernel.kallsyms] [k] cpu_startup_entry | + 53.51% 0.00% swapper [kernel.kallsyms] [k] start_secondary | + 53.02% 0.08% swapper [kernel.kallsyms] [k] call_cpuidle | + 52.94% 0.02% swapper [kernel.kallsyms] [k] cpuidle_enter | + 31.81% 0.67% swapper [kernel.kallsyms] [k] cpuidle_enter_state | + 29.59% 0.12% swapper [kernel.kallsyms] [k] acpi_idle_enter | + 29.45% 0.05% swapper [kernel.kallsyms] [k] acpi_idle_do_entry | + 29.43%29.43% swapper [kernel.kallsyms] [k] acpi_processor_ffh_cstate_enter | + 20.51% 0.04% swapper [kernel.kallsyms] [k] ret_from_intr | + 20.47% 0.12% swapper [kernel.kallsyms] [k] do_IRQ | + 19.30% 0.07% swapper [kernel.kallsyms] [k] irq_exit | + 19.18% 0.07% apache2 [kernel.kallsyms] [k] entry_SYSCALL_64_fastpath | + 18.80% 0.17% swapper [kernel.kallsyms] [k] __do_softirq | + 16.45% 0.11% swapper [kernel.kallsyms] [k] net_rx_action | + 16.25% 0.43% swapper [kernel.kallsyms] [k] be_poll | + 14.74% 0.21% swapper [kernel.kallsyms] [k] be_process_rx | + 13.61% 0.07% swapper [kernel.kallsyms] [k] napi_gro_frags | + 12.58% 0.04% swapper [kernel.kallsyms] [k] netif_receive_skb_internal | + 12.48% 0.03% swapper [kernel.kallsyms] [k] __netif_receive_skb | + 12.42% 0.24% swapper [kernel.kallsyms] [k] __netif_receive_skb_core | + 12.41% 0.00% apache2 [unknown] [k] 0x7f27983b5028 | + 12.41% 0.00% apache2 [unknown] [k] 0x7f2798369028 | + 11.49% 0.16% swapper [kernel.kallsyms] [k] ip_rcv | + 11.29% 0.09% swapper [kernel.kallsyms] [k] ip_rcv_finish | + 10.77% 0.05% swapper [kernel.kallsyms] [k] ip_local_deliver | + 10.70% 0.06% swapper [kernel.kallsyms] [k] ip_local_deliver_finish | + 10.55% 0.22% swapper [kernel.kallsyms] [k] tcp_v4_rcv | + 10.10% 0.00% apache2 [unknown] [k] | + 10.01% 0.04% swapper [kernel.kallsyms] [k] tcp_v4_do_rcv Expanding in a few of those, you'll see: | - 55.24% 0.20% swapper [kernel.kallsyms] [k] cpu_startup_entry |- 55.04% cpu_startup_entry | - 52.98% call_cpuidle | + 52.93% cpuidle_enter | + 0.00% ret_from_intr |0.00% cpuidle_enter_state |0.00% irq_entries_start | + 1.14% cpuidle_select | + 0.47% schedule_preempt_disabled | 0.10% rcu_idle_enter | 0.09% rcu_idle_exit | + 0.05% ret_from_intr | + 0.05% tick_nohz_idle_enter | + 0.04% arch_cpu_idle_enter | 0.02% cpuidle_enter | 0.02% tick_check_broadcast_expired | + 0.01% cpuidle_reflect | 0.01% menu_reflect | 0.01% atomic_notifier_call_chain | 0.01% local_touch_nmi | 0.01% cpuidle_not_available | 0.01% menu_select | 0.01% cpuidle_get_cpu_driver | + 0.01% tick_nohz_idle_exit | + 0.01% sched_ttwu_pending | 0.00% set_cpu_sd_state_idle | 0.00% native_irq_retur
[Kernel-packages] [Bug 1579278] Re: Keep powersave CPU frequency scaling governor for CPUs that support intel_pstate
>> The preferred governor with the intel_pstate driver is powersave. > Do you have some references/proof for that? This is contrary to what > kernel developers say, see comment 1. In my opinion, that reference is obsolete. While I don't work for Intel, the intel_pstate CPU frequency driver is pretty much the only thing that I work on, and for a few years now, and through 3 maintainers. Why would Intel expend so much effort on powersave mode, if it were better to merely set performance mode and forget about it? The objective with powersave mode is best energy/ performance tradeoff. There are still issues, yes. For example whenever clock modulation becomes involved. Also, there can be a tendency to incorrectly drive up the CPU frequency. There is work in progress that should address these issues (see: http://askubuntu.com/questions/812530 /cpu-frequency-scaling-not-working-as-intended-on-vanilla- ubuntu-16-04/812721#812721 and the reference links therein.) I'll try to come back, maybe this weekend, with some energy comparison numbers. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1579278 Title: Keep powersave CPU frequency scaling governor for CPUs that support intel_pstate Status in linux package in Ubuntu: Invalid Status in systemd package in Ubuntu: Fix Released Status in sysvinit package in Ubuntu: Invalid Status in linux source package in Xenial: Invalid Status in systemd source package in Xenial: Invalid Status in sysvinit source package in Xenial: Triaged Bug description: Hi, With the new Ubuntu archive servers, we saw constantly high load and after some tinkering, we found that it was mostly CPUs being woken up to see if they should enter idle states. Changing the CPU frequency scaling governor to "performance" saw a considerable drop. Perf report using the following commands: | perf record -g -a sleep 10 | perf report | Samples: 287K of event 'cycles:pp', Event count (approx.): 124776998906 | Children Self Command Shared Object Symbol | + 55.24% 0.20% swapper [kernel.kallsyms] [k] cpu_startup_entry | + 53.51% 0.00% swapper [kernel.kallsyms] [k] start_secondary | + 53.02% 0.08% swapper [kernel.kallsyms] [k] call_cpuidle | + 52.94% 0.02% swapper [kernel.kallsyms] [k] cpuidle_enter | + 31.81% 0.67% swapper [kernel.kallsyms] [k] cpuidle_enter_state | + 29.59% 0.12% swapper [kernel.kallsyms] [k] acpi_idle_enter | + 29.45% 0.05% swapper [kernel.kallsyms] [k] acpi_idle_do_entry | + 29.43%29.43% swapper [kernel.kallsyms] [k] acpi_processor_ffh_cstate_enter | + 20.51% 0.04% swapper [kernel.kallsyms] [k] ret_from_intr | + 20.47% 0.12% swapper [kernel.kallsyms] [k] do_IRQ | + 19.30% 0.07% swapper [kernel.kallsyms] [k] irq_exit | + 19.18% 0.07% apache2 [kernel.kallsyms] [k] entry_SYSCALL_64_fastpath | + 18.80% 0.17% swapper [kernel.kallsyms] [k] __do_softirq | + 16.45% 0.11% swapper [kernel.kallsyms] [k] net_rx_action | + 16.25% 0.43% swapper [kernel.kallsyms] [k] be_poll | + 14.74% 0.21% swapper [kernel.kallsyms] [k] be_process_rx | + 13.61% 0.07% swapper [kernel.kallsyms] [k] napi_gro_frags | + 12.58% 0.04% swapper [kernel.kallsyms] [k] netif_receive_skb_internal | + 12.48% 0.03% swapper [kernel.kallsyms] [k] __netif_receive_skb | + 12.42% 0.24% swapper [kernel.kallsyms] [k] __netif_receive_skb_core | + 12.41% 0.00% apache2 [unknown] [k] 0x7f27983b5028 | + 12.41% 0.00% apache2 [unknown] [k] 0x7f2798369028 | + 11.49% 0.16% swapper [kernel.kallsyms] [k] ip_rcv | + 11.29% 0.09% swapper [kernel.kallsyms] [k] ip_rcv_finish | + 10.77% 0.05% swapper [kernel.kallsyms] [k] ip_local_deliver | + 10.70% 0.06% swapper [kernel.kallsyms] [k] ip_local_deliver_finish | + 10.55% 0.22% swapper [kernel.kallsyms] [k] tcp_v4_rcv | + 10.10% 0.00% apache2 [unknown] [k] | + 10.01% 0.04% swapper [kernel.kallsyms] [k] tcp_v4_do_rcv Expanding in a few of those, you'll see: | - 55.24% 0.20% swapper [kernel.kallsyms] [k] cpu_startup_entry |- 55.04% cpu_startup_entry | - 52.98% call_cpuidle |
[Kernel-packages] [Bug 1579278] Re: Keep powersave CPU frequency scaling governor for CPUs that support intel_pstate
Using kernel 4.8-rc5 I did some energy tests, using a SpecPower simulator that I made one time. I reused intel_pstate powersave data from a previous test. Reference: http://marc.info/?l=linux-kernel&m=147326197513427&w=2 Big numbers are Joules (package Joules from turbostat) Smaller numbers are watts, 1500 Seconds test run time. X ~= 18.2% on a real SpecPower. Load: idle0.5XX 2X 3X 4X 5X 100% powersave 570811037 16075 29147 45913 61165 76650 81695 3.817.3610.72 19.43 30.61 40.78 51.10 54.46 performance 574914536 22841 38475 51757 64425 77000 81733 3.839.6915.23 25.65 34.50 42.95 51.33 54.49 0.7%31.7% 42.1% 32.0% 12.7% 5.3%0.5%0.0% Note: A real SpecPower test-bed tends to get less dramatic results than my simulator. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1579278 Title: Keep powersave CPU frequency scaling governor for CPUs that support intel_pstate Status in linux package in Ubuntu: Invalid Status in systemd package in Ubuntu: Fix Released Status in sysvinit package in Ubuntu: Invalid Status in linux source package in Xenial: Invalid Status in systemd source package in Xenial: Invalid Status in sysvinit source package in Xenial: Triaged Bug description: Hi, With the new Ubuntu archive servers, we saw constantly high load and after some tinkering, we found that it was mostly CPUs being woken up to see if they should enter idle states. Changing the CPU frequency scaling governor to "performance" saw a considerable drop. Perf report using the following commands: | perf record -g -a sleep 10 | perf report | Samples: 287K of event 'cycles:pp', Event count (approx.): 124776998906 | Children Self Command Shared Object Symbol | + 55.24% 0.20% swapper [kernel.kallsyms] [k] cpu_startup_entry | + 53.51% 0.00% swapper [kernel.kallsyms] [k] start_secondary | + 53.02% 0.08% swapper [kernel.kallsyms] [k] call_cpuidle | + 52.94% 0.02% swapper [kernel.kallsyms] [k] cpuidle_enter | + 31.81% 0.67% swapper [kernel.kallsyms] [k] cpuidle_enter_state | + 29.59% 0.12% swapper [kernel.kallsyms] [k] acpi_idle_enter | + 29.45% 0.05% swapper [kernel.kallsyms] [k] acpi_idle_do_entry | + 29.43%29.43% swapper [kernel.kallsyms] [k] acpi_processor_ffh_cstate_enter | + 20.51% 0.04% swapper [kernel.kallsyms] [k] ret_from_intr | + 20.47% 0.12% swapper [kernel.kallsyms] [k] do_IRQ | + 19.30% 0.07% swapper [kernel.kallsyms] [k] irq_exit | + 19.18% 0.07% apache2 [kernel.kallsyms] [k] entry_SYSCALL_64_fastpath | + 18.80% 0.17% swapper [kernel.kallsyms] [k] __do_softirq | + 16.45% 0.11% swapper [kernel.kallsyms] [k] net_rx_action | + 16.25% 0.43% swapper [kernel.kallsyms] [k] be_poll | + 14.74% 0.21% swapper [kernel.kallsyms] [k] be_process_rx | + 13.61% 0.07% swapper [kernel.kallsyms] [k] napi_gro_frags | + 12.58% 0.04% swapper [kernel.kallsyms] [k] netif_receive_skb_internal | + 12.48% 0.03% swapper [kernel.kallsyms] [k] __netif_receive_skb | + 12.42% 0.24% swapper [kernel.kallsyms] [k] __netif_receive_skb_core | + 12.41% 0.00% apache2 [unknown] [k] 0x7f27983b5028 | + 12.41% 0.00% apache2 [unknown] [k] 0x7f2798369028 | + 11.49% 0.16% swapper [kernel.kallsyms] [k] ip_rcv | + 11.29% 0.09% swapper [kernel.kallsyms] [k] ip_rcv_finish | + 10.77% 0.05% swapper [kernel.kallsyms] [k] ip_local_deliver | + 10.70% 0.06% swapper [kernel.kallsyms] [k] ip_local_deliver_finish | + 10.55% 0.22% swapper [kernel.kallsyms] [k] tcp_v4_rcv | + 10.10% 0.00% apache2 [unknown] [k] | + 10.01% 0.04% swapper [kernel.kallsyms] [k] tcp_v4_do_rcv Expanding in a few of those, you'll see: | - 55.24% 0.20% swapper [kernel.kallsyms] [k] cpu_startup_entry |- 55.04% cpu_startup_entry | - 52.98% call_cpuidle | + 52.93% cpuidle_enter | + 0.00% ret_from_intr |0.00% cpuidle_enter_state |0.00% irq_ent
[Kernel-packages] [Bug 1579278] Re: Keep powersave CPU frequency scaling governor for CPUs that support intel_pstate
The table formatting got messed up, trying again: Load: idle0.5XX 2X 3X 4X 5X 100% powersave 570811037 16075 29147 45913 61165 76650 81695 3.817.3610.72 19.43 30.61 40.78 51.10 54.46 performance 574914536 22841 38475 51757 64425 77000 81733 3.839.6915.23 25.65 34.50 42.95 51.33 54.49 0.7%31.7% 42.1% 32.0% 12.7% 5.3%0.5%0.0% -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1579278 Title: Keep powersave CPU frequency scaling governor for CPUs that support intel_pstate Status in linux package in Ubuntu: Invalid Status in systemd package in Ubuntu: Fix Released Status in sysvinit package in Ubuntu: Invalid Status in linux source package in Xenial: Invalid Status in systemd source package in Xenial: Invalid Status in sysvinit source package in Xenial: Triaged Bug description: Hi, With the new Ubuntu archive servers, we saw constantly high load and after some tinkering, we found that it was mostly CPUs being woken up to see if they should enter idle states. Changing the CPU frequency scaling governor to "performance" saw a considerable drop. Perf report using the following commands: | perf record -g -a sleep 10 | perf report | Samples: 287K of event 'cycles:pp', Event count (approx.): 124776998906 | Children Self Command Shared Object Symbol | + 55.24% 0.20% swapper [kernel.kallsyms] [k] cpu_startup_entry | + 53.51% 0.00% swapper [kernel.kallsyms] [k] start_secondary | + 53.02% 0.08% swapper [kernel.kallsyms] [k] call_cpuidle | + 52.94% 0.02% swapper [kernel.kallsyms] [k] cpuidle_enter | + 31.81% 0.67% swapper [kernel.kallsyms] [k] cpuidle_enter_state | + 29.59% 0.12% swapper [kernel.kallsyms] [k] acpi_idle_enter | + 29.45% 0.05% swapper [kernel.kallsyms] [k] acpi_idle_do_entry | + 29.43%29.43% swapper [kernel.kallsyms] [k] acpi_processor_ffh_cstate_enter | + 20.51% 0.04% swapper [kernel.kallsyms] [k] ret_from_intr | + 20.47% 0.12% swapper [kernel.kallsyms] [k] do_IRQ | + 19.30% 0.07% swapper [kernel.kallsyms] [k] irq_exit | + 19.18% 0.07% apache2 [kernel.kallsyms] [k] entry_SYSCALL_64_fastpath | + 18.80% 0.17% swapper [kernel.kallsyms] [k] __do_softirq | + 16.45% 0.11% swapper [kernel.kallsyms] [k] net_rx_action | + 16.25% 0.43% swapper [kernel.kallsyms] [k] be_poll | + 14.74% 0.21% swapper [kernel.kallsyms] [k] be_process_rx | + 13.61% 0.07% swapper [kernel.kallsyms] [k] napi_gro_frags | + 12.58% 0.04% swapper [kernel.kallsyms] [k] netif_receive_skb_internal | + 12.48% 0.03% swapper [kernel.kallsyms] [k] __netif_receive_skb | + 12.42% 0.24% swapper [kernel.kallsyms] [k] __netif_receive_skb_core | + 12.41% 0.00% apache2 [unknown] [k] 0x7f27983b5028 | + 12.41% 0.00% apache2 [unknown] [k] 0x7f2798369028 | + 11.49% 0.16% swapper [kernel.kallsyms] [k] ip_rcv | + 11.29% 0.09% swapper [kernel.kallsyms] [k] ip_rcv_finish | + 10.77% 0.05% swapper [kernel.kallsyms] [k] ip_local_deliver | + 10.70% 0.06% swapper [kernel.kallsyms] [k] ip_local_deliver_finish | + 10.55% 0.22% swapper [kernel.kallsyms] [k] tcp_v4_rcv | + 10.10% 0.00% apache2 [unknown] [k] | + 10.01% 0.04% swapper [kernel.kallsyms] [k] tcp_v4_do_rcv Expanding in a few of those, you'll see: | - 55.24% 0.20% swapper [kernel.kallsyms] [k] cpu_startup_entry |- 55.04% cpu_startup_entry | - 52.98% call_cpuidle | + 52.93% cpuidle_enter | + 0.00% ret_from_intr |0.00% cpuidle_enter_state |0.00% irq_entries_start | + 1.14% cpuidle_select | + 0.47% schedule_preempt_disabled | 0.10% rcu_idle_enter | 0.09% rcu_idle_exit | + 0.05% ret_from_intr | + 0.05% tick_nohz_idle_enter | + 0.04% arch_cpu_idle_enter | 0.02% cpuidle_enter | 0.02% tick_check_broadcast_expired | + 0.01% cpuidle_reflect | 0.01% menu_reflec
[Kernel-packages] [Bug 1579278] Re: Keep powersave CPU frequency scaling governor for CPUs that support intel_pstate
Thanks Doug! Ack, I'll change it to use "powersave" again then. ** Changed in: systemd (Ubuntu) Status: Fix Released => In Progress -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1579278 Title: Keep powersave CPU frequency scaling governor for CPUs that support intel_pstate Status in linux package in Ubuntu: Invalid Status in systemd package in Ubuntu: In Progress Status in sysvinit package in Ubuntu: Invalid Status in linux source package in Xenial: Invalid Status in systemd source package in Xenial: Invalid Status in sysvinit source package in Xenial: Triaged Bug description: Hi, With the new Ubuntu archive servers, we saw constantly high load and after some tinkering, we found that it was mostly CPUs being woken up to see if they should enter idle states. Changing the CPU frequency scaling governor to "performance" saw a considerable drop. Perf report using the following commands: | perf record -g -a sleep 10 | perf report | Samples: 287K of event 'cycles:pp', Event count (approx.): 124776998906 | Children Self Command Shared Object Symbol | + 55.24% 0.20% swapper [kernel.kallsyms] [k] cpu_startup_entry | + 53.51% 0.00% swapper [kernel.kallsyms] [k] start_secondary | + 53.02% 0.08% swapper [kernel.kallsyms] [k] call_cpuidle | + 52.94% 0.02% swapper [kernel.kallsyms] [k] cpuidle_enter | + 31.81% 0.67% swapper [kernel.kallsyms] [k] cpuidle_enter_state | + 29.59% 0.12% swapper [kernel.kallsyms] [k] acpi_idle_enter | + 29.45% 0.05% swapper [kernel.kallsyms] [k] acpi_idle_do_entry | + 29.43%29.43% swapper [kernel.kallsyms] [k] acpi_processor_ffh_cstate_enter | + 20.51% 0.04% swapper [kernel.kallsyms] [k] ret_from_intr | + 20.47% 0.12% swapper [kernel.kallsyms] [k] do_IRQ | + 19.30% 0.07% swapper [kernel.kallsyms] [k] irq_exit | + 19.18% 0.07% apache2 [kernel.kallsyms] [k] entry_SYSCALL_64_fastpath | + 18.80% 0.17% swapper [kernel.kallsyms] [k] __do_softirq | + 16.45% 0.11% swapper [kernel.kallsyms] [k] net_rx_action | + 16.25% 0.43% swapper [kernel.kallsyms] [k] be_poll | + 14.74% 0.21% swapper [kernel.kallsyms] [k] be_process_rx | + 13.61% 0.07% swapper [kernel.kallsyms] [k] napi_gro_frags | + 12.58% 0.04% swapper [kernel.kallsyms] [k] netif_receive_skb_internal | + 12.48% 0.03% swapper [kernel.kallsyms] [k] __netif_receive_skb | + 12.42% 0.24% swapper [kernel.kallsyms] [k] __netif_receive_skb_core | + 12.41% 0.00% apache2 [unknown] [k] 0x7f27983b5028 | + 12.41% 0.00% apache2 [unknown] [k] 0x7f2798369028 | + 11.49% 0.16% swapper [kernel.kallsyms] [k] ip_rcv | + 11.29% 0.09% swapper [kernel.kallsyms] [k] ip_rcv_finish | + 10.77% 0.05% swapper [kernel.kallsyms] [k] ip_local_deliver | + 10.70% 0.06% swapper [kernel.kallsyms] [k] ip_local_deliver_finish | + 10.55% 0.22% swapper [kernel.kallsyms] [k] tcp_v4_rcv | + 10.10% 0.00% apache2 [unknown] [k] | + 10.01% 0.04% swapper [kernel.kallsyms] [k] tcp_v4_do_rcv Expanding in a few of those, you'll see: | - 55.24% 0.20% swapper [kernel.kallsyms] [k] cpu_startup_entry |- 55.04% cpu_startup_entry | - 52.98% call_cpuidle | + 52.93% cpuidle_enter | + 0.00% ret_from_intr |0.00% cpuidle_enter_state |0.00% irq_entries_start | + 1.14% cpuidle_select | + 0.47% schedule_preempt_disabled | 0.10% rcu_idle_enter | 0.09% rcu_idle_exit | + 0.05% ret_from_intr | + 0.05% tick_nohz_idle_enter | + 0.04% arch_cpu_idle_enter | 0.02% cpuidle_enter | 0.02% tick_check_broadcast_expired | + 0.01% cpuidle_reflect | 0.01% menu_reflect | 0.01% atomic_notifier_call_chain | 0.01% local_touch_nmi | 0.01% cpuidle_not_available | 0.01% menu_select | 0.01% cpuidle_get_cpu_driver | + 0.01% tick_nohz_idle_exit | + 0.01% sched_ttwu_pending | 0.00% set_cpu_sd_state_idle | 0.00% native_irq_return_iret | 0.00% schedule |
[Kernel-packages] [Bug 1579278] Re: Keep powersave CPU frequency scaling governor for CPUs that support intel_pstate
pitti, can we please have a config option to disable "ondemand"? As originally reported, one some workloads, it seems setting to powersave/ondemand causes high load with CPUs checking to see if they need to enter powersave state. While the original report included perf report for the acpi-cpufreq driver, it was also seen with intel_pstate (and was why we decided to try switch to acpi-cpufreq). -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1579278 Title: Keep powersave CPU frequency scaling governor for CPUs that support intel_pstate Status in linux package in Ubuntu: Invalid Status in systemd package in Ubuntu: In Progress Status in sysvinit package in Ubuntu: Invalid Status in linux source package in Xenial: Invalid Status in systemd source package in Xenial: Invalid Status in sysvinit source package in Xenial: Triaged Bug description: Hi, With the new Ubuntu archive servers, we saw constantly high load and after some tinkering, we found that it was mostly CPUs being woken up to see if they should enter idle states. Changing the CPU frequency scaling governor to "performance" saw a considerable drop. Perf report using the following commands: | perf record -g -a sleep 10 | perf report | Samples: 287K of event 'cycles:pp', Event count (approx.): 124776998906 | Children Self Command Shared Object Symbol | + 55.24% 0.20% swapper [kernel.kallsyms] [k] cpu_startup_entry | + 53.51% 0.00% swapper [kernel.kallsyms] [k] start_secondary | + 53.02% 0.08% swapper [kernel.kallsyms] [k] call_cpuidle | + 52.94% 0.02% swapper [kernel.kallsyms] [k] cpuidle_enter | + 31.81% 0.67% swapper [kernel.kallsyms] [k] cpuidle_enter_state | + 29.59% 0.12% swapper [kernel.kallsyms] [k] acpi_idle_enter | + 29.45% 0.05% swapper [kernel.kallsyms] [k] acpi_idle_do_entry | + 29.43%29.43% swapper [kernel.kallsyms] [k] acpi_processor_ffh_cstate_enter | + 20.51% 0.04% swapper [kernel.kallsyms] [k] ret_from_intr | + 20.47% 0.12% swapper [kernel.kallsyms] [k] do_IRQ | + 19.30% 0.07% swapper [kernel.kallsyms] [k] irq_exit | + 19.18% 0.07% apache2 [kernel.kallsyms] [k] entry_SYSCALL_64_fastpath | + 18.80% 0.17% swapper [kernel.kallsyms] [k] __do_softirq | + 16.45% 0.11% swapper [kernel.kallsyms] [k] net_rx_action | + 16.25% 0.43% swapper [kernel.kallsyms] [k] be_poll | + 14.74% 0.21% swapper [kernel.kallsyms] [k] be_process_rx | + 13.61% 0.07% swapper [kernel.kallsyms] [k] napi_gro_frags | + 12.58% 0.04% swapper [kernel.kallsyms] [k] netif_receive_skb_internal | + 12.48% 0.03% swapper [kernel.kallsyms] [k] __netif_receive_skb | + 12.42% 0.24% swapper [kernel.kallsyms] [k] __netif_receive_skb_core | + 12.41% 0.00% apache2 [unknown] [k] 0x7f27983b5028 | + 12.41% 0.00% apache2 [unknown] [k] 0x7f2798369028 | + 11.49% 0.16% swapper [kernel.kallsyms] [k] ip_rcv | + 11.29% 0.09% swapper [kernel.kallsyms] [k] ip_rcv_finish | + 10.77% 0.05% swapper [kernel.kallsyms] [k] ip_local_deliver | + 10.70% 0.06% swapper [kernel.kallsyms] [k] ip_local_deliver_finish | + 10.55% 0.22% swapper [kernel.kallsyms] [k] tcp_v4_rcv | + 10.10% 0.00% apache2 [unknown] [k] | + 10.01% 0.04% swapper [kernel.kallsyms] [k] tcp_v4_do_rcv Expanding in a few of those, you'll see: | - 55.24% 0.20% swapper [kernel.kallsyms] [k] cpu_startup_entry |- 55.04% cpu_startup_entry | - 52.98% call_cpuidle | + 52.93% cpuidle_enter | + 0.00% ret_from_intr |0.00% cpuidle_enter_state |0.00% irq_entries_start | + 1.14% cpuidle_select | + 0.47% schedule_preempt_disabled | 0.10% rcu_idle_enter | 0.09% rcu_idle_exit | + 0.05% ret_from_intr | + 0.05% tick_nohz_idle_enter | + 0.04% arch_cpu_idle_enter | 0.02% cpuidle_enter | 0.02% tick_check_broadcast_expired | + 0.01% cpuidle_reflect | 0.01% menu_reflect | 0.01% atomic_notifier_call_chain | 0.01% local_touch_nmi | 0.01% cpuidle_not_availabl
[Kernel-packages] [Bug 1579278] Re: Keep powersave CPU frequency scaling governor for CPUs that support intel_pstate
With https://anonscm.debian.org/cgit/pkg- systemd/systemd.git/commit/?id=94a70093 it uses powersave on intel_pstate again, like in xenial. > pitti, can we please have a config option to disable "ondemand"? Indeed, right now it is statically enabled, so it can only be disabled with "systemctl mask ondemand". I'll make it dynamically enabled so that it can be disabled with the usual systemctl enable/disable. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1579278 Title: Keep powersave CPU frequency scaling governor for CPUs that support intel_pstate Status in linux package in Ubuntu: Invalid Status in systemd package in Ubuntu: In Progress Status in sysvinit package in Ubuntu: Invalid Status in linux source package in Xenial: Invalid Status in systemd source package in Xenial: Invalid Status in sysvinit source package in Xenial: Triaged Bug description: Hi, With the new Ubuntu archive servers, we saw constantly high load and after some tinkering, we found that it was mostly CPUs being woken up to see if they should enter idle states. Changing the CPU frequency scaling governor to "performance" saw a considerable drop. Perf report using the following commands: | perf record -g -a sleep 10 | perf report | Samples: 287K of event 'cycles:pp', Event count (approx.): 124776998906 | Children Self Command Shared Object Symbol | + 55.24% 0.20% swapper [kernel.kallsyms] [k] cpu_startup_entry | + 53.51% 0.00% swapper [kernel.kallsyms] [k] start_secondary | + 53.02% 0.08% swapper [kernel.kallsyms] [k] call_cpuidle | + 52.94% 0.02% swapper [kernel.kallsyms] [k] cpuidle_enter | + 31.81% 0.67% swapper [kernel.kallsyms] [k] cpuidle_enter_state | + 29.59% 0.12% swapper [kernel.kallsyms] [k] acpi_idle_enter | + 29.45% 0.05% swapper [kernel.kallsyms] [k] acpi_idle_do_entry | + 29.43%29.43% swapper [kernel.kallsyms] [k] acpi_processor_ffh_cstate_enter | + 20.51% 0.04% swapper [kernel.kallsyms] [k] ret_from_intr | + 20.47% 0.12% swapper [kernel.kallsyms] [k] do_IRQ | + 19.30% 0.07% swapper [kernel.kallsyms] [k] irq_exit | + 19.18% 0.07% apache2 [kernel.kallsyms] [k] entry_SYSCALL_64_fastpath | + 18.80% 0.17% swapper [kernel.kallsyms] [k] __do_softirq | + 16.45% 0.11% swapper [kernel.kallsyms] [k] net_rx_action | + 16.25% 0.43% swapper [kernel.kallsyms] [k] be_poll | + 14.74% 0.21% swapper [kernel.kallsyms] [k] be_process_rx | + 13.61% 0.07% swapper [kernel.kallsyms] [k] napi_gro_frags | + 12.58% 0.04% swapper [kernel.kallsyms] [k] netif_receive_skb_internal | + 12.48% 0.03% swapper [kernel.kallsyms] [k] __netif_receive_skb | + 12.42% 0.24% swapper [kernel.kallsyms] [k] __netif_receive_skb_core | + 12.41% 0.00% apache2 [unknown] [k] 0x7f27983b5028 | + 12.41% 0.00% apache2 [unknown] [k] 0x7f2798369028 | + 11.49% 0.16% swapper [kernel.kallsyms] [k] ip_rcv | + 11.29% 0.09% swapper [kernel.kallsyms] [k] ip_rcv_finish | + 10.77% 0.05% swapper [kernel.kallsyms] [k] ip_local_deliver | + 10.70% 0.06% swapper [kernel.kallsyms] [k] ip_local_deliver_finish | + 10.55% 0.22% swapper [kernel.kallsyms] [k] tcp_v4_rcv | + 10.10% 0.00% apache2 [unknown] [k] | + 10.01% 0.04% swapper [kernel.kallsyms] [k] tcp_v4_do_rcv Expanding in a few of those, you'll see: | - 55.24% 0.20% swapper [kernel.kallsyms] [k] cpu_startup_entry |- 55.04% cpu_startup_entry | - 52.98% call_cpuidle | + 52.93% cpuidle_enter | + 0.00% ret_from_intr |0.00% cpuidle_enter_state |0.00% irq_entries_start | + 1.14% cpuidle_select | + 0.47% schedule_preempt_disabled | 0.10% rcu_idle_enter | 0.09% rcu_idle_exit | + 0.05% ret_from_intr | + 0.05% tick_nohz_idle_enter | + 0.04% arch_cpu_idle_enter | 0.02% cpuidle_enter | 0.02% tick_check_broadcast_expired | + 0.01% cpuidle_reflect | 0.01% menu_reflect | 0.01% atomic_notifier_call_chain | 0.01% local_touch_nmi | 0.01% cpuidle
[Kernel-packages] [Bug 1579278] Re: Keep powersave CPU frequency scaling governor for CPUs that support intel_pstate
> I'll make it dynamically enabled https://anonscm.debian.org/cgit/pkg- systemd/systemd.git/commit/?id=19e67c70 ** Changed in: systemd (Ubuntu) Status: In Progress => Fix Committed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1579278 Title: Keep powersave CPU frequency scaling governor for CPUs that support intel_pstate Status in linux package in Ubuntu: Invalid Status in systemd package in Ubuntu: Fix Committed Status in sysvinit package in Ubuntu: Invalid Status in linux source package in Xenial: Invalid Status in systemd source package in Xenial: Invalid Status in sysvinit source package in Xenial: Triaged Bug description: Hi, With the new Ubuntu archive servers, we saw constantly high load and after some tinkering, we found that it was mostly CPUs being woken up to see if they should enter idle states. Changing the CPU frequency scaling governor to "performance" saw a considerable drop. Perf report using the following commands: | perf record -g -a sleep 10 | perf report | Samples: 287K of event 'cycles:pp', Event count (approx.): 124776998906 | Children Self Command Shared Object Symbol | + 55.24% 0.20% swapper [kernel.kallsyms] [k] cpu_startup_entry | + 53.51% 0.00% swapper [kernel.kallsyms] [k] start_secondary | + 53.02% 0.08% swapper [kernel.kallsyms] [k] call_cpuidle | + 52.94% 0.02% swapper [kernel.kallsyms] [k] cpuidle_enter | + 31.81% 0.67% swapper [kernel.kallsyms] [k] cpuidle_enter_state | + 29.59% 0.12% swapper [kernel.kallsyms] [k] acpi_idle_enter | + 29.45% 0.05% swapper [kernel.kallsyms] [k] acpi_idle_do_entry | + 29.43%29.43% swapper [kernel.kallsyms] [k] acpi_processor_ffh_cstate_enter | + 20.51% 0.04% swapper [kernel.kallsyms] [k] ret_from_intr | + 20.47% 0.12% swapper [kernel.kallsyms] [k] do_IRQ | + 19.30% 0.07% swapper [kernel.kallsyms] [k] irq_exit | + 19.18% 0.07% apache2 [kernel.kallsyms] [k] entry_SYSCALL_64_fastpath | + 18.80% 0.17% swapper [kernel.kallsyms] [k] __do_softirq | + 16.45% 0.11% swapper [kernel.kallsyms] [k] net_rx_action | + 16.25% 0.43% swapper [kernel.kallsyms] [k] be_poll | + 14.74% 0.21% swapper [kernel.kallsyms] [k] be_process_rx | + 13.61% 0.07% swapper [kernel.kallsyms] [k] napi_gro_frags | + 12.58% 0.04% swapper [kernel.kallsyms] [k] netif_receive_skb_internal | + 12.48% 0.03% swapper [kernel.kallsyms] [k] __netif_receive_skb | + 12.42% 0.24% swapper [kernel.kallsyms] [k] __netif_receive_skb_core | + 12.41% 0.00% apache2 [unknown] [k] 0x7f27983b5028 | + 12.41% 0.00% apache2 [unknown] [k] 0x7f2798369028 | + 11.49% 0.16% swapper [kernel.kallsyms] [k] ip_rcv | + 11.29% 0.09% swapper [kernel.kallsyms] [k] ip_rcv_finish | + 10.77% 0.05% swapper [kernel.kallsyms] [k] ip_local_deliver | + 10.70% 0.06% swapper [kernel.kallsyms] [k] ip_local_deliver_finish | + 10.55% 0.22% swapper [kernel.kallsyms] [k] tcp_v4_rcv | + 10.10% 0.00% apache2 [unknown] [k] | + 10.01% 0.04% swapper [kernel.kallsyms] [k] tcp_v4_do_rcv Expanding in a few of those, you'll see: | - 55.24% 0.20% swapper [kernel.kallsyms] [k] cpu_startup_entry |- 55.04% cpu_startup_entry | - 52.98% call_cpuidle | + 52.93% cpuidle_enter | + 0.00% ret_from_intr |0.00% cpuidle_enter_state |0.00% irq_entries_start | + 1.14% cpuidle_select | + 0.47% schedule_preempt_disabled | 0.10% rcu_idle_enter | 0.09% rcu_idle_exit | + 0.05% ret_from_intr | + 0.05% tick_nohz_idle_enter | + 0.04% arch_cpu_idle_enter | 0.02% cpuidle_enter | 0.02% tick_check_broadcast_expired | + 0.01% cpuidle_reflect | 0.01% menu_reflect | 0.01% atomic_notifier_call_chain | 0.01% local_touch_nmi | 0.01% cpuidle_not_available | 0.01% menu_select | 0.01% cpuidle_get_cpu_driver | + 0.01% tick_nohz_idle_exit | + 0.01% sched_ttwu_pending | 0.00% set_cpu_sd_state_idle | 0.00% n
[Kernel-packages] [Bug 1579278] Re: Keep powersave CPU frequency scaling governor for CPUs that support intel_pstate
@dsmythies, sorry, missed your question. We're actually seeing performance issues/load on a couple of our Ubuntu Archive servers (archive.ubuntu.com). They're high traffic servers with 10GbE NICs with just apache2 serving .deb packages from disk. We've recently upgraded a few to 4.11.0-14-generic from linux-image-generic-hwe-16.04-edge (for TCP BBR). ** Attachment added: "cpu usage 2017-08-23 02-11-45" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1579278/+attachment/4969405/+files/cpu%20usage%202017-08-23%2002-11-45.png -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1579278 Title: Keep powersave CPU frequency scaling governor for CPUs that support intel_pstate Status in linux package in Ubuntu: Invalid Status in systemd package in Ubuntu: Fix Released Status in sysvinit package in Ubuntu: Invalid Status in linux source package in Xenial: Invalid Status in systemd source package in Xenial: Invalid Status in sysvinit source package in Xenial: Triaged Bug description: Hi, With the new Ubuntu archive servers, we saw constantly high load and after some tinkering, we found that it was mostly CPUs being woken up to see if they should enter idle states. Changing the CPU frequency scaling governor to "performance" saw a considerable drop. Perf report using the following commands: | perf record -g -a sleep 10 | perf report | Samples: 287K of event 'cycles:pp', Event count (approx.): 124776998906 | Children Self Command Shared Object Symbol | + 55.24% 0.20% swapper [kernel.kallsyms] [k] cpu_startup_entry | + 53.51% 0.00% swapper [kernel.kallsyms] [k] start_secondary | + 53.02% 0.08% swapper [kernel.kallsyms] [k] call_cpuidle | + 52.94% 0.02% swapper [kernel.kallsyms] [k] cpuidle_enter | + 31.81% 0.67% swapper [kernel.kallsyms] [k] cpuidle_enter_state | + 29.59% 0.12% swapper [kernel.kallsyms] [k] acpi_idle_enter | + 29.45% 0.05% swapper [kernel.kallsyms] [k] acpi_idle_do_entry | + 29.43%29.43% swapper [kernel.kallsyms] [k] acpi_processor_ffh_cstate_enter | + 20.51% 0.04% swapper [kernel.kallsyms] [k] ret_from_intr | + 20.47% 0.12% swapper [kernel.kallsyms] [k] do_IRQ | + 19.30% 0.07% swapper [kernel.kallsyms] [k] irq_exit | + 19.18% 0.07% apache2 [kernel.kallsyms] [k] entry_SYSCALL_64_fastpath | + 18.80% 0.17% swapper [kernel.kallsyms] [k] __do_softirq | + 16.45% 0.11% swapper [kernel.kallsyms] [k] net_rx_action | + 16.25% 0.43% swapper [kernel.kallsyms] [k] be_poll | + 14.74% 0.21% swapper [kernel.kallsyms] [k] be_process_rx | + 13.61% 0.07% swapper [kernel.kallsyms] [k] napi_gro_frags | + 12.58% 0.04% swapper [kernel.kallsyms] [k] netif_receive_skb_internal | + 12.48% 0.03% swapper [kernel.kallsyms] [k] __netif_receive_skb | + 12.42% 0.24% swapper [kernel.kallsyms] [k] __netif_receive_skb_core | + 12.41% 0.00% apache2 [unknown] [k] 0x7f27983b5028 | + 12.41% 0.00% apache2 [unknown] [k] 0x7f2798369028 | + 11.49% 0.16% swapper [kernel.kallsyms] [k] ip_rcv | + 11.29% 0.09% swapper [kernel.kallsyms] [k] ip_rcv_finish | + 10.77% 0.05% swapper [kernel.kallsyms] [k] ip_local_deliver | + 10.70% 0.06% swapper [kernel.kallsyms] [k] ip_local_deliver_finish | + 10.55% 0.22% swapper [kernel.kallsyms] [k] tcp_v4_rcv | + 10.10% 0.00% apache2 [unknown] [k] | + 10.01% 0.04% swapper [kernel.kallsyms] [k] tcp_v4_do_rcv Expanding in a few of those, you'll see: | - 55.24% 0.20% swapper [kernel.kallsyms] [k] cpu_startup_entry |- 55.04% cpu_startup_entry | - 52.98% call_cpuidle | + 52.93% cpuidle_enter | + 0.00% ret_from_intr |0.00% cpuidle_enter_state |0.00% irq_entries_start | + 1.14% cpuidle_select | + 0.47% schedule_preempt_disabled | 0.10% rcu_idle_enter | 0.09% rcu_idle_exit | + 0.05% ret_from_intr | + 0.05% tick_nohz_idle_enter | + 0.04% arch_cpu_idle_enter | 0.02% cpuidle_enter | 0.02% tick_check_broadcast_expired | + 0.01% cpuidle_reflect |
[Kernel-packages] [Bug 1579278] Re: Keep powersave CPU frequency scaling governor for CPUs that support intel_pstate
** Attachment added: "load 15m 2017-08-23 02-10-53" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1579278/+attachment/4969406/+files/load%2015m%202017-08-23%2002-10-53.png -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1579278 Title: Keep powersave CPU frequency scaling governor for CPUs that support intel_pstate Status in linux package in Ubuntu: Invalid Status in systemd package in Ubuntu: Fix Released Status in sysvinit package in Ubuntu: Invalid Status in linux source package in Xenial: Invalid Status in systemd source package in Xenial: Invalid Status in sysvinit source package in Xenial: Triaged Bug description: Hi, With the new Ubuntu archive servers, we saw constantly high load and after some tinkering, we found that it was mostly CPUs being woken up to see if they should enter idle states. Changing the CPU frequency scaling governor to "performance" saw a considerable drop. Perf report using the following commands: | perf record -g -a sleep 10 | perf report | Samples: 287K of event 'cycles:pp', Event count (approx.): 124776998906 | Children Self Command Shared Object Symbol | + 55.24% 0.20% swapper [kernel.kallsyms] [k] cpu_startup_entry | + 53.51% 0.00% swapper [kernel.kallsyms] [k] start_secondary | + 53.02% 0.08% swapper [kernel.kallsyms] [k] call_cpuidle | + 52.94% 0.02% swapper [kernel.kallsyms] [k] cpuidle_enter | + 31.81% 0.67% swapper [kernel.kallsyms] [k] cpuidle_enter_state | + 29.59% 0.12% swapper [kernel.kallsyms] [k] acpi_idle_enter | + 29.45% 0.05% swapper [kernel.kallsyms] [k] acpi_idle_do_entry | + 29.43%29.43% swapper [kernel.kallsyms] [k] acpi_processor_ffh_cstate_enter | + 20.51% 0.04% swapper [kernel.kallsyms] [k] ret_from_intr | + 20.47% 0.12% swapper [kernel.kallsyms] [k] do_IRQ | + 19.30% 0.07% swapper [kernel.kallsyms] [k] irq_exit | + 19.18% 0.07% apache2 [kernel.kallsyms] [k] entry_SYSCALL_64_fastpath | + 18.80% 0.17% swapper [kernel.kallsyms] [k] __do_softirq | + 16.45% 0.11% swapper [kernel.kallsyms] [k] net_rx_action | + 16.25% 0.43% swapper [kernel.kallsyms] [k] be_poll | + 14.74% 0.21% swapper [kernel.kallsyms] [k] be_process_rx | + 13.61% 0.07% swapper [kernel.kallsyms] [k] napi_gro_frags | + 12.58% 0.04% swapper [kernel.kallsyms] [k] netif_receive_skb_internal | + 12.48% 0.03% swapper [kernel.kallsyms] [k] __netif_receive_skb | + 12.42% 0.24% swapper [kernel.kallsyms] [k] __netif_receive_skb_core | + 12.41% 0.00% apache2 [unknown] [k] 0x7f27983b5028 | + 12.41% 0.00% apache2 [unknown] [k] 0x7f2798369028 | + 11.49% 0.16% swapper [kernel.kallsyms] [k] ip_rcv | + 11.29% 0.09% swapper [kernel.kallsyms] [k] ip_rcv_finish | + 10.77% 0.05% swapper [kernel.kallsyms] [k] ip_local_deliver | + 10.70% 0.06% swapper [kernel.kallsyms] [k] ip_local_deliver_finish | + 10.55% 0.22% swapper [kernel.kallsyms] [k] tcp_v4_rcv | + 10.10% 0.00% apache2 [unknown] [k] | + 10.01% 0.04% swapper [kernel.kallsyms] [k] tcp_v4_do_rcv Expanding in a few of those, you'll see: | - 55.24% 0.20% swapper [kernel.kallsyms] [k] cpu_startup_entry |- 55.04% cpu_startup_entry | - 52.98% call_cpuidle | + 52.93% cpuidle_enter | + 0.00% ret_from_intr |0.00% cpuidle_enter_state |0.00% irq_entries_start | + 1.14% cpuidle_select | + 0.47% schedule_preempt_disabled | 0.10% rcu_idle_enter | 0.09% rcu_idle_exit | + 0.05% ret_from_intr | + 0.05% tick_nohz_idle_enter | + 0.04% arch_cpu_idle_enter | 0.02% cpuidle_enter | 0.02% tick_check_broadcast_expired | + 0.01% cpuidle_reflect | 0.01% menu_reflect | 0.01% atomic_notifier_call_chain | 0.01% local_touch_nmi | 0.01% cpuidle_not_available | 0.01% menu_select | 0.01% cpuidle_get_cpu_driver | + 0.01% tick_nohz_idle_exit | + 0.01% sched_ttwu_pending | 0.00% set_cpu_sd_state_idle | 0.00% native_ir
[Kernel-packages] [Bug 1579278] Re: Keep powersave CPU frequency scaling governor for CPUs that support intel_pstate
NOTE: That these are systems using the pcc-cpufreq and acpi-cpufreq drivers. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1579278 Title: Keep powersave CPU frequency scaling governor for CPUs that support intel_pstate Status in linux package in Ubuntu: Invalid Status in systemd package in Ubuntu: Fix Released Status in sysvinit package in Ubuntu: Invalid Status in linux source package in Xenial: Invalid Status in systemd source package in Xenial: Invalid Status in sysvinit source package in Xenial: Triaged Bug description: Hi, With the new Ubuntu archive servers, we saw constantly high load and after some tinkering, we found that it was mostly CPUs being woken up to see if they should enter idle states. Changing the CPU frequency scaling governor to "performance" saw a considerable drop. Perf report using the following commands: | perf record -g -a sleep 10 | perf report | Samples: 287K of event 'cycles:pp', Event count (approx.): 124776998906 | Children Self Command Shared Object Symbol | + 55.24% 0.20% swapper [kernel.kallsyms] [k] cpu_startup_entry | + 53.51% 0.00% swapper [kernel.kallsyms] [k] start_secondary | + 53.02% 0.08% swapper [kernel.kallsyms] [k] call_cpuidle | + 52.94% 0.02% swapper [kernel.kallsyms] [k] cpuidle_enter | + 31.81% 0.67% swapper [kernel.kallsyms] [k] cpuidle_enter_state | + 29.59% 0.12% swapper [kernel.kallsyms] [k] acpi_idle_enter | + 29.45% 0.05% swapper [kernel.kallsyms] [k] acpi_idle_do_entry | + 29.43%29.43% swapper [kernel.kallsyms] [k] acpi_processor_ffh_cstate_enter | + 20.51% 0.04% swapper [kernel.kallsyms] [k] ret_from_intr | + 20.47% 0.12% swapper [kernel.kallsyms] [k] do_IRQ | + 19.30% 0.07% swapper [kernel.kallsyms] [k] irq_exit | + 19.18% 0.07% apache2 [kernel.kallsyms] [k] entry_SYSCALL_64_fastpath | + 18.80% 0.17% swapper [kernel.kallsyms] [k] __do_softirq | + 16.45% 0.11% swapper [kernel.kallsyms] [k] net_rx_action | + 16.25% 0.43% swapper [kernel.kallsyms] [k] be_poll | + 14.74% 0.21% swapper [kernel.kallsyms] [k] be_process_rx | + 13.61% 0.07% swapper [kernel.kallsyms] [k] napi_gro_frags | + 12.58% 0.04% swapper [kernel.kallsyms] [k] netif_receive_skb_internal | + 12.48% 0.03% swapper [kernel.kallsyms] [k] __netif_receive_skb | + 12.42% 0.24% swapper [kernel.kallsyms] [k] __netif_receive_skb_core | + 12.41% 0.00% apache2 [unknown] [k] 0x7f27983b5028 | + 12.41% 0.00% apache2 [unknown] [k] 0x7f2798369028 | + 11.49% 0.16% swapper [kernel.kallsyms] [k] ip_rcv | + 11.29% 0.09% swapper [kernel.kallsyms] [k] ip_rcv_finish | + 10.77% 0.05% swapper [kernel.kallsyms] [k] ip_local_deliver | + 10.70% 0.06% swapper [kernel.kallsyms] [k] ip_local_deliver_finish | + 10.55% 0.22% swapper [kernel.kallsyms] [k] tcp_v4_rcv | + 10.10% 0.00% apache2 [unknown] [k] | + 10.01% 0.04% swapper [kernel.kallsyms] [k] tcp_v4_do_rcv Expanding in a few of those, you'll see: | - 55.24% 0.20% swapper [kernel.kallsyms] [k] cpu_startup_entry |- 55.04% cpu_startup_entry | - 52.98% call_cpuidle | + 52.93% cpuidle_enter | + 0.00% ret_from_intr |0.00% cpuidle_enter_state |0.00% irq_entries_start | + 1.14% cpuidle_select | + 0.47% schedule_preempt_disabled | 0.10% rcu_idle_enter | 0.09% rcu_idle_exit | + 0.05% ret_from_intr | + 0.05% tick_nohz_idle_enter | + 0.04% arch_cpu_idle_enter | 0.02% cpuidle_enter | 0.02% tick_check_broadcast_expired | + 0.01% cpuidle_reflect | 0.01% menu_reflect | 0.01% atomic_notifier_call_chain | 0.01% local_touch_nmi | 0.01% cpuidle_not_available | 0.01% menu_select | 0.01% cpuidle_get_cpu_driver | + 0.01% tick_nohz_idle_exit | + 0.01% sched_ttwu_pending | 0.00% set_cpu_sd_state_idle | 0.00% native_irq_return_iret | 0.00% schedule | + 0.00% arch_cpu_idle_exit | 0.00% __tick_noh
[Kernel-packages] [Bug 1579278] Re: Keep powersave CPU frequency scaling governor for CPUs that support intel_pstate
On my Dell XPS 13 (i7-6500U) the only governors available are performance and powersave. Ubuntu 16.04 seems to set this always to 'powersave' for me. This causes the CPU to clock down to 300-400Mhz when idle. Unfortunately it's not clocking up again when under load sometimes. Setting the governor to performance 'fixes' it. I don't know if this is *this* issue or another one, so happy to open a new issue if you prefer. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1579278 Title: Keep powersave CPU frequency scaling governor for CPUs that support intel_pstate Status in linux package in Ubuntu: Invalid Status in systemd package in Ubuntu: Fix Released Status in sysvinit package in Ubuntu: Invalid Status in linux source package in Xenial: Invalid Status in systemd source package in Xenial: Invalid Status in sysvinit source package in Xenial: Triaged Bug description: Hi, With the new Ubuntu archive servers, we saw constantly high load and after some tinkering, we found that it was mostly CPUs being woken up to see if they should enter idle states. Changing the CPU frequency scaling governor to "performance" saw a considerable drop. Perf report using the following commands: | perf record -g -a sleep 10 | perf report | Samples: 287K of event 'cycles:pp', Event count (approx.): 124776998906 | Children Self Command Shared Object Symbol | + 55.24% 0.20% swapper [kernel.kallsyms] [k] cpu_startup_entry | + 53.51% 0.00% swapper [kernel.kallsyms] [k] start_secondary | + 53.02% 0.08% swapper [kernel.kallsyms] [k] call_cpuidle | + 52.94% 0.02% swapper [kernel.kallsyms] [k] cpuidle_enter | + 31.81% 0.67% swapper [kernel.kallsyms] [k] cpuidle_enter_state | + 29.59% 0.12% swapper [kernel.kallsyms] [k] acpi_idle_enter | + 29.45% 0.05% swapper [kernel.kallsyms] [k] acpi_idle_do_entry | + 29.43%29.43% swapper [kernel.kallsyms] [k] acpi_processor_ffh_cstate_enter | + 20.51% 0.04% swapper [kernel.kallsyms] [k] ret_from_intr | + 20.47% 0.12% swapper [kernel.kallsyms] [k] do_IRQ | + 19.30% 0.07% swapper [kernel.kallsyms] [k] irq_exit | + 19.18% 0.07% apache2 [kernel.kallsyms] [k] entry_SYSCALL_64_fastpath | + 18.80% 0.17% swapper [kernel.kallsyms] [k] __do_softirq | + 16.45% 0.11% swapper [kernel.kallsyms] [k] net_rx_action | + 16.25% 0.43% swapper [kernel.kallsyms] [k] be_poll | + 14.74% 0.21% swapper [kernel.kallsyms] [k] be_process_rx | + 13.61% 0.07% swapper [kernel.kallsyms] [k] napi_gro_frags | + 12.58% 0.04% swapper [kernel.kallsyms] [k] netif_receive_skb_internal | + 12.48% 0.03% swapper [kernel.kallsyms] [k] __netif_receive_skb | + 12.42% 0.24% swapper [kernel.kallsyms] [k] __netif_receive_skb_core | + 12.41% 0.00% apache2 [unknown] [k] 0x7f27983b5028 | + 12.41% 0.00% apache2 [unknown] [k] 0x7f2798369028 | + 11.49% 0.16% swapper [kernel.kallsyms] [k] ip_rcv | + 11.29% 0.09% swapper [kernel.kallsyms] [k] ip_rcv_finish | + 10.77% 0.05% swapper [kernel.kallsyms] [k] ip_local_deliver | + 10.70% 0.06% swapper [kernel.kallsyms] [k] ip_local_deliver_finish | + 10.55% 0.22% swapper [kernel.kallsyms] [k] tcp_v4_rcv | + 10.10% 0.00% apache2 [unknown] [k] | + 10.01% 0.04% swapper [kernel.kallsyms] [k] tcp_v4_do_rcv Expanding in a few of those, you'll see: | - 55.24% 0.20% swapper [kernel.kallsyms] [k] cpu_startup_entry |- 55.04% cpu_startup_entry | - 52.98% call_cpuidle | + 52.93% cpuidle_enter | + 0.00% ret_from_intr |0.00% cpuidle_enter_state |0.00% irq_entries_start | + 1.14% cpuidle_select | + 0.47% schedule_preempt_disabled | 0.10% rcu_idle_enter | 0.09% rcu_idle_exit | + 0.05% ret_from_intr | + 0.05% tick_nohz_idle_enter | + 0.04% arch_cpu_idle_enter | 0.02% cpuidle_enter | 0.02% tick_check_broadcast_expired | + 0.01% cpuidle_reflect | 0.01% menu_reflect | 0.01% atomic_notifier_call_chain | 0.01% local_touch_nmi | 0
[Kernel-packages] [Bug 1579278] Re: Keep powersave CPU frequency scaling governor for CPUs that support intel_pstate
@fish : Yes your issue does not belong here. To figure out where your issue actually belongs, some details are missing: If your stuck at low frequency issue only occurs after a suspend and on battery power, then your issue is covered by: https://bugzilla.kernel.org/show_bug.cgi?id=90041 (and it is not resolved quite yet, even though that is the status). If your issue always occurs and you are not using the proper Dell AC adapter, then that is why. Otherwise, I don't know. Try the acpi_cpufreq CPU frequency scaling driver instead. Also have a look at: https://bugzilla.kernel.org/show_bug.cgi?id=189861 ** Bug watch added: Linux Kernel Bug Tracker #90041 http://bugzilla.kernel.org/show_bug.cgi?id=90041 ** Bug watch added: Linux Kernel Bug Tracker #189861 http://bugzilla.kernel.org/show_bug.cgi?id=189861 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1579278 Title: Keep powersave CPU frequency scaling governor for CPUs that support intel_pstate Status in linux package in Ubuntu: Invalid Status in systemd package in Ubuntu: Fix Released Status in sysvinit package in Ubuntu: Invalid Status in linux source package in Xenial: Invalid Status in systemd source package in Xenial: Invalid Status in sysvinit source package in Xenial: Triaged Bug description: Hi, With the new Ubuntu archive servers, we saw constantly high load and after some tinkering, we found that it was mostly CPUs being woken up to see if they should enter idle states. Changing the CPU frequency scaling governor to "performance" saw a considerable drop. Perf report using the following commands: | perf record -g -a sleep 10 | perf report | Samples: 287K of event 'cycles:pp', Event count (approx.): 124776998906 | Children Self Command Shared Object Symbol | + 55.24% 0.20% swapper [kernel.kallsyms] [k] cpu_startup_entry | + 53.51% 0.00% swapper [kernel.kallsyms] [k] start_secondary | + 53.02% 0.08% swapper [kernel.kallsyms] [k] call_cpuidle | + 52.94% 0.02% swapper [kernel.kallsyms] [k] cpuidle_enter | + 31.81% 0.67% swapper [kernel.kallsyms] [k] cpuidle_enter_state | + 29.59% 0.12% swapper [kernel.kallsyms] [k] acpi_idle_enter | + 29.45% 0.05% swapper [kernel.kallsyms] [k] acpi_idle_do_entry | + 29.43%29.43% swapper [kernel.kallsyms] [k] acpi_processor_ffh_cstate_enter | + 20.51% 0.04% swapper [kernel.kallsyms] [k] ret_from_intr | + 20.47% 0.12% swapper [kernel.kallsyms] [k] do_IRQ | + 19.30% 0.07% swapper [kernel.kallsyms] [k] irq_exit | + 19.18% 0.07% apache2 [kernel.kallsyms] [k] entry_SYSCALL_64_fastpath | + 18.80% 0.17% swapper [kernel.kallsyms] [k] __do_softirq | + 16.45% 0.11% swapper [kernel.kallsyms] [k] net_rx_action | + 16.25% 0.43% swapper [kernel.kallsyms] [k] be_poll | + 14.74% 0.21% swapper [kernel.kallsyms] [k] be_process_rx | + 13.61% 0.07% swapper [kernel.kallsyms] [k] napi_gro_frags | + 12.58% 0.04% swapper [kernel.kallsyms] [k] netif_receive_skb_internal | + 12.48% 0.03% swapper [kernel.kallsyms] [k] __netif_receive_skb | + 12.42% 0.24% swapper [kernel.kallsyms] [k] __netif_receive_skb_core | + 12.41% 0.00% apache2 [unknown] [k] 0x7f27983b5028 | + 12.41% 0.00% apache2 [unknown] [k] 0x7f2798369028 | + 11.49% 0.16% swapper [kernel.kallsyms] [k] ip_rcv | + 11.29% 0.09% swapper [kernel.kallsyms] [k] ip_rcv_finish | + 10.77% 0.05% swapper [kernel.kallsyms] [k] ip_local_deliver | + 10.70% 0.06% swapper [kernel.kallsyms] [k] ip_local_deliver_finish | + 10.55% 0.22% swapper [kernel.kallsyms] [k] tcp_v4_rcv | + 10.10% 0.00% apache2 [unknown] [k] | + 10.01% 0.04% swapper [kernel.kallsyms] [k] tcp_v4_do_rcv Expanding in a few of those, you'll see: | - 55.24% 0.20% swapper [kernel.kallsyms] [k] cpu_startup_entry |- 55.04% cpu_startup_entry | - 52.98% call_cpuidle | + 52.93% cpuidle_enter | + 0.00% ret_from_intr |0.00% cpuidle_enter_state |0.00% irq_entries_start | + 1.14% cpuidle_select | + 0.47% schedule_preempt_disabled |