Re: [PATCH 1/5] random: fix crng_ready() test
On 4/13/18 10:00 AM, Theodore Y. Ts'o wrote: > On Fri, Apr 13, 2018 at 03:05:01PM +0200, Stephan Mueller wrote: >> >> What I would like to point out that more and more folks change to >> getrandom(2). As this call will now unblock much later in the boot cycle, >> these systems see a significant departure from the current system behavior. >> >> E.g. an sshd using getrandom(2) would be ready shortly after the boot >> finishes >> as of now. Now it can be a matter minutes before it responds. Thus, is such >> change in the kernel behavior something for stable? [...] > I was a little worried that on VM's this could end up causing things > to block for a long time, but an experiment on a GCE VM shows that > isn't a problem: > > [0.00] Linux version 4.16.0-rc3-ext4-9-gf6b302ebca85 (tytso@cwcc) > (gcc version 7.3.0 (Debian 7.3.0-15)) #16 SMP Thu Apr 12 16:57:17 EDT 2018 > [1.282220] random: fast init done > [3.987092] random: crng init done > [4.376787] EXT4-fs (sda1): re-mounted. Opts: (null) > > There are some desktops where the "crng_init done" report doesn't > happen until 45-90 seconds into the boot. I don't think I've seen > reports where it takes _minutes_ however. Can you give me some > examples of such cases? On a Photon OS VM running on VMware ESXi, this patch causes a boot speed regression of 5 minutes :-( [ The VM doesn't have haveged or rng-tools (rngd) installed. ] [1.420246] EXT4-fs (sda2): re-mounted. Opts: barrier,noacl,data=ordered [1.469722] tsc: Refined TSC clocksource calibration: 1900.002 MHz [1.470707] clocksource: tsc: mask: 0x max_cycles: 0x36c65c1a9e1, max_idle_ns: 881590695311 ns [1.474249] clocksource: Switched to clocksource tsc [1.584427] systemd-journald[216]: Received request to flush runtime journal from PID 1 [ 346.620718] random: crng init done Interestingly, the boot delay is exacerbated on VMs with large amounts of RAM. For example, the delay is not so noticeable (< 30 seconds) on a VM with 2GB memory, but goes up to 5 minutes on an 8GB VM. Also, cloud-init-local.service seems to be the one blocking for entropy here. systemd-analyze critical-chain shows: The time after the unit is active or started is printed after the "@" character. The time the unit takes to start is printed after the "+" character. multi-user.target @6min 1.283s └─vmtoolsd.service @6min 1.282s └─cloud-final.service @6min 366ms +914ms └─cloud-config.service @5min 59.174s +1.190s └─cloud-config.target @5min 59.172s └─cloud-init.service @5min 47.423s +11.744s └─systemd-networkd-wait-online.service @5min 45.999s +1.420s └─systemd-networkd.service @5min 45.975s +21ms └─network-pre.target @5min 45.973s └─cloud-init-local.service @241ms +5min 45.687s └─systemd-remount-fs.service @222ms +13ms └─systemd-fsck-root.service @193ms +26ms └─systemd-journald.socket @188ms └─-.mount @151ms └─system.slice @161ms └─-.slice @151ms It would be great if this CVE can be fixed somehow without causing boot speed to spike from ~20 seconds to 5 minutes, as that makes the system pretty much unusable. I can workaround this by installing haveged, but ideally an in-kernel fix would be better. If you need any other info about my setup or if you have a patch that I can test, please let me know! Thank you very much! Regards, Srivatsa VMware Photon OS
Re: Deadlock on poweroff
On 10/07/2012 10:20 PM, Kirill A. Shutemov wrote: On Sun, Oct 07, 2012 at 09:03:11AM -0700, Paul E. McKenney wrote: On Sun, Oct 07, 2012 at 05:47:11AM +0300, Kirill A. Shutemov wrote: Hi Paul and all, With commit 755609a9087fa983f567dc5452b2fa7b089b591f I've got deadlock on poweroff. It guess it happens because of race for cpu_hotplug.lock: CPU A CPU B disable_nonboot_cpus() _cpu_down() cpu_hotplug_begin() mutex_lock(cpu_hotplug.lock); __cpu_notify() padata_cpu_callback() __padata_remove_cpu() padata_replace() synchronize_rcu() rcu_gp_kthread() get_online_cpus(); mutex_lock(cpu_hotplug.lock); Have you seen the issue before? This is a new one for me. Does the following (very lightly tested) patch help? Works for me. Thanks. Could you share the patch please? Looks like it didn't hit the mailing lists.. Regards, Srivatsa S. Bhat -- To unsubscribe from this list: send the line unsubscribe linux-crypto in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Deadlock on poweroff
On 10/07/2012 10:41 PM, Kirill A. Shutemov wrote: On Sun, Oct 07, 2012 at 10:35:01PM +0530, Srivatsa S. Bhat wrote: On 10/07/2012 10:20 PM, Kirill A. Shutemov wrote: On Sun, Oct 07, 2012 at 09:03:11AM -0700, Paul E. McKenney wrote: On Sun, Oct 07, 2012 at 05:47:11AM +0300, Kirill A. Shutemov wrote: Hi Paul and all, With commit 755609a9087fa983f567dc5452b2fa7b089b591f I've got deadlock on poweroff. It guess it happens because of race for cpu_hotplug.lock: CPU A CPU B disable_nonboot_cpus() _cpu_down() cpu_hotplug_begin() mutex_lock(cpu_hotplug.lock); __cpu_notify() padata_cpu_callback() __padata_remove_cpu() padata_replace() synchronize_rcu() rcu_gp_kthread() get_online_cpus(); mutex_lock(cpu_hotplug.lock); Have you seen the issue before? This is a new one for me. Does the following (very lightly tested) patch help? Works for me. Thanks. Could you share the patch please? Looks like it didn't hit the mailing lists.. Sure. Here's original mail from Paul: Ah, great! Thanks! Regards, Srivatsa S. Bhat Date: Sun, 7 Oct 2012 09:03:11 -0700 From: Paul E. McKenney paul...@linux.vnet.ibm.com To: Kirill A. Shutemov kir...@shutemov.name Cc: linux-ker...@vger.kernel.org, Dipankar Sarma dipan...@in.ibm.com, Thomas Gleixner t...@linutronix.de, Andrew Morton a...@linux-foundation.org, Steffen Klassert steffen.klass...@secunet.com, .linux-crypto@vger.kernel.org Subject: Re: Deadlock on poweroff Message-ID: 20121007160311.ge2...@linux.vnet.ibm.com Reply-To: paul...@linux.vnet.ibm.com References: 20121007024711.ga21...@shutemov.name MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: 20121007024711.ga21...@shutemov.name User-Agent: Mutt/1.5.21 (2010-09-15) X-Content-Scanned: Fidelis XPS MAILER x-cbid: 12100716-7408---09194B17 Status: RO Content-Length: 6055 Lines: 173 On Sun, Oct 07, 2012 at 05:47:11AM +0300, Kirill A. Shutemov wrote: Hi Paul and all, With commit 755609a9087fa983f567dc5452b2fa7b089b591f I've got deadlock on poweroff. It guess it happens because of race for cpu_hotplug.lock: CPU A CPU B disable_nonboot_cpus() _cpu_down() cpu_hotplug_begin() mutex_lock(cpu_hotplug.lock); __cpu_notify() padata_cpu_callback() __padata_remove_cpu() padata_replace() synchronize_rcu() rcu_gp_kthread() get_online_cpus(); mutex_lock(cpu_hotplug.lock); Have you seen the issue before? This is a new one for me. Does the following (very lightly tested) patch help? Thanx, Paul rcu: Grace-period initialization excludes only RCU notifier Kirill noted the following deadlock cycle on shutdown involving padata: With commit 755609a9087fa983f567dc5452b2fa7b089b591f I've got deadlock on poweroff. It guess it happens because of race for cpu_hotplug.lock: CPU A CPU B disable_nonboot_cpus() _cpu_down() cpu_hotplug_begin() mutex_lock(cpu_hotplug.lock); __cpu_notify() padata_cpu_callback() __padata_remove_cpu() padata_replace() synchronize_rcu() rcu_gp_kthread() get_online_cpus(); mutex_lock(cpu_hotplug.lock); It would of course be good to eliminate grace-period delays from CPU-hotplug notifiers, but that is a separate issue. Deadlock is not an appropriate diagnostic for excessive CPU-hotplug latency. Fortunately, grace-period initialization does not actually need to exclude all of the CPU-hotplug operation, but rather only RCU's own CPU_UP_PREPARE and CPU_DEAD CPU-hotplug notifiers. This commit therefore introduces a new per-rcu_state onoff_mutex that provides the required concurrency control in place of the get_online_cpus() that was previously in rcu_gp_init(). Reported-by: Kirill A. Shutemov kir...@shutemov.name Signed-off-by: Paul E. McKenney paul...@linux.vnet.ibm.com diff --git a/kernel/rcutree.c b/kernel/rcutree.c index fb63d7b..5eece12 100644 --- a/kernel/rcutree.c +++ b/kernel/rcutree.c @@ -74,6 +74,7 @@ static struct lock_class_key rcu_fqs_class[RCU_NUM_LVLS]; .orphan_nxttail = sname##_state.orphan_nxtlist, \ .orphan_donetail = sname##_state.orphan_donelist, \ .barrier_mutex = __MUTEX_INITIALIZER(sname##_state.barrier_mutex), \ + .onoff_mutex = __MUTEX_INITIALIZER(sname##_state.onoff_mutex), \ .name = #sname, \ } @@ -1229,7 +1230,7 @@ static int rcu_gp_init(struct rcu_state *rsp
