Re: struct ifnet locking [was Re: IFEF_MPSAFE]
On Fri, Dec 22, 2017 at 11:33 AM, Ryota Ozakiwrote: > On Thu, Dec 21, 2017 at 7:45 PM, Frank Kardel wrote: >> Hi, >> >> does this panic (with -current/amd64 as of 20171221) ring a bell with >> anyone? > > Yes. Could you try the patch below? Oops. The patch was wrong... Please use the new one below. ozaki-r diff --git a/sys/net/if.c b/sys/net/if.c index 05ac3b2273c..6c867cdce0a 100644 --- a/sys/net/if.c +++ b/sys/net/if.c @@ -3598,7 +3598,7 @@ if_mcast_op(ifnet_t *ifp, const unsigned long cmd, const struct sockaddr *sa) struct ifreq ifr; /* CARP still doesn't deal with the lock yet */ -#if !defined(NCARP) || (NCARP == 0) +#if (!defined(NCARP) || (NCARP == 0)) && !defined(MROUTER) KASSERT(IFNET_LOCKED(ifp)); #endif if (ifp->if_mcastop != NULL)
Re: struct ifnet locking [was Re: IFEF_MPSAFE]
On Thu, Dec 21, 2017 at 7:58 PM, Nick Hudson <sk...@netbsd.org> wrote: > On 12/19/17 08:21, Ryota Ozaki wrote: >> >> On Tue, Dec 19, 2017 at 4:52 PM, Nick Hudson <nick.hud...@gmx.co.uk> >> wrote: >>> >>> On 19/12/2017 03:43, Ryota Ozaki wrote: >>> >>> >>>> BTW I committed a change that disables IFEF_MPSAFE by default on >>>> all interfaces because it seems that IFEF_MPSAFE requires additional >>>> changes to work safely with it. We should enable it by default if an >>>> interface is guaranteed to be safe. >>> >>> What additional changes? >> >> For example, avoid updating if_flags (and reading it and depending on >> the result) in Tx/Rx paths and using if_watchdog. > > > Ah, I see. > > Do we have a model driver yet? Not yet, but ixg and wm are close. ozaki-r
Re: struct ifnet locking [was Re: IFEF_MPSAFE]
On Thu, Dec 21, 2017 at 7:45 PM, Frank Kardelwrote: > Hi, > > does this panic (with -current/amd64 as of 20171221) ring a bell with > anyone? Yes. Could you try the patch below? Thanks, ozaki-r diff --git a/sys/net/if.c b/sys/net/if.c index 05ac3b2273c..fc3e144373f 100644 --- a/sys/net/if.c +++ b/sys/net/if.c @@ -3598,7 +3598,7 @@ if_mcast_op(ifnet_t *ifp, const unsigned long cmd, const struct sockaddr *sa) struct ifreq ifr; /* CARP still doesn't deal with the lock yet */ -#if !defined(NCARP) || (NCARP == 0) +#if !defined(NCARP) || (NCARP == 0) || !defined(MROUTER) KASSERT(IFNET_LOCKED(ifp)); #endif if (ifp->if_mcastop != NULL)
Re: struct ifnet locking [was Re: IFEF_MPSAFE]
On 12/19/17 08:21, Ryota Ozaki wrote: On Tue, Dec 19, 2017 at 4:52 PM, Nick Hudson <nick.hud...@gmx.co.uk> wrote: On 19/12/2017 03:43, Ryota Ozaki wrote: BTW I committed a change that disables IFEF_MPSAFE by default on all interfaces because it seems that IFEF_MPSAFE requires additional changes to work safely with it. We should enable it by default if an interface is guaranteed to be safe. What additional changes? For example, avoid updating if_flags (and reading it and depending on the result) in Tx/Rx paths and using if_watchdog. Ah, I see. Do we have a model driver yet? Thanks, NIck
Re: struct ifnet locking [was Re: IFEF_MPSAFE]
Hi, does this panic (with -current/amd64 as of 20171221) ring a bell with anyone? shortly after starting mrouted and ntpd: panic: kernel diagnostic assertion "IFNET_LOCKED(ifp)" failed: file "/src/NetBSD/cur/src/sys/net/if.c", line 3602 fatal breakpoint trap in supervisor mode trap type 1 code 0 rip 0x8021d2d5 cs 0x8 rflags 0x246 cr2 0x80013dd84000 ilevel 0 rsp 0xe4013bb5aae0 curlwp 0xe404743f15c0 pid 916.1 lowest kstack 0xe4013bb572c0 Stopped in pid 916.1 (mrouted) at netbsd:breakpoint+0x5: leave db{3}> bt breakpoint() at netbsd:breakpoint+0x5 vpanic() at netbsd:vpanic+0x140 ch_voltag_convert_in() at netbsd:ch_voltag_convert_in if_mcast_op() at netbsd:if_mcast_op+0xb3 ip_mrouter_set() at netbsd:ip_mrouter_set+0xdef rip_ctloutput() at netbsd:rip_ctloutput+0x11f rip_ctloutput_wrapper() at netbsd:rip_ctloutput_wrapper+0x2c sosetopt() at netbsd:sosetopt+0x67 sys_setsockopt() at netbsd:sys_setsockopt+0x91 syscall() at netbsd:syscall+0x235 Best regards, Frank On 12/19/17 09:21, Ryota Ozaki wrote: On Tue, Dec 19, 2017 at 4:52 PM, Nick Hudson <nick.hud...@gmx.co.uk> wrote: On 19/12/2017 03:43, Ryota Ozaki wrote: BTW I committed a change that disables IFEF_MPSAFE by default on all interfaces because it seems that IFEF_MPSAFE requires additional changes to work safely with it. We should enable it by default if an interface is guaranteed to be safe. What additional changes? For example, avoid updating if_flags (and reading it and depending on the result) in Tx/Rx paths and using if_watchdog. ozaki-r
Re: struct ifnet locking [was Re: IFEF_MPSAFE]
On Thu, Dec 14, 2017 at 10:41 PM, Nick Hudson <nick.hud...@gmx.co.uk> wrote: > > > On 14/12/2017 10:48, Ryota Ozaki wrote: >> >> On Fri, Dec 8, 2017 at 7:53 PM, Nick Hudson <sk...@netbsd.org> wrote: >> > >>> Not sure I follow this. I think we agree that the driver should not use >>> if_flags in the rx/tx path (anymore). >> >> Yes. Drivers should provide their own method. > > > Great. > >> >> Anyway I updated the document that reflects recent changes: >>http://www.netbsd.org/~ozaki-r/ifnet-locks.v2.diff > > > Some wording improvement suggestions... > > @@ -391,11 +429,33 @@ typedef struct ifnet { > #defineIFEF_NO_LINK_STATE_CHANGE __BIT(1)/* doesn't > use link state interrupts */ > /* > - * The following if_XXX() handlers don't take KERNEL_LOCK if the interface > - * is set IFEF_MPSAFE: > - * - if_start > - * - if_output > - * - if_ioctl > + * The guidline to enable IFEF_MPSAFE on an interface > > The guidelines for converting an interface to IFEF_MPSAFE are as > follows > > + * Enabling IFEF_MPSAFE on an interface suppress taking KERNEL_LOCK > + * on the following handlers: > > Enabling IFEF_MPSAFE on an interface suppresses taking KERNEL_LOCK > when > calling the following handlers: Thanks. I reflected the suggestions and committed the updated document. BTW I committed a change that disables IFEF_MPSAFE by default on all interfaces because it seems that IFEF_MPSAFE requires additional changes to work safely with it. We should enable it by default if an interface is guaranteed to be safe. ozaki-r
Re: struct ifnet locking [was Re: IFEF_MPSAFE]
On Tue, Dec 19, 2017 at 4:52 PM, Nick Hudson <nick.hud...@gmx.co.uk> wrote: > On 19/12/2017 03:43, Ryota Ozaki wrote: > > >> BTW I committed a change that disables IFEF_MPSAFE by default on >> all interfaces because it seems that IFEF_MPSAFE requires additional >> changes to work safely with it. We should enable it by default if an >> interface is guaranteed to be safe. > > What additional changes? For example, avoid updating if_flags (and reading it and depending on the result) in Tx/Rx paths and using if_watchdog. ozaki-r
Re: struct ifnet locking [was Re: IFEF_MPSAFE]
On 19/12/2017 03:43, Ryota Ozaki wrote: BTW I committed a change that disables IFEF_MPSAFE by default on all interfaces because it seems that IFEF_MPSAFE requires additional changes to work safely with it. We should enable it by default if an interface is guaranteed to be safe. What additional changes? Thanks, Nick
Re: struct ifnet locking [was Re: IFEF_MPSAFE]
On 14/12/2017 10:48, Ryota Ozaki wrote: On Fri, Dec 8, 2017 at 7:53 PM, Nick Hudson <sk...@netbsd.org> wrote: Not sure I follow this. I think we agree that the driver should not use if_flags in the rx/tx path (anymore). Yes. Drivers should provide their own method. Great. Anyway I updated the document that reflects recent changes: http://www.netbsd.org/~ozaki-r/ifnet-locks.v2.diff Some wording improvement suggestions... @@ -391,11 +429,33 @@ typedef struct ifnet { #defineIFEF_NO_LINK_STATE_CHANGE __BIT(1)/* doesn't use link state interrupts */ /* - * The following if_XXX() handlers don't take KERNEL_LOCK if the interface - * is set IFEF_MPSAFE: - * - if_start - * - if_output - * - if_ioctl + * The guidline to enable IFEF_MPSAFE on an interface The guidelines for converting an interface to IFEF_MPSAFE are as follows + * Enabling IFEF_MPSAFE on an interface suppress taking KERNEL_LOCK + * on the following handlers: Enabling IFEF_MPSAFE on an interface suppresses taking KERNEL_LOCK when calling the following handlers: Thanks for working on this. Nick
Re: struct ifnet locking [was Re: IFEF_MPSAFE]
On Fri, Dec 8, 2017 at 7:53 PM, Nick Hudsonwrote: > On 12/06/17 11:11, Ryota Ozaki wrote: >> >> On Tue, Nov 28, 2017 at 6:40 PM, Ryota Ozaki >> wrote: >>> >>> On Mon, Nov 27, 2017 at 12:24 AM, Nick Hudson wrote: On 11/17/17 07:44, Ryota Ozaki wrote: > > > [snip] Hi, >>> >>> (Sorry for late replying. I was sick in bed for days...) >>> Can you document the protection requirements of the struct ifnet members, e.g. if_flags. Ideally by annotating it like struct proc http://src.illumos.org/source/xref/netbsd-src/sys/sys/proc.h#230 >>> >>> OK. I'm trying to add annotations like that. > > > It's my turn to say sorry for a late reply... Sorry and thanks for working > on this. NP. I tend to be late... > > >> Some more changes are needed though, I fixed some race conditions on ifnet >> and wrote the lock requirements of ifnet: >> http://www.netbsd.org/~ozaki-r/ifnet-locks.diff >> >> There remain some unsafe members. Those for carp, agr and pf should be >> fixed >> when making them MP-safe. Another remainder is a set of statistic >> counters, >> which will be MP-safe (by making them per-CPU counters) sooner or later. >> >> if_flags should be modified with holding if_ioct_lock. If an interface >> enables >> IEFE_MPSAFE, the interface must meet the contract, which enforces the >> interface to update if_flags only in XXX_ioctl and also the interface has >> to >> give up using IFF_OACTIVE flag which doesn't suit for multi-core systems >> and hardware multi-queue NICs. > > > I'd pretty much come to the same conclusion, but wanted to make sure we > shared the same understanding. > >> I'm not sure yet we have to do >> synchronization between updates and reads on if_flags. (Anyway we should >> NOT do that because the synchronization will hurt performance.) > > > Not sure I follow this. I think we agree that the driver should not use > if_flags in the rx/tx path (anymore). Yes. Drivers should provide their own method. Anyway I updated the document that reflects recent changes: http://www.netbsd.org/~ozaki-r/ifnet-locks.v2.diff ozaki-r
Re: struct ifnet locking [was Re: IFEF_MPSAFE]
On 12/06/17 11:11, Ryota Ozaki wrote: On Tue, Nov 28, 2017 at 6:40 PM, Ryota Ozakiwrote: On Mon, Nov 27, 2017 at 12:24 AM, Nick Hudson wrote: On 11/17/17 07:44, Ryota Ozaki wrote: [snip] Hi, (Sorry for late replying. I was sick in bed for days...) Can you document the protection requirements of the struct ifnet members, e.g. if_flags. Ideally by annotating it like struct proc http://src.illumos.org/source/xref/netbsd-src/sys/sys/proc.h#230 OK. I'm trying to add annotations like that. It's my turn to say sorry for a late reply... Sorry and thanks for working on this. Some more changes are needed though, I fixed some race conditions on ifnet and wrote the lock requirements of ifnet: http://www.netbsd.org/~ozaki-r/ifnet-locks.diff There remain some unsafe members. Those for carp, agr and pf should be fixed when making them MP-safe. Another remainder is a set of statistic counters, which will be MP-safe (by making them per-CPU counters) sooner or later. if_flags should be modified with holding if_ioct_lock. If an interface enables IEFE_MPSAFE, the interface must meet the contract, which enforces the interface to update if_flags only in XXX_ioctl and also the interface has to give up using IFF_OACTIVE flag which doesn't suit for multi-core systems and hardware multi-queue NICs. I'd pretty much come to the same conclusion, but wanted to make sure we shared the same understanding. I'm not sure yet we have to do synchronization between updates and reads on if_flags. (Anyway we should NOT do that because the synchronization will hurt performance.) Not sure I follow this. I think we agree that the driver should not use if_flags in the rx/tx path (anymore). Thanks, Nick
Re: struct ifnet locking [was Re: IFEF_MPSAFE]
On Tue, Nov 28, 2017 at 6:40 PM, Ryota Ozaki <ozaki.ry...@gmail.com> wrote: > On Mon, Nov 27, 2017 at 12:24 AM, Nick Hudson <sk...@netbsd.org> wrote: >> On 11/17/17 07:44, Ryota Ozaki wrote: >>> >>> On Wed, Nov 15, 2017 at 1:53 PM, Ryota Ozaki <ozaki.ry...@gmail.com> >>> wrote: >>>> >>>> On Fri, Nov 10, 2017 at 6:35 PM, Ryota Ozaki <ozaki.ry...@gmail.com> >>>> wrote: >>>>> >>>>> Hi, >>>>> >>>>> http://www.netbsd.org/~ozaki-r/IFEF_MPSAFE.diff >>>>> >>>>> I'm going to commit the above change that integrates >>>>> IFEF_OUTPUT_MPSAFE and IFEF_START_MPSAFE flags into >>>>> IFEF_MPSAFE. >>>>> >>>>> The motivation is to not waste if_extflags bits. I'm now >>>>> trying to make if_ioctl() hold KERNEL_LOCK selectively >>>>> for some reasons as well as if_start() and if_output(). >>>> >>>> BTW this is a patch for this plan: >>>>http://www.netbsd.org/~ozaki-r/if_ioctl-no-KERNEL_LOCK.diff >>>> >>>> It removes KERNEL_LOCK for if_ioctl from soo_ioctl and >>>> selectively takes it in doifioctl. To this end, some fine-grain >>>> KERNEL_LOCKs have to be added where calling components/functions >>>> that aren't MP-safe. >>> >>> Revised rebased on latest NET_MPSAFE macros. The new patch also provides >>> similar macros: >>>http://www.netbsd.org/~ozaki-r/if_ioctl-no-KERNEL_LOCK.revised.diff >>> >>>ozaki-r >>> >> Hi, > > (Sorry for late replying. I was sick in bed for days...) > >> >> Can you document the protection requirements of the struct ifnet members, >> e.g. if_flags. >> >> Ideally by annotating it like struct proc >> >> http://src.illumos.org/source/xref/netbsd-src/sys/sys/proc.h#230 > > OK. I'm trying to add annotations like that. Some more changes are needed though, I fixed some race conditions on ifnet and wrote the lock requirements of ifnet: http://www.netbsd.org/~ozaki-r/ifnet-locks.diff There remain some unsafe members. Those for carp, agr and pf should be fixed when making them MP-safe. Another remainder is a set of statistic counters, which will be MP-safe (by making them per-CPU counters) sooner or later. if_flags should be modified with holding if_ioct_lock. If an interface enables IEFE_MPSAFE, the interface must meet the contract, which enforces the interface to update if_flags only in XXX_ioctl and also the interface has to give up using IFF_OACTIVE flag which doesn't suit for multi-core systems and hardware multi-queue NICs. I'm not sure yet we have to do synchronization between updates and reads on if_flags. (Anyway we should NOT do that because the synchronization will hurt performance.) Thanks, ozaki-r
Re: struct ifnet locking [was Re: IFEF_MPSAFE]
On Mon, Nov 27, 2017 at 12:24 AM, Nick Hudson <sk...@netbsd.org> wrote: > On 11/17/17 07:44, Ryota Ozaki wrote: >> >> On Wed, Nov 15, 2017 at 1:53 PM, Ryota Ozaki <ozaki.ry...@gmail.com> >> wrote: >>> >>> On Fri, Nov 10, 2017 at 6:35 PM, Ryota Ozaki <ozaki.ry...@gmail.com> >>> wrote: >>>> >>>> Hi, >>>> >>>> http://www.netbsd.org/~ozaki-r/IFEF_MPSAFE.diff >>>> >>>> I'm going to commit the above change that integrates >>>> IFEF_OUTPUT_MPSAFE and IFEF_START_MPSAFE flags into >>>> IFEF_MPSAFE. >>>> >>>> The motivation is to not waste if_extflags bits. I'm now >>>> trying to make if_ioctl() hold KERNEL_LOCK selectively >>>> for some reasons as well as if_start() and if_output(). >>> >>> BTW this is a patch for this plan: >>>http://www.netbsd.org/~ozaki-r/if_ioctl-no-KERNEL_LOCK.diff >>> >>> It removes KERNEL_LOCK for if_ioctl from soo_ioctl and >>> selectively takes it in doifioctl. To this end, some fine-grain >>> KERNEL_LOCKs have to be added where calling components/functions >>> that aren't MP-safe. >> >> Revised rebased on latest NET_MPSAFE macros. The new patch also provides >> similar macros: >>http://www.netbsd.org/~ozaki-r/if_ioctl-no-KERNEL_LOCK.revised.diff >> >>ozaki-r >> > Hi, (Sorry for late replying. I was sick in bed for days...) > > Can you document the protection requirements of the struct ifnet members, > e.g. if_flags. > > Ideally by annotating it like struct proc > > http://src.illumos.org/source/xref/netbsd-src/sys/sys/proc.h#230 OK. I'm trying to add annotations like that. ozaki-r
struct ifnet locking [was Re: IFEF_MPSAFE]
On 11/17/17 07:44, Ryota Ozaki wrote: On Wed, Nov 15, 2017 at 1:53 PM, Ryota Ozaki <ozaki.ry...@gmail.com> wrote: On Fri, Nov 10, 2017 at 6:35 PM, Ryota Ozaki <ozaki.ry...@gmail.com> wrote: Hi, http://www.netbsd.org/~ozaki-r/IFEF_MPSAFE.diff I'm going to commit the above change that integrates IFEF_OUTPUT_MPSAFE and IFEF_START_MPSAFE flags into IFEF_MPSAFE. The motivation is to not waste if_extflags bits. I'm now trying to make if_ioctl() hold KERNEL_LOCK selectively for some reasons as well as if_start() and if_output(). BTW this is a patch for this plan: http://www.netbsd.org/~ozaki-r/if_ioctl-no-KERNEL_LOCK.diff It removes KERNEL_LOCK for if_ioctl from soo_ioctl and selectively takes it in doifioctl. To this end, some fine-grain KERNEL_LOCKs have to be added where calling components/functions that aren't MP-safe. Revised rebased on latest NET_MPSAFE macros. The new patch also provides similar macros: http://www.netbsd.org/~ozaki-r/if_ioctl-no-KERNEL_LOCK.revised.diff ozaki-r Hi, Can you document the protection requirements of the struct ifnet members, e.g. if_flags. Ideally by annotating it like struct proc http://src.illumos.org/source/xref/netbsd-src/sys/sys/proc.h#230 Thanks, Nick
Re: IFEF_MPSAFE
On Wed, Nov 15, 2017 at 1:53 PM, Ryota Ozaki <ozaki.ry...@gmail.com> wrote: > On Fri, Nov 10, 2017 at 6:35 PM, Ryota Ozaki <ozaki.ry...@gmail.com> wrote: >> Hi, >> >> http://www.netbsd.org/~ozaki-r/IFEF_MPSAFE.diff >> >> I'm going to commit the above change that integrates >> IFEF_OUTPUT_MPSAFE and IFEF_START_MPSAFE flags into >> IFEF_MPSAFE. >> >> The motivation is to not waste if_extflags bits. I'm now >> trying to make if_ioctl() hold KERNEL_LOCK selectively >> for some reasons as well as if_start() and if_output(). > > BTW this is a patch for this plan: > http://www.netbsd.org/~ozaki-r/if_ioctl-no-KERNEL_LOCK.diff > > It removes KERNEL_LOCK for if_ioctl from soo_ioctl and > selectively takes it in doifioctl. To this end, some fine-grain > KERNEL_LOCKs have to be added where calling components/functions > that aren't MP-safe. Revised rebased on latest NET_MPSAFE macros. The new patch also provides similar macros: http://www.netbsd.org/~ozaki-r/if_ioctl-no-KERNEL_LOCK.revised.diff ozaki-r
Re: IFEF_MPSAFE
On Fri, Nov 10, 2017 at 6:35 PM, Ryota Ozaki <ozaki.ry...@gmail.com> wrote: > Hi, > > http://www.netbsd.org/~ozaki-r/IFEF_MPSAFE.diff > > I'm going to commit the above change that integrates > IFEF_OUTPUT_MPSAFE and IFEF_START_MPSAFE flags into > IFEF_MPSAFE. > > The motivation is to not waste if_extflags bits. I'm now > trying to make if_ioctl() hold KERNEL_LOCK selectively > for some reasons as well as if_start() and if_output(). BTW this is a patch for this plan: http://www.netbsd.org/~ozaki-r/if_ioctl-no-KERNEL_LOCK.diff It removes KERNEL_LOCK for if_ioctl from soo_ioctl and selectively takes it in doifioctl. To this end, some fine-grain KERNEL_LOCKs have to be added where calling components/functions that aren't MP-safe. ozaki-r > But adding one more flag for if_ioctl() is I think > wasteful. Also there are other functions such as if_init() > and if_slowtimo() that would also need a flag. > > So I propose to have just one flag for indications of > MP-safe. If an interface have both MP-safe and non-MP-safe > operations at a time, we have to set the IFEF_MPSAFE flag > and let callees of non-MP-safe operations take KERNEL_LOCK. > > This change breaks ABI and need a kernel version bump, > however, IFEF_*_MPSAFE flags are new to netbsd-8 so it > doesn't break backward compatibility. > > Any comments or objections? > > Thanks, > ozaki-r
IFEF_MPSAFE
Hi, http://www.netbsd.org/~ozaki-r/IFEF_MPSAFE.diff I'm going to commit the above change that integrates IFEF_OUTPUT_MPSAFE and IFEF_START_MPSAFE flags into IFEF_MPSAFE. The motivation is to not waste if_extflags bits. I'm now trying to make if_ioctl() hold KERNEL_LOCK selectively for some reasons as well as if_start() and if_output(). But adding one more flag for if_ioctl() is I think wasteful. Also there are other functions such as if_init() and if_slowtimo() that would also need a flag. So I propose to have just one flag for indications of MP-safe. If an interface have both MP-safe and non-MP-safe operations at a time, we have to set the IFEF_MPSAFE flag and let callees of non-MP-safe operations take KERNEL_LOCK. This change breaks ABI and need a kernel version bump, however, IFEF_*_MPSAFE flags are new to netbsd-8 so it doesn't break backward compatibility. Any comments or objections? Thanks, ozaki-r